[jira] [Updated] (HIVE-17240) function ACOS(2) and ASIN(2) should be null

[Yuming Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuming Wang updated HIVE-17240:
---
Attachment: HIVE-17240.2.patch

> function ACOS(2) and ASIN(2) should be null
> ---
>
> Key: HIVE-17240
> URL: https://issues.apache.org/jira/browse/HIVE-17240
> Project: Hive
>  Issue Type: Bug
>  Components: UDF
>Affects Versions: 1.1.1, 1.2.2, 2.2.0
>Reporter: Yuming Wang
>Assignee: Yuming Wang
> Attachments: HIVE-17240.1.patch, HIVE-17240.2.patch
>
>
> {{acos(2)}} should be null, same as MySQL:
> {code:sql}
> hive> desc function extended acos;
> OK
> acos(x) - returns the arc cosine of x if -1<=x<=1 or NULL otherwise
> Example:
>   > SELECT acos(1) FROM src LIMIT 1;
>   0
>   > SELECT acos(2) FROM src LIMIT 1;
>   NULL
> Time taken: 0.009 seconds, Fetched: 6 row(s)
> hive> select acos(2);
> OK
> NaN
> Time taken: 0.437 seconds, Fetched: 1 row(s)
> {code}
> {code:sql}
> mysql>  select acos(2);
> +-+
> | acos(2) |
> +-+
> |NULL |
> +-+
> 1 row in set (0.00 sec)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17235) Add ORC Decimal64 Serialization/Deserialization

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-17235:

Attachment: HIVE-17235.05.patch

> Add ORC Decimal64 Serialization/Deserialization
> ---
>
> Key: HIVE-17235
> URL: https://issues.apache.org/jira/browse/HIVE-17235
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-17235.03.patch, HIVE-17235.04.patch, 
> HIVE-17235.05.patch
>
>
> The storage-api changes for ORC-209.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17240) function ACOS(2) and ASIN(2) should be null

[Yuming Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuming Wang updated HIVE-17240:
---
Summary: function ACOS(2) and ASIN(2) should be null  (was: function 
acos(2) should be null)

> function ACOS(2) and ASIN(2) should be null
> ---
>
> Key: HIVE-17240
> URL: https://issues.apache.org/jira/browse/HIVE-17240
> Project: Hive
>  Issue Type: Bug
>  Components: UDF
>Affects Versions: 1.1.1, 1.2.2, 2.2.0
>Reporter: Yuming Wang
>Assignee: Yuming Wang
> Attachments: HIVE-17240.1.patch
>
>
> {{acos(2)}} should be null, same as MySQL:
> {code:sql}
> hive> desc function extended acos;
> OK
> acos(x) - returns the arc cosine of x if -1<=x<=1 or NULL otherwise
> Example:
>   > SELECT acos(1) FROM src LIMIT 1;
>   0
>   > SELECT acos(2) FROM src LIMIT 1;
>   NULL
> Time taken: 0.009 seconds, Fetched: 6 row(s)
> hive> select acos(2);
> OK
> NaN
> Time taken: 0.437 seconds, Fetched: 1 row(s)
> {code}
> {code:sql}
> mysql>  select acos(2);
> +-+
> | acos(2) |
> +-+
> |NULL |
> +-+
> 1 row in set (0.00 sec)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17148) Incorrect result for Hive join query with COALESCE in WHERE condition

[Vlad Gudikov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vlad Gudikov updated HIVE-17148:

Status: Open  (was: Patch Available)

> Incorrect result for Hive join query with COALESCE in WHERE condition
> -
>
> Key: HIVE-17148
> URL: https://issues.apache.org/jira/browse/HIVE-17148
> Project: Hive
>  Issue Type: Bug
>  Components: CBO
>Affects Versions: 2.1.1
>Reporter: Vlad Gudikov
>Assignee: Vlad Gudikov
> Attachments: HIVE-17148.1.patch, HIVE-17148.patch
>
>
> The issue exists in Hive-2.1. In Hive-1.2 the query works fine with cbo 
> enabled:
> STEPS TO REPRODUCE:
> {code}
> Step 1: Create a table ct1
> create table ct1 (a1 string,b1 string);
> Step 2: Create a table ct2
> create table ct2 (a2 string);
> Step 3 : Insert following data into table ct1
> insert into table ct1 (a1) values ('1');
> Step 4 : Insert following data into table ct2
> insert into table ct2 (a2) values ('1');
> Step 5 : Execute the following query 
> select * from ct1 c1, ct2 c2 where COALESCE(a1,b1)=a2;
> {code}
> ACTUAL RESULT:
> {code}
> The query returns nothing;
> {code}
> EXPECTED RESULT:
> {code}
> 1   NULL1
> {code}
> The issue seems to be because of the incorrect query plan. In the plan we can 
> see:
> predicate:(a1 is not null and b1 is not null)
> which does not look correct. As a result, it is filtering out all the rows is 
> any column mentioned in the COALESCE has null value.
> Please find the query plan below:
> {code}
> Plan optimized by CBO.
> Vertex dependency in root stage
> Map 1 <- Map 2 (BROADCAST_EDGE)
> Stage-0
>   Fetch Operator
> limit:-1
> Stage-1
>   Map 1
>   File Output Operator [FS_10]
> Map Join Operator [MAPJOIN_15] (rows=1 width=4)
>   
> Conds:SEL_2.COALESCE(_col0,_col1)=RS_7._col0(Inner),HybridGraceHashJoin:true,Output:["_col0","_col1","_col2"]
> <-Map 2 [BROADCAST_EDGE]
>   BROADCAST [RS_7]
> PartitionCols:_col0
> Select Operator [SEL_5] (rows=1 width=1)
>   Output:["_col0"]
>   Filter Operator [FIL_14] (rows=1 width=1)
> predicate:a2 is not null
> TableScan [TS_3] (rows=1 width=1)
>   default@ct2,c2,Tbl:COMPLETE,Col:NONE,Output:["a2"]
> <-Select Operator [SEL_2] (rows=1 width=4)
> Output:["_col0","_col1"]
> Filter Operator [FIL_13] (rows=1 width=4)
>   predicate:(a1 is not null and b1 is not null)
>   TableScan [TS_0] (rows=1 width=4)
> default@ct1,c1,Tbl:COMPLETE,Col:NONE,Output:["a1","b1"]
> {code}
> This happens only if join is inner type, otherwise HiveJoinAddNotRule which 
> creates this problem is skipped.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17148) Incorrect result for Hive join query with COALESCE in WHERE condition

[Vlad Gudikov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vlad Gudikov updated HIVE-17148:

Status: Patch Available  (was: Open)

> Incorrect result for Hive join query with COALESCE in WHERE condition
> -
>
> Key: HIVE-17148
> URL: https://issues.apache.org/jira/browse/HIVE-17148
> Project: Hive
>  Issue Type: Bug
>  Components: CBO
>Affects Versions: 2.1.1
>Reporter: Vlad Gudikov
>Assignee: Vlad Gudikov
> Attachments: HIVE-17148.1.patch, HIVE-17148.patch
>
>
> The issue exists in Hive-2.1. In Hive-1.2 the query works fine with cbo 
> enabled:
> STEPS TO REPRODUCE:
> {code}
> Step 1: Create a table ct1
> create table ct1 (a1 string,b1 string);
> Step 2: Create a table ct2
> create table ct2 (a2 string);
> Step 3 : Insert following data into table ct1
> insert into table ct1 (a1) values ('1');
> Step 4 : Insert following data into table ct2
> insert into table ct2 (a2) values ('1');
> Step 5 : Execute the following query 
> select * from ct1 c1, ct2 c2 where COALESCE(a1,b1)=a2;
> {code}
> ACTUAL RESULT:
> {code}
> The query returns nothing;
> {code}
> EXPECTED RESULT:
> {code}
> 1   NULL1
> {code}
> The issue seems to be because of the incorrect query plan. In the plan we can 
> see:
> predicate:(a1 is not null and b1 is not null)
> which does not look correct. As a result, it is filtering out all the rows is 
> any column mentioned in the COALESCE has null value.
> Please find the query plan below:
> {code}
> Plan optimized by CBO.
> Vertex dependency in root stage
> Map 1 <- Map 2 (BROADCAST_EDGE)
> Stage-0
>   Fetch Operator
> limit:-1
> Stage-1
>   Map 1
>   File Output Operator [FS_10]
> Map Join Operator [MAPJOIN_15] (rows=1 width=4)
>   
> Conds:SEL_2.COALESCE(_col0,_col1)=RS_7._col0(Inner),HybridGraceHashJoin:true,Output:["_col0","_col1","_col2"]
> <-Map 2 [BROADCAST_EDGE]
>   BROADCAST [RS_7]
> PartitionCols:_col0
> Select Operator [SEL_5] (rows=1 width=1)
>   Output:["_col0"]
>   Filter Operator [FIL_14] (rows=1 width=1)
> predicate:a2 is not null
> TableScan [TS_3] (rows=1 width=1)
>   default@ct2,c2,Tbl:COMPLETE,Col:NONE,Output:["a2"]
> <-Select Operator [SEL_2] (rows=1 width=4)
> Output:["_col0","_col1"]
> Filter Operator [FIL_13] (rows=1 width=4)
>   predicate:(a1 is not null and b1 is not null)
>   TableScan [TS_0] (rows=1 width=4)
> default@ct1,c1,Tbl:COMPLETE,Col:NONE,Output:["a1","b1"]
> {code}
> This happens only if join is inner type, otherwise HiveJoinAddNotRule which 
> creates this problem is skipped.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17250) Avoid manually deploy changes to Jenkins server in testutils/ptest2/conf

[Chao Sun (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113920#comment-16113920
 ] 

Chao Sun commented on HIVE-17250:
-

[~stakiar] FYI.

> Avoid manually deploy changes to Jenkins server in testutils/ptest2/conf
> 
>
> Key: HIVE-17250
> URL: https://issues.apache.org/jira/browse/HIVE-17250
> Project: Hive
>  Issue Type: Bug
>  Components: Testing Infrastructure
>Reporter: Chao Sun
>
> At the moment it seems all changes to test properties under 
> testutils/ptest2/conf have to be manually deployed to the Jenkins server to 
> take effect. Ideally the Jenkins job should pick up the change in patches and 
> apply them before triggering tests.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17250) Avoid manually deploy changes to Jenkins server in testutils/ptest2/conf

[Chao Sun (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chao Sun updated HIVE-17250:

Issue Type: Improvement  (was: Bug)

> Avoid manually deploy changes to Jenkins server in testutils/ptest2/conf
> 
>
> Key: HIVE-17250
> URL: https://issues.apache.org/jira/browse/HIVE-17250
> Project: Hive
>  Issue Type: Improvement
>  Components: Testing Infrastructure
>Reporter: Chao Sun
>
> At the moment it seems all changes to test properties under 
> testutils/ptest2/conf have to be manually deployed to the Jenkins server to 
> take effect. Ideally the Jenkins job should pick up the change in patches and 
> apply them before triggering tests.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17249) Concurrent appendPartition calls lead to data loss

[Hans Zeller (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hans Zeller updated HIVE-17249:
---
Description: 
We are running into a problem with data getting lost when loading data in 
parallel into a partitioned Hive table. The data loader runs on multiple nodes 
and it dynamically creates partitions as it needs them, using the 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.appendPartition(String,
String, String) interface. We assume that if multiple processes try to create 
the same partition at the same time, only one of them succeeds while the others 
fail.

What we are seeing is that the partition gets created, but a few of the created 
files end up in the .Trash folder in HDFS. From the metastore log, we assume 
the following is happening in the threads of the metastore server:

- Thread 1: A first process tries to create a partition.
- Thread 1: The org.apache.hadoop.hive.metastore.HiveMetaStore.append_common() 
method
creates the HDFS directory.
- Thread 2: A second process tries to create the same partition.
- Thread 2: Notices that the directory already exists and skips the step of 
creating it.
- Thread 2: Update the metastore.
- Thread 2: Return success to the caller.
- Caller 2: Create a file in the partition directory and start inserting.
- Thread 1: Try to update the metastore, but this fails, since thread 2 already 
has inserted the partition. Retry the operation, but it still fails.
- Thread 1: Abort the transaction and move the HDFS directory to the trash, 
since it knows that it created the directory.
- Thread 1: Return failure to the caller.

The second caller can now continue to load data successfully, but the file it 
loads is actually already in the trash. It returns success, but the data is not 
inserted and not visible in the table.

Note that in our case, the caller(s) that got an error continue as well - they 
ignore the error. I think they automatically create the HDFS partition 
directory when they create their output files. These processes can insert data 
successfully, the data that is lost is from the process that successfully 
created the partition, we believe.

  was:
We are running into a problem with data getting lost when loading data in 
parallel into a partitioned Hive table. The data loader runs on multiple nodes 
and it dynamically creates partitions as it needs them, using the 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.appendPartition(String,
String, String) interface. We assume that if multiple processes try to create 
the same partition at the same time, only one of them succeeds while the others 
fail.

What we are seeing is that the partition gets created, but a few of the created 
files end up in the .Trash folder in HDFS. From the metastore log, we assume 
the following is happening in the threads of the metastore server:

- Thread 1: A first process tries to create a partition.
- Thread 1: The org.apache.hadoop.hive.metastore.HiveMetaStore.append_common() 
method
creates the HDFS directory.
- Thread 2: A second process tries to create the same partition.
- Thread 2: Notices that the directory already exists and skips the step of 
creating it.
- Thread 2: Update the metastore.
- Thread 2: Return success to the caller.
- Caller 2: Create a file in the partition directory and start inserting.
- Thread 1: Try to update the metastore, but this fails, since thread 2 already 
has inserted the partition. Retry the operation, but it still fails.
- Thread 1: Abort the transaction and move the HDFS directory to the trash, 
since it knows that it created the directory.
- Thread 1: Return failure to the caller.

The first caller can now continue to load data successfully, but the file it 
loads is actually already in the trash. It returns success, but the data is not 
inserted and not visible in the table.

Note that in our case, the callers that got an error continue as well - they 
ignore the error. I think they automatically create the HDFS partition 
directory when they create their output files. These processes can insert data 
successfully, the data that is lost is from the process that successfully 
created the partition, we believe.


> Concurrent appendPartition calls lead to data loss
> --
>
> Key: HIVE-17249
> URL: https://issues.apache.org/jira/browse/HIVE-17249
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 1.2.1
> Environment: Hortonworks HDP 2.4.
> MySQL metastore.
>Reporter: Hans Zeller
> Attachments: partition1log.txt
>
>
> We are running into a problem with data getting lost when loading data in 
> parallel into a partitioned Hive table. The data loader runs on multiple 
> nodes and it dynamically creates partitions as it needs them, using the 
> 

[jira] [Updated] (HIVE-17249) Concurrent appendPartition calls lead to data loss

[Hans Zeller (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hans Zeller updated HIVE-17249:
---
Attachment: partition1log.txt

Attached is an extract of messages relating to one partition from the
hivemetastore.log file.

> Concurrent appendPartition calls lead to data loss
> --
>
> Key: HIVE-17249
> URL: https://issues.apache.org/jira/browse/HIVE-17249
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 1.2.1
> Environment: Hortonworks HDP 2.4.
> MySQL metastore.
>Reporter: Hans Zeller
> Attachments: partition1log.txt
>
>
> We are running into a problem with data getting lost when loading data in 
> parallel into a partitioned Hive table. The data loader runs on multiple 
> nodes and it dynamically creates partitions as it needs them, using the 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.appendPartition(String,
> String, String) interface. We assume that if multiple processes try to create 
> the same partition at the same time, only one of them succeeds while the 
> others fail.
> What we are seeing is that the partition gets created, but a few of the 
> created files end up in the .Trash folder in HDFS. From the metastore log, we 
> assume the following is happening in the threads of the metastore server:
> - Thread 1: A first process tries to create a partition.
> - Thread 1: The 
> org.apache.hadoop.hive.metastore.HiveMetaStore.append_common() method
> creates the HDFS directory.
> - Thread 2: A second process tries to create the same partition.
> - Thread 2: Notices that the directory already exists and skips the step of 
> creating it.
> - Thread 2: Update the metastore.
> - Thread 2: Return success to the caller.
> - Caller 2: Create a file in the partition directory and start inserting.
> - Thread 1: Try to update the metastore, but this fails, since thread 2 
> already has inserted the partition. Retry the operation, but it still fails.
> - Thread 1: Abort the transaction and move the HDFS directory to the trash, 
> since it knows that it created the directory.
> - Thread 1: Return failure to the caller.
> The first caller can now continue to load data successfully, but the file it 
> loads is actually already in the trash. It returns success, but the data is 
> not inserted and not visible in the table.
> Note that in our case, the callers that got an error continue as well - they 
> ignore the error. I think they automatically create the HDFS partition 
> directory when they create their output files. These processes can insert 
> data successfully, the data that is lost is from the process that 
> successfully created the partition, we believe.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-14380) Queries on tables with remote HDFS paths fail in "encryption" checks.

[Mithun Radhakrishnan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mithun Radhakrishnan updated HIVE-14380:

Attachment: HIVE-14380.branch-1.2.patch

Sorry I missed your message, [~cltlfcjin]. Here's a patch that applies to 
{{branch-1.2}}. I've had to pull in part of HIVE-7476 for this.

> Queries on tables with remote HDFS paths fail in "encryption" checks.
> -
>
> Key: HIVE-14380
> URL: https://issues.apache.org/jira/browse/HIVE-14380
> Project: Hive
>  Issue Type: Bug
>  Components: Encryption
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
> Fix For: 2.2.0
>
> Attachments: HIVE-14380.1.patch, HIVE-14380.branch-1.2.patch
>
>
> If a table has table/partition locations set to remote HDFS paths, querying 
> them will cause the following IAException:
> {noformat}
> 2016-07-26 01:16:27,471 ERROR parse.CalcitePlanner 
> (SemanticAnalyzer.java:getMetaData(1867)) - 
> org.apache.hadoop.hive.ql.metadata.HiveException: Unable to determine if 
> hdfs://foo.ygrid.yahoo.com:8020/projects/my_db/my_table is encrypted: 
> java.lang.IllegalArgumentException: Wrong FS: 
> hdfs://foo.ygrid.yahoo.com:8020/projects/my_db/my_table, expected: 
> hdfs://bar.ygrid.yahoo.com:8020
> at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.isPathEncrypted(SemanticAnalyzer.java:2204)
> at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getStrongestEncryptedTablePath(SemanticAnalyzer.java:2274)
> ...
> {noformat}
> This is because of the following code in {{SessionState}}:
> {code:title=SessionState.java|borderStyle=solid}
>  public HadoopShims.HdfsEncryptionShim getHdfsEncryptionShim() throws 
> HiveException {
> if (hdfsEncryptionShim == null) {
>   try {
> FileSystem fs = FileSystem.get(sessionConf);
> if ("hdfs".equals(fs.getUri().getScheme())) {
>   hdfsEncryptionShim = 
> ShimLoader.getHadoopShims().createHdfsEncryptionShim(fs, sessionConf);
> } else {
>   LOG.debug("Could not get hdfsEncryptionShim, it is only applicable 
> to hdfs filesystem.");
> }
>   } catch (Exception e) {
> throw new HiveException(e);
>   }
> }
> return hdfsEncryptionShim;
>   }
> {code}
> When the {{FileSystem}} instance is created, using the {{sessionConf}} 
> implies that the current HDFS is going to be used. This call should instead 
> fetch the {{FileSystem}} instance corresponding to the path being checked.
> A fix is forthcoming...



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17178) Spark Partition Pruning Sink Operator can't target multiple Works

[Sahil Takiar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113761#comment-16113761
 ] 

Sahil Takiar commented on HIVE-17178:
-

On a related note, when there are multiple columns from the same table that can 
target the same map work, separate DPP subtrees need to be created - e.g. the 
below is possible:

{code}
Select Operator
  expressions: ds (type: string), hr (type: string)
  outputColumnNames: _col0, _col2
  Statistics: Num rows: 2 Data size: 54 Basic stats: 
COMPLETE Column stats: NONE
  Spark HashTable Sink Operator
keys:
  0 _col0 (type: string), UDFToDouble(_col1) (type: 
double)
  1 _col0 (type: string), UDFToDouble(_col2) (type: 
double)
  Select Operator
expressions: _col0 (type: string)
outputColumnNames: _col0
Statistics: Num rows: 2 Data size: 54 Basic stats: 
COMPLETE Column stats: NONE
Group By Operator
  keys: _col0 (type: string)
  mode: hash
  outputColumnNames: _col0
  Statistics: Num rows: 2 Data size: 54 Basic stats: 
COMPLETE Column stats: NONE
  Spark Partition Pruning Sink Operator
partition key expr: ds
Statistics: Num rows: 2 Data size: 54 Basic stats: 
COMPLETE Column stats: NONE
target column name: ds
target work: Map 1
  Select Operator
expressions: UDFToDouble(_col2) (type: double)
outputColumnNames: _col0
Statistics: Num rows: 2 Data size: 54 Basic stats: 
COMPLETE Column stats: NONE
Group By Operator
  keys: _col0 (type: double)
  mode: hash
  outputColumnNames: _col0
  Statistics: Num rows: 2 Data size: 54 Basic stats: 
COMPLETE Column stats: NONE
  Spark Partition Pruning Sink Operator
partition key expr: UDFToDouble(hr)
Statistics: Num rows: 2 Data size: 54 Basic stats: 
COMPLETE Column stats: NONE
target column name: hr
target work: Map 1
{code}

> Spark Partition Pruning Sink Operator can't target multiple Works
> -
>
> Key: HIVE-17178
> URL: https://issues.apache.org/jira/browse/HIVE-17178
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>
> A Spark Partition Pruning Sink Operator cannot be used to target multiple Map 
> Work objects. The entire DPP subtree (SEL-GBY-SPARKPRUNINGSINK) is duplicated 
> if a single table needs to be used to target multiple Map Works.
> The following query shows the issue:
> {code}
> set hive.spark.dynamic.partition.pruning=true;
> set hive.auto.convert.join=true;
> create table part_table_1 (col int) partitioned by (part_col int);
> create table part_table_2 (col int) partitioned by (part_col int);
> create table regular_table (col int);
> insert into table regular_table values (1);
> alter table part_table_1 add partition (part_col=1);
> insert into table part_table_1 partition (part_col=1) values (1), (2), (3), 
> (4);
> alter table part_table_1 add partition (part_col=2);
> insert into table part_table_1 partition (part_col=2) values (1), (2), (3), 
> (4);
> alter table part_table_2 add partition (part_col=1);
> insert into table part_table_2 partition (part_col=1) values (1), (2), (3), 
> (4);
> alter table part_table_2 add partition (part_col=2);
> insert into table part_table_2 partition (part_col=2) values (1), (2), (3), 
> (4);
> explain select * from regular_table, part_table_1, part_table_2 where 
> regular_table.col = part_table_1.part_col and regular_table.col = 
> part_table_2.part_col;
> {code}
> The explain plan is
> {code}
> STAGE DEPENDENCIES:
>   Stage-2 is a root stage
>   Stage-1 depends on stages: Stage-2
>   Stage-0 depends on stages: Stage-1
> STAGE PLANS:
>   Stage: Stage-2
> Spark
>  A masked pattern was here 
>   Vertices:
> Map 1 
> Map Operator Tree:
> TableScan
>   alias: regular_table
>   Statistics: Num rows: 1 Data size: 1 Basic stats: COMPLETE 
> Column stats: NONE
>   Filter Operator
> predicate: col is not null (type: 

[jira] [Updated] (HIVE-17248) DPP isn't triggered if static pruning is done for one of the partition columns

[Sahil Takiar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Takiar updated HIVE-17248:

Description: 
Queries such as:

{code}
EXPLAIN select count(*) from srcpart join srcpart_date on (srcpart.ds = 
srcpart_date.ds) join srcpart_hour on (srcpart.hr = srcpart_hour.hr) 
where srcpart_date.`date` = '2008-04-08' and srcpart.hr = 13
{code}

Don't trigger DPP. {{srcpart}} is the partitioned by {{ds}} and {{hr}}. DPP 
could be triggered for the join {{(srcpart.ds = srcpart_date.ds)}}. I'm 
guessing its because static pruning is triggered for the {{(srcpart.hr = 
srcpart_hour.hr)}} join condition.

Affects Hive-on-Tez and Hive-on-Spark.

  was:
Queries such as:

{code}
EXPLAIN select count(*) from srcpart join srcpart_date on (srcpart.ds = 
srcpart_date.ds) join srcpart_hour on (srcpart.hr = srcpart_hour.hr) 
where srcpart_date.`date` = '2008-04-08' and srcpart.hr = 13
{code}

Where {{srcpart}} is the partitioned by {{ds}} and {{hr}}. DPP isn't triggered 
from for the join {{(srcpart.ds = srcpart_date.ds)}}, even though it could be. 
I'm guessing its because static pruning is triggered for the {{(srcpart.hr = 
srcpart_hour.hr)}} join condition.

Affects Hive-on-Tez and Hive-on-Spark.


> DPP isn't triggered if static pruning is done for one of the partition columns
> --
>
> Key: HIVE-17248
> URL: https://issues.apache.org/jira/browse/HIVE-17248
> Project: Hive
>  Issue Type: Bug
>Reporter: Sahil Takiar
>
> Queries such as:
> {code}
> EXPLAIN select count(*) from srcpart join srcpart_date on (srcpart.ds = 
> srcpart_date.ds) join srcpart_hour on (srcpart.hr = srcpart_hour.hr) 
> where srcpart_date.`date` = '2008-04-08' and srcpart.hr = 13
> {code}
> Don't trigger DPP. {{srcpart}} is the partitioned by {{ds}} and {{hr}}. DPP 
> could be triggered for the join {{(srcpart.ds = srcpart_date.ds)}}. I'm 
> guessing its because static pruning is triggered for the {{(srcpart.hr = 
> srcpart_hour.hr)}} join condition.
> Affects Hive-on-Tez and Hive-on-Spark.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-16811) Estimate statistics in absence of stats

[Vineet Garg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-16811:
---
Attachment: HIVE-16811.5.patch

> Estimate statistics in absence of stats
> ---
>
> Key: HIVE-16811
> URL: https://issues.apache.org/jira/browse/HIVE-16811
> Project: Hive
>  Issue Type: Improvement
>Reporter: Vineet Garg
>Assignee: Vineet Garg
> Attachments: HIVE-16811.1.patch, HIVE-16811.2.patch, 
> HIVE-16811.3.patch, HIVE-16811.4.patch, HIVE-16811.5.patch
>
>
> Currently Join ordering completely bails out in absence of statistics and 
> this could lead to bad joins such as cross joins.
> e.g. following select query will produce cross join.
> {code:sql}
> create table supplier (S_SUPPKEY INT, S_NAME STRING, S_ADDRESS STRING, 
> S_NATIONKEY INT, 
> S_PHONE STRING, S_ACCTBAL DOUBLE, S_COMMENT STRING)
> CREATE TABLE lineitem (L_ORDERKEY  INT,
> L_PARTKEY   INT,
> L_SUPPKEY   INT,
> L_LINENUMBERINT,
> L_QUANTITY  DOUBLE,
> L_EXTENDEDPRICE DOUBLE,
> L_DISCOUNT  DOUBLE,
> L_TAX   DOUBLE,
> L_RETURNFLAGSTRING,
> L_LINESTATUSSTRING,
> l_shipdate  STRING,
> L_COMMITDATESTRING,
> L_RECEIPTDATE   STRING,
> L_SHIPINSTRUCT  STRING,
> L_SHIPMODE  STRING,
> L_COMMENT   STRING) partitioned by (dl 
> int)
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY '|';
> CREATE TABLE part(
> p_partkey INT,
> p_name STRING,
> p_mfgr STRING,
> p_brand STRING,
> p_type STRING,
> p_size INT,
> p_container STRING,
> p_retailprice DOUBLE,
> p_comment STRING
> );
> explain select count(1) from part,supplier,lineitem where p_partkey = 
> l_partkey and s_suppkey = l_suppkey;
> {code}
> Estimating stats will prevent join ordering algorithm to bail out and come up 
> with join at least better than cross join 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-16811) Estimate statistics in absence of stats

[Vineet Garg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-16811:
---
Status: Open  (was: Patch Available)

> Estimate statistics in absence of stats
> ---
>
> Key: HIVE-16811
> URL: https://issues.apache.org/jira/browse/HIVE-16811
> Project: Hive
>  Issue Type: Improvement
>Reporter: Vineet Garg
>Assignee: Vineet Garg
> Attachments: HIVE-16811.1.patch, HIVE-16811.2.patch, 
> HIVE-16811.3.patch, HIVE-16811.4.patch
>
>
> Currently Join ordering completely bails out in absence of statistics and 
> this could lead to bad joins such as cross joins.
> e.g. following select query will produce cross join.
> {code:sql}
> create table supplier (S_SUPPKEY INT, S_NAME STRING, S_ADDRESS STRING, 
> S_NATIONKEY INT, 
> S_PHONE STRING, S_ACCTBAL DOUBLE, S_COMMENT STRING)
> CREATE TABLE lineitem (L_ORDERKEY  INT,
> L_PARTKEY   INT,
> L_SUPPKEY   INT,
> L_LINENUMBERINT,
> L_QUANTITY  DOUBLE,
> L_EXTENDEDPRICE DOUBLE,
> L_DISCOUNT  DOUBLE,
> L_TAX   DOUBLE,
> L_RETURNFLAGSTRING,
> L_LINESTATUSSTRING,
> l_shipdate  STRING,
> L_COMMITDATESTRING,
> L_RECEIPTDATE   STRING,
> L_SHIPINSTRUCT  STRING,
> L_SHIPMODE  STRING,
> L_COMMENT   STRING) partitioned by (dl 
> int)
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY '|';
> CREATE TABLE part(
> p_partkey INT,
> p_name STRING,
> p_mfgr STRING,
> p_brand STRING,
> p_type STRING,
> p_size INT,
> p_container STRING,
> p_retailprice DOUBLE,
> p_comment STRING
> );
> explain select count(1) from part,supplier,lineitem where p_partkey = 
> l_partkey and s_suppkey = l_suppkey;
> {code}
> Estimating stats will prevent join ordering algorithm to bail out and come up 
> with join at least better than cross join 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17246) Add having related blobstore query test

[Taklon Stephen Wu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Taklon Stephen Wu updated HIVE-17246:
-
Status: Patch Available  (was: In Progress)

> Add having related blobstore query test
> ---
>
> Key: HIVE-17246
> URL: https://issues.apache.org/jira/browse/HIVE-17246
> Project: Hive
>  Issue Type: Test
>Affects Versions: 2.1.1
>Reporter: Taklon Stephen Wu
>Assignee: Taklon Stephen Wu
> Attachments: HIVE-17246-2.patch, HIVE-17246.patch
>
>
> This patch introduces the following regression test into the hive-blobstore 
> qtest module:
> * having.q -> Test Having clause for aggregation functions such as COUNT(), 
> MAX(), and min()
> ** test HAVING with aggregation as alias
> ** test HAVING with aggregation as field
> ** test HAVING with aggregation as a function does not exist in field



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HIVE-17247) HoS DPP: UDFs on the partition column side does not evaluate correctly

[Sahil Takiar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Takiar reassigned HIVE-17247:
---


> HoS DPP: UDFs on the partition column side does not evaluate correctly
> --
>
> Key: HIVE-17247
> URL: https://issues.apache.org/jira/browse/HIVE-17247
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>
> Same problem as HIVE-12473 and HIVE-12667.
> The query below (uses tables from {{spark_dynamic_partition_pruning.q}}) 
> returns incorrect results:
> {code}
> select count(*) from srcpart join srcpart_date on (day(srcpart.ds) = 
> day(srcpart_date.ds)) where srcpart_date.`date` = '2008-04-08';
> {code}
> It returns a value of 0 when DPP is on, when it is disabled it returns 1000



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17208) Repl dump should pass in db/table information to authorization API

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated HIVE-17208:
--
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 3.0.0
   Status: Resolved  (was: Patch Available)

Test failures are not related. Patch pushed to master.

> Repl dump should pass in db/table information to authorization API
> --
>
> Key: HIVE-17208
> URL: https://issues.apache.org/jira/browse/HIVE-17208
> Project: Hive
>  Issue Type: Bug
>  Components: Authorization
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Fix For: 3.0.0
>
> Attachments: HIVE-17208.1.patch, HIVE-17208.2.patch, 
> HIVE-17208.3.patch
>
>
> "repl dump" does not provide db/table information. That is necessary for 
> authorization replication in ranger.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17246) Add having related blobstore query test

[Taklon Stephen Wu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Taklon Stephen Wu updated HIVE-17246:
-
Attachment: HIVE-17246-2.patch

> Add having related blobstore query test
> ---
>
> Key: HIVE-17246
> URL: https://issues.apache.org/jira/browse/HIVE-17246
> Project: Hive
>  Issue Type: Test
>Affects Versions: 2.1.1
>Reporter: Taklon Stephen Wu
>Assignee: Taklon Stephen Wu
> Attachments: HIVE-17246-2.patch, HIVE-17246.patch
>
>
> This patch introduces the following regression test into the hive-blobstore 
> qtest module:
> * having.q -> Test Having clause for aggregation functions such as COUNT(), 
> MAX(), and min()
> ** test HAVING with aggregation as alias
> ** test HAVING with aggregation as field
> ** test HAVING with aggregation as a function does not exist in field



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17246) Add having related blobstore query test

[Taklon Stephen Wu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Taklon Stephen Wu updated HIVE-17246:
-
Attachment: (was: HIVE-17246.patch)

> Add having related blobstore query test
> ---
>
> Key: HIVE-17246
> URL: https://issues.apache.org/jira/browse/HIVE-17246
> Project: Hive
>  Issue Type: Test
>Affects Versions: 2.1.1
>Reporter: Taklon Stephen Wu
>Assignee: Taklon Stephen Wu
> Attachments: HIVE-17246-2.patch, HIVE-17246.patch
>
>
> This patch introduces the following regression test into the hive-blobstore 
> qtest module:
> * having.q -> Test Having clause for aggregation functions such as COUNT(), 
> MAX(), and min()
> ** test HAVING with aggregation as alias
> ** test HAVING with aggregation as field
> ** test HAVING with aggregation as a function does not exist in field



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17246) Add having related blobstore query test

[Taklon Stephen Wu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Taklon Stephen Wu updated HIVE-17246:
-
Status: In Progress  (was: Patch Available)

> Add having related blobstore query test
> ---
>
> Key: HIVE-17246
> URL: https://issues.apache.org/jira/browse/HIVE-17246
> Project: Hive
>  Issue Type: Test
>Affects Versions: 2.1.1
>Reporter: Taklon Stephen Wu
>Assignee: Taklon Stephen Wu
> Attachments: HIVE-17246-2.patch, HIVE-17246.patch
>
>
> This patch introduces the following regression test into the hive-blobstore 
> qtest module:
> * having.q -> Test Having clause for aggregation functions such as COUNT(), 
> MAX(), and min()
> ** test HAVING with aggregation as alias
> ** test HAVING with aggregation as field
> ** test HAVING with aggregation as a function does not exist in field



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17246) Add having related blobstore query test

[Taklon Stephen Wu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Taklon Stephen Wu updated HIVE-17246:
-
Attachment: HIVE-17246.patch

> Add having related blobstore query test
> ---
>
> Key: HIVE-17246
> URL: https://issues.apache.org/jira/browse/HIVE-17246
> Project: Hive
>  Issue Type: Test
>Affects Versions: 2.1.1
>Reporter: Taklon Stephen Wu
>Assignee: Taklon Stephen Wu
> Attachments: HIVE-17246.patch, HIVE-17246.patch
>
>
> This patch introduces the following regression test into the hive-blobstore 
> qtest module:
> * having.q -> Test Having clause for aggregation functions such as COUNT(), 
> MAX(), and min()
> ** test HAVING with aggregation as alias
> ** test HAVING with aggregation as field
> ** test HAVING with aggregation as a function does not exist in field



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17246) Add having related blobstore query test

[Taklon Stephen Wu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Taklon Stephen Wu updated HIVE-17246:
-
Description: 
This patch introduces the following regression test into the hive-blobstore 
qtest module:
* having.q -> Test Having clause for aggregation functions such as COUNT(), 
MAX(), and min()
** test HAVING with aggregation as alias
** test HAVING with aggregation as field
** test HAVING with aggregation as a function does not exist in field

  was:
This patch introduces the following regression test into the hive-blobstore 
qtest module:
* having.q -> Test Having clause for aggregation functions such as count(), 
max(), and min()
** test Having with aggregation as alias
** test Having with aggregation as field
** test Having with aggregation as a function does not exist in field


> Add having related blobstore query test
> ---
>
> Key: HIVE-17246
> URL: https://issues.apache.org/jira/browse/HIVE-17246
> Project: Hive
>  Issue Type: Test
>Affects Versions: 2.1.1
>Reporter: Taklon Stephen Wu
>Assignee: Taklon Stephen Wu
> Attachments: HIVE-17246.patch, HIVE-17246.patch
>
>
> This patch introduces the following regression test into the hive-blobstore 
> qtest module:
> * having.q -> Test Having clause for aggregation functions such as COUNT(), 
> MAX(), and min()
> ** test HAVING with aggregation as alias
> ** test HAVING with aggregation as field
> ** test HAVING with aggregation as a function does not exist in field



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17213) HoS: file merging doesn't work for union all

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113741#comment-16113741
 ] 

Hive QA commented on HIVE-17213:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12880115/HIVE-17213.5.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 11132 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[insert_overwrite_local_directory_1]
 (batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_uncompressed] 
(batchId=56)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_union_merge]
 (batchId=169)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3] 
(batchId=99)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] 
(batchId=235)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
 (batchId=178)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
 (batchId=178)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation 
(batchId=178)
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testHttpRetryOnServerIdleTimeout 
(batchId=228)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6255/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6255/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6255/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12880115 - PreCommit-HIVE-Build

> HoS: file merging doesn't work for union all
> 
>
> Key: HIVE-17213
> URL: https://issues.apache.org/jira/browse/HIVE-17213
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-17213.0.patch, HIVE-17213.1.patch, 
> HIVE-17213.2.patch, HIVE-17213.3.patch, HIVE-17213.4.patch, HIVE-17213.5.patch
>
>
> HoS file merging doesn't work properly since it doesn't set linked file sinks 
> properly which is used to generate move tasks.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17242) Vectorized query execution for parquet tables on S3 fail with Timeout waiting for connection from pool exception

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113728#comment-16113728
 ] 

Vihang Karajgaonkar commented on HIVE-17242:


HI [~Ferd] I investigated this and in the end found that it a problem with my 
local dev environment (git conflict issue due to which close() method for 
VectorizedParquetRecordReader was missed.). When I fixed that, I don't see the 
issue anymore. Thanks for responding.

> Vectorized query execution for parquet tables on S3 fail with Timeout waiting 
> for connection from pool exception
> 
>
> Key: HIVE-17242
> URL: https://issues.apache.org/jira/browse/HIVE-17242
> Project: Hive
>  Issue Type: Bug
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>
> When I turn vectorization on with tables on S3 and using Hive-on-Spark many 
> of the TPCDS queries fail due to the error "Timeout waiting for connection 
> from pool exception" from S3AClient. This does not happen when I turn 
> vectorization off.
> Here is the exception trace I am seeing ..
> {noformat}
> Driver stacktrace:
>   at 
> org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1452)
>   at 
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1440)
>   at 
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1439)
>   at 
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
>   at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
>   at 
> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1439)
>   at 
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:811)
>   at 
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:811)
>   at scala.Option.foreach(Option.scala:257)
>   at 
> org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:811)
>   at 
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1665)
>   at 
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1620)
>   at 
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1609)
>   at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
> Caused by: java.io.IOException: java.lang.reflect.InvocationTargetException
>   at 
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
>   at 
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
>   at 
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:269)
>   at 
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.(HadoopShimsSecure.java:216)
>   at 
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getRecordReader(HadoopShimsSecure.java:343)
>   at 
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:681)
>   at org.apache.spark.rdd.HadoopRDD$$anon$1.(HadoopRDD.scala:245)
>   at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:208)
>   at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:101)
>   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
>   at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
>   at 
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
>   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
>   at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
>   at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:105)
>   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
>   at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
>   at 
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79)
>   at 
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)
>   at org.apache.spark.scheduler.Task.run(Task.scala:86)
>   at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.reflect.InvocationTargetException
>   at 

[jira] [Resolved] (HIVE-17242) Vectorized query execution for parquet tables on S3 fail with Timeout waiting for connection from pool exception

[Vihang Karajgaonkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vihang Karajgaonkar resolved HIVE-17242.

Resolution: Not A Problem

> Vectorized query execution for parquet tables on S3 fail with Timeout waiting 
> for connection from pool exception
> 
>
> Key: HIVE-17242
> URL: https://issues.apache.org/jira/browse/HIVE-17242
> Project: Hive
>  Issue Type: Bug
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>
> When I turn vectorization on with tables on S3 and using Hive-on-Spark many 
> of the TPCDS queries fail due to the error "Timeout waiting for connection 
> from pool exception" from S3AClient. This does not happen when I turn 
> vectorization off.
> Here is the exception trace I am seeing ..
> {noformat}
> Driver stacktrace:
>   at 
> org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1452)
>   at 
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1440)
>   at 
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1439)
>   at 
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
>   at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
>   at 
> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1439)
>   at 
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:811)
>   at 
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:811)
>   at scala.Option.foreach(Option.scala:257)
>   at 
> org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:811)
>   at 
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1665)
>   at 
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1620)
>   at 
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1609)
>   at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
> Caused by: java.io.IOException: java.lang.reflect.InvocationTargetException
>   at 
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
>   at 
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
>   at 
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:269)
>   at 
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.(HadoopShimsSecure.java:216)
>   at 
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getRecordReader(HadoopShimsSecure.java:343)
>   at 
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:681)
>   at org.apache.spark.rdd.HadoopRDD$$anon$1.(HadoopRDD.scala:245)
>   at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:208)
>   at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:101)
>   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
>   at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
>   at 
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
>   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
>   at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
>   at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:105)
>   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
>   at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
>   at 
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79)
>   at 
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)
>   at org.apache.spark.scheduler.Task.run(Task.scala:86)
>   at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.reflect.InvocationTargetException
>   at sun.reflect.GeneratedConstructorAccessor43.newInstance(Unknown 
> Source)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>   at 
> 

[jira] [Commented] (HIVE-17242) Vectorized query execution for parquet tables on S3 fail with Timeout waiting for connection from pool exception

[Ferdinand Xu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113721#comment-16113721
 ] 

Ferdinand Xu commented on HIVE-17242:
-

Not likely to cause by Vectorization, will try to reproduce it locally.

> Vectorized query execution for parquet tables on S3 fail with Timeout waiting 
> for connection from pool exception
> 
>
> Key: HIVE-17242
> URL: https://issues.apache.org/jira/browse/HIVE-17242
> Project: Hive
>  Issue Type: Bug
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>
> When I turn vectorization on with tables on S3 and using Hive-on-Spark many 
> of the TPCDS queries fail due to the error "Timeout waiting for connection 
> from pool exception" from S3AClient. This does not happen when I turn 
> vectorization off.
> Here is the exception trace I am seeing ..
> {noformat}
> Driver stacktrace:
>   at 
> org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1452)
>   at 
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1440)
>   at 
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1439)
>   at 
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
>   at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
>   at 
> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1439)
>   at 
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:811)
>   at 
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:811)
>   at scala.Option.foreach(Option.scala:257)
>   at 
> org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:811)
>   at 
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1665)
>   at 
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1620)
>   at 
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1609)
>   at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
> Caused by: java.io.IOException: java.lang.reflect.InvocationTargetException
>   at 
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
>   at 
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
>   at 
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:269)
>   at 
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.(HadoopShimsSecure.java:216)
>   at 
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getRecordReader(HadoopShimsSecure.java:343)
>   at 
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:681)
>   at org.apache.spark.rdd.HadoopRDD$$anon$1.(HadoopRDD.scala:245)
>   at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:208)
>   at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:101)
>   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
>   at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
>   at 
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
>   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
>   at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
>   at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:105)
>   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
>   at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
>   at 
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79)
>   at 
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)
>   at org.apache.spark.scheduler.Task.run(Task.scala:86)
>   at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.reflect.InvocationTargetException
>   at sun.reflect.GeneratedConstructorAccessor43.newInstance(Unknown 
> Source)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at 

[jira] [Commented] (HIVE-17220) Bloomfilter probing in semijoin reduction is thrashing L1 dcache

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113697#comment-16113697
 ] 

Hive QA commented on HIVE-17220:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12880158/HIVE-17220.3.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 11165 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_uncompressed] 
(batchId=56)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
 (batchId=168)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3] 
(batchId=99)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] 
(batchId=236)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
 (batchId=179)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
 (batchId=179)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation 
(batchId=179)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6254/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6254/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6254/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12880158 - PreCommit-HIVE-Build

> Bloomfilter probing in semijoin reduction is thrashing L1 dcache
> 
>
> Key: HIVE-17220
> URL: https://issues.apache.org/jira/browse/HIVE-17220
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-17220.1.patch, HIVE-17220.2.patch, 
> HIVE-17220.3.patch, HIVE-17220.WIP.patch
>
>
> [~gopalv] observed perf profiles showing bloomfilter probes as bottleneck for 
> some of the TPC-DS queries and resulted L1 data cache thrashing. 
> This is because of the huge bitset in bloom filter that doesn't fit in any 
> levels of cache, also the hash bits corresponding to a single key map to 
> different segments of bitset which are spread out. This can result in K-1 
> memory access (K being number of hash functions) in worst case for every key 
> that gets probed because of locality miss in L1 cache. 
> Ran a JMH microbenchmark to verify the same. Following is the JMH perf 
> profile for bloom filter probing
> {code}
> Perf stats:
> --
>5101.935637  task-clock (msec) #0.461 CPUs utilized
>346  context-switches  #0.068 K/sec
>336  cpu-migrations#0.066 K/sec
>  6,207  page-faults   #0.001 M/sec
> 10,016,486,301  cycles#1.963 GHz  
> (26.90%)
>  5,751,692,176  stalled-cycles-frontend   #   57.42% frontend cycles 
> idle (27.05%)
>  stalled-cycles-backend
> 14,359,914,397  instructions  #1.43  insns per cycle
>   #0.40  stalled cycles 
> per insn  (33.78%)
>  2,200,632,861  branches  #  431.333 M/sec
> (33.84%)
>  1,162,860  branch-misses #0.05% of all branches  
> (33.97%)
>  1,025,992,254  L1-dcache-loads   #  201.099 M/sec
> (26.56%)
>432,663,098  L1-dcache-load-misses #   42.17% of all L1-dcache 
> hits(14.49%)
>331,383,297  LLC-loads #   64.952 M/sec
> (14.47%)
>203,524  LLC-load-misses   #0.06% of all LL-cache 
> hits (21.67%)
>  L1-icache-loads
>  1,633,821  L1-icache-load-misses #0.320 M/sec
> (28.85%)
>950,368,796  dTLB-loads#  186.276 M/sec
> (28.61%)
>246,813,393  dTLB-load-misses  #   25.97% of all dTLB 
> cache hits   (14.53%)
> 25,451  iTLB-loads#0.005 M/sec
> (14.48%)
> 35,415  iTLB-load-misses  #  139.15% of all iTLB 
> cache hits   (21.73%)
>  L1-dcache-prefetches
>175,958  

[jira] [Commented] (HIVE-12369) Native Vector GroupBy

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113621#comment-16113621
 ] 

Sergey Shelukhin commented on HIVE-12369:
-

Finished the first iteration of the diff #3. 
My main comment is that the patch needs more comments, esp. for things that are 
used elsewhere and hard to understand without context.
Also left some logic-specific/etc. comments on RB. I didn't read all the code 
in great detail in the parts that look very similar for different types and 
cases. 

> Native Vector GroupBy
> -
>
> Key: HIVE-12369
> URL: https://issues.apache.org/jira/browse/HIVE-12369
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-12369.01.patch, HIVE-12369.02.patch, 
> HIVE-12369.05.patch, HIVE-12369.06.patch
>
>
> Implement Native Vector GroupBy using fast hash table technology developed 
> for Native Vector MapJoin, etc.
> Patch is currently limited to a single Long key, aggregation on Long columns, 
> no more than 31 columns.
> 3 new classes introduces that stored the count in the slot table and don't 
> allocate hash elements:
> {noformat}
>   COUNT(column)  VectorGroupByHashOneLongKeyCountColumnOperator  
>   COUNT(key) VectorGroupByHashOneLongKeyCountKeyOperator
>   COUNT(*)   VectorGroupByHashOneLongKeyCountStarOperator   
> {noformat}
> And a new class that aggregates a single Long key:
> {noformat}
>   VectorGroupByHashOneLongKeyOperator
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-15705) Event replication for constraints

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113616#comment-16113616
 ] 

Hive QA commented on HIVE-15705:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12880168/HIVE-15705.5.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6253/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6253/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6253/

Messages:
{noformat}
 This message was trimmed, see log for full details 
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/2.8.0/hadoop-common-2.8.0.jar(org/apache/hadoop/util/ToolRunner.class)]]
[loading 
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/CancellationException.class)]]
[loading 
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/RejectedExecutionException.class)]]
[loading 
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/SynchronousQueue.class)]]
[loading 
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/ThreadPoolExecutor.class)]]
[loading 
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/TimeUnit.class)]]
[loading 
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/Future.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/2.8.0/hadoop-common-2.8.0.jar(org/apache/hadoop/conf/Configured.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/2.8.0/hadoop-common-2.8.0.jar(org/apache/hadoop/io/NullWritable.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/2.8.0/hadoop-common-2.8.0.jar(org/apache/hadoop/io/Text.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-mapreduce-client-core/2.8.0/hadoop-mapreduce-client-core-2.8.0.jar(org/apache/hadoop/mapred/JobClient.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-mapreduce-client-core/2.8.0/hadoop-mapreduce-client-core-2.8.0.jar(org/apache/hadoop/mapred/JobConf.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-mapreduce-client-core/2.8.0/hadoop-mapreduce-client-core-2.8.0.jar(org/apache/hadoop/mapreduce/Job.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-mapreduce-client-core/2.8.0/hadoop-mapreduce-client-core-2.8.0.jar(org/apache/hadoop/mapreduce/JobID.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-mapreduce-client-core/2.8.0/hadoop-mapreduce-client-core-2.8.0.jar(org/apache/hadoop/mapreduce/lib/output/NullOutputFormat.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-mapreduce-client-core/2.8.0/hadoop-mapreduce-client-core-2.8.0.jar(org/apache/hadoop/mapreduce/security/token/delegation/DelegationTokenIdentifier.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/2.8.0/hadoop-common-2.8.0.jar(org/apache/hadoop/security/token/Token.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/2.8.0/hadoop-common-2.8.0.jar(org/apache/hadoop/util/Tool.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/thrift/libthrift/0.9.3/libthrift-0.9.3.jar(org/apache/thrift/TException.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/2.8.0/hadoop-common-2.8.0.jar(org/apache/hadoop/conf/Configurable.class)]]
[loading 
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/Callable.class)]]
[loading 
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/lang/InterruptedException.class)]]
[loading 
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/lang/Boolean.class)]]
[loading 
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/lang/ClassNotFoundException.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/ql/target/hive-exec-3.0.0-SNAPSHOT.jar(org/apache/hadoop/hive/ql/ErrorMsg.class)]]
[loading 
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/lang/Integer.class)]]
[loading 

[jira] [Commented] (HIVE-17208) Repl dump should pass in db/table information to authorization API

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113613#comment-16113613
 ] 

Hive QA commented on HIVE-17208:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12880082/HIVE-17208.3.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 11144 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_uncompressed] 
(batchId=56)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
 (batchId=168)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] 
(batchId=100)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] 
(batchId=236)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
 (batchId=179)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
 (batchId=179)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation 
(batchId=179)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testConnection (batchId=242)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testIsValid (batchId=242)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testIsValidNeg (batchId=242)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testNegativeProxyAuth 
(batchId=242)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testNegativeTokenAuth 
(batchId=242)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testProxyAuth (batchId=242)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testTokenAuth (batchId=242)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6252/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6252/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6252/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12880082 - PreCommit-HIVE-Build

> Repl dump should pass in db/table information to authorization API
> --
>
> Key: HIVE-17208
> URL: https://issues.apache.org/jira/browse/HIVE-17208
> Project: Hive
>  Issue Type: Bug
>  Components: Authorization
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Attachments: HIVE-17208.1.patch, HIVE-17208.2.patch, 
> HIVE-17208.3.patch
>
>
> "repl dump" does not provide db/table information. That is necessary for 
> authorization replication in ranger.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17243) Replace Hive's JavaDataModel with Java Object Layout

[Prasanth Jayachandran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113573#comment-16113573
 ] 

Prasanth Jayachandran commented on HIVE-17243:
--

Can't do this because of licensing issue of jol-core.

> Replace Hive's JavaDataModel with Java Object Layout
> 
>
> Key: HIVE-17243
> URL: https://issues.apache.org/jira/browse/HIVE-17243
> Project: Hive
>  Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>
> org.apache.hadoop.hive.ql.util.JavaDataModel is used for estimating memory of 
> objects in many places. The problem with this approach is that manual 
> accounting for all fields and references has to be done. This will also be 
> problematic for cases where shallow object size vs deep object sizes are 
> required. 
> Hash table memory monitoring does accounting for size of the hash tables. The 
> estimated sizes of hash tables are often very different for non-vector 
> operators whereas the estimates are close to actual object size for 
> vectorized operators. Also addition of fields requires manual changes to 
> memory monitoring. 
> Java object layout is openjdk project that can provide shallow and deep 
> object sizes without using java agents. We can leverage that for much more 
> accurate memory estimates.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17246) Add having related blobstore query test

[Taklon Stephen Wu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Taklon Stephen Wu updated HIVE-17246:
-
Status: Patch Available  (was: Open)

> Add having related blobstore query test
> ---
>
> Key: HIVE-17246
> URL: https://issues.apache.org/jira/browse/HIVE-17246
> Project: Hive
>  Issue Type: Test
>Affects Versions: 2.1.1
>Reporter: Taklon Stephen Wu
>Assignee: Taklon Stephen Wu
> Attachments: HIVE-17246.patch
>
>
> This patch introduces the following regression test into the hive-blobstore 
> qtest module:
> * having.q -> Test Having clause for aggregation functions such as count(), 
> max(), and min()
> ** test Having with aggregation as alias
> ** test Having with aggregation as field
> ** test Having with aggregation as a function does not exist in field



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17246) Add having related blobstore query test

[Taklon Stephen Wu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Taklon Stephen Wu updated HIVE-17246:
-
Attachment: HIVE-17246.patch

> Add having related blobstore query test
> ---
>
> Key: HIVE-17246
> URL: https://issues.apache.org/jira/browse/HIVE-17246
> Project: Hive
>  Issue Type: Test
>Affects Versions: 2.1.1
>Reporter: Taklon Stephen Wu
>Assignee: Taklon Stephen Wu
> Attachments: HIVE-17246.patch
>
>
> This patch introduces the following regression test into the hive-blobstore 
> qtest module:
> * having.q -> Test Having clause for aggregation functions such as count(), 
> max(), and min()
> ** test Having with aggregation as alias
> ** test Having with aggregation as field
> ** test Having with aggregation as a function does not exist in field



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17246) Add having related blobstore query test

[Taklon Stephen Wu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Taklon Stephen Wu updated HIVE-17246:
-
Status: Open  (was: Patch Available)

> Add having related blobstore query test
> ---
>
> Key: HIVE-17246
> URL: https://issues.apache.org/jira/browse/HIVE-17246
> Project: Hive
>  Issue Type: Test
>Affects Versions: 2.1.1
>Reporter: Taklon Stephen Wu
>Assignee: Taklon Stephen Wu
> Attachments: HIVE-17246.patch
>
>
> This patch introduces the following regression test into the hive-blobstore 
> qtest module:
> * having.q -> Test Having clause for aggregation functions such as count(), 
> max(), and min()
> ** test Having with aggregation as alias
> ** test Having with aggregation as field
> ** test Having with aggregation as a function does not exist in field



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17246) Add having related blobstore query test

[Taklon Stephen Wu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Taklon Stephen Wu updated HIVE-17246:
-
Fix Version/s: (was: 3.0.0)

> Add having related blobstore query test
> ---
>
> Key: HIVE-17246
> URL: https://issues.apache.org/jira/browse/HIVE-17246
> Project: Hive
>  Issue Type: Test
>Affects Versions: 2.1.1
>Reporter: Taklon Stephen Wu
>Assignee: Taklon Stephen Wu
> Attachments: HIVE-17246.patch
>
>
> This patch introduces the following regression test into the hive-blobstore 
> qtest module:
> * having.q -> Test Having clause for aggregation functions such as count(), 
> max(), and min()
> ** test Having with aggregation as alias
> ** test Having with aggregation as field
> ** test Having with aggregation as a function does not exist in field



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-16294) Support snapshot for truncate table

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113518#comment-16113518
 ] 

Hive QA commented on HIVE-16294:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12880218/HIVE-16294.03.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 11143 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_uncompressed] 
(batchId=56)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_ppd_decimal] 
(batchId=9)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
 (batchId=168)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3] 
(batchId=99)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] 
(batchId=236)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] 
(batchId=236)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
 (batchId=179)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
 (batchId=179)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation 
(batchId=179)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6251/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6251/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6251/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12880218 - PreCommit-HIVE-Build

> Support snapshot for truncate table
> ---
>
> Key: HIVE-16294
> URL: https://issues.apache.org/jira/browse/HIVE-16294
> Project: Hive
>  Issue Type: Sub-task
>  Components: Query Processor
>Reporter: Vihang Karajgaonkar
>Assignee: Barna Zsombor Klara
> Attachments: HIVE-16294.01.patch, HIVE-16294.02.patch, 
> HIVE-16294.03.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17246) Add having related blobstore query test

[Taklon Stephen Wu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Taklon Stephen Wu updated HIVE-17246:
-
Fix Version/s: 3.0.0
   Status: Patch Available  (was: Open)

> Add having related blobstore query test
> ---
>
> Key: HIVE-17246
> URL: https://issues.apache.org/jira/browse/HIVE-17246
> Project: Hive
>  Issue Type: Test
>Affects Versions: 2.1.1
>Reporter: Taklon Stephen Wu
>Assignee: Taklon Stephen Wu
> Fix For: 3.0.0
>
>
> This patch introduces the following regression test into the hive-blobstore 
> qtest module:
> * having.q -> Test Having clause for aggregation functions such as count(), 
> max(), and min()
> ** test Having with aggregation as alias
> ** test Having with aggregation as field
> ** test Having with aggregation as a function does not exist in field



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17246) Add having related blobstore query test

[Taklon Stephen Wu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Taklon Stephen Wu updated HIVE-17246:
-
Description: 
This patch introduces the following regression test into the hive-blobstore 
qtest module:
* having.q -> Test Having clause for aggregation functions such as count(), 
max(), and min()
** test Having with aggregation as alias
** test Having with aggregation as field
** test Having with aggregation as a function does not exist in field

> Add having related blobstore query test
> ---
>
> Key: HIVE-17246
> URL: https://issues.apache.org/jira/browse/HIVE-17246
> Project: Hive
>  Issue Type: Test
>Affects Versions: 2.1.1
>Reporter: Taklon Stephen Wu
>Assignee: Taklon Stephen Wu
>
> This patch introduces the following regression test into the hive-blobstore 
> qtest module:
> * having.q -> Test Having clause for aggregation functions such as count(), 
> max(), and min()
> ** test Having with aggregation as alias
> ** test Having with aggregation as field
> ** test Having with aggregation as a function does not exist in field



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17246) Add having related blobstore query test

[Taklon Stephen Wu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Taklon Stephen Wu updated HIVE-17246:
-
Docs Text:   (was: This patch introduces the following regression test into 
the hive-blobstore qtest module:
* having.q -> Test Having clause for aggregation functions such as count(), 
max(), and min()
** test Having with aggregation as alias
** test Having with aggregation as field
** test Having with aggregation as a function does not exist in field)

> Add having related blobstore query test
> ---
>
> Key: HIVE-17246
> URL: https://issues.apache.org/jira/browse/HIVE-17246
> Project: Hive
>  Issue Type: Test
>Affects Versions: 2.1.1
>Reporter: Taklon Stephen Wu
>Assignee: Taklon Stephen Wu
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HIVE-17246) Add having related blobstore query test

[Taklon Stephen Wu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Taklon Stephen Wu reassigned HIVE-17246:



> Add having related blobstore query test
> ---
>
> Key: HIVE-17246
> URL: https://issues.apache.org/jira/browse/HIVE-17246
> Project: Hive
>  Issue Type: Test
>Affects Versions: 2.1.1
>Reporter: Taklon Stephen Wu
>Assignee: Taklon Stephen Wu
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HIVE-17245) Replace MemoryEstimate interface with JOL

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran reassigned HIVE-17245:



> Replace MemoryEstimate interface with JOL
> -
>
> Key: HIVE-17245
> URL: https://issues.apache.org/jira/browse/HIVE-17245
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>
> Replace MemoryEstimate interface used by hash table loading memory monitor 
> with JOL for better memory estimation. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17243) Replace Hive's JavaDataModel with Java Object Layout

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-17243:
-
Summary: Replace Hive's JavaDataModel with Java Object Layout  (was: 
Replace Hive's DataModel with Java Object Layout)

> Replace Hive's JavaDataModel with Java Object Layout
> 
>
> Key: HIVE-17243
> URL: https://issues.apache.org/jira/browse/HIVE-17243
> Project: Hive
>  Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>
> org.apache.hadoop.hive.ql.util.JavaDataModel is used for estimating memory of 
> objects in many places. The problem with this approach is that manual 
> accounting for all fields and references has to be done. This will also be 
> problematic for cases where shallow object size vs deep object sizes are 
> required. 
> Hash table memory monitoring does accounting for size of the hash tables. The 
> estimated sizes of hash tables are often very different for non-vector 
> operators whereas the estimates are close to actual object size for 
> vectorized operators. Also addition of fields requires manual changes to 
> memory monitoring. 
> Java object layout is openjdk project that can provide shallow and deep 
> object sizes without using java agents. We can leverage that for much more 
> accurate memory estimates.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HIVE-17243) Replace Hive's DataModel with Java Object Layout

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran reassigned HIVE-17243:



> Replace Hive's DataModel with Java Object Layout
> 
>
> Key: HIVE-17243
> URL: https://issues.apache.org/jira/browse/HIVE-17243
> Project: Hive
>  Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>
> org.apache.hadoop.hive.ql.util.JavaDataModel is used for estimating memory of 
> objects in many places. The problem with this approach is that manual 
> accounting for all fields and references has to be done. This will also be 
> problematic for cases where shallow object size vs deep object sizes are 
> required. 
> Hash table memory monitoring does accounting for size of the hash tables. The 
> estimated sizes of hash tables are often very different for non-vector 
> operators whereas the estimates are close to actual object size for 
> vectorized operators. Also addition of fields requires manual changes to 
> memory monitoring. 
> Java object layout is openjdk project that can provide shallow and deep 
> object sizes without using java agents. We can leverage that for much more 
> accurate memory estimates.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-16294) Support snapshot for truncate table

[Andrew Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113423#comment-16113423
 ] 

Andrew Wang commented on HIVE-16294:


Thanks for the ping. Yes, looks like an oversight in the HDFS API. We expose 
this listing API on the CLI but not via a programmatic interface. We should add 
it to HdfsAdmin alongside the other snapshot-related APIs.

Filed HDFS-12257 for tracking, let's try and find someone to pick it up.

> Support snapshot for truncate table
> ---
>
> Key: HIVE-16294
> URL: https://issues.apache.org/jira/browse/HIVE-16294
> Project: Hive
>  Issue Type: Sub-task
>  Components: Query Processor
>Reporter: Vihang Karajgaonkar
>Assignee: Barna Zsombor Klara
> Attachments: HIVE-16294.01.patch, HIVE-16294.02.patch, 
> HIVE-16294.03.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17242) Vectorized query execution for parquet tables on S3 fail with Timeout waiting for connection from pool exception

[Sahil Takiar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113387#comment-16113387
 ] 

Sahil Takiar commented on HIVE-17242:
-

Haven't seen this before, not sure why it would be vectorization specific.

> Vectorized query execution for parquet tables on S3 fail with Timeout waiting 
> for connection from pool exception
> 
>
> Key: HIVE-17242
> URL: https://issues.apache.org/jira/browse/HIVE-17242
> Project: Hive
>  Issue Type: Bug
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>
> When I turn vectorization on with tables on S3 and using Hive-on-Spark many 
> of the TPCDS queries fail due to the error "Timeout waiting for connection 
> from pool exception" from S3AClient. This does not happen when I turn 
> vectorization off.
> Here is the exception trace I am seeing ..
> {noformat}
> Driver stacktrace:
>   at 
> org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1452)
>   at 
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1440)
>   at 
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1439)
>   at 
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
>   at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
>   at 
> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1439)
>   at 
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:811)
>   at 
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:811)
>   at scala.Option.foreach(Option.scala:257)
>   at 
> org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:811)
>   at 
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1665)
>   at 
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1620)
>   at 
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1609)
>   at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
> Caused by: java.io.IOException: java.lang.reflect.InvocationTargetException
>   at 
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
>   at 
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
>   at 
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:269)
>   at 
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.(HadoopShimsSecure.java:216)
>   at 
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getRecordReader(HadoopShimsSecure.java:343)
>   at 
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:681)
>   at org.apache.spark.rdd.HadoopRDD$$anon$1.(HadoopRDD.scala:245)
>   at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:208)
>   at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:101)
>   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
>   at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
>   at 
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
>   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
>   at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
>   at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:105)
>   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
>   at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
>   at 
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79)
>   at 
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)
>   at org.apache.spark.scheduler.Task.run(Task.scala:86)
>   at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.reflect.InvocationTargetException
>   at sun.reflect.GeneratedConstructorAccessor43.newInstance(Unknown 
> Source)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at 

[jira] [Commented] (HIVE-16820) TezTask may not shut down correctly before submit

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113361#comment-16113361
 ] 

Sergey Shelukhin commented on HIVE-16820:
-

Thanks! I'd be happy to review

> TezTask may not shut down correctly before submit
> -
>
> Key: HIVE-16820
> URL: https://issues.apache.org/jira/browse/HIVE-16820
> Project: Hive
>  Issue Type: Bug
>Reporter: Visakh Nair
>Assignee: Sergey Shelukhin
> Fix For: 3.0.0
>
> Attachments: HIVE-16820.01.patch, HIVE-16820.patch
>
>
> The query will run and only fail at the very end when the driver checks its 
> own shutdown flag.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-16820) TezTask may not shut down correctly before submit

[Mithun Radhakrishnan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113354#comment-16113354
 ] 

Mithun Radhakrishnan commented on HIVE-16820:
-

bq. Yeah, what I was saying is that we don't need the same bugfix for that task 
...
Right. Sorry, I didn't mean to imply otherwise.

bq. It probably does need an implementation (without bugs like this).
I'm working on it. Our fix for this was not as complete/correct as yours. I'll 
extend this for {{MergeFileTask}}.

> TezTask may not shut down correctly before submit
> -
>
> Key: HIVE-16820
> URL: https://issues.apache.org/jira/browse/HIVE-16820
> Project: Hive
>  Issue Type: Bug
>Reporter: Visakh Nair
>Assignee: Sergey Shelukhin
> Fix For: 3.0.0
>
> Attachments: HIVE-16820.01.patch, HIVE-16820.patch
>
>
> The query will run and only fail at the very end when the driver checks its 
> own shutdown flag.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17188) ObjectStore runs out of memory for large batches of addPartitions().

[Mithun Radhakrishnan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113351#comment-16113351
 ] 

Mithun Radhakrishnan commented on HIVE-17188:
-

[~leftylev]: Thank you for correcting me. I've added 2.4, as per your advice.

bq. would probably make sense to merge this in branch-2.3 as well...
A fair point, [~vihangk1]. [~pxiong], do I have your permission to merge this 
into {{branch-2.3}}?

> ObjectStore runs out of memory for large batches of addPartitions().
> 
>
> Key: HIVE-17188
> URL: https://issues.apache.org/jira/browse/HIVE-17188
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 2.2.0
>Reporter: Mithun Radhakrishnan
>Assignee: Chris Drome
> Fix For: 2.2.0, 3.0.0, 2.4.0
>
> Attachments: HIVE-17188.1.patch
>
>
> For large batches (e.g. hundreds) of {{addPartitions()}}, the {{ObjectStore}} 
> runs out of memory. Flushing the {{PersistenceManager}} alleviates the 
> problem.
> Note: The problem being addressed here isn't so much with the size of the 
> hundreds of Partition objects, but the cruft that builds with the 
> PersistenceManager, in the JDO layer, as confirmed through memory-profiling.
> (Raising this on behalf of [~cdrome] and [~thiruvel].)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17220) Bloomfilter probing in semijoin reduction is thrashing L1 dcache

[Prasanth Jayachandran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113332#comment-16113332
 ] 

Prasanth Jayachandran commented on HIVE-17220:
--

Thanks Gopal.
Scheduled ptest again for retry. 

> Bloomfilter probing in semijoin reduction is thrashing L1 dcache
> 
>
> Key: HIVE-17220
> URL: https://issues.apache.org/jira/browse/HIVE-17220
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-17220.1.patch, HIVE-17220.2.patch, 
> HIVE-17220.3.patch, HIVE-17220.WIP.patch
>
>
> [~gopalv] observed perf profiles showing bloomfilter probes as bottleneck for 
> some of the TPC-DS queries and resulted L1 data cache thrashing. 
> This is because of the huge bitset in bloom filter that doesn't fit in any 
> levels of cache, also the hash bits corresponding to a single key map to 
> different segments of bitset which are spread out. This can result in K-1 
> memory access (K being number of hash functions) in worst case for every key 
> that gets probed because of locality miss in L1 cache. 
> Ran a JMH microbenchmark to verify the same. Following is the JMH perf 
> profile for bloom filter probing
> {code}
> Perf stats:
> --
>5101.935637  task-clock (msec) #0.461 CPUs utilized
>346  context-switches  #0.068 K/sec
>336  cpu-migrations#0.066 K/sec
>  6,207  page-faults   #0.001 M/sec
> 10,016,486,301  cycles#1.963 GHz  
> (26.90%)
>  5,751,692,176  stalled-cycles-frontend   #   57.42% frontend cycles 
> idle (27.05%)
>  stalled-cycles-backend
> 14,359,914,397  instructions  #1.43  insns per cycle
>   #0.40  stalled cycles 
> per insn  (33.78%)
>  2,200,632,861  branches  #  431.333 M/sec
> (33.84%)
>  1,162,860  branch-misses #0.05% of all branches  
> (33.97%)
>  1,025,992,254  L1-dcache-loads   #  201.099 M/sec
> (26.56%)
>432,663,098  L1-dcache-load-misses #   42.17% of all L1-dcache 
> hits(14.49%)
>331,383,297  LLC-loads #   64.952 M/sec
> (14.47%)
>203,524  LLC-load-misses   #0.06% of all LL-cache 
> hits (21.67%)
>  L1-icache-loads
>  1,633,821  L1-icache-load-misses #0.320 M/sec
> (28.85%)
>950,368,796  dTLB-loads#  186.276 M/sec
> (28.61%)
>246,813,393  dTLB-load-misses  #   25.97% of all dTLB 
> cache hits   (14.53%)
> 25,451  iTLB-loads#0.005 M/sec
> (14.48%)
> 35,415  iTLB-load-misses  #  139.15% of all iTLB 
> cache hits   (21.73%)
>  L1-dcache-prefetches
>175,958  L1-dcache-prefetch-misses #0.034 M/sec
> (28.94%)
>   11.064783140 seconds time elapsed
> {code}
> This shows 42.17% of L1 data cache misses. 
> This jira is to use cache efficient bloom filter for semijoin probing.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17240) function acos(2) should be null

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113316#comment-16113316
 ] 

Hive QA commented on HIVE-17240:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12880205/HIVE-17240.1.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 13 failed/errored test(s), 11143 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[materialized_view_create_rewrite]
 (batchId=241)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[decimal_udf2] 
(batchId=84)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_uncompressed] 
(batchId=56)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal_udf2] 
(batchId=70)
org.apache.hadoop.hive.cli.TestCompareCliDriver.testCliDriver[vectorized_math_funcs]
 (batchId=237)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_schema_evol_3a]
 (batchId=142)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_decimal_udf2]
 (batchId=159)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[schemeAuthority]
 (batchId=169)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
 (batchId=168)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3] 
(batchId=99)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
 (batchId=179)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
 (batchId=179)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation 
(batchId=179)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6249/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6249/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6249/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 13 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12880205 - PreCommit-HIVE-Build

> function acos(2) should be null
> ---
>
> Key: HIVE-17240
> URL: https://issues.apache.org/jira/browse/HIVE-17240
> Project: Hive
>  Issue Type: Bug
>  Components: UDF
>Affects Versions: 1.1.1, 1.2.2, 2.2.0
>Reporter: Yuming Wang
>Assignee: Yuming Wang
> Attachments: HIVE-17240.1.patch
>
>
> {{acos(2)}} should be null, same as MySQL:
> {code:sql}
> hive> desc function extended acos;
> OK
> acos(x) - returns the arc cosine of x if -1<=x<=1 or NULL otherwise
> Example:
>   > SELECT acos(1) FROM src LIMIT 1;
>   0
>   > SELECT acos(2) FROM src LIMIT 1;
>   NULL
> Time taken: 0.009 seconds, Fetched: 6 row(s)
> hive> select acos(2);
> OK
> NaN
> Time taken: 0.437 seconds, Fetched: 1 row(s)
> {code}
> {code:sql}
> mysql>  select acos(2);
> +-+
> | acos(2) |
> +-+
> |NULL |
> +-+
> 1 row in set (0.00 sec)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17220) Bloomfilter probing in semijoin reduction is thrashing L1 dcache

[Gopal V (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113279#comment-16113279
 ] 

Gopal V commented on HIVE-17220:


{code}
[ERROR] symbol:   class CopyUtils
[ERROR] location: class 
org.apache.hadoop.hive.ql.parse.repl.dump.io.FileOperations
{code}

Build failure unrelated to this patch.

> Bloomfilter probing in semijoin reduction is thrashing L1 dcache
> 
>
> Key: HIVE-17220
> URL: https://issues.apache.org/jira/browse/HIVE-17220
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-17220.1.patch, HIVE-17220.2.patch, 
> HIVE-17220.3.patch, HIVE-17220.WIP.patch
>
>
> [~gopalv] observed perf profiles showing bloomfilter probes as bottleneck for 
> some of the TPC-DS queries and resulted L1 data cache thrashing. 
> This is because of the huge bitset in bloom filter that doesn't fit in any 
> levels of cache, also the hash bits corresponding to a single key map to 
> different segments of bitset which are spread out. This can result in K-1 
> memory access (K being number of hash functions) in worst case for every key 
> that gets probed because of locality miss in L1 cache. 
> Ran a JMH microbenchmark to verify the same. Following is the JMH perf 
> profile for bloom filter probing
> {code}
> Perf stats:
> --
>5101.935637  task-clock (msec) #0.461 CPUs utilized
>346  context-switches  #0.068 K/sec
>336  cpu-migrations#0.066 K/sec
>  6,207  page-faults   #0.001 M/sec
> 10,016,486,301  cycles#1.963 GHz  
> (26.90%)
>  5,751,692,176  stalled-cycles-frontend   #   57.42% frontend cycles 
> idle (27.05%)
>  stalled-cycles-backend
> 14,359,914,397  instructions  #1.43  insns per cycle
>   #0.40  stalled cycles 
> per insn  (33.78%)
>  2,200,632,861  branches  #  431.333 M/sec
> (33.84%)
>  1,162,860  branch-misses #0.05% of all branches  
> (33.97%)
>  1,025,992,254  L1-dcache-loads   #  201.099 M/sec
> (26.56%)
>432,663,098  L1-dcache-load-misses #   42.17% of all L1-dcache 
> hits(14.49%)
>331,383,297  LLC-loads #   64.952 M/sec
> (14.47%)
>203,524  LLC-load-misses   #0.06% of all LL-cache 
> hits (21.67%)
>  L1-icache-loads
>  1,633,821  L1-icache-load-misses #0.320 M/sec
> (28.85%)
>950,368,796  dTLB-loads#  186.276 M/sec
> (28.61%)
>246,813,393  dTLB-load-misses  #   25.97% of all dTLB 
> cache hits   (14.53%)
> 25,451  iTLB-loads#0.005 M/sec
> (14.48%)
> 35,415  iTLB-load-misses  #  139.15% of all iTLB 
> cache hits   (21.73%)
>  L1-dcache-prefetches
>175,958  L1-dcache-prefetch-misses #0.034 M/sec
> (28.94%)
>   11.064783140 seconds time elapsed
> {code}
> This shows 42.17% of L1 data cache misses. 
> This jira is to use cache efficient bloom filter for semijoin probing.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-15305) Add tests for METASTORE_EVENT_LISTENERS

[Mohit Sabharwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113276#comment-16113276
 ] 

Mohit Sabharwal commented on HIVE-15305:


Thanks, [~vgumashta]. Re-attaching the patch. Appreciate any insights you have 
on the sqlInsertPartition failure, which is passing for me locally. 
{{assertEquals(24, rsp.getEventsSize());}} is failing with rsp.getEventsSize() 
returning 17.

> Add tests for METASTORE_EVENT_LISTENERS
> ---
>
> Key: HIVE-15305
> URL: https://issues.apache.org/jira/browse/HIVE-15305
> Project: Hive
>  Issue Type: Bug
>Reporter: Mohit Sabharwal
>Assignee: Mohit Sabharwal
> Attachments: HIVE-15305.1.patch, HIVE-15305.1.patch, 
> HIVE-15305.1.patch, HIVE-15305.patch
>
>
> HIVE-15232 reused TestDbNotificationListener to test 
> METASTORE_TRANSACTIONAL_EVENT_LISTENERS and removed unit testing of 
> METASTORE_EVENT_LISTENERS config. We should test both. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-15305) Add tests for METASTORE_EVENT_LISTENERS

[Mohit Sabharwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mohit Sabharwal updated HIVE-15305:
---
Attachment: HIVE-15305.1.patch

> Add tests for METASTORE_EVENT_LISTENERS
> ---
>
> Key: HIVE-15305
> URL: https://issues.apache.org/jira/browse/HIVE-15305
> Project: Hive
>  Issue Type: Bug
>Reporter: Mohit Sabharwal
>Assignee: Mohit Sabharwal
> Attachments: HIVE-15305.1.patch, HIVE-15305.1.patch, 
> HIVE-15305.1.patch, HIVE-15305.patch
>
>
> HIVE-15232 reused TestDbNotificationListener to test 
> METASTORE_TRANSACTIONAL_EVENT_LISTENERS and removed unit testing of 
> METASTORE_EVENT_LISTENERS config. We should test both. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-16895) Multi-threaded execution of bootstrap dump of partitions

[Daniel Dai (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113265#comment-16113265
 ] 

Daniel Dai commented on HIVE-16895:
---

The new file needs a license header.

>  Multi-threaded execution of bootstrap dump of partitions
> -
>
> Key: HIVE-16895
> URL: https://issues.apache.org/jira/browse/HIVE-16895
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: anishek
>Assignee: anishek
> Fix For: 3.0.0
>
> Attachments: HIVE-16895.1.patch
>
>
> to allow faster execution of bootstrap dump phase we dump multiple partitions 
> from same table simultaneously. 
> even though dumping  functions is  not going to be a blocker, moving to 
> similar execution modes for all metastore objects will make code more 
> coherent. 
> Bootstrap dump at db level does :
> * boostrap of all tables
> ** boostrap of all partitions in a table.  (scope of current jira) 
> * boostrap of all functions 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17234) Remove HBase metastore from master

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113260#comment-16113260
 ] 

Sergey Shelukhin commented on HIVE-17234:
-

+1, I'm assuming any green in the pull request is just the code that was 
relocated

> Remove HBase metastore from master
> --
>
> Key: HIVE-17234
> URL: https://issues.apache.org/jira/browse/HIVE-17234
> Project: Hive
>  Issue Type: Task
>  Components: HBase Metastore
>Affects Versions: 3.0.0
>Reporter: Alan Gates
>Assignee: Alan Gates
> Attachments: HIVE-17234.patch
>
>
> No new development has been done on the HBase metastore in at least a year, 
> and to my knowledge no one is using it (nor is it even in a state to be fully 
> usable).  Given the lack of interest in continuing to develop it, we should 
> remove it rather than leave dead code hanging around and extra tests taking 
> up time in test runs.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17241) Change metastore classes to not use the shims

[Vaibhav Gumashta (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113256#comment-16113256
 ] 

Vaibhav Gumashta commented on HIVE-17241:
-

[~alangates] My opinion would be to not let Hive (HS2) depend on metastore for 
this. Would be better to keep it separate and cleaner.

> Change metastore classes to not use the shims
> -
>
> Key: HIVE-17241
> URL: https://issues.apache.org/jira/browse/HIVE-17241
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>
> As part of moving the metastore into a standalone package, it will no longer 
> have access to the shims.  This means we need to either copy them or access 
> the underlying Hadoop operations directly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-15305) Add tests for METASTORE_EVENT_LISTENERS

[Vaibhav Gumashta (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113246#comment-16113246
 ] 

Vaibhav Gumashta commented on HIVE-15305:
-

[~mohitsabharwal] Thanks for the update, the changes look good, but the test 
output seems to be gone now (wanted to take a look at 
TestTransactionalDbNotificationListener.sqlInsertPartition). Can you submit for 
another run?

> Add tests for METASTORE_EVENT_LISTENERS
> ---
>
> Key: HIVE-15305
> URL: https://issues.apache.org/jira/browse/HIVE-15305
> Project: Hive
>  Issue Type: Bug
>Reporter: Mohit Sabharwal
>Assignee: Mohit Sabharwal
> Attachments: HIVE-15305.1.patch, HIVE-15305.1.patch, HIVE-15305.patch
>
>
> HIVE-15232 reused TestDbNotificationListener to test 
> METASTORE_TRANSACTIONAL_EVENT_LISTENERS and removed unit testing of 
> METASTORE_EVENT_LISTENERS config. We should test both. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17234) Remove HBase metastore from master

[Daniel Dai (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113242#comment-16113242
 ] 

Daniel Dai commented on HIVE-17234:
---

+1

> Remove HBase metastore from master
> --
>
> Key: HIVE-17234
> URL: https://issues.apache.org/jira/browse/HIVE-17234
> Project: Hive
>  Issue Type: Task
>  Components: HBase Metastore
>Affects Versions: 3.0.0
>Reporter: Alan Gates
>Assignee: Alan Gates
> Attachments: HIVE-17234.patch
>
>
> No new development has been done on the HBase metastore in at least a year, 
> and to my knowledge no one is using it (nor is it even in a state to be fully 
> usable).  Given the lack of interest in continuing to develop it, we should 
> remove it rather than leave dead code hanging around and extra tests taking 
> up time in test runs.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17240) function acos(2) should be null

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113227#comment-16113227
 ] 

Sergey Shelukhin commented on HIVE-17240:
-

I didn't realize NaN was a thing in Hive. 
+1 cc [~ashutoshc]

> function acos(2) should be null
> ---
>
> Key: HIVE-17240
> URL: https://issues.apache.org/jira/browse/HIVE-17240
> Project: Hive
>  Issue Type: Bug
>  Components: UDF
>Affects Versions: 1.1.1, 1.2.2, 2.2.0
>Reporter: Yuming Wang
>Assignee: Yuming Wang
> Attachments: HIVE-17240.1.patch
>
>
> {{acos(2)}} should be null, same as MySQL:
> {code:sql}
> hive> desc function extended acos;
> OK
> acos(x) - returns the arc cosine of x if -1<=x<=1 or NULL otherwise
> Example:
>   > SELECT acos(1) FROM src LIMIT 1;
>   0
>   > SELECT acos(2) FROM src LIMIT 1;
>   NULL
> Time taken: 0.009 seconds, Fetched: 6 row(s)
> hive> select acos(2);
> OK
> NaN
> Time taken: 0.437 seconds, Fetched: 1 row(s)
> {code}
> {code:sql}
> mysql>  select acos(2);
> +-+
> | acos(2) |
> +-+
> |NULL |
> +-+
> 1 row in set (0.00 sec)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-14786) Beeline displays binary column data as string instead of byte array

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113212#comment-16113212
 ] 

Hive QA commented on HIVE-14786:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12880221/HIVE-14786.03.patch

{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 11145 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_uncompressed] 
(batchId=56)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_schema_evol_3a]
 (batchId=142)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
 (batchId=168)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3] 
(batchId=99)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] 
(batchId=236)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] 
(batchId=236)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
 (batchId=179)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
 (batchId=179)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation 
(batchId=179)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6248/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6248/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6248/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12880221 - PreCommit-HIVE-Build

> Beeline displays binary column data as string instead of byte array
> ---
>
> Key: HIVE-14786
> URL: https://issues.apache.org/jira/browse/HIVE-14786
> Project: Hive
>  Issue Type: Improvement
>  Components: Beeline
>Affects Versions: 1.2.1
>Reporter: Ram Mettu
>Assignee: Barna Zsombor Klara
>Priority: Minor
> Attachments: HIVE-14786.01.patch, HIVE-14786.02.patch, 
> HIVE-14786.03.patch
>
>
> In Beeline, doing a SELECT binaryColName FROM tableName; results in displays 
> data as string type (which looks corrupted due to unprintable chars). Instead 
> Beeline should display binary columns as byte array. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17242) Vectorized query execution for parquet tables on S3 fail with Timeout waiting for connection from pool exception

[Vihang Karajgaonkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vihang Karajgaonkar updated HIVE-17242:
---
Description: 
When I turn vectorization on with tables on S3 and using Hive-on-Spark many of 
the TPCDS queries fail due to the error "Timeout waiting for connection from 
pool exception" from S3AClient. This does not happen when I turn vectorization 
off.

Here is the exception trace I am seeing ..

{noformat}
Driver stacktrace:
at 
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1452)
at 
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1440)
at 
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1439)
at 
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at 
org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1439)
at 
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:811)
at 
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:811)
at scala.Option.foreach(Option.scala:257)
at 
org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:811)
at 
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1665)
at 
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1620)
at 
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1609)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
Caused by: java.io.IOException: java.lang.reflect.InvocationTargetException
at 
org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
at 
org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
at 
org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:269)
at 
org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.(HadoopShimsSecure.java:216)
at 
org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getRecordReader(HadoopShimsSecure.java:343)
at 
org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:681)
at org.apache.spark.rdd.HadoopRDD$$anon$1.(HadoopRDD.scala:245)
at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:208)
at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:101)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
at 
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:105)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
at 
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79)
at 
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)
at org.apache.spark.scheduler.Task.run(Task.scala:86)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedConstructorAccessor43.newInstance(Unknown 
Source)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at 
org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:255)
... 21 more
Caused by: java.lang.RuntimeException: java.io.InterruptedIOException: 
getFileStatus on 
/tpcds_30_decimal_parquet/store_sales/ss_sold_date_sk=2452583/2b4b7a8a5573cc3a-682f12ce0982ee04_852204394_data.0.parq:
 com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout 
waiting for connection from pool
at 

[jira] [Commented] (HIVE-17144) export of temporary tables not working and it seems to be using distcp rather than filesystem copy

[Daniel Dai (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113175#comment-16113175
 ] 

Daniel Dai commented on HIVE-17144:
---

Thanks Barna/Peter to fixing this quickly! I have put back the license header.

> export of temporary tables not working and it seems to be using distcp rather 
> than filesystem copy
> --
>
> Key: HIVE-17144
> URL: https://issues.apache.org/jira/browse/HIVE-17144
> Project: Hive
>  Issue Type: Bug
>  Components: Hive, HiveServer2
>Affects Versions: 3.0.0
>Reporter: anishek
>Assignee: anishek
> Fix For: 3.0.0
>
> Attachments: HIVE-17144.1.patch
>
>
> create temporary table t1 (i int);
> insert into t1 values (3);
> export table t1 to 'hdfs://somelocation';
> above fails. additionally it should use filesystem copy and not distcp to do 
> the job.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17242) Vectorized query execution for parquet tables on S3 fail with Timeout waiting for connection from pool exception

[Vihang Karajgaonkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vihang Karajgaonkar updated HIVE-17242:
---
Description: 
When I turn vectorization on with tables on S3 and using Hive-on-Spark may of 
the TPCDS queries fail due to the error "Timeout waiting for connection from 
pool exception" from S3AClient. This does not happen when I turn vectorization 
off.

Here is the exception trace I am seeing ..

{noformat}
Driver stacktrace:
at 
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1452)
at 
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1440)
at 
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1439)
at 
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at 
org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1439)
at 
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:811)
at 
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:811)
at scala.Option.foreach(Option.scala:257)
at 
org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:811)
at 
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1665)
at 
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1620)
at 
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1609)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
Caused by: java.io.IOException: java.lang.reflect.InvocationTargetException
at 
org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
at 
org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
at 
org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:269)
at 
org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.(HadoopShimsSecure.java:216)
at 
org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getRecordReader(HadoopShimsSecure.java:343)
at 
org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:681)
at org.apache.spark.rdd.HadoopRDD$$anon$1.(HadoopRDD.scala:245)
at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:208)
at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:101)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
at 
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:105)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
at 
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79)
at 
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)
at org.apache.spark.scheduler.Task.run(Task.scala:86)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedConstructorAccessor43.newInstance(Unknown 
Source)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at 
org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:255)
... 21 more
Caused by: java.lang.RuntimeException: java.io.InterruptedIOException: 
getFileStatus on 
/tpcds_30_decimal_parquet/store_sales/ss_sold_date_sk=2452583/2b4b7a8a5573cc3a-682f12ce0982ee04_852204394_data.0.parq:
 com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout 
waiting for connection from pool
at 
org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.(VectorizedParquetRecordReader.java:129)

[jira] [Commented] (HIVE-17242) Vectorized query execution for parquet tables on S3 fail with Timeout waiting for connection from pool exception

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113171#comment-16113171
 ] 

Vihang Karajgaonkar commented on HIVE-17242:


[~stakiar] [~Ferd] Have you seen this error before?

> Vectorized query execution for parquet tables on S3 fail with Timeout waiting 
> for connection from pool exception
> 
>
> Key: HIVE-17242
> URL: https://issues.apache.org/jira/browse/HIVE-17242
> Project: Hive
>  Issue Type: Bug
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>
> When I turn vectorization on with tables on S3 and using Hive-on-Spark may of 
> the TPCDS queries fail due to the error "Timeout waiting for connection from 
> pool exception" from S3AClient. This does not happen when I turn 
> vectorization off.
> Here is the exception trace I am seeing ..
> {noformat}
> Driver stacktrace:
>   at 
> org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1452)
>   at 
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1440)
>   at 
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1439)
>   at 
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
>   at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
>   at 
> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1439)
>   at 
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:811)
>   at 
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:811)
>   at scala.Option.foreach(Option.scala:257)
>   at 
> org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:811)
>   at 
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1665)
>   at 
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1620)
>   at 
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1609)
>   at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
> Caused by: java.io.IOException: java.lang.reflect.InvocationTargetException
>   at 
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
>   at 
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
>   at 
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:269)
>   at 
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.(HadoopShimsSecure.java:216)
>   at 
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getRecordReader(HadoopShimsSecure.java:343)
>   at 
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:681)
>   at org.apache.spark.rdd.HadoopRDD$$anon$1.(HadoopRDD.scala:245)
>   at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:208)
>   at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:101)
>   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
>   at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
>   at 
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
>   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
>   at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
>   at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:105)
>   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
>   at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
>   at 
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79)
>   at 
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)
>   at org.apache.spark.scheduler.Task.run(Task.scala:86)
>   at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.reflect.InvocationTargetException
>   at sun.reflect.GeneratedConstructorAccessor43.newInstance(Unknown 
> Source)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at 

[jira] [Assigned] (HIVE-17242) Vectorized query execution for parquet tables on S3 fail with Timeout waiting for connection from pool exception

[Vihang Karajgaonkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vihang Karajgaonkar reassigned HIVE-17242:
--


> Vectorized query execution for parquet tables on S3 fail with Timeout waiting 
> for connection from pool exception
> 
>
> Key: HIVE-17242
> URL: https://issues.apache.org/jira/browse/HIVE-17242
> Project: Hive
>  Issue Type: Bug
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>
> When I turn vectorization on with tables on S3 and using Hive-on-Spark may of 
> the TPCDS queries fail due to the error "Timeout waiting for connection from 
> pool exception" from S3AClient. This does not happen when I turn 
> vectorization off.
> Here is the exception trace I am seeing ..
> {noformat}
> Driver stacktrace:
>   at 
> org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1452)
>   at 
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1440)
>   at 
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1439)
>   at 
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
>   at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
>   at 
> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1439)
>   at 
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:811)
>   at 
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:811)
>   at scala.Option.foreach(Option.scala:257)
>   at 
> org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:811)
>   at 
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1665)
>   at 
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1620)
>   at 
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1609)
>   at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
> Caused by: java.io.IOException: java.lang.reflect.InvocationTargetException
>   at 
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
>   at 
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
>   at 
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:269)
>   at 
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.(HadoopShimsSecure.java:216)
>   at 
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getRecordReader(HadoopShimsSecure.java:343)
>   at 
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:681)
>   at org.apache.spark.rdd.HadoopRDD$$anon$1.(HadoopRDD.scala:245)
>   at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:208)
>   at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:101)
>   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
>   at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
>   at 
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
>   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
>   at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
>   at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:105)
>   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
>   at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
>   at 
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79)
>   at 
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)
>   at org.apache.spark.scheduler.Task.run(Task.scala:86)
>   at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.reflect.InvocationTargetException
>   at sun.reflect.GeneratedConstructorAccessor43.newInstance(Unknown 
> Source)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>   at 
> 

[jira] [Updated] (HIVE-17188) ObjectStore runs out of memory for large batches of addPartitions().

[Mithun Radhakrishnan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mithun Radhakrishnan updated HIVE-17188:

Fix Version/s: 2.4.0

> ObjectStore runs out of memory for large batches of addPartitions().
> 
>
> Key: HIVE-17188
> URL: https://issues.apache.org/jira/browse/HIVE-17188
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 2.2.0
>Reporter: Mithun Radhakrishnan
>Assignee: Chris Drome
> Fix For: 2.2.0, 3.0.0, 2.4.0
>
> Attachments: HIVE-17188.1.patch
>
>
> For large batches (e.g. hundreds) of {{addPartitions()}}, the {{ObjectStore}} 
> runs out of memory. Flushing the {{PersistenceManager}} alleviates the 
> problem.
> Note: The problem being addressed here isn't so much with the size of the 
> hundreds of Partition objects, but the cruft that builds with the 
> PersistenceManager, in the JDO layer, as confirmed through memory-profiling.
> (Raising this on behalf of [~cdrome] and [~thiruvel].)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17241) Change metastore classes to not use the shims

[Alan Gates (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113131#comment-16113131
 ] 

Alan Gates commented on HIVE-17241:
---

I propose a hybrid solution.

First, even in Hive 2 we no longer support Hadoop 1, so we are fine to drop it 
completely from the standalone metastore.

In looking through the metastore shims three features are used:
# HDFS operations
# Secure password access from config files
# HadoopThriftAuthBridge and related code

None of these areas change from Hadoop 2 to 3, so we do not need to shim for 
them.  HDFS operations can be done directly in utility classes.  
MetastoreConf.getPassword() already handles the secure password access.

HadoopThriftAuthBridge is harder.  The only Hive module being referenced by the 
stand alone metastore is the storage-api.  It does not feel like 
HadoopThriftAuthBridge fits in there.  Talking to Vaibhav (who has spent the 
most time with this code), the code has not changed significantly over the last 
couple of years.  Thus copying it is an option.  This is about 1KLoC in size.  

The other alternative is to move it into the standalone metastore (rather than 
just copy it) and let HiveServer2 use it from there.

[~thejas], [~vgumashta], feedback?

> Change metastore classes to not use the shims
> -
>
> Key: HIVE-17241
> URL: https://issues.apache.org/jira/browse/HIVE-17241
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>
> As part of moving the metastore into a standalone package, it will no longer 
> have access to the shims.  This means we need to either copy them or access 
> the underlying Hadoop operations directly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HIVE-17241) Change metastore classes to not use the shims

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates reassigned HIVE-17241:
-


> Change metastore classes to not use the shims
> -
>
> Key: HIVE-17241
> URL: https://issues.apache.org/jira/browse/HIVE-17241
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>
> As part of moving the metastore into a standalone package, it will no longer 
> have access to the shims.  This means we need to either copy them or access 
> the underlying Hadoop operations directly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17224) Move JDO classes to standalone metastore

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-17224:
--
Attachment: HIVE-17224.patch

This patch moves the M* classes and package.jdo into the standalone metastore.

> Move JDO classes to standalone metastore
> 
>
> Key: HIVE-17224
> URL: https://issues.apache.org/jira/browse/HIVE-17224
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
> Attachments: HIVE-17224.patch
>
>
> The JDO model classes (MDatabase, MTable, etc.) and the package.jdo file that 
> defines the DB mapping need to be moved to the standalone metastore.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17224) Move JDO classes to standalone metastore

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-17224:
--
Status: Patch Available  (was: Open)

> Move JDO classes to standalone metastore
> 
>
> Key: HIVE-17224
> URL: https://issues.apache.org/jira/browse/HIVE-17224
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
> Attachments: HIVE-17224.patch
>
>
> The JDO model classes (MDatabase, MTable, etc.) and the package.jdo file that 
> defines the DB mapping need to be moved to the standalone metastore.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17224) Move JDO classes to standalone metastore

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113100#comment-16113100
 ] 

ASF GitHub Bot commented on HIVE-17224:
---

GitHub user alanfgates opened a pull request:

https://github.com/apache/hive/pull/220

HIVE-17224 Moved all the JDO classes and package.jdo



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/alanfgates/hive hive17224

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/220.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #220


commit cd70b9ce7798251d5dd8b247bba0fb8211ebce40
Author: Alan Gates 
Date:   2017-07-27T21:27:57Z

Moved all the JDO classes and package.jdo




> Move JDO classes to standalone metastore
> 
>
> Key: HIVE-17224
> URL: https://issues.apache.org/jira/browse/HIVE-17224
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>
> The JDO model classes (MDatabase, MTable, etc.) and the package.jdo file that 
> defines the DB mapping need to be moved to the standalone metastore.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17168) Create separate module for stand alone metastore

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113094#comment-16113094
 ] 

ASF GitHub Bot commented on HIVE-17168:
---

Github user alanfgates closed the pull request at:

https://github.com/apache/hive/pull/210


> Create separate module for stand alone metastore
> 
>
> Key: HIVE-17168
> URL: https://issues.apache.org/jira/browse/HIVE-17168
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
> Fix For: 3.0.0
>
> Attachments: HIVE-17168.patch
>
>
> We need to create a separate maven module for the stand alone metastore.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17167) Create metastore specific configuration tool

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113093#comment-16113093
 ] 

ASF GitHub Bot commented on HIVE-17167:
---

Github user alanfgates closed the pull request at:

https://github.com/apache/hive/pull/211


> Create metastore specific configuration tool
> 
>
> Key: HIVE-17167
> URL: https://issues.apache.org/jira/browse/HIVE-17167
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
> Fix For: 3.0.0
>
> Attachments: HIVE-17167.2.patch, HIVE-17167.patch
>
>
> As part of making the metastore a separately releasable module we need 
> configuration tools that are specific to that module.  It cannot use or 
> extend HiveConf as that is in hive common.  But it must take a HiveConf 
> object and be able to operate on it.
> The best way to achieve this is using Hadoop's Configuration object (which 
> HiveConf extends) together with enums and static methods.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17170) Move thrift generated code to stand alone metastore

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113092#comment-16113092
 ] 

ASF GitHub Bot commented on HIVE-17170:
---

Github user alanfgates closed the pull request at:

https://github.com/apache/hive/pull/216


> Move thrift generated code to stand alone metastore
> ---
>
> Key: HIVE-17170
> URL: https://issues.apache.org/jira/browse/HIVE-17170
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
> Fix For: 3.0.0
>
> Attachments: HIVE-17170.2.patch, HIVE-17170.patch
>
>
> hive_metastore.thrift and the code it generates needs to be moved into the 
> standalone metastore module.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17170) Move thrift generated code to stand alone metastore

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-17170:
--
   Resolution: Fixed
Fix Version/s: 3.0.0
   Status: Resolved  (was: Patch Available)

Patch committed.  Thanks Vihang for the review.

> Move thrift generated code to stand alone metastore
> ---
>
> Key: HIVE-17170
> URL: https://issues.apache.org/jira/browse/HIVE-17170
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
> Fix For: 3.0.0
>
> Attachments: HIVE-17170.2.patch, HIVE-17170.patch
>
>
> hive_metastore.thrift and the code it generates needs to be moved into the 
> standalone metastore module.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-15794) Support get hdfsEncryptionShim if FileSystem is ViewFileSystem

[Yuming Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuming Wang updated HIVE-15794:
---
Status: Patch Available  (was: In Progress)

> Support get hdfsEncryptionShim if FileSystem is ViewFileSystem
> --
>
> Key: HIVE-15794
> URL: https://issues.apache.org/jira/browse/HIVE-15794
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Affects Versions: 2.2.0, 1.1.0, 1.2.0
>Reporter: Yuming Wang
>Assignee: Yuming Wang
> Attachments: HIVE-15794.1.patch, HIVE-15794.2.patch, 
> HIVE-15794.3.patch
>
>
> *SQL*:
> {code:sql}
> hive> create table table2 as select * from table1;
> hive> show create table table2;
> OK
> CREATE TABLE `table2`(
>   `id` string)
> ROW FORMAT SERDE 
>   'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' 
> STORED AS INPUTFORMAT 
>   'org.apache.hadoop.mapred.TextInputFormat' 
> OUTPUTFORMAT 
>   'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
> LOCATION
>   'viewfs://cluster4/user/hive/warehouse/table2'
> TBLPROPERTIES (
>   'transient_lastDdlTime'='1486050317')
> {code}
> *LOG*:
> {noformat}
> 2017-02-02T20:12:49,738  INFO [99374b82-e9ca-4654-b803-93b194b9331b main] 
> session.SessionState: Could not get hdfsEncryptionShim, it is only applicable 
> to hdfs filesystem.
> 2017-02-02T20:12:49,738  INFO [99374b82-e9ca-4654-b803-93b194b9331b main] 
> session.SessionState: Could not get hdfsEncryptionShim, it is only applicable 
> to hdfs filesystem.
> {noformat}
> Can’t get hdfsEncryptionShim if {{FileSystem}} is 
> [ViewFileSystem|http://hadoop.apache.org/docs/r2.6.5/hadoop-project-dist/hadoop-hdfs/ViewFs.html],
>  we should support it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-15794) Support get hdfsEncryptionShim if FileSystem is ViewFileSystem

[Yuming Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuming Wang updated HIVE-15794:
---
Status: In Progress  (was: Patch Available)

> Support get hdfsEncryptionShim if FileSystem is ViewFileSystem
> --
>
> Key: HIVE-15794
> URL: https://issues.apache.org/jira/browse/HIVE-15794
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Affects Versions: 2.2.0, 1.1.0, 1.2.0
>Reporter: Yuming Wang
>Assignee: Yuming Wang
> Attachments: HIVE-15794.1.patch, HIVE-15794.2.patch, 
> HIVE-15794.3.patch
>
>
> *SQL*:
> {code:sql}
> hive> create table table2 as select * from table1;
> hive> show create table table2;
> OK
> CREATE TABLE `table2`(
>   `id` string)
> ROW FORMAT SERDE 
>   'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' 
> STORED AS INPUTFORMAT 
>   'org.apache.hadoop.mapred.TextInputFormat' 
> OUTPUTFORMAT 
>   'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
> LOCATION
>   'viewfs://cluster4/user/hive/warehouse/table2'
> TBLPROPERTIES (
>   'transient_lastDdlTime'='1486050317')
> {code}
> *LOG*:
> {noformat}
> 2017-02-02T20:12:49,738  INFO [99374b82-e9ca-4654-b803-93b194b9331b main] 
> session.SessionState: Could not get hdfsEncryptionShim, it is only applicable 
> to hdfs filesystem.
> 2017-02-02T20:12:49,738  INFO [99374b82-e9ca-4654-b803-93b194b9331b main] 
> session.SessionState: Could not get hdfsEncryptionShim, it is only applicable 
> to hdfs filesystem.
> {noformat}
> Can’t get hdfsEncryptionShim if {{FileSystem}} is 
> [ViewFileSystem|http://hadoop.apache.org/docs/r2.6.5/hadoop-project-dist/hadoop-hdfs/ViewFs.html],
>  we should support it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17170) Move thrift generated code to stand alone metastore

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112950#comment-16112950
 ] 

Vihang Karajgaonkar commented on HIVE-17170:


+1 LGTM. Thanks [~alangates]

> Move thrift generated code to stand alone metastore
> ---
>
> Key: HIVE-17170
> URL: https://issues.apache.org/jira/browse/HIVE-17170
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
> Attachments: HIVE-17170.2.patch, HIVE-17170.patch
>
>
> hive_metastore.thrift and the code it generates needs to be moved into the 
> standalone metastore module.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17167) Create metastore specific configuration tool

[Alan Gates (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112945#comment-16112945
 ] 

Alan Gates commented on HIVE-17167:
---

Do we have a "wait and see" label for docs?  If we eventually decide to split 
out the metastore it doesn't make sense to me to spend time in the Hive docs. 

> Create metastore specific configuration tool
> 
>
> Key: HIVE-17167
> URL: https://issues.apache.org/jira/browse/HIVE-17167
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
> Fix For: 3.0.0
>
> Attachments: HIVE-17167.2.patch, HIVE-17167.patch
>
>
> As part of making the metastore a separately releasable module we need 
> configuration tools that are specific to that module.  It cannot use or 
> extend HiveConf as that is in hive common.  But it must take a HiveConf 
> object and be able to operate on it.
> The best way to achieve this is using Hadoop's Configuration object (which 
> HiveConf extends) together with enums and static methods.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17188) ObjectStore runs out of memory for large batches of addPartitions().

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112917#comment-16112917
 ] 

Vihang Karajgaonkar commented on HIVE-17188:


Yes, would probably make sense to merge this in branch-2.3 as well if this is 
available in 2.2.0 release. Otherwise, this patch would be available in 2.2.0 
and 2.4.0 but not in 2.3.0

> ObjectStore runs out of memory for large batches of addPartitions().
> 
>
> Key: HIVE-17188
> URL: https://issues.apache.org/jira/browse/HIVE-17188
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 2.2.0
>Reporter: Mithun Radhakrishnan
>Assignee: Chris Drome
> Fix For: 2.2.0, 3.0.0
>
> Attachments: HIVE-17188.1.patch
>
>
> For large batches (e.g. hundreds) of {{addPartitions()}}, the {{ObjectStore}} 
> runs out of memory. Flushing the {{PersistenceManager}} alleviates the 
> problem.
> Note: The problem being addressed here isn't so much with the size of the 
> hundreds of Partition objects, but the cruft that builds with the 
> PersistenceManager, in the JDO layer, as confirmed through memory-profiling.
> (Raising this on behalf of [~cdrome] and [~thiruvel].)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17235) Add ORC Decimal64 Serialization/Deserialization

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-17235:

Attachment: HIVE-17235.04.patch

> Add ORC Decimal64 Serialization/Deserialization
> ---
>
> Key: HIVE-17235
> URL: https://issues.apache.org/jira/browse/HIVE-17235
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-17235.03.patch, HIVE-17235.04.patch
>
>
> The storage-api changes for ORC-209.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17234) Remove HBase metastore from master

[Alan Gates (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112892#comment-16112892
 ] 

Alan Gates commented on HIVE-17234:
---

[~sershe], [~daijy] can you take a look and make sure I didn't break anything 
you are using or planning to use?

> Remove HBase metastore from master
> --
>
> Key: HIVE-17234
> URL: https://issues.apache.org/jira/browse/HIVE-17234
> Project: Hive
>  Issue Type: Task
>  Components: HBase Metastore
>Affects Versions: 3.0.0
>Reporter: Alan Gates
>Assignee: Alan Gates
> Attachments: HIVE-17234.patch
>
>
> No new development has been done on the HBase metastore in at least a year, 
> and to my knowledge no one is using it (nor is it even in a state to be fully 
> usable).  Given the lack of interest in continuing to develop it, we should 
> remove it rather than leave dead code hanging around and extra tests taking 
> up time in test runs.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17234) Remove HBase metastore from master

[Alan Gates (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112891#comment-16112891
 ] 

Alan Gates commented on HIVE-17234:
---

Me too :(

Thanks, I forgot to check the docs.

> Remove HBase metastore from master
> --
>
> Key: HIVE-17234
> URL: https://issues.apache.org/jira/browse/HIVE-17234
> Project: Hive
>  Issue Type: Task
>  Components: HBase Metastore
>Affects Versions: 3.0.0
>Reporter: Alan Gates
>Assignee: Alan Gates
> Attachments: HIVE-17234.patch
>
>
> No new development has been done on the HBase metastore in at least a year, 
> and to my knowledge no one is using it (nor is it even in a state to be fully 
> usable).  Given the lack of interest in continuing to develop it, we should 
> remove it rather than leave dead code hanging around and extra tests taking 
> up time in test runs.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17160) Adding kerberos Authorization to the Druid hive integration

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112883#comment-16112883
 ] 

Ashutosh Chauhan commented on HIVE-17160:
-

+1

> Adding kerberos Authorization to the Druid hive integration
> ---
>
> Key: HIVE-17160
> URL: https://issues.apache.org/jira/browse/HIVE-17160
> Project: Hive
>  Issue Type: New Feature
>  Components: Druid integration
>Reporter: slim bouguerra
>Assignee: slim bouguerra
> Attachments: HIVE-17160.2.patch, HIVE-17160.patch
>
>
> This goal of this feature is to allow hive querying a secured druid cluster 
> using kerberos credentials.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-16895) Multi-threaded execution of bootstrap dump of partitions

[Sankar Hariappan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112726#comment-16112726
 ] 

Sankar Hariappan commented on HIVE-16895:
-

+1 
Patch looks good to me

Request [~daijy] to review/commit to master!

>  Multi-threaded execution of bootstrap dump of partitions
> -
>
> Key: HIVE-16895
> URL: https://issues.apache.org/jira/browse/HIVE-16895
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: anishek
>Assignee: anishek
> Fix For: 3.0.0
>
> Attachments: HIVE-16895.1.patch
>
>
> to allow faster execution of bootstrap dump phase we dump multiple partitions 
> from same table simultaneously. 
> even though dumping  functions is  not going to be a blocker, moving to 
> similar execution modes for all metastore objects will make code more 
> coherent. 
> Bootstrap dump at db level does :
> * boostrap of all tables
> ** boostrap of all partitions in a table.  (scope of current jira) 
> * boostrap of all functions 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-16845) INSERT OVERWRITE a table with dynamic partitions on S3 fails with NPE

[Marta Kuczora (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112665#comment-16112665
 ] 

Marta Kuczora commented on HIVE-16845:
--

Thanks a lot [~pvary] for committing the patch!

> INSERT OVERWRITE a table with dynamic partitions on S3 fails with NPE
> -
>
> Key: HIVE-16845
> URL: https://issues.apache.org/jira/browse/HIVE-16845
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 2.1.1
>Reporter: Marta Kuczora
>Assignee: Marta Kuczora
> Fix For: 3.0.0
>
> Attachments: HIVE-16845.1.patch, HIVE-16845.2.patch, 
> HIVE-16845.3.patch, HIVE-16845.4.patch
>
>
> *How to reproduce*
> - Create a partitioned table on S3:
> {noformat}
> CREATE EXTERNAL TABLE s3table(user_id string COMMENT '', event_name string 
> COMMENT '') PARTITIONED BY (reported_date string, product_id int) LOCATION 
> 's3a://'; 
> {noformat}
> - Create a temp table:
> {noformat}
> create table tmp_table (id string, name string, date string, pid int) row 
> format delimited fields terminated by '\t' lines terminated by '\n' stored as 
> textfile;
> {noformat}
> - Load the following rows to the tmp table:
> {noformat}
> u1value1  2017-04-10  1
> u2value2  2017-04-10  1
> u3value3  2017-04-10  10001
> {noformat}
> - Set the following parameters:
> -- hive.exec.dynamic.partition.mode=nonstrict
> -- mapreduce.input.fileinputformat.split.maxsize=10
> -- hive.blobstore.optimizations.enabled=true
> -- hive.blobstore.use.blobstore.as.scratchdir=false
> -- hive.merge.mapfiles=true
> - Insert the rows from the temp table into the s3 table:
> {noformat}
> INSERT OVERWRITE TABLE s3table
> PARTITION (reported_date, product_id)
> SELECT
>   t.id as user_id,
>   t.name as event_name,
>   t.date as reported_date,
>   t.pid as product_id
> FROM tmp_table t;
> {noformat}
> A NPE will occur with the following stacktrace:
> {noformat}
> 2017-05-08 21:32:50,607 ERROR 
> org.apache.hive.service.cli.operation.Operation: 
> [HiveServer2-Background-Pool: Thread-184028]: Error running hive query: 
> org.apache.hive.service.cli.HiveSQLException: Error while processing 
> statement: FAILED: Execution Error, return code -101 from 
> org.apache.hadoop.hive.ql.exec.ConditionalTask. null
> at 
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:400)
> at 
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:239)
> at 
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:88)
> at 
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:293)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
> at 
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:306)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
> at 
> org.apache.hadoop.hive.ql.plan.ConditionalResolverMergeFiles.generateActualTasks(ConditionalResolverMergeFiles.java:290)
> at 
> org.apache.hadoop.hive.ql.plan.ConditionalResolverMergeFiles.getTasks(ConditionalResolverMergeFiles.java:175)
> at 
> org.apache.hadoop.hive.ql.exec.ConditionalTask.execute(ConditionalTask.java:81)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
> at 
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1977)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1690)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1422)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1206)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1201)
> at 
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:237)
> ... 11 more 
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-14786) Beeline displays binary column data as string instead of byte array

[Barna Zsombor Klara (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Barna Zsombor Klara updated HIVE-14786:
---
Attachment: HIVE-14786.03.patch

Added the option description to the help text.

> Beeline displays binary column data as string instead of byte array
> ---
>
> Key: HIVE-14786
> URL: https://issues.apache.org/jira/browse/HIVE-14786
> Project: Hive
>  Issue Type: Improvement
>  Components: Beeline
>Affects Versions: 1.2.1
>Reporter: Ram Mettu
>Assignee: Barna Zsombor Klara
>Priority: Minor
> Attachments: HIVE-14786.01.patch, HIVE-14786.02.patch, 
> HIVE-14786.03.patch
>
>
> In Beeline, doing a SELECT binaryColName FROM tableName; results in displays 
> data as string type (which looks corrupted due to unprintable chars). Instead 
> Beeline should display binary columns as byte array. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-16895) Multi-threaded execution of bootstrap dump of partitions

[anishek (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112612#comment-16112612
 ] 

anishek commented on HIVE-16895:


* 
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning]
 : runs fine on local machine 
* 
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_combine_equivalent_work]
 : runs fine on local machine 
* 
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_use_op_stats]
 : runs fine on local machine 
* 
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[orc_merge_diff_fs]
 : runs fine on local machine 
* 
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[quotedid_smb]
 : runs fine on local machine 
* 
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[truncate_column_buckets]
 : runs fine on local machine 
* 
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[orc_merge3]
 : runs fine on local machine

other tests are failing from old builds.


[~thejas]/[~sankarh]/[~daijy] please review

>  Multi-threaded execution of bootstrap dump of partitions
> -
>
> Key: HIVE-16895
> URL: https://issues.apache.org/jira/browse/HIVE-16895
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: anishek
>Assignee: anishek
> Fix For: 3.0.0
>
> Attachments: HIVE-16895.1.patch
>
>
> to allow faster execution of bootstrap dump phase we dump multiple partitions 
> from same table simultaneously. 
> even though dumping  functions is  not going to be a blocker, moving to 
> similar execution modes for all metastore objects will make code more 
> coherent. 
> Bootstrap dump at db level does :
> * boostrap of all tables
> ** boostrap of all partitions in a table.  (scope of current jira) 
> * boostrap of all functions 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-16294) Support snapshot for truncate table

[Barna Zsombor Klara (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Barna Zsombor Klara updated HIVE-16294:
---
Attachment: HIVE-16294.03.patch

Added the comment until we are waiting for further feedback.

> Support snapshot for truncate table
> ---
>
> Key: HIVE-16294
> URL: https://issues.apache.org/jira/browse/HIVE-16294
> Project: Hive
>  Issue Type: Sub-task
>  Components: Query Processor
>Reporter: Vihang Karajgaonkar
>Assignee: Barna Zsombor Klara
> Attachments: HIVE-16294.01.patch, HIVE-16294.02.patch, 
> HIVE-16294.03.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-14786) Beeline displays binary column data as string instead of byte array

[Ram Mettu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112582#comment-16112582
 ] 

Ram Mettu commented on HIVE-14786:
--

[~zsombor.klara] Thanks for the PR. Also please update the user documentation 
as needed for this new config option. 

> Beeline displays binary column data as string instead of byte array
> ---
>
> Key: HIVE-14786
> URL: https://issues.apache.org/jira/browse/HIVE-14786
> Project: Hive
>  Issue Type: Improvement
>  Components: Beeline
>Affects Versions: 1.2.1
>Reporter: Ram Mettu
>Assignee: Barna Zsombor Klara
>Priority: Minor
> Attachments: HIVE-14786.01.patch, HIVE-14786.02.patch
>
>
> In Beeline, doing a SELECT binaryColName FROM tableName; results in displays 
> data as string type (which looks corrupted due to unprintable chars). Instead 
> Beeline should display binary columns as byte array. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-15794) Support get hdfsEncryptionShim if FileSystem is ViewFileSystem

[Yuming Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuming Wang updated HIVE-15794:
---
Status: Patch Available  (was: In Progress)

> Support get hdfsEncryptionShim if FileSystem is ViewFileSystem
> --
>
> Key: HIVE-15794
> URL: https://issues.apache.org/jira/browse/HIVE-15794
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Affects Versions: 2.2.0, 1.1.0, 1.2.0
>Reporter: Yuming Wang
>Assignee: Yuming Wang
> Attachments: HIVE-15794.1.patch, HIVE-15794.2.patch, 
> HIVE-15794.3.patch
>
>
> *SQL*:
> {code:sql}
> hive> create table table2 as select * from table1;
> hive> show create table table2;
> OK
> CREATE TABLE `table2`(
>   `id` string)
> ROW FORMAT SERDE 
>   'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' 
> STORED AS INPUTFORMAT 
>   'org.apache.hadoop.mapred.TextInputFormat' 
> OUTPUTFORMAT 
>   'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
> LOCATION
>   'viewfs://cluster4/user/hive/warehouse/table2'
> TBLPROPERTIES (
>   'transient_lastDdlTime'='1486050317')
> {code}
> *LOG*:
> {noformat}
> 2017-02-02T20:12:49,738  INFO [99374b82-e9ca-4654-b803-93b194b9331b main] 
> session.SessionState: Could not get hdfsEncryptionShim, it is only applicable 
> to hdfs filesystem.
> 2017-02-02T20:12:49,738  INFO [99374b82-e9ca-4654-b803-93b194b9331b main] 
> session.SessionState: Could not get hdfsEncryptionShim, it is only applicable 
> to hdfs filesystem.
> {noformat}
> Can’t get hdfsEncryptionShim if {{FileSystem}} is 
> [ViewFileSystem|http://hadoop.apache.org/docs/r2.6.5/hadoop-project-dist/hadoop-hdfs/ViewFs.html],
>  we should support it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-15794) Support get hdfsEncryptionShim if FileSystem is ViewFileSystem

[Yuming Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuming Wang updated HIVE-15794:
---
Status: In Progress  (was: Patch Available)

> Support get hdfsEncryptionShim if FileSystem is ViewFileSystem
> --
>
> Key: HIVE-15794
> URL: https://issues.apache.org/jira/browse/HIVE-15794
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Affects Versions: 2.2.0, 1.1.0, 1.2.0
>Reporter: Yuming Wang
>Assignee: Yuming Wang
> Attachments: HIVE-15794.1.patch, HIVE-15794.2.patch, 
> HIVE-15794.3.patch
>
>
> *SQL*:
> {code:sql}
> hive> create table table2 as select * from table1;
> hive> show create table table2;
> OK
> CREATE TABLE `table2`(
>   `id` string)
> ROW FORMAT SERDE 
>   'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' 
> STORED AS INPUTFORMAT 
>   'org.apache.hadoop.mapred.TextInputFormat' 
> OUTPUTFORMAT 
>   'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
> LOCATION
>   'viewfs://cluster4/user/hive/warehouse/table2'
> TBLPROPERTIES (
>   'transient_lastDdlTime'='1486050317')
> {code}
> *LOG*:
> {noformat}
> 2017-02-02T20:12:49,738  INFO [99374b82-e9ca-4654-b803-93b194b9331b main] 
> session.SessionState: Could not get hdfsEncryptionShim, it is only applicable 
> to hdfs filesystem.
> 2017-02-02T20:12:49,738  INFO [99374b82-e9ca-4654-b803-93b194b9331b main] 
> session.SessionState: Could not get hdfsEncryptionShim, it is only applicable 
> to hdfs filesystem.
> {noformat}
> Can’t get hdfsEncryptionShim if {{FileSystem}} is 
> [ViewFileSystem|http://hadoop.apache.org/docs/r2.6.5/hadoop-project-dist/hadoop-hdfs/ViewFs.html],
>  we should support it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-14786) Beeline displays binary column data as string instead of byte array

[Barna Zsombor Klara (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Barna Zsombor Klara updated HIVE-14786:
---
Attachment: HIVE-14786.02.patch

Fixed regressions caused in 2 unit tests.

> Beeline displays binary column data as string instead of byte array
> ---
>
> Key: HIVE-14786
> URL: https://issues.apache.org/jira/browse/HIVE-14786
> Project: Hive
>  Issue Type: Improvement
>  Components: Beeline
>Affects Versions: 1.2.1
>Reporter: Ram Mettu
>Assignee: Barna Zsombor Klara
>Priority: Minor
> Attachments: HIVE-14786.01.patch, HIVE-14786.02.patch
>
>
> In Beeline, doing a SELECT binaryColName FROM tableName; results in displays 
> data as string type (which looks corrupted due to unprintable chars). Instead 
> Beeline should display binary columns as byte array. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-15794) Support get hdfsEncryptionShim if FileSystem is ViewFileSystem

[Yuming Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuming Wang updated HIVE-15794:
---
Attachment: HIVE-15794.3.patch

> Support get hdfsEncryptionShim if FileSystem is ViewFileSystem
> --
>
> Key: HIVE-15794
> URL: https://issues.apache.org/jira/browse/HIVE-15794
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Affects Versions: 1.2.0, 1.1.0, 2.2.0
>Reporter: Yuming Wang
>Assignee: Yuming Wang
> Attachments: HIVE-15794.1.patch, HIVE-15794.2.patch, 
> HIVE-15794.3.patch
>
>
> *SQL*:
> {code:sql}
> hive> create table table2 as select * from table1;
> hive> show create table table2;
> OK
> CREATE TABLE `table2`(
>   `id` string)
> ROW FORMAT SERDE 
>   'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' 
> STORED AS INPUTFORMAT 
>   'org.apache.hadoop.mapred.TextInputFormat' 
> OUTPUTFORMAT 
>   'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
> LOCATION
>   'viewfs://cluster4/user/hive/warehouse/table2'
> TBLPROPERTIES (
>   'transient_lastDdlTime'='1486050317')
> {code}
> *LOG*:
> {noformat}
> 2017-02-02T20:12:49,738  INFO [99374b82-e9ca-4654-b803-93b194b9331b main] 
> session.SessionState: Could not get hdfsEncryptionShim, it is only applicable 
> to hdfs filesystem.
> 2017-02-02T20:12:49,738  INFO [99374b82-e9ca-4654-b803-93b194b9331b main] 
> session.SessionState: Could not get hdfsEncryptionShim, it is only applicable 
> to hdfs filesystem.
> {noformat}
> Can’t get hdfsEncryptionShim if {{FileSystem}} is 
> [ViewFileSystem|http://hadoop.apache.org/docs/r2.6.5/hadoop-project-dist/hadoop-hdfs/ViewFs.html],
>  we should support it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-15794) Support get hdfsEncryptionShim if FileSystem is ViewFileSystem

[Yuming Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112549#comment-16112549
 ] 

Yuming Wang commented on HIVE-15794:


missing the CopyUtils, see:
https://issues.apache.org/jira/browse/HIVE-17144?focusedCommentId=16112539=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16112539


> Support get hdfsEncryptionShim if FileSystem is ViewFileSystem
> --
>
> Key: HIVE-15794
> URL: https://issues.apache.org/jira/browse/HIVE-15794
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Affects Versions: 1.2.0, 1.1.0, 2.2.0
>Reporter: Yuming Wang
>Assignee: Yuming Wang
> Attachments: HIVE-15794.1.patch, HIVE-15794.2.patch
>
>
> *SQL*:
> {code:sql}
> hive> create table table2 as select * from table1;
> hive> show create table table2;
> OK
> CREATE TABLE `table2`(
>   `id` string)
> ROW FORMAT SERDE 
>   'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' 
> STORED AS INPUTFORMAT 
>   'org.apache.hadoop.mapred.TextInputFormat' 
> OUTPUTFORMAT 
>   'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
> LOCATION
>   'viewfs://cluster4/user/hive/warehouse/table2'
> TBLPROPERTIES (
>   'transient_lastDdlTime'='1486050317')
> {code}
> *LOG*:
> {noformat}
> 2017-02-02T20:12:49,738  INFO [99374b82-e9ca-4654-b803-93b194b9331b main] 
> session.SessionState: Could not get hdfsEncryptionShim, it is only applicable 
> to hdfs filesystem.
> 2017-02-02T20:12:49,738  INFO [99374b82-e9ca-4654-b803-93b194b9331b main] 
> session.SessionState: Could not get hdfsEncryptionShim, it is only applicable 
> to hdfs filesystem.
> {noformat}
> Can’t get hdfsEncryptionShim if {{FileSystem}} is 
> [ViewFileSystem|http://hadoop.apache.org/docs/r2.6.5/hadoop-project-dist/hadoop-hdfs/ViewFs.html],
>  we should support it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-16845) INSERT OVERWRITE a table with dynamic partitions on S3 fails with NPE

[Peter Vary (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Vary updated HIVE-16845:
--
   Resolution: Fixed
Fix Version/s: 3.0.0
   Status: Resolved  (was: Patch Available)

Pushed to master.
Thanks [~kuczoram] for the patch!

> INSERT OVERWRITE a table with dynamic partitions on S3 fails with NPE
> -
>
> Key: HIVE-16845
> URL: https://issues.apache.org/jira/browse/HIVE-16845
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 2.1.1
>Reporter: Marta Kuczora
>Assignee: Marta Kuczora
> Fix For: 3.0.0
>
> Attachments: HIVE-16845.1.patch, HIVE-16845.2.patch, 
> HIVE-16845.3.patch, HIVE-16845.4.patch
>
>
> *How to reproduce*
> - Create a partitioned table on S3:
> {noformat}
> CREATE EXTERNAL TABLE s3table(user_id string COMMENT '', event_name string 
> COMMENT '') PARTITIONED BY (reported_date string, product_id int) LOCATION 
> 's3a://'; 
> {noformat}
> - Create a temp table:
> {noformat}
> create table tmp_table (id string, name string, date string, pid int) row 
> format delimited fields terminated by '\t' lines terminated by '\n' stored as 
> textfile;
> {noformat}
> - Load the following rows to the tmp table:
> {noformat}
> u1value1  2017-04-10  1
> u2value2  2017-04-10  1
> u3value3  2017-04-10  10001
> {noformat}
> - Set the following parameters:
> -- hive.exec.dynamic.partition.mode=nonstrict
> -- mapreduce.input.fileinputformat.split.maxsize=10
> -- hive.blobstore.optimizations.enabled=true
> -- hive.blobstore.use.blobstore.as.scratchdir=false
> -- hive.merge.mapfiles=true
> - Insert the rows from the temp table into the s3 table:
> {noformat}
> INSERT OVERWRITE TABLE s3table
> PARTITION (reported_date, product_id)
> SELECT
>   t.id as user_id,
>   t.name as event_name,
>   t.date as reported_date,
>   t.pid as product_id
> FROM tmp_table t;
> {noformat}
> A NPE will occur with the following stacktrace:
> {noformat}
> 2017-05-08 21:32:50,607 ERROR 
> org.apache.hive.service.cli.operation.Operation: 
> [HiveServer2-Background-Pool: Thread-184028]: Error running hive query: 
> org.apache.hive.service.cli.HiveSQLException: Error while processing 
> statement: FAILED: Execution Error, return code -101 from 
> org.apache.hadoop.hive.ql.exec.ConditionalTask. null
> at 
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:400)
> at 
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:239)
> at 
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:88)
> at 
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:293)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
> at 
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:306)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
> at 
> org.apache.hadoop.hive.ql.plan.ConditionalResolverMergeFiles.generateActualTasks(ConditionalResolverMergeFiles.java:290)
> at 
> org.apache.hadoop.hive.ql.plan.ConditionalResolverMergeFiles.getTasks(ConditionalResolverMergeFiles.java:175)
> at 
> org.apache.hadoop.hive.ql.exec.ConditionalTask.execute(ConditionalTask.java:81)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
> at 
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1977)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1690)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1422)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1206)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1201)
> at 
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:237)
> ... 11 more 
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17144) export of temporary tables not working and it seems to be using distcp rather than filesystem copy

[Peter Vary (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112539#comment-16112539
 ] 

Peter Vary commented on HIVE-17144:
---

Thanks [~zsombor.klara] for localizing this issue.
Committed the remaining files using the attached patch file. I hope there is no 
difference between the pull request and the patch file :)

[~anishek]: Could you please file a jira and add the remaining apache headers 
to the new files? I was not sure if I am allowed to change the patch, so 
committed it without change to unblock others.

> export of temporary tables not working and it seems to be using distcp rather 
> than filesystem copy
> --
>
> Key: HIVE-17144
> URL: https://issues.apache.org/jira/browse/HIVE-17144
> Project: Hive
>  Issue Type: Bug
>  Components: Hive, HiveServer2
>Affects Versions: 3.0.0
>Reporter: anishek
>Assignee: anishek
> Fix For: 3.0.0
>
> Attachments: HIVE-17144.1.patch
>
>
> create temporary table t1 (i int);
> insert into t1 values (3);
> export table t1 to 'hdfs://somelocation';
> above fails. additionally it should use filesystem copy and not distcp to do 
> the job.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17240) function acos(2) should be null

[Yuming Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112537#comment-16112537
 ] 

Yuming Wang commented on HIVE-17240:


CC [~sershe]

> function acos(2) should be null
> ---
>
> Key: HIVE-17240
> URL: https://issues.apache.org/jira/browse/HIVE-17240
> Project: Hive
>  Issue Type: Bug
>  Components: UDF
>Affects Versions: 1.1.1, 1.2.2, 2.2.0
>Reporter: Yuming Wang
>Assignee: Yuming Wang
> Attachments: HIVE-17240.1.patch
>
>
> {{acos(2)}} should be null, same as MySQL:
> {code:sql}
> hive> desc function extended acos;
> OK
> acos(x) - returns the arc cosine of x if -1<=x<=1 or NULL otherwise
> Example:
>   > SELECT acos(1) FROM src LIMIT 1;
>   0
>   > SELECT acos(2) FROM src LIMIT 1;
>   NULL
> Time taken: 0.009 seconds, Fetched: 6 row(s)
> hive> select acos(2);
> OK
> NaN
> Time taken: 0.437 seconds, Fetched: 1 row(s)
> {code}
> {code:sql}
> mysql>  select acos(2);
> +-+
> | acos(2) |
> +-+
> |NULL |
> +-+
> 1 row in set (0.00 sec)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-16896) move replication load related work in semantic analysis phase to execution phase using a task

[anishek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

anishek updated HIVE-16896:
---
Attachment: HIVE-16896.3.patch

> move replication load related work in semantic analysis phase to execution 
> phase using a task
> -
>
> Key: HIVE-16896
> URL: https://issues.apache.org/jira/browse/HIVE-16896
> Project: Hive
>  Issue Type: Sub-task
>Reporter: anishek
>Assignee: anishek
> Attachments: HIVE-16896.1.patch, HIVE-16896.2.patch, 
> HIVE-16896.3.patch
>
>
> we want to not create too many tasks in memory in the analysis phase while 
> loading data. Currently we load all the files in the bootstrap dump location 
> as {{FileStatus[]}} and then iterate over it to load objects, we should 
> rather move to 
> {code}
> org.apache.hadoop.fs.RemoteIteratorlistFiles(Path 
> f, boolean recursive)
> {code}
> which would internally batch and return values. 
> additionally since we cant hand off partial tasks from analysis pahse => 
> execution phase, we are going to move the whole repl load functionality to 
> execution phase so we can better control creation/execution of tasks (not 
> related to hive {{Task}}, we may get rid of ReplCopyTask)
> Additional consideration to take into account at the end of this jira is to 
> see if we want to specifically do a multi threaded load of bootstrap dump.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17240) function acos(2) should be null

[Yuming Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuming Wang updated HIVE-17240:
---
Assignee: Yuming Wang
  Status: Patch Available  (was: Open)

> function acos(2) should be null
> ---
>
> Key: HIVE-17240
> URL: https://issues.apache.org/jira/browse/HIVE-17240
> Project: Hive
>  Issue Type: Bug
>  Components: UDF
>Affects Versions: 2.2.0, 1.2.2, 1.1.1
>Reporter: Yuming Wang
>Assignee: Yuming Wang
> Attachments: HIVE-17240.1.patch
>
>
> {{acos(2)}} should be null, same as MySQL:
> {code:sql}
> hive> desc function extended acos;
> OK
> acos(x) - returns the arc cosine of x if -1<=x<=1 or NULL otherwise
> Example:
>   > SELECT acos(1) FROM src LIMIT 1;
>   0
>   > SELECT acos(2) FROM src LIMIT 1;
>   NULL
> Time taken: 0.009 seconds, Fetched: 6 row(s)
> hive> select acos(2);
> OK
> NaN
> Time taken: 0.437 seconds, Fetched: 1 row(s)
> {code}
> {code:sql}
> mysql>  select acos(2);
> +-+
> | acos(2) |
> +-+
> |NULL |
> +-+
> 1 row in set (0.00 sec)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)