[jira] [Created] (HIVE-24289) RetryingMetaStoreClient should not retry connecting to HMS on genuine errors

2020-10-19 Thread Rajesh Balamohan (Jira) 
Rajesh Balamohan created HIVE-24289:
---

 Summary: RetryingMetaStoreClient should not retry connecting to 
HMS on genuine errors
 Key: HIVE-24289
 URL: https://issues.apache.org/jira/browse/HIVE-24289
 Project: Hive
  Issue Type: Improvement
Reporter: Rajesh Balamohan


When there is genuine error from HMS, it should not be retried in 
RetryingMetaStoreClient. 

For e.g, following query would be retried multiple times (~20+ times) in HMS 
causing huge delay in processing, even though this constraint is available in 
HMS. 

It should just throw exception to client and stop retrying in such cases.

{noformat}
alter table web_sales add constraint tpcds_bin_partitioned_orc_1_ws_s_hd 
foreign key  (ws_ship_hdemo_sk) references household_demographics (hd_demo_sk) 
disable novalidate rely;

org.apache.hadoop.hive.ql.metadata.HiveException: 
org.apache.thrift.TApplicationException: Internal error processing 
add_foreign_key
at org.apache.hadoop.hive.ql.metadata.Hive.addForeignKey(Hive.java:5914)
..
...
Caused by: org.apache.thrift.TApplicationException: Internal error processing 
add_foreign_key
   at 
org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
   at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
   at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_add_foreign_key(ThriftHiveMetastore.java:1872)
{noformat}

https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/RetryingMetaStoreClient.java#L256

For e.g, if exception contains "Internal error processing ", it could stop 
retrying all over again.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-24288) Files created by CompileProcessor have incorrect permissions

2020-10-19 Thread Naveen Gangam (Jira) 
Naveen Gangam created HIVE-24288:


 Summary: Files created by CompileProcessor have incorrect 
permissions
 Key: HIVE-24288
 URL: https://issues.apache.org/jira/browse/HIVE-24288
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 3.1.0
Reporter: Naveen Gangam
Assignee: Naveen Gangam


Compile processor generates some temporary files as part of processing.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-24287) Cookie Signer class should use SHA-512 instead SHA-256 for cookie signature

2020-10-19 Thread Sai Hemanth Gantasala (Jira) 
Sai Hemanth Gantasala created HIVE-24287:


 Summary: Cookie Signer class should use SHA-512 instead SHA-256 
for cookie signature
 Key: HIVE-24287
 URL: https://issues.apache.org/jira/browse/HIVE-24287
 Project: Hive
  Issue Type: Bug
Reporter: Sai Hemanth Gantasala
Assignee: Sai Hemanth Gantasala


private static final String SHA_STRING = "SHA-256"; should use SHA-512 instead



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-24286) Render date and time with progress of Hive on Tez

2020-10-19 Thread okumin (Jira) 
okumin created HIVE-24286:
-

 Summary: Render date and time with progress of Hive on Tez
 Key: HIVE-24286
 URL: https://issues.apache.org/jira/browse/HIVE-24286
 Project: Hive
  Issue Type: Improvement
Affects Versions: 4.0.0
Reporter: okumin
Assignee: okumin


Add date/time to each line written by RenderStrategy like MapReduce and Spark.

 
 * 
[https://github.com/apache/hive/blob/31c1658d9884eb4f31b06eaa718dfef8b1d92d22/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/HadoopJobExecHelper.java#L350]
 * 
[https://github.com/apache/hive/blob/31c1658d9884eb4f31b06eaa718dfef8b1d92d22/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/RenderStrategy.java#L64-L67]

 

This ticket would add the current time to the head of each line.

 
{code:java}
2020-10-19 13:32:41,162 Map 1: 0/1  Reducer 2: 0/1  
2020-10-19 13:32:44,231 Map 1: 0/1  Reducer 2: 0/1  
2020-10-19 13:32:46,813 Map 1: 0(+1)/1  Reducer 2: 0/1  
2020-10-19 13:32:49,878 Map 1: 0(+1)/1  Reducer 2: 0/1  
2020-10-19 13:32:51,416 Map 1: 1/1  Reducer 2: 0/1  
2020-10-19 13:32:51,936 Map 1: 1/1  Reducer 2: 0(+1)/1  
2020-10-19 13:32:52,877 Map 1: 1/1  Reducer 2: 1/1  
{code}
 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-24285) GlobalLimitOptimizer not working

2020-10-19 Thread lucusguo (Jira) 
lucusguo created HIVE-24285:
---

 Summary: GlobalLimitOptimizer not working
 Key: HIVE-24285
 URL: https://issues.apache.org/jira/browse/HIVE-24285
 Project: Hive
  Issue Type: Bug
  Components: Query Processor
Affects Versions: 3.1.2, 2.0.1
Reporter: lucusguo
Assignee: Jesus Camacho Rodriguez
 Attachments: image-2020-10-19-18-59-05-302.png, 
image-2020-10-19-19-00-05-123.png, image-2020-10-19-19-00-23-487.png

In our environment，we have set 
hive.limit.optimize.enable=true,hive.limit.row.max.size=1000,hive.limit.optimize.fetch.max=1000,hive.fetch.task.conversion.threshold=256,hive.fetch.task.conversion=more,hive.auto.convert.join=true

when we execute the sql like this,GlobalLimitOptimizer not working.

select db_name,concat('22',cnt) from (select * from lb1 limit 5) t1;

because in this sql there is one LimitOperator, but logs count 2

the code and logs below

 

!image-2020-10-19-19-00-23-487.png!

 

!image-2020-10-19-19-00-05-123.png!

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)