[jira] [Resolved] (HIVE-25857) Replication fails in case of Control Character in the table description

2022-01-17 Thread Anishek Agarwal (Jira) 


 [ 
https://issues.apache.org/jira/browse/HIVE-25857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anishek Agarwal resolved HIVE-25857.

Resolution: Fixed

+1,  Merged to master. Thanks for the patch [~ayushtkn]

> Replication fails in case of Control Character in the table description
> ---
>
> Key: HIVE-25857
> URL: https://issues.apache.org/jira/browse/HIVE-25857
> Project: Hive
>  Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> In case there is a control character in the table metadata. The LOAD fails 
> while decoding the JSON.
> *Exception:*
> {noformat}
> Caused by: com.fasterxml.jackson.core.JsonParseException: Illegal unquoted 
> character ((CTRL-CHAR, code 24)): has to be escaped using backslash to be 
> included in string value
>  at [Source: 
> (String)"{"server":"","servicePrincipal":"","db":"sampletestreplic","table":"testlmo","tableType":"MANAGED_TABLE","tableObjBeforeJson":"{\"1\":{\"str\":\"testlmo\"},\"2\":{\"str\":\"sampletestreplic\"},\"3\":{\"str\":\"hive\"},\"4\":{\"i32\":1641717786},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"dc_codeacteurcandidat\"},\"2\":{\"str\":\"string\"},\"3\":{\"str\":\"Code
>  de l'acteur de candidature (^XA' a dterminer, ^XC' conseiller ou ^XD' 
> candidat)\"}}]},\"[truncated 3054 chars]; line: 1, column: 445]
>         at 
> com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1840) 
> ~[jackson-core-2.10.5.jar:2.10.5]
>         at 
> com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:712)
>  ~[jackson-core-2.10.5.jar:2.10.5]
>         at 
> com.fasterxml.jackson.core.base.ParserBase._throwUnquotedSpace(ParserBase.java:1046)
>  ~[jackson-core-2.10.5.jar:2.10.5]
>         at 
> com.fasterxml.jackson.core.json.ReaderBasedJsonParser._finishString2(ReaderBasedJsonParser.java:2073)
>  ~[jackson-core-2.10.5.jar:2.10.5]
>         at 
> com.fasterxml.jackson.core.json.ReaderBasedJsonParser._finishString(ReaderBasedJsonParser.java:2044)
>  ~[jackson-core-2.10.5.jar:2.10.5]
>         at 
> com.fasterxml.jackson.core.json.ReaderBasedJsonParser.getText(ReaderBasedJsonParser.java:293)
>  ~[jackson-core-2.10.5.jar:2.10.5]
>         at 
> com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:35)
>  ~[jackson-databind-2.10.5.1.jar:2.10.5.1]
>         at 
> com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:10)
>  ~[jackson-databind-2.10.5.1.jar:2.10.5.1]
>         at 
> com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:138)
>  ~[jackson-databind-2.10.5.1.jar:2.10.5.1]
>         at 
> com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:288)
>  ~[jackson-databind-2.10.5.1.jar:2.10.5.1]
>         at 
> com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:151)
>  ~[jackson-databind-2.10.5.1.jar:2.10.5.1]
>         at 
> com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4218)
>  ~[jackson-databind-2.10.5.1.jar:2.10.5.1]
>         at 
> com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3214) 
>         at 
> com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3182) 
>         at 
> org.apache.hadoop.hive.metastore.messaging.json.JSONMessageDeserializer.getAlterTableMessage(JSONMessageDeserializer.java:111)
>  
>         at 
> org.apache.hadoop.hive.ql.parse.repl.load.message.TableHandler.extract(TableHandler.java:111)]
>         at 
> org.apache.hadoop.hive.ql.parse.repl.load.message.TableHandler.handle(TableHandler.java:51)
>  
>         at 
> org.apache.hadoop.hive.ql.exec.repl.incremental.IncrementalLoadTasksBuilder.analyzeEventLoad(IncrementalLoadTasksBuilder.java:213){noformat}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Assigned] (HIVE-25873) Fix nested partition statements in Explain DDL

2022-01-17 Thread Harshit Gupta (Jira) 


 [ 
https://issues.apache.org/jira/browse/HIVE-25873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harshit Gupta reassigned HIVE-25873:



> Fix nested partition statements in Explain DDL
> --
>
> Key: HIVE-25873
> URL: https://issues.apache.org/jira/browse/HIVE-25873
> Project: Hive
>  Issue Type: Bug
>Reporter: Harshit Gupta
>Assignee: Harshit Gupta
>Priority: Major
>
> Explain ddl doesn't generate proper statements for nested partitions.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (HIVE-25857) Replication fails in case of Control Character in the table description

2022-01-17 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HIVE-25857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated HIVE-25857:
--
Labels: pull-request-available  (was: )

> Replication fails in case of Control Character in the table description
> ---
>
> Key: HIVE-25857
> URL: https://issues.apache.org/jira/browse/HIVE-25857
> Project: Hive
>  Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> In case there is a control character in the table metadata. The LOAD fails 
> while decoding the JSON.
> *Exception:*
> {noformat}
> Caused by: com.fasterxml.jackson.core.JsonParseException: Illegal unquoted 
> character ((CTRL-CHAR, code 24)): has to be escaped using backslash to be 
> included in string value
>  at [Source: 
> (String)"{"server":"","servicePrincipal":"","db":"sampletestreplic","table":"testlmo","tableType":"MANAGED_TABLE","tableObjBeforeJson":"{\"1\":{\"str\":\"testlmo\"},\"2\":{\"str\":\"sampletestreplic\"},\"3\":{\"str\":\"hive\"},\"4\":{\"i32\":1641717786},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"dc_codeacteurcandidat\"},\"2\":{\"str\":\"string\"},\"3\":{\"str\":\"Code
>  de l'acteur de candidature (^XA' a dterminer, ^XC' conseiller ou ^XD' 
> candidat)\"}}]},\"[truncated 3054 chars]; line: 1, column: 445]
>         at 
> com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1840) 
> ~[jackson-core-2.10.5.jar:2.10.5]
>         at 
> com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:712)
>  ~[jackson-core-2.10.5.jar:2.10.5]
>         at 
> com.fasterxml.jackson.core.base.ParserBase._throwUnquotedSpace(ParserBase.java:1046)
>  ~[jackson-core-2.10.5.jar:2.10.5]
>         at 
> com.fasterxml.jackson.core.json.ReaderBasedJsonParser._finishString2(ReaderBasedJsonParser.java:2073)
>  ~[jackson-core-2.10.5.jar:2.10.5]
>         at 
> com.fasterxml.jackson.core.json.ReaderBasedJsonParser._finishString(ReaderBasedJsonParser.java:2044)
>  ~[jackson-core-2.10.5.jar:2.10.5]
>         at 
> com.fasterxml.jackson.core.json.ReaderBasedJsonParser.getText(ReaderBasedJsonParser.java:293)
>  ~[jackson-core-2.10.5.jar:2.10.5]
>         at 
> com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:35)
>  ~[jackson-databind-2.10.5.1.jar:2.10.5.1]
>         at 
> com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:10)
>  ~[jackson-databind-2.10.5.1.jar:2.10.5.1]
>         at 
> com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:138)
>  ~[jackson-databind-2.10.5.1.jar:2.10.5.1]
>         at 
> com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:288)
>  ~[jackson-databind-2.10.5.1.jar:2.10.5.1]
>         at 
> com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:151)
>  ~[jackson-databind-2.10.5.1.jar:2.10.5.1]
>         at 
> com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4218)
>  ~[jackson-databind-2.10.5.1.jar:2.10.5.1]
>         at 
> com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3214) 
>         at 
> com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3182) 
>         at 
> org.apache.hadoop.hive.metastore.messaging.json.JSONMessageDeserializer.getAlterTableMessage(JSONMessageDeserializer.java:111)
>  
>         at 
> org.apache.hadoop.hive.ql.parse.repl.load.message.TableHandler.extract(TableHandler.java:111)]
>         at 
> org.apache.hadoop.hive.ql.parse.repl.load.message.TableHandler.handle(TableHandler.java:51)
>  
>         at 
> org.apache.hadoop.hive.ql.exec.repl.incremental.IncrementalLoadTasksBuilder.analyzeEventLoad(IncrementalLoadTasksBuilder.java:213){noformat}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Work logged] (HIVE-25857) Replication fails in case of Control Character in the table description

2022-01-17 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HIVE-25857?focusedWorklogId=710283=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-710283
 ]

ASF GitHub Bot logged work on HIVE-25857:
-

Author: ASF GitHub Bot
Created on: 18/Jan/22 06:12
Start Date: 18/Jan/22 06:12
Worklog Time Spent: 10m 
  Work Description: anishek commented on pull request #2935:
URL: https://github.com/apache/hive/pull/2935#issuecomment-1015100649


   +1
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 710283)
Remaining Estimate: 0h
Time Spent: 10m

> Replication fails in case of Control Character in the table description
> ---
>
> Key: HIVE-25857
> URL: https://issues.apache.org/jira/browse/HIVE-25857
> Project: Hive
>  Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> In case there is a control character in the table metadata. The LOAD fails 
> while decoding the JSON.
> *Exception:*
> {noformat}
> Caused by: com.fasterxml.jackson.core.JsonParseException: Illegal unquoted 
> character ((CTRL-CHAR, code 24)): has to be escaped using backslash to be 
> included in string value
>  at [Source: 
> (String)"{"server":"","servicePrincipal":"","db":"sampletestreplic","table":"testlmo","tableType":"MANAGED_TABLE","tableObjBeforeJson":"{\"1\":{\"str\":\"testlmo\"},\"2\":{\"str\":\"sampletestreplic\"},\"3\":{\"str\":\"hive\"},\"4\":{\"i32\":1641717786},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"dc_codeacteurcandidat\"},\"2\":{\"str\":\"string\"},\"3\":{\"str\":\"Code
>  de l'acteur de candidature (^XA' a dterminer, ^XC' conseiller ou ^XD' 
> candidat)\"}}]},\"[truncated 3054 chars]; line: 1, column: 445]
>         at 
> com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1840) 
> ~[jackson-core-2.10.5.jar:2.10.5]
>         at 
> com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:712)
>  ~[jackson-core-2.10.5.jar:2.10.5]
>         at 
> com.fasterxml.jackson.core.base.ParserBase._throwUnquotedSpace(ParserBase.java:1046)
>  ~[jackson-core-2.10.5.jar:2.10.5]
>         at 
> com.fasterxml.jackson.core.json.ReaderBasedJsonParser._finishString2(ReaderBasedJsonParser.java:2073)
>  ~[jackson-core-2.10.5.jar:2.10.5]
>         at 
> com.fasterxml.jackson.core.json.ReaderBasedJsonParser._finishString(ReaderBasedJsonParser.java:2044)
>  ~[jackson-core-2.10.5.jar:2.10.5]
>         at 
> com.fasterxml.jackson.core.json.ReaderBasedJsonParser.getText(ReaderBasedJsonParser.java:293)
>  ~[jackson-core-2.10.5.jar:2.10.5]
>         at 
> com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:35)
>  ~[jackson-databind-2.10.5.1.jar:2.10.5.1]
>         at 
> com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:10)
>  ~[jackson-databind-2.10.5.1.jar:2.10.5.1]
>         at 
> com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:138)
>  ~[jackson-databind-2.10.5.1.jar:2.10.5.1]
>         at 
> com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:288)
>  ~[jackson-databind-2.10.5.1.jar:2.10.5.1]
>         at 
> com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:151)
>  ~[jackson-databind-2.10.5.1.jar:2.10.5.1]
>         at 
> com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4218)
>  ~[jackson-databind-2.10.5.1.jar:2.10.5.1]
>         at 
> com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3214) 
>         at 
> com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3182) 
>         at 
> org.apache.hadoop.hive.metastore.messaging.json.JSONMessageDeserializer.getAlterTableMessage(JSONMessageDeserializer.java:111)
>  
>         at 
> org.apache.hadoop.hive.ql.parse.repl.load.message.TableHandler.extract(TableHandler.java:111)]
>         at 
> org.apache.hadoop.hive.ql.parse.repl.load.message.TableHandler.handle(TableHandler.java:51)
>  
>         at 
> org.apache.hadoop.hive.ql.exec.repl.incremental.IncrementalLoadTasksBuilder.analyzeEventLoad(IncrementalLoadTasksBuilder.java:213){noformat}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Commented] (HIVE-25857) Replication fails in case of Control Character in the table description

2022-01-17 Thread Anishek Agarwal (Jira) 


[ 
https://issues.apache.org/jira/browse/HIVE-25857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17477593#comment-17477593
 ] 

Anishek Agarwal commented on HIVE-25857:


not sure why the git change didnt gete linked automatically 
https://github.com/apache/hive/pull/2935

> Replication fails in case of Control Character in the table description
> ---
>
> Key: HIVE-25857
> URL: https://issues.apache.org/jira/browse/HIVE-25857
> Project: Hive
>  Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
>
> In case there is a control character in the table metadata. The LOAD fails 
> while decoding the JSON.
> *Exception:*
> {noformat}
> Caused by: com.fasterxml.jackson.core.JsonParseException: Illegal unquoted 
> character ((CTRL-CHAR, code 24)): has to be escaped using backslash to be 
> included in string value
>  at [Source: 
> (String)"{"server":"","servicePrincipal":"","db":"sampletestreplic","table":"testlmo","tableType":"MANAGED_TABLE","tableObjBeforeJson":"{\"1\":{\"str\":\"testlmo\"},\"2\":{\"str\":\"sampletestreplic\"},\"3\":{\"str\":\"hive\"},\"4\":{\"i32\":1641717786},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"dc_codeacteurcandidat\"},\"2\":{\"str\":\"string\"},\"3\":{\"str\":\"Code
>  de l'acteur de candidature (^XA' a dterminer, ^XC' conseiller ou ^XD' 
> candidat)\"}}]},\"[truncated 3054 chars]; line: 1, column: 445]
>         at 
> com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1840) 
> ~[jackson-core-2.10.5.jar:2.10.5]
>         at 
> com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:712)
>  ~[jackson-core-2.10.5.jar:2.10.5]
>         at 
> com.fasterxml.jackson.core.base.ParserBase._throwUnquotedSpace(ParserBase.java:1046)
>  ~[jackson-core-2.10.5.jar:2.10.5]
>         at 
> com.fasterxml.jackson.core.json.ReaderBasedJsonParser._finishString2(ReaderBasedJsonParser.java:2073)
>  ~[jackson-core-2.10.5.jar:2.10.5]
>         at 
> com.fasterxml.jackson.core.json.ReaderBasedJsonParser._finishString(ReaderBasedJsonParser.java:2044)
>  ~[jackson-core-2.10.5.jar:2.10.5]
>         at 
> com.fasterxml.jackson.core.json.ReaderBasedJsonParser.getText(ReaderBasedJsonParser.java:293)
>  ~[jackson-core-2.10.5.jar:2.10.5]
>         at 
> com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:35)
>  ~[jackson-databind-2.10.5.1.jar:2.10.5.1]
>         at 
> com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:10)
>  ~[jackson-databind-2.10.5.1.jar:2.10.5.1]
>         at 
> com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:138)
>  ~[jackson-databind-2.10.5.1.jar:2.10.5.1]
>         at 
> com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:288)
>  ~[jackson-databind-2.10.5.1.jar:2.10.5.1]
>         at 
> com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:151)
>  ~[jackson-databind-2.10.5.1.jar:2.10.5.1]
>         at 
> com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4218)
>  ~[jackson-databind-2.10.5.1.jar:2.10.5.1]
>         at 
> com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3214) 
>         at 
> com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3182) 
>         at 
> org.apache.hadoop.hive.metastore.messaging.json.JSONMessageDeserializer.getAlterTableMessage(JSONMessageDeserializer.java:111)
>  
>         at 
> org.apache.hadoop.hive.ql.parse.repl.load.message.TableHandler.extract(TableHandler.java:111)]
>         at 
> org.apache.hadoop.hive.ql.parse.repl.load.message.TableHandler.handle(TableHandler.java:51)
>  
>         at 
> org.apache.hadoop.hive.ql.exec.repl.incremental.IncrementalLoadTasksBuilder.analyzeEventLoad(IncrementalLoadTasksBuilder.java:213){noformat}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Assigned] (HIVE-25872) Skip tracking of alterDatabase events for replication specific property (repl.last.id).

2022-01-17 Thread Haymant Mangla (Jira) 


 [ 
https://issues.apache.org/jira/browse/HIVE-25872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haymant Mangla reassigned HIVE-25872:
-


> Skip tracking of alterDatabase events for replication specific property 
> (repl.last.id).
> ---
>
> Key: HIVE-25872
> URL: https://issues.apache.org/jira/browse/HIVE-25872
> Project: Hive
>  Issue Type: Improvement
>Reporter: Haymant Mangla
>Assignee: Haymant Mangla
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Work logged] (HIVE-25143) Improve ERROR Logging in QL Package

2022-01-17 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HIVE-25143?focusedWorklogId=710196=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-710196
 ]

ASF GitHub Bot logged work on HIVE-25143:
-

Author: ASF GitHub Bot
Created on: 18/Jan/22 00:13
Start Date: 18/Jan/22 00:13
Worklog Time Spent: 10m 
  Work Description: github-actions[bot] commented on pull request #2301:
URL: https://github.com/apache/hive/pull/2301#issuecomment-1014967189


   This pull request has been automatically marked as stale because it has not 
had recent activity. It will be closed if no further activity occurs.
   Feel free to reach out on the d...@hive.apache.org list if the patch is in 
need of reviews.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 710196)
Time Spent: 40m  (was: 0.5h)

> Improve ERROR Logging in QL Package
> ---
>
> Key: HIVE-25143
> URL: https://issues.apache.org/jira/browse/HIVE-25143
> Project: Hive
>  Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> I went through and reviewed all of the ERROR logging in the HS2 {{ql}} module 
> and I removed (most of) the following bad habits:
>  
>  * Log-and-Throw (log or throw, not both)
>  * Pass in the Exception to the logging framework instead of logging its 
> toString() : LOG.error("alter table update columns: {}", e);
>  * Add additional context instead of copying the message from the wrapped 
> Exception : throw new SemanticException(e.getMessage(), e);
>  * The wrapped exception is being lost in some case, though the message 
> survives :  throw new HiveException(e.getMessage());
>  * Remove new-lines from Exception messages, this is annoying as log messages 
> should all be on a single line for GREP
>  * Not logging the Exception stack trace :  LOG.error("Error in close loader: 
> " + ie);
>  * Logging information but not passing it into an Exception for bubbling up:  
> LOG.error("Failed to return session: {} to pool", session, e); throw e;
>  * Other miscellaneous improvements



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Work logged] (HIVE-25871) Hive should set name mapping table property for migrated Iceberg tables

2022-01-17 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HIVE-25871?focusedWorklogId=710112=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-710112
 ]

ASF GitHub Bot logged work on HIVE-25871:
-

Author: ASF GitHub Bot
Created on: 17/Jan/22 19:01
Start Date: 17/Jan/22 19:01
Worklog Time Spent: 10m 
  Work Description: boroknagyz opened a new pull request #2948:
URL: https://github.com/apache/hive/pull/2948


   …Iceberg tables
   
   
   
   ### What changes were proposed in this pull request?
   With this PR, Hive will set table property 'schema.name-mapping.default' for 
migrated Iceberg tables.
   The value of this property contains a mapping between Iceberg field ids and 
column names.
   
   ### Why are the changes needed?
   This table property is useful for column projection of legacy data files:
   https://iceberg.apache.org/#spec/#column-projection
   
   ### Does this PR introduce _any_ user-facing change?
   When users migrate a legacy table to Iceberg, this new table property will 
be set.
   
   ### How was this patch tested?
   Extended unit test TestHiveIcebergMigration.
   Q tests.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 710112)
Remaining Estimate: 0h
Time Spent: 10m

> Hive should set name mapping table property for migrated Iceberg tables
> ---
>
> Key: HIVE-25871
> URL: https://issues.apache.org/jira/browse/HIVE-25871
> Project: Hive
>  Issue Type: Bug
>Reporter: Zoltán Borók-Nagy
>Assignee: Zoltán Borók-Nagy
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Hive should  set the name-mapping table property during table migration.
> It would be useful for [column 
> projection|https://iceberg.apache.org/#spec/#column-projection] for files 
> without field ids.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (HIVE-25871) Hive should set name mapping table property for migrated Iceberg tables

2022-01-17 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HIVE-25871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated HIVE-25871:
--
Labels: pull-request-available  (was: )

> Hive should set name mapping table property for migrated Iceberg tables
> ---
>
> Key: HIVE-25871
> URL: https://issues.apache.org/jira/browse/HIVE-25871
> Project: Hive
>  Issue Type: Bug
>Reporter: Zoltán Borók-Nagy
>Assignee: Zoltán Borók-Nagy
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Hive should  set the name-mapping table property during table migration.
> It would be useful for [column 
> projection|https://iceberg.apache.org/#spec/#column-projection] for files 
> without field ids.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Assigned] (HIVE-25871) Hive should set name mapping table property for migrated Iceberg tables

2022-01-17 Thread Jira 


 [ 
https://issues.apache.org/jira/browse/HIVE-25871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zoltán Borók-Nagy reassigned HIVE-25871:


Assignee: Zoltán Borók-Nagy

> Hive should set name mapping table property for migrated Iceberg tables
> ---
>
> Key: HIVE-25871
> URL: https://issues.apache.org/jira/browse/HIVE-25871
> Project: Hive
>  Issue Type: Bug
>Reporter: Zoltán Borók-Nagy
>Assignee: Zoltán Borók-Nagy
>Priority: Major
>
> Hive should  set the name-mapping table property during table migration.
> It would be useful for [column 
> projection|https://iceberg.apache.org/#spec/#column-projection] for files 
> without field ids.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (HIVE-25870) Avoid simplification in HivePointLookupOptimizerRule, convert only

2022-01-17 Thread Alessandro Solimando (Jira) 


 [ 
https://issues.apache.org/jira/browse/HIVE-25870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alessandro Solimando updated HIVE-25870:

Summary: Avoid simplification in HivePointLookupOptimizerRule, convert only 
 (was: Make HivePointLookupOptimizerRule to just convert OR to IN, no 
simplification)

> Avoid simplification in HivePointLookupOptimizerRule, convert only
> --
>
> Key: HIVE-25870
> URL: https://issues.apache.org/jira/browse/HIVE-25870
> Project: Hive
>  Issue Type: Improvement
>  Components: CBO
>Affects Versions: 4.0.0
>Reporter: Alessandro Solimando
>Priority: Major
>
> _HivePointLookupOptimizerRule_ has been introduced to improve simplifications 
> and improve statistics/estimations (see 
> https://issues.apache.org/jira/browse/HIVE-11424?focusedCommentId=15197407=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15197407).
> At the time, _RexSimplify_ could not simplify such OR/AND expressions (as 
> reported in the JIRA above):
> {noformat}
> simplify(unknown as unknown): AND(true, OR(=(?0.int0, 1), =(?0.int0, 2), 
> =(?0.int0, 3)), OR(AND(true, =(?0.int0, 1)), AND(true, =(?0.int0, 2
> {noformat}
> For Calcite <= 1.25, simplification is still missed:
> {noformat}
> Expected: "OR(?0.int0=1, ?0.int0=2)"
>  but: was "AND(OR(=(?0.int0, 1), =(?0.int0, 2), =(?0.int0, 3)), 
> OR(=(?0.int0, 1), =(?0.int0, 2)))"
> {noformat}
> From Calcite >= 1.26, the simplifications happens:
> {noformat}
> Expected: "OR(?0.int0=1, ?0.int0=2)"
>  but: was "SEARCH(?0.int0, Sarg[1, 2])"
> {noformat}
> For this reason, once we upgrade to Calcite >= 1.26, we could drop all the 
> simplifications within the rule, just keep the OR -> IN conversion, and move 
> the rule to the very last planning stage.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (HIVE-25870) Make HivePointLookupOptimizerRule to just convert OR to IN, no simplification

2022-01-17 Thread Alessandro Solimando (Jira) 


 [ 
https://issues.apache.org/jira/browse/HIVE-25870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alessandro Solimando updated HIVE-25870:

Description: 
_HivePointLookupOptimizerRule_ has been introduced to improve simplifications 
and improve statistics/estimations (see 
https://issues.apache.org/jira/browse/HIVE-11424?focusedCommentId=15197407=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15197407).

At the time, _RexSimplify_ could not simplify such OR/AND expressions (as 
reported in the JIRA above):

{noformat}
simplify(unknown as unknown): AND(true, OR(=(?0.int0, 1), =(?0.int0, 2), 
=(?0.int0, 3)), OR(AND(true, =(?0.int0, 1)), AND(true, =(?0.int0, 2
{noformat}

For Calcite <= 1.25, simplification is still missed:
{noformat}
Expected: "OR(?0.int0=1, ?0.int0=2)"
 but: was "AND(OR(=(?0.int0, 1), =(?0.int0, 2), =(?0.int0, 3)), 
OR(=(?0.int0, 1), =(?0.int0, 2)))"
{noformat}

>From Calcite >= 1.26, the simplifications happens:
{noformat}
Expected: "OR(?0.int0=1, ?0.int0=2)"
 but: was "SEARCH(?0.int0, Sarg[1, 2])"
{noformat}

For this reason, once we upgrade to Calcite >= 1.26, we could drop all the 
simplifications within the rule, just keep the OR -> IN conversion, and move 
the rule to the very last planning stage.

  was:
_HivePointLookupOptimizerRule_ has been introduced to improve simplifications 
and improve statistics/estimations (see 
https://issues.apache.org/jira/browse/HIVE-11424?focusedCommentId=15197407=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15197407).

At the time, _RexSimplify_ could not simplify such OR/AND expressions (as 
reported in the JIRA above):

{noformat}
simplify(unknown as unknown): AND(true, OR(=(?0.int0, 1), =(?0.int0, 2), 
=(?0.int0, 3)), OR(AND(true, =(?0.int0, 1)), AND(true, =(?0.int0, 2
{noformat}

For Calcite <= 1.25, simplification is still missed:
{noformat}
Expected: "OR(?0.int0=1, ?0.int0=2)"
 but: was "AND(OR(=(?0.int0, 1), =(?0.int0, 2), =(?0.int0, 3)), 
OR(=(?0.int0, 1), =(?0.int0, 2)))"
{noformat}

>From Calcite >= 1.26, the simplifications happens:
{noformat}
Expected: "OR(?0.int0=1, ?0.int0=2)"
 but: was "SEARCH(?0.int0, Sarg[1, 2])"
{noformat}

For this reason, we could drop all the simplifications within the rule, just 
keep the OR -> IN conversion, and move the rule to the very last planning stage.


> Make HivePointLookupOptimizerRule to just convert OR to IN, no simplification
> -
>
> Key: HIVE-25870
> URL: https://issues.apache.org/jira/browse/HIVE-25870
> Project: Hive
>  Issue Type: Improvement
>  Components: CBO
>Affects Versions: 4.0.0
>Reporter: Alessandro Solimando
>Priority: Major
>
> _HivePointLookupOptimizerRule_ has been introduced to improve simplifications 
> and improve statistics/estimations (see 
> https://issues.apache.org/jira/browse/HIVE-11424?focusedCommentId=15197407=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15197407).
> At the time, _RexSimplify_ could not simplify such OR/AND expressions (as 
> reported in the JIRA above):
> {noformat}
> simplify(unknown as unknown): AND(true, OR(=(?0.int0, 1), =(?0.int0, 2), 
> =(?0.int0, 3)), OR(AND(true, =(?0.int0, 1)), AND(true, =(?0.int0, 2
> {noformat}
> For Calcite <= 1.25, simplification is still missed:
> {noformat}
> Expected: "OR(?0.int0=1, ?0.int0=2)"
>  but: was "AND(OR(=(?0.int0, 1), =(?0.int0, 2), =(?0.int0, 3)), 
> OR(=(?0.int0, 1), =(?0.int0, 2)))"
> {noformat}
> From Calcite >= 1.26, the simplifications happens:
> {noformat}
> Expected: "OR(?0.int0=1, ?0.int0=2)"
>  but: was "SEARCH(?0.int0, Sarg[1, 2])"
> {noformat}
> For this reason, once we upgrade to Calcite >= 1.26, we could drop all the 
> simplifications within the rule, just keep the OR -> IN conversion, and move 
> the rule to the very last planning stage.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Work logged] (HIVE-25842) Reimplement delta file metric collection

2022-01-17 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HIVE-25842?focusedWorklogId=709960=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-709960
 ]

ASF GitHub Bot logged work on HIVE-25842:
-

Author: ASF GitHub Bot
Created on: 17/Jan/22 14:34
Start Date: 17/Jan/22 14:34
Worklog Time Spent: 10m 
  Work Description: klcopp commented on a change in pull request #2916:
URL: https://github.com/apache/hive/pull/2916#discussion_r782811556



##
File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java
##
@@ -142,6 +144,9 @@ public void init(AtomicBoolean stop) throws Exception {
 super.init(stop);
 this.workerName = getWorkerId();
 setName(workerName);
+metricsEnabled = MetastoreConf.getBoolVar(conf, 
MetastoreConf.ConfVars.METRICS_ENABLED) &&
+MetastoreConf.getBoolVar(conf, 
MetastoreConf.ConfVars.METASTORE_ACIDMETRICS_EXT_ON) &&
+MetastoreConf.getBoolVar(conf, 
MetastoreConf.ConfVars.COMPACTOR_INITIATOR_ON);

Review comment:
   COMPACTOR_INITIATOR_ON could be false the HS2 running this Worker, but 
metrics should still be collected.

##
File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java
##
@@ -87,14 +106,15 @@ public void init(AtomicBoolean stop) throws Exception {
 cleanerExecutor = CompactorUtil.createExecutorWithThreadFactory(
 
conf.getIntVar(HiveConf.ConfVars.HIVE_COMPACTOR_CLEANER_THREADS_NUM),
 COMPACTOR_CLEANER_THREAD_NAME_FORMAT);
+metricsEnabled = MetastoreConf.getBoolVar(conf, 
MetastoreConf.ConfVars.METRICS_ENABLED) &&
+MetastoreConf.getBoolVar(conf, 
MetastoreConf.ConfVars.METASTORE_ACIDMETRICS_EXT_ON) &&
+MetastoreConf.getBoolVar(conf, 
MetastoreConf.ConfVars.COMPACTOR_INITIATOR_ON);

Review comment:
   COMPACTOR_INITIATOR_ON also controls whether the Cleaner runs, so this 
line is unnecessary... but if you want to leave it in for posterity/the future, 
I understand.

##
File path: 
standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift
##
@@ -2926,11 +2945,15 @@ PartitionsResponse 
get_partitions_req(1:PartitionsRequest req)
   void mark_cleaned(1:CompactionInfoStruct cr) throws(1:MetaException o1)
   void mark_compacted(1: CompactionInfoStruct cr) throws(1:MetaException o1)
   void mark_failed(1: CompactionInfoStruct cr) throws(1:MetaException o1)
+  CompactionMetricsDataResponse get_compaction_metrics_data(1: string dbName, 
2: string tblName, 3: string partitionName, 4: CompactionMetricsMetricType 
type) throws(1:MetaException o1)

Review comment:
   If there's any doubt that the parameters might be changed, I recommend 
introducing a CompactionMetricsDataRequest object

##
File path: 
ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/TestDeltaFilesMetrics.java
##
@@ -39,199 +45,396 @@
 import javax.management.MBeanServer;
 import javax.management.ObjectName;
 import java.lang.management.ManagementFactory;
-import java.util.EnumMap;
+import java.util.ArrayList;
 import java.util.HashMap;
+import java.util.List;
 import java.util.Map;
-import java.util.Queue;
 import java.util.concurrent.TimeUnit;
 
-import static 
org.apache.hadoop.hive.ql.txn.compactor.metrics.DeltaFilesMetricReporter.DeltaFilesMetricType.NUM_DELTAS;
-import static 
org.apache.hadoop.hive.ql.txn.compactor.metrics.DeltaFilesMetricReporter.DeltaFilesMetricType.NUM_OBSOLETE_DELTAS;
-import static 
org.apache.hadoop.hive.ql.txn.compactor.metrics.DeltaFilesMetricReporter.DeltaFilesMetricType.NUM_SMALL_DELTAS;
-import static org.mockito.Mockito.mock;
-import static org.mockito.Mockito.when;
-
 public class TestDeltaFilesMetrics extends CompactorTest  {
 
   private void setUpHiveConf() {
-HiveConf.setBoolVar(conf, HiveConf.ConfVars.HIVE_SERVER2_METRICS_ENABLED, 
true);
-HiveConf.setIntVar(conf, 
HiveConf.ConfVars.HIVE_TXN_ACID_METRICS_MAX_CACHE_SIZE, 2);
-HiveConf.setTimeVar(conf, 
HiveConf.ConfVars.HIVE_TXN_ACID_METRICS_CACHE_DURATION, 7200, TimeUnit.SECONDS);
-HiveConf.setIntVar(conf, 
HiveConf.ConfVars.HIVE_TXN_ACID_METRICS_OBSOLETE_DELTA_NUM_THRESHOLD, 100);
-HiveConf.setIntVar(conf, 
HiveConf.ConfVars.HIVE_TXN_ACID_METRICS_DELTA_NUM_THRESHOLD, 100);
-HiveConf.setTimeVar(conf, 
HiveConf.ConfVars.HIVE_TXN_ACID_METRICS_REPORTING_INTERVAL, 1, 
TimeUnit.SECONDS);
+MetastoreConf.setLongVar(conf, 
MetastoreConf.ConfVars.METASTORE_DELTAMETRICS_DELTA_NUM_THRESHOLD, 1);
+MetastoreConf.setLongVar(conf, 
MetastoreConf.ConfVars.METASTORE_DELTAMETRICS_OBSOLETE_DELTA_NUM_THRESHOLD, 1);
+MetastoreConf.setTimeVar(conf, 
MetastoreConf.ConfVars.METASTORE_DELTAMETRICS_REPORTING_INTERVAL, 1,
+TimeUnit.SECONDS);
+MetastoreConf.setDoubleVar(conf, 
MetastoreConf.ConfVars.METASTORE_DELTAMETRICS_DELTA_PCT_THRESHOLD, 0.15f);
+MetastoreConf.setBoolVar(conf,

[jira] [Resolved] (HIVE-25862) Persist the time of last run of the initiator

2022-01-17 Thread Antal Sinkovits (Jira) 


 [ 
https://issues.apache.org/jira/browse/HIVE-25862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Antal Sinkovits resolved HIVE-25862.

Fix Version/s: 4.0.0
   Resolution: Fixed

Pushed to master. Thanks for the. review [~dkuzmenko].

> Persist the time of last run of the initiator
> -
>
> Key: HIVE-25862
> URL: https://issues.apache.org/jira/browse/HIVE-25862
> Project: Hive
>  Issue Type: Improvement
>Reporter: Antal Sinkovits
>Assignee: Antal Sinkovits
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> The time of last run is used as a filter when finding compaction candidates.
> Because its only stored in memory, we lose this filtering capability if the 
> service restarts, so it would make sense to persist it.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Work logged] (HIVE-25862) Persist the time of last run of the initiator

2022-01-17 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HIVE-25862?focusedWorklogId=709901=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-709901
 ]

ASF GitHub Bot logged work on HIVE-25862:
-

Author: ASF GitHub Bot
Created on: 17/Jan/22 12:30
Start Date: 17/Jan/22 12:30
Worklog Time Spent: 10m 
  Work Description: asinkovits merged pull request #2945:
URL: https://github.com/apache/hive/pull/2945


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 709901)
Time Spent: 1h  (was: 50m)

> Persist the time of last run of the initiator
> -
>
> Key: HIVE-25862
> URL: https://issues.apache.org/jira/browse/HIVE-25862
> Project: Hive
>  Issue Type: Improvement
>Reporter: Antal Sinkovits
>Assignee: Antal Sinkovits
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> The time of last run is used as a filter when finding compaction candidates.
> Because its only stored in memory, we lose this filtering capability if the 
> service restarts, so it would make sense to persist it.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (HIVE-25869) Add GitHub Action job to publish snapshot

2022-01-17 Thread Yuming Wang (Jira) 


 [ 
https://issues.apache.org/jira/browse/HIVE-25869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuming Wang updated HIVE-25869:
---
Description: 
Publish Hive snapshots:
https://repository.apache.org/content/repositories/snapshots/org/apache/hive/

> Add GitHub Action job to publish snapshot
> -
>
> Key: HIVE-25869
> URL: https://issues.apache.org/jira/browse/HIVE-25869
> Project: Hive
>  Issue Type: Improvement
>Reporter: Yuming Wang
>Priority: Major
>
> Publish Hive snapshots:
> https://repository.apache.org/content/repositories/snapshots/org/apache/hive/



--
This message was sent by Atlassian Jira
(v8.20.1#820001)