[jira] [Updated] (HIVE-25011) Concurrency: Do not acquire locks for EXPLAIN and initiate implicit transaction for SHOW commands
[
https://issues.apache.org/jira/browse/HIVE-25011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-25011:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Concurrency: Do not acquire locks for EXPLAIN and initiate implicit
> transaction for SHOW commands
> -
>
> Key: HIVE-25011
> URL: https://issues.apache.org/jira/browse/HIVE-25011
> Project: Hive
> Issue Type: Improvement
> Components: Locking, Transactions
>Affects Versions: 4.0.0
>Reporter: Gopal Vijayaraghavan
>Assignee: Gopal Vijayaraghavan
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-25011.1.patch, HIVE-25011.2.patch
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> {code}
> EXPLAIN UPDATE ...
> {code}
> should not be in conflict with another active ongoing UPDATE operation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25011) Concurrency: Do not acquire locks for EXPLAIN and initiate implicit transaction for SHOW commands
[
https://issues.apache.org/jira/browse/HIVE-25011?focusedWorklogId=660718=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-660718
]
ASF GitHub Bot logged work on HIVE-25011:
-
Author: ASF GitHub Bot
Created on: 06/Oct/21 05:05
Start Date: 06/Oct/21 05:05
Worklog Time Spent: 10m
Work Description: deniskuzZ merged pull request #2660:
URL: https://github.com/apache/hive/pull/2660
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 660718)
Time Spent: 1h 40m (was: 1.5h)
> Concurrency: Do not acquire locks for EXPLAIN and initiate implicit
> transaction for SHOW commands
> -
>
> Key: HIVE-25011
> URL: https://issues.apache.org/jira/browse/HIVE-25011
> Project: Hive
> Issue Type: Improvement
> Components: Locking, Transactions
>Affects Versions: 4.0.0
>Reporter: Gopal Vijayaraghavan
>Assignee: Gopal Vijayaraghavan
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-25011.1.patch, HIVE-25011.2.patch
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> {code}
> EXPLAIN UPDATE ...
> {code}
> should not be in conflict with another active ongoing UPDATE operation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25571) Fix Metastore script for Oracle Database
[
https://issues.apache.org/jira/browse/HIVE-25571?focusedWorklogId=660713=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-660713
]
ASF GitHub Bot logged work on HIVE-25571:
-
Author: ASF GitHub Bot
Created on: 06/Oct/21 04:45
Start Date: 06/Oct/21 04:45
Worklog Time Spent: 10m
Work Description: ayushtkn closed pull request #2681:
URL: https://github.com/apache/hive/pull/2681
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 660713)
Time Spent: 0.5h (was: 20m)
> Fix Metastore script for Oracle Database
>
>
> Key: HIVE-25571
> URL: https://issues.apache.org/jira/browse/HIVE-25571
> Project: Hive
> Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Error:1
> {noformat}
> 354/359 CREATE UNIQUE INDEX DBPRIVILEGEINDEX ON DC_PRIVS
> (AUTHORIZER,NAME,PRINCIPAL_NAME,PRINCIPAL_TYPE,DC_PRIV,GRANTOR,GRANTOR_TYPE);
> Error: ORA-00955: name is already used by an existing object
> (state=42000,code=955)
> Aborting command set because "force" is false and command failed: "CREATE
> UNIQUE INDEX DBPRIVILEGEINDEX ON DC_PRIVS
> (AUTHORIZER,NAME,PRINCIPAL_NAME,PRINCIPAL_TYPE,DC_PRIV,GRANTOR,GRANTOR_TYPE);"
> [ERROR] 2021-09-29 09:18:59.075 [main] MetastoreSchemaTool - Schema
> initialization FAILED! Metastore state would be inconsistent!
> Schema initialization FAILED! Metastore state would be inconsistent!{noformat}
> Error:2
> {noformat}
> Error: ORA-00900: invalid SQL statement (state=42000,code=900)
> Aborting command set because "force" is false and command failed: "===
> -- HIVE-24396
> -- Create DataCo{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25497) Bump ORC to 1.7.0
[ https://issues.apache.org/jira/browse/HIVE-25497?focusedWorklogId=660710=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-660710 ] ASF GitHub Bot logged work on HIVE-25497: - Author: ASF GitHub Bot Created on: 06/Oct/21 04:26 Start Date: 06/Oct/21 04:26 Worklog Time Spent: 10m Work Description: dongjoon-hyun commented on pull request #2615: URL: https://github.com/apache/hive/pull/2615#issuecomment-935417928 Gentle ping, @pgaref . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 660710) Time Spent: 1h 20m (was: 1h 10m) > Bump ORC to 1.7.0 > - > > Key: HIVE-25497 > URL: https://issues.apache.org/jira/browse/HIVE-25497 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0 >Reporter: William Hyun >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25355) EXPLAIN statement for write transactions with hive.txn.readonly.enabled fails
[ https://issues.apache.org/jira/browse/HIVE-25355?focusedWorklogId=660707=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-660707 ] ASF GitHub Bot logged work on HIVE-25355: - Author: ASF GitHub Bot Created on: 06/Oct/21 04:13 Start Date: 06/Oct/21 04:13 Worklog Time Spent: 10m Work Description: aasha merged pull request #2698: URL: https://github.com/apache/hive/pull/2698 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 660707) Time Spent: 0.5h (was: 20m) > EXPLAIN statement for write transactions with hive.txn.readonly.enabled fails > - > > Key: HIVE-25355 > URL: https://issues.apache.org/jira/browse/HIVE-25355 > Project: Hive > Issue Type: Bug >Reporter: Pravin Sinha >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-25570) Hive should send full URL path for authorization for the command insert overwrite location
[
https://issues.apache.org/jira/browse/HIVE-25570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-25570 started by Sai Hemanth Gantasala.
> Hive should send full URL path for authorization for the command insert
> overwrite location
> --
>
> Key: HIVE-25570
> URL: https://issues.apache.org/jira/browse/HIVE-25570
> Project: Hive
> Issue Type: Bug
> Components: Authorization, HiveServer2
>Affects Versions: 4.0.0
>Reporter: Sai Hemanth Gantasala
>Assignee: Sai Hemanth Gantasala
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> For authorization, Hive is currently sending the path given as input from the
> user for the command, for eg
> {code:java}
> insert overwrite directory
> '/user/warehouse/tablespace/external/something/new/test_new_tb1' select *
> from test_tb1;
> {code}
> Hive is sending the path as
> '/user/warehouse/tablespace/external/something/new/test_new_tb1'
> Instead, Hive should send a fully qualified path for authorization, for e.g:
> 'hdfs://hostname:port_name/user/warehouse/tablespace/external/something/new/test_new_tb1'
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HIVE-25570) Hive should send full URL path for authorization for the command insert overwrite location
[
https://issues.apache.org/jira/browse/HIVE-25570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sai Hemanth Gantasala resolved HIVE-25570.
--
Fix Version/s: 4.0.0
Resolution: Fixed
> Hive should send full URL path for authorization for the command insert
> overwrite location
> --
>
> Key: HIVE-25570
> URL: https://issues.apache.org/jira/browse/HIVE-25570
> Project: Hive
> Issue Type: Bug
> Components: Authorization, HiveServer2
>Affects Versions: 4.0.0
>Reporter: Sai Hemanth Gantasala
>Assignee: Sai Hemanth Gantasala
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> For authorization, Hive is currently sending the path given as input from the
> user for the command, for eg
> {code:java}
> insert overwrite directory
> '/user/warehouse/tablespace/external/something/new/test_new_tb1' select *
> from test_tb1;
> {code}
> Hive is sending the path as
> '/user/warehouse/tablespace/external/something/new/test_new_tb1'
> Instead, Hive should send a fully qualified path for authorization, for e.g:
> 'hdfs://hostname:port_name/user/warehouse/tablespace/external/something/new/test_new_tb1'
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25570) Hive should send full URL path for authorization for the command insert overwrite location
[
https://issues.apache.org/jira/browse/HIVE-25570?focusedWorklogId=660674=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-660674
]
ASF GitHub Bot logged work on HIVE-25570:
-
Author: ASF GitHub Bot
Created on: 06/Oct/21 01:08
Start Date: 06/Oct/21 01:08
Worklog Time Spent: 10m
Work Description: saihemanth-cloudera closed pull request #2684:
URL: https://github.com/apache/hive/pull/2684
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 660674)
Time Spent: 1h (was: 50m)
> Hive should send full URL path for authorization for the command insert
> overwrite location
> --
>
> Key: HIVE-25570
> URL: https://issues.apache.org/jira/browse/HIVE-25570
> Project: Hive
> Issue Type: Bug
> Components: Authorization, HiveServer2
>Affects Versions: 4.0.0
>Reporter: Sai Hemanth Gantasala
>Assignee: Sai Hemanth Gantasala
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> For authorization, Hive is currently sending the path given as input from the
> user for the command, for eg
> {code:java}
> insert overwrite directory
> '/user/warehouse/tablespace/external/something/new/test_new_tb1' select *
> from test_tb1;
> {code}
> Hive is sending the path as
> '/user/warehouse/tablespace/external/something/new/test_new_tb1'
> Instead, Hive should send a fully qualified path for authorization, for e.g:
> 'hdfs://hostname:port_name/user/warehouse/tablespace/external/something/new/test_new_tb1'
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25570) Hive should send full URL path for authorization for the command insert overwrite location
[
https://issues.apache.org/jira/browse/HIVE-25570?focusedWorklogId=660670=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-660670
]
ASF GitHub Bot logged work on HIVE-25570:
-
Author: ASF GitHub Bot
Created on: 06/Oct/21 00:47
Start Date: 06/Oct/21 00:47
Worklog Time Spent: 10m
Work Description: nrg4878 commented on pull request #2684:
URL: https://github.com/apache/hive/pull/2684#issuecomment-935165762
Fix has been pushed to master. Please close the PR. Thank you
@saihemanth-cloudera
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 660670)
Time Spent: 50m (was: 40m)
> Hive should send full URL path for authorization for the command insert
> overwrite location
> --
>
> Key: HIVE-25570
> URL: https://issues.apache.org/jira/browse/HIVE-25570
> Project: Hive
> Issue Type: Bug
> Components: Authorization, HiveServer2
>Affects Versions: 4.0.0
>Reporter: Sai Hemanth Gantasala
>Assignee: Sai Hemanth Gantasala
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> For authorization, Hive is currently sending the path given as input from the
> user for the command, for eg
> {code:java}
> insert overwrite directory
> '/user/warehouse/tablespace/external/something/new/test_new_tb1' select *
> from test_tb1;
> {code}
> Hive is sending the path as
> '/user/warehouse/tablespace/external/something/new/test_new_tb1'
> Instead, Hive should send a fully qualified path for authorization, for e.g:
> 'hdfs://hostname:port_name/user/warehouse/tablespace/external/something/new/test_new_tb1'
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25396) Improve Exception handling in Hive Server 2
[ https://issues.apache.org/jira/browse/HIVE-25396?focusedWorklogId=660655=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-660655 ] ASF GitHub Bot logged work on HIVE-25396: - Author: ASF GitHub Bot Created on: 06/Oct/21 00:10 Start Date: 06/Oct/21 00:10 Worklog Time Spent: 10m Work Description: github-actions[bot] commented on pull request #2565: URL: https://github.com/apache/hive/pull/2565#issuecomment-935121235 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Feel free to reach out on the d...@hive.apache.org list if the patch is in need of reviews. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 660655) Time Spent: 1h 40m (was: 1.5h) > Improve Exception handling in Hive Server 2 > --- > > Key: HIVE-25396 > URL: https://issues.apache.org/jira/browse/HIVE-25396 > Project: Hive > Issue Type: Bug >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Major > Labels: pull-request-available > Attachments: hive-improve-exception-handling-04-master.patch > > Time Spent: 1h 40m > Remaining Estimate: 0h > > Hive's org.apache.hive.service.thrift.ThriftHttpServlet.doPost method does > not handle all Exception kinds. This leaves uncaught Exception handling > choices to the Jetty HTTP library. We fix that. > Also, a Thread.UncaughtExceptionHandler is added to Hive Server 2 so uncaught > Exception are handled uniformly, including making them logged and not just > printed to stderr. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25541) JsonSerDe: TBLPROPERTY treating nested json as String
[
https://issues.apache.org/jira/browse/HIVE-25541?focusedWorklogId=660646=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-660646
]
ASF GitHub Bot logged work on HIVE-25541:
-
Author: ASF GitHub Bot
Created on: 05/Oct/21 23:53
Start Date: 05/Oct/21 23:53
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on a change in pull request #2664:
URL: https://github.com/apache/hive/pull/2664#discussion_r722787680
##
File path: serde/src/java/org/apache/hadoop/hive/serde2/json/HiveJsonReader.java
##
@@ -393,7 +402,16 @@ private Object visitLeafNode(final JsonNode leafNode,
case DOUBLE:
return Double.valueOf(leafNode.asDouble());
case STRING:
- return leafNode.asText();
+ if (leafNode.isValueNode()) {
+return leafNode.asText();
+ } else {
+if (isEnabled(Feature.STRINGIFY_COMPLEX_FIELDS)) {
+ return leafNode.toString();
+} else {
+ throw new SerDeException(
+ "Complex field found in JSON does not match table definition: "
+ typeInfo.getTypeName());
Review comment:
Got it. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 660646)
Time Spent: 2h 20m (was: 2h 10m)
> JsonSerDe: TBLPROPERTY treating nested json as String
> -
>
> Key: HIVE-25541
> URL: https://issues.apache.org/jira/browse/HIVE-25541
> Project: Hive
> Issue Type: Bug
>Reporter: Panagiotis Garefalakis
>Assignee: Panagiotis Garefalakis
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> Native Jsonserde 'org.apache.hive.hcatalog.data.JsonSerDe' currently does not
> support loading nested json into a string type directly. It requires the
> declaring the column as complex type (struct, map, array) to unpack nested
> json data.
> Even though the data field is not a valid JSON String type there is value
> treating it as plain String instead of throwing an exception as we currently
> do.
> {code:java}
> create table json_table(data string, messageid string, publish_time bigint,
> attributes string);
> {"data":{"H":{"event":"track_active","platform":"Android"},"B":{"device_type":"Phone","uuid":"[36ffec24-f6a4-4f5d-aa39-72e5513d2cae,11883bee-a7aa-4010-8a66-6c3c63a73f16]"}},"messageId":"2475185636801962","publish_time":1622514629783,"attributes":{"region":"IN"}}"}}
> {code}
> This JIRA introduces an extra Table Property allowing to Stringify Complex
> JSON values instead of forcing the User to define the complete nested
> structure
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24975) Bug in ValidWriteIdList comparison in TxnIdUtils
[
https://issues.apache.org/jira/browse/HIVE-24975?focusedWorklogId=660511=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-660511
]
ASF GitHub Bot logged work on HIVE-24975:
-
Author: ASF GitHub Bot
Created on: 05/Oct/21 18:34
Start Date: 05/Oct/21 18:34
Worklog Time Spent: 10m
Work Description: sourabh912 commented on pull request #2641:
URL: https://github.com/apache/hive/pull/2641#issuecomment-934664670
@nrg4878 @kishendas @vihangk1 : Can you please take a look and provide your
feedback.
Thanks !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 660511)
Time Spent: 0.5h (was: 20m)
> Bug in ValidWriteIdList comparison in TxnIdUtils
>
>
> Key: HIVE-24975
> URL: https://issues.apache.org/jira/browse/HIVE-24975
> Project: Hive
> Issue Type: Bug
>Reporter: Sourabh Goyal
>Assignee: Sourabh Goyal
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> TxnIdUtils's
> [compare|https://github.com/apache/hive/blob/master/storage-api/src/java/org/apache/hive/common/util/TxnIdUtils.java#L38]
> method returns incorrect result for the following validWriteIdLists
> ValidWriteIdList a = new ValidReaderWriteIdList("default.test:1:1:1:");
> ValidWriteIdList b = new
> ValidReaderWriteIdList("default.test:1:9223372036854775807::");
> TxnIdUtils.compare(a, b) returns +1 whereas the expected response is -1 since
> b is more recent.
> cc - [~kishendas] [~vihangk1]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-25592) Remove scheme less path comparisons when creating external table in HMS
[ https://issues.apache.org/jira/browse/HIVE-25592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sourabh Goyal reassigned HIVE-25592: > Remove scheme less path comparisons when creating external table in HMS > --- > > Key: HIVE-25592 > URL: https://issues.apache.org/jira/browse/HIVE-25592 > Project: Hive > Issue Type: Improvement >Reporter: Sourabh Goyal >Assignee: Sourabh Goyal >Priority: Major > > In validateTablePaths() in MetastoreDefaultTransformer.java , we are > currently doing [scheme > less|https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDefaultTransformer.java#L928] > comparison. This can cause issues if warehouse default root and table > location's scheme are different but the rest of the path is same. For example > warehouse if pointing to a location in hdfs and table is pointing to a > location in s3. > The proposed fix is to include scheme when comparing warehouse root and table > location paths. > > cc - [~ngangam] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25590) Able to create views referencing temporary tables and materialized views
[
https://issues.apache.org/jira/browse/HIVE-25590?focusedWorklogId=660437=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-660437
]
ASF GitHub Bot logged work on HIVE-25590:
-
Author: ASF GitHub Bot
Created on: 05/Oct/21 16:39
Start Date: 05/Oct/21 16:39
Worklog Time Spent: 10m
Work Description: kasakrisz commented on pull request #2697:
URL: https://github.com/apache/hive/pull/2697#issuecomment-934575532
I usually use `-Dtest.output.overwrite` and then `git status` to see if
there were any change because if any of the q.out files changed I can do a
better diff.
Anyway I removed `-Dtest.output.overwrite` from the description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 660437)
Time Spent: 40m (was: 0.5h)
> Able to create views referencing temporary tables and materialized views
>
>
> Key: HIVE-25590
> URL: https://issues.apache.org/jira/browse/HIVE-25590
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Creating views/materialized views referencing temporary tables and
> materialized views are disabled in Hive. However the verification algorithm
> fails to recognize temporary tables and materialized views in subqueries.
> The verification also fails when the view definition contains joins because
> CBO transforms join branches to subqueries.
> Example1:
> {code}
> create temporary table tmp1 (c1 string, c2 string);
> create view tmp1_view as
> select subq.c1 from (select c1, c2 from tmp1) subq;
> {code}
> Example2:
> {code}
> create table t1 (a int) stored as orc tblproperties ('transactional'='true');
> create table t2 (a int) stored as orc tblproperties ('transactional'='true');
> create materialized view mv1 as
> select a from t1 where a = 10;
> create materialized view mv2 as
> select t2.a from mv1
> join t2 on (mv1.a = t2.a);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25590) Able to create views referencing temporary tables and materialized views
[
https://issues.apache.org/jira/browse/HIVE-25590?focusedWorklogId=660438=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-660438
]
ASF GitHub Bot logged work on HIVE-25590:
-
Author: ASF GitHub Bot
Created on: 05/Oct/21 16:39
Start Date: 05/Oct/21 16:39
Worklog Time Spent: 10m
Work Description: kasakrisz edited a comment on pull request #2697:
URL: https://github.com/apache/hive/pull/2697#issuecomment-934575532
@asolimando
I usually use `-Dtest.output.overwrite` and then `git status` to see if
there were any change because if any of the q.out files changed I can do a
better diff.
Anyway I removed `-Dtest.output.overwrite` from the description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 660438)
Time Spent: 50m (was: 40m)
> Able to create views referencing temporary tables and materialized views
>
>
> Key: HIVE-25590
> URL: https://issues.apache.org/jira/browse/HIVE-25590
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Creating views/materialized views referencing temporary tables and
> materialized views are disabled in Hive. However the verification algorithm
> fails to recognize temporary tables and materialized views in subqueries.
> The verification also fails when the view definition contains joins because
> CBO transforms join branches to subqueries.
> Example1:
> {code}
> create temporary table tmp1 (c1 string, c2 string);
> create view tmp1_view as
> select subq.c1 from (select c1, c2 from tmp1) subq;
> {code}
> Example2:
> {code}
> create table t1 (a int) stored as orc tblproperties ('transactional'='true');
> create table t2 (a int) stored as orc tblproperties ('transactional'='true');
> create materialized view mv1 as
> select a from t1 where a = 10;
> create materialized view mv2 as
> select t2.a from mv1
> join t2 on (mv1.a = t2.a);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25397) Snapshot support for controlled failover
[ https://issues.apache.org/jira/browse/HIVE-25397?focusedWorklogId=660388=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-660388 ] ASF GitHub Bot logged work on HIVE-25397: - Author: ASF GitHub Bot Created on: 05/Oct/21 15:22 Start Date: 05/Oct/21 15:22 Worklog Time Spent: 10m Work Description: ArkoSharma commented on a change in pull request #2539: URL: https://github.com/apache/hive/pull/2539#discussion_r722353424 ## File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/DirCopyTask.java ## @@ -270,8 +278,9 @@ boolean copyUsingDistCpSnapshots(Path sourcePath, Path targetPath, UserGroupInfo // snapshots. SnapshotUtils.allowSnapshot(targetFs, work.getFullyQualifiedTargetPath(), clonedConf); // Attempt to delete the snapshot, in case this is a bootstrap post a failed incremental, Since in case of - // bootstrap we go from start, so delete any pre-existing snapshot. + // bootstrap we go from start, so delete any pre-existing snapshot, (both snapshots can exist in case of failback) Review comment: Changed to read 'reverse replication' -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 660388) Time Spent: 2h (was: 1h 50m) > Snapshot support for controlled failover > > > Key: HIVE-25397 > URL: https://issues.apache.org/jira/browse/HIVE-25397 > Project: Hive > Issue Type: Bug >Reporter: Arko Sharma >Assignee: Arko Sharma >Priority: Major > Labels: pull-request-available > Time Spent: 2h > Remaining Estimate: 0h > > In case the same locations are used for external tables on the source and > target, then the snapshots created during replication can be re-used during > reverse replication. This patch enables re-using the snapshots during > reverse replication using a configuration. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25397) Snapshot support for controlled failover
[
https://issues.apache.org/jira/browse/HIVE-25397?focusedWorklogId=660386=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-660386
]
ASF GitHub Bot logged work on HIVE-25397:
-
Author: ASF GitHub Bot
Created on: 05/Oct/21 15:22
Start Date: 05/Oct/21 15:22
Worklog Time Spent: 10m
Work Description: ArkoSharma commented on a change in pull request #2539:
URL: https://github.com/apache/hive/pull/2539#discussion_r722352700
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplExternalTables.java
##
@@ -192,64 +196,135 @@ private void dirLocationToCopy(String tableName,
FileList fileList, Path sourceP
fileList.add(new DirCopyWork(tableName, sourcePath, targetPath, copyMode,
snapshotPrefix).convertToString());
}
- private SnapshotUtils.SnapshotCopyMode createSnapshotsAtSource(Path
sourcePath, String snapshotPrefix,
- boolean isSnapshotEnabled, HiveConf conf,
SnapshotUtils.ReplSnapshotCount replSnapshotCount, FileList snapPathFileList,
- ArrayList prevSnaps, boolean isBootstrap) throws IOException {
+ private Map
createSnapshotsAtSource(Path sourcePath, Path targetPath, String snapshotPrefix,
+
boolean isSnapshotEnabled, HiveConf conf, SnapshotUtils.ReplSnapshotCount
replSnapshotCount, FileList snapPathFileList,
+
ArrayList prevSnaps, boolean isBootstrap) throws IOException {
+Map ret = new HashMap<>();
+ret.put(snapshotPrefix, FALLBACK_COPY);
if (!isSnapshotEnabled) {
LOG.info("Snapshot copy not enabled for path {} Will use normal distCp
for copying data.", sourcePath);
- return FALLBACK_COPY;
+ return ret;
}
+String prefix = snapshotPrefix;
+SnapshotUtils.SnapshotCopyMode copyMode = FALLBACK_COPY;
DistributedFileSystem sourceDfs = SnapshotUtils.getDFS(sourcePath, conf);
try {
- if(isBootstrap) {
+ if(conf.getBoolVar(HiveConf.ConfVars.REPL_REUSE_SNAPSHOTS)) {
+try {
+ FileStatus[] listing = sourceDfs.listStatus(new Path(sourcePath,
".snapshot"));
+ for (FileStatus elem : listing) {
+String snapShotName = elem.getPath().getName();
+if (snapShotName.contains(OLD_SNAPSHOT)) {
+ prefix = snapShotName.substring(0,
snapShotName.lastIndexOf(OLD_SNAPSHOT));
+ break;
+}
+if (snapShotName.contains(NEW_SNAPSHOT)) {
+ prefix = snapShotName.substring(0,
snapShotName.lastIndexOf(NEW_SNAPSHOT));
+ break;
+}
+ }
+ ret.clear();
+ ret.put(prefix, copyMode);
+ snapshotPrefix = prefix;
+} catch (SnapshotException e) {
+ //dir not snapshottable, continue
+}
+ }
+ boolean isFirstSnapshotAvl =
+ SnapshotUtils.isSnapshotAvailable(sourceDfs, sourcePath,
snapshotPrefix, OLD_SNAPSHOT, conf);
+ boolean isSecondSnapAvl =
+ SnapshotUtils.isSnapshotAvailable(sourceDfs, sourcePath,
snapshotPrefix, NEW_SNAPSHOT, conf);
+ //for bootstrap and non - failback case, use initial_copy
+ if(isBootstrap && !(!isSecondSnapAvl && isFirstSnapshotAvl)) {
// Delete any pre existing snapshots.
SnapshotUtils.deleteSnapshotIfExists(sourceDfs, sourcePath,
firstSnapshot(snapshotPrefix), conf);
SnapshotUtils.deleteSnapshotIfExists(sourceDfs, sourcePath,
secondSnapshot(snapshotPrefix), conf);
allowAndCreateInitialSnapshot(sourcePath, snapshotPrefix, conf,
replSnapshotCount, snapPathFileList, sourceDfs);
-return INITIAL_COPY;
+ret.put(prefix, INITIAL_COPY);
+return ret;
}
+ //While resuming a failed replication
if (prevSnaps.contains(sourcePath.toString())) {
Review comment:
Added.
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/DirCopyTask.java
##
@@ -243,18 +244,25 @@ boolean copyUsingDistCpSnapshots(Path sourcePath, Path
targetPath, UserGroupInfo
DistributedFileSystem targetFs = SnapshotUtils.getDFS(targetPath,
clonedConf);
boolean result = false;
+boolean isSecondSnapshotAvl =
Review comment:
Renamed.
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/DirCopyTask.java
##
@@ -243,18 +244,25 @@ boolean copyUsingDistCpSnapshots(Path sourcePath, Path
targetPath, UserGroupInfo
DistributedFileSystem targetFs = SnapshotUtils.getDFS(targetPath,
clonedConf);
boolean result = false;
+boolean isSecondSnapshotAvl =
+SnapshotUtils.isSnapshotAvailable(targetFs, targetPath,
work.getSnapshotPrefix(), NEW_SNAPSHOT, clonedConf);
if
(getWork().getCopyMode().equals(SnapshotUtils.SnapshotCopyMode.DIFF_COPY)) {
+
[jira] [Work logged] (HIVE-25397) Snapshot support for controlled failover
[
https://issues.apache.org/jira/browse/HIVE-25397?focusedWorklogId=660385=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-660385
]
ASF GitHub Bot logged work on HIVE-25397:
-
Author: ASF GitHub Bot
Created on: 05/Oct/21 15:21
Start Date: 05/Oct/21 15:21
Worklog Time Spent: 10m
Work Description: ArkoSharma commented on a change in pull request #2539:
URL: https://github.com/apache/hive/pull/2539#discussion_r722352460
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosUsingSnapshots.java
##
@@ -79,11 +80,11 @@ public static void classLevelSetup() throws Exception {
overrides.put(HiveConf.ConfVars.REPL_INCLUDE_EXTERNAL_TABLES.varname,
"true");
overrides.put(HiveConf.ConfVars.HIVE_DISTCP_DOAS_USER.varname,
UserGroupInformation.getCurrentUser().getUserName());
-
overrides.put(HiveConf.ConfVars.REPL_RUN_DATA_COPY_TASKS_ON_TARGET.varname,
"false");
+
overrides.put(HiveConf.ConfVars.REPL_RUN_DATA_COPY_TASKS_ON_TARGET.varname,
"true");
overrides.put(HiveConf.ConfVars.REPL_EXTERNAL_WAREHOUSE_SINGLE_COPY_TASK.varname,
"true");
overrides.put(REPL_SNAPSHOT_DIFF_FOR_EXTERNAL_TABLE_COPY.varname, "true");
-internalBeforeClassSetup(overrides, TestReplicationScenarios.class);
+internalBeforeClassSetupExclusiveReplica(overrides, overrides,
TestReplicationScenarios.class);
Review comment:
The tests are modified to use different filesystems for src and tgt. A
few wrongFS related errors were fixed and this required testing with different
fs for src and tgt.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 660385)
Time Spent: 1h 40m (was: 1.5h)
> Snapshot support for controlled failover
>
>
> Key: HIVE-25397
> URL: https://issues.apache.org/jira/browse/HIVE-25397
> Project: Hive
> Issue Type: Bug
>Reporter: Arko Sharma
>Assignee: Arko Sharma
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> In case the same locations are used for external tables on the source and
> target, then the snapshots created during replication can be re-used during
> reverse replication. This patch enables re-using the snapshots during
> reverse replication using a configuration.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25397) Snapshot support for controlled failover
[
https://issues.apache.org/jira/browse/HIVE-25397?focusedWorklogId=660383=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-660383
]
ASF GitHub Bot logged work on HIVE-25397:
-
Author: ASF GitHub Bot
Created on: 05/Oct/21 15:19
Start Date: 05/Oct/21 15:19
Worklog Time Spent: 10m
Work Description: ArkoSharma commented on a change in pull request #2539:
URL: https://github.com/apache/hive/pull/2539#discussion_r722350281
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplExternalTables.java
##
@@ -192,64 +196,135 @@ private void dirLocationToCopy(String tableName,
FileList fileList, Path sourceP
fileList.add(new DirCopyWork(tableName, sourcePath, targetPath, copyMode,
snapshotPrefix).convertToString());
}
- private SnapshotUtils.SnapshotCopyMode createSnapshotsAtSource(Path
sourcePath, String snapshotPrefix,
- boolean isSnapshotEnabled, HiveConf conf,
SnapshotUtils.ReplSnapshotCount replSnapshotCount, FileList snapPathFileList,
- ArrayList prevSnaps, boolean isBootstrap) throws IOException {
+ private Map
createSnapshotsAtSource(Path sourcePath, Path targetPath, String snapshotPrefix,
+
boolean isSnapshotEnabled, HiveConf conf, SnapshotUtils.ReplSnapshotCount
replSnapshotCount, FileList snapPathFileList,
+
ArrayList prevSnaps, boolean isBootstrap) throws IOException {
+Map ret = new HashMap<>();
+ret.put(snapshotPrefix, FALLBACK_COPY);
if (!isSnapshotEnabled) {
LOG.info("Snapshot copy not enabled for path {} Will use normal distCp
for copying data.", sourcePath);
- return FALLBACK_COPY;
+ return ret;
}
+String prefix = snapshotPrefix;
+SnapshotUtils.SnapshotCopyMode copyMode = FALLBACK_COPY;
DistributedFileSystem sourceDfs = SnapshotUtils.getDFS(sourcePath, conf);
try {
- if(isBootstrap) {
+ if(conf.getBoolVar(HiveConf.ConfVars.REPL_REUSE_SNAPSHOTS)) {
+try {
+ FileStatus[] listing = sourceDfs.listStatus(new Path(sourcePath,
".snapshot"));
+ for (FileStatus elem : listing) {
+String snapShotName = elem.getPath().getName();
+if (snapShotName.contains(OLD_SNAPSHOT)) {
+ prefix = snapShotName.substring(0,
snapShotName.lastIndexOf(OLD_SNAPSHOT));
+ break;
+}
+if (snapShotName.contains(NEW_SNAPSHOT)) {
+ prefix = snapShotName.substring(0,
snapShotName.lastIndexOf(NEW_SNAPSHOT));
+ break;
+}
+ }
+ ret.clear();
+ ret.put(prefix, copyMode);
+ snapshotPrefix = prefix;
+} catch (SnapshotException e) {
+ //dir not snapshottable, continue
+}
+ }
+ boolean isFirstSnapshotAvl =
+ SnapshotUtils.isSnapshotAvailable(sourceDfs, sourcePath,
snapshotPrefix, OLD_SNAPSHOT, conf);
+ boolean isSecondSnapAvl =
+ SnapshotUtils.isSnapshotAvailable(sourceDfs, sourcePath,
snapshotPrefix, NEW_SNAPSHOT, conf);
+ //for bootstrap and non - failback case, use initial_copy
+ if(isBootstrap && !(!isSecondSnapAvl && isFirstSnapshotAvl)) {
Review comment:
Considered making this change, but realised this would then require a
similar listing and name-checking for snapshot on the target side. This can
occur in case of reverse replication after failover with different names of src
and tgt dbs.
Hence decided to proceed with this implementation itself as it allows for
identifying which snapshots are being re-used. Also makes sense to include this
work with dump considering that in general, dump should take lesser time than
load (except possibly cases with just external tables in db with data-copy on
src).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 660383)
Time Spent: 1.5h (was: 1h 20m)
> Snapshot support for controlled failover
>
>
> Key: HIVE-25397
> URL: https://issues.apache.org/jira/browse/HIVE-25397
> Project: Hive
> Issue Type: Bug
>Reporter: Arko Sharma
>Assignee: Arko Sharma
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> In case the same locations are used for external tables on the source and
> target, then the snapshots
[jira] [Work logged] (HIVE-25590) Able to create views referencing temporary tables and materialized views
[
https://issues.apache.org/jira/browse/HIVE-25590?focusedWorklogId=660307=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-660307
]
ASF GitHub Bot logged work on HIVE-25590:
-
Author: ASF GitHub Bot
Created on: 05/Oct/21 13:08
Start Date: 05/Oct/21 13:08
Worklog Time Spent: 10m
Work Description: asolimando edited a comment on pull request #2697:
URL: https://github.com/apache/hive/pull/2697#issuecomment-934396405
I did run tests with `-Dtest.output.overwrite` like in the PR description,
and I did not notice the discrepancies in the `.q.out` files.
Maybe it's better to remove the option from the "How it was tested" example
to avoid somebody else getting fooled by this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 660307)
Time Spent: 0.5h (was: 20m)
> Able to create views referencing temporary tables and materialized views
>
>
> Key: HIVE-25590
> URL: https://issues.apache.org/jira/browse/HIVE-25590
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Creating views/materialized views referencing temporary tables and
> materialized views are disabled in Hive. However the verification algorithm
> fails to recognize temporary tables and materialized views in subqueries.
> The verification also fails when the view definition contains joins because
> CBO transforms join branches to subqueries.
> Example1:
> {code}
> create temporary table tmp1 (c1 string, c2 string);
> create view tmp1_view as
> select subq.c1 from (select c1, c2 from tmp1) subq;
> {code}
> Example2:
> {code}
> create table t1 (a int) stored as orc tblproperties ('transactional'='true');
> create table t2 (a int) stored as orc tblproperties ('transactional'='true');
> create materialized view mv1 as
> select a from t1 where a = 10;
> create materialized view mv2 as
> select t2.a from mv1
> join t2 on (mv1.a = t2.a);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25590) Able to create views referencing temporary tables and materialized views
[
https://issues.apache.org/jira/browse/HIVE-25590?focusedWorklogId=660306=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-660306
]
ASF GitHub Bot logged work on HIVE-25590:
-
Author: ASF GitHub Bot
Created on: 05/Oct/21 13:07
Start Date: 05/Oct/21 13:07
Worklog Time Spent: 10m
Work Description: asolimando commented on pull request #2697:
URL: https://github.com/apache/hive/pull/2697#issuecomment-934396405
I did run it with `-Dtest.output.overwrite` like in the PR title, and I did
not notice the discrepancies in the `.q.out` files.
Maybe it's better to remove the option from the "How it was tested" example
to avoid somebody else getting fooled by this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 660306)
Time Spent: 20m (was: 10m)
> Able to create views referencing temporary tables and materialized views
>
>
> Key: HIVE-25590
> URL: https://issues.apache.org/jira/browse/HIVE-25590
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Creating views/materialized views referencing temporary tables and
> materialized views are disabled in Hive. However the verification algorithm
> fails to recognize temporary tables and materialized views in subqueries.
> The verification also fails when the view definition contains joins because
> CBO transforms join branches to subqueries.
> Example1:
> {code}
> create temporary table tmp1 (c1 string, c2 string);
> create view tmp1_view as
> select subq.c1 from (select c1, c2 from tmp1) subq;
> {code}
> Example2:
> {code}
> create table t1 (a int) stored as orc tblproperties ('transactional'='true');
> create table t2 (a int) stored as orc tblproperties ('transactional'='true');
> create materialized view mv1 as
> select a from t1 where a = 10;
> create materialized view mv2 as
> select t2.a from mv1
> join t2 on (mv1.a = t2.a);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25541) JsonSerDe: TBLPROPERTY treating nested json as String
[
https://issues.apache.org/jira/browse/HIVE-25541?focusedWorklogId=660289=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-660289
]
ASF GitHub Bot logged work on HIVE-25541:
-
Author: ASF GitHub Bot
Created on: 05/Oct/21 12:39
Start Date: 05/Oct/21 12:39
Worklog Time Spent: 10m
Work Description: pgaref commented on a change in pull request #2664:
URL: https://github.com/apache/hive/pull/2664#discussion_r722198077
##
File path: serde/src/java/org/apache/hadoop/hive/serde2/json/HiveJsonReader.java
##
@@ -393,7 +402,16 @@ private Object visitLeafNode(final JsonNode leafNode,
case DOUBLE:
return Double.valueOf(leafNode.asDouble());
case STRING:
- return leafNode.asText();
+ if (leafNode.isValueNode()) {
+return leafNode.asText();
+ } else {
+if (isEnabled(Feature.STRINGIFY_COMPLEX_FIELDS)) {
+ return leafNode.toString();
+} else {
+ throw new SerDeException(
+ "Complex field found in JSON does not match table definition: "
+ typeInfo.getTypeName());
Review comment:
Hey @dengzhhu653 -- this could be indeed extended to support more types
like varchar but I would do that as a follow up. Lets open a new ticket for
supporting more types, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 660289)
Time Spent: 2h 10m (was: 2h)
> JsonSerDe: TBLPROPERTY treating nested json as String
> -
>
> Key: HIVE-25541
> URL: https://issues.apache.org/jira/browse/HIVE-25541
> Project: Hive
> Issue Type: Bug
>Reporter: Panagiotis Garefalakis
>Assignee: Panagiotis Garefalakis
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> Native Jsonserde 'org.apache.hive.hcatalog.data.JsonSerDe' currently does not
> support loading nested json into a string type directly. It requires the
> declaring the column as complex type (struct, map, array) to unpack nested
> json data.
> Even though the data field is not a valid JSON String type there is value
> treating it as plain String instead of throwing an exception as we currently
> do.
> {code:java}
> create table json_table(data string, messageid string, publish_time bigint,
> attributes string);
> {"data":{"H":{"event":"track_active","platform":"Android"},"B":{"device_type":"Phone","uuid":"[36ffec24-f6a4-4f5d-aa39-72e5513d2cae,11883bee-a7aa-4010-8a66-6c3c63a73f16]"}},"messageId":"2475185636801962","publish_time":1622514629783,"attributes":{"region":"IN"}}"}}
> {code}
> This JIRA introduces an extra Table Property allowing to Stringify Complex
> JSON values instead of forcing the User to define the complete nested
> structure
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-25591) CREATE EXTERNAL TABLE fails for JDBC tables stored in non-default schema
[
https://issues.apache.org/jira/browse/HIVE-25591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17424419#comment-17424419
]
Stamatis Zampetakis commented on HIVE-25591:
HIVE-21383 started the work for supporting JDBC tables in different
catalogs/schemas but currently there are no tests in the repository.
Moreover there are various places in the codebase which do not take the
{{hive.sql.schema}} table property into account leading to failures as the one
mentioned in the description.
> CREATE EXTERNAL TABLE fails for JDBC tables stored in non-default schema
>
>
> Key: HIVE-25591
> URL: https://issues.apache.org/jira/browse/HIVE-25591
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
>
> Consider the following use case where tables reside in some user-defined
> schema in some JDBC compliant database:
> +Postgres+
> {code:sql}
> create schema world;
> create table if not exists world.country (name varchar(80) not null);
> insert into world.country (name) values ('India');
> insert into world.country (name) values ('Russia');
> insert into world.country (name) values ('USA');
> {code}
> The following DDL statement in Hive fails:
> +Hive+
> {code:sql}
> CREATE EXTERNAL TABLE country (name varchar(80))
> STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
> "hive.sql.database.type" = "POSTGRES",
> "hive.sql.jdbc.driver" = "org.postgresql.Driver",
> "hive.sql.jdbc.url" = "jdbc:postgresql://localhost:5432/test",
> "hive.sql.dbcp.username" = "user",
> "hive.sql.dbcp.password" = "pwd",
> "hive.sql.schema" = "world",
> "hive.sql.table" = "country");
> {code}
> The exception is the following:
> {noformat}
> org.postgresql.util.PSQLException: ERROR: relation "country" does not exist
> Position: 15
> at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2532)
> ~[postgresql-42.2.14.jar:42.2.14]
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2267)
> ~[postgresql-42.2.14.jar:42.2.14]
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:312)
> ~[postgresql-42.2.14.jar:42.2.14]
> at
> org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:448)
> ~[postgresql-42.2.14.jar:42.2.14]
> at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:369)
> ~[postgresql-42.2.14.jar:42.2.14]
> at
> org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:153)
> ~[postgresql-42.2.14.jar:42.2.14]
> at
> org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:103)
> ~[postgresql-42.2.14.jar:42.2.14]
> at
> org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:122)
> ~[commons-dbcp2-2.7.0.jar:2.7.0]
> at
> org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:122)
> ~[commons-dbcp2-2.7.0.jar:2.7.0]
> at
> org.apache.hive.storage.jdbc.dao.GenericJdbcDatabaseAccessor.getColumnNames(GenericJdbcDatabaseAccessor.java:83)
> [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hive.storage.jdbc.JdbcSerDe.initialize(JdbcSerDe.java:98)
> [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:95)
> [hive-metastore-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:78)
> [hive-metastore-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:342)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:324)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getColsInternal(Table.java:734)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:717)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.ddl.table.create.CreateTableDesc.toTable(CreateTableDesc.java:933)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.ddl.table.create.CreateTableOperation.execute(CreateTableOperation.java:59)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.ddl.DDLTask.execute(DDLTask.java:84)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
[jira] [Assigned] (HIVE-25591) CREATE EXTERNAL TABLE fails for JDBC tables stored in non-default schema
[
https://issues.apache.org/jira/browse/HIVE-25591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stamatis Zampetakis reassigned HIVE-25591:
--
> CREATE EXTERNAL TABLE fails for JDBC tables stored in non-default schema
>
>
> Key: HIVE-25591
> URL: https://issues.apache.org/jira/browse/HIVE-25591
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
>
> Consider the following use case where tables reside in some user-defined
> schema in some JDBC compliant database:
> +Postgres+
> {code:sql}
> create schema world;
> create table if not exists world.country (name varchar(80) not null);
> insert into world.country (name) values ('India');
> insert into world.country (name) values ('Russia');
> insert into world.country (name) values ('USA');
> {code}
> The following DDL statement in Hive fails:
> +Hive+
> {code:sql}
> CREATE EXTERNAL TABLE country (name varchar(80))
> STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
> "hive.sql.database.type" = "POSTGRES",
> "hive.sql.jdbc.driver" = "org.postgresql.Driver",
> "hive.sql.jdbc.url" = "jdbc:postgresql://localhost:5432/test",
> "hive.sql.dbcp.username" = "user",
> "hive.sql.dbcp.password" = "pwd",
> "hive.sql.schema" = "world",
> "hive.sql.table" = "country");
> {code}
> The exception is the following:
> {noformat}
> org.postgresql.util.PSQLException: ERROR: relation "country" does not exist
> Position: 15
> at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2532)
> ~[postgresql-42.2.14.jar:42.2.14]
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2267)
> ~[postgresql-42.2.14.jar:42.2.14]
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:312)
> ~[postgresql-42.2.14.jar:42.2.14]
> at
> org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:448)
> ~[postgresql-42.2.14.jar:42.2.14]
> at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:369)
> ~[postgresql-42.2.14.jar:42.2.14]
> at
> org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:153)
> ~[postgresql-42.2.14.jar:42.2.14]
> at
> org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:103)
> ~[postgresql-42.2.14.jar:42.2.14]
> at
> org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:122)
> ~[commons-dbcp2-2.7.0.jar:2.7.0]
> at
> org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:122)
> ~[commons-dbcp2-2.7.0.jar:2.7.0]
> at
> org.apache.hive.storage.jdbc.dao.GenericJdbcDatabaseAccessor.getColumnNames(GenericJdbcDatabaseAccessor.java:83)
> [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hive.storage.jdbc.JdbcSerDe.initialize(JdbcSerDe.java:98)
> [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:95)
> [hive-metastore-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:78)
> [hive-metastore-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:342)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:324)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getColsInternal(Table.java:734)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:717)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.ddl.table.create.CreateTableDesc.toTable(CreateTableDesc.java:933)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.ddl.table.create.CreateTableOperation.execute(CreateTableOperation.java:59)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.ddl.DDLTask.execute(DDLTask.java:84)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:212)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:361)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
[jira] [Work logged] (HIVE-25355) EXPLAIN statement for write transactions with hive.txn.readonly.enabled fails
[ https://issues.apache.org/jira/browse/HIVE-25355?focusedWorklogId=660236=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-660236 ] ASF GitHub Bot logged work on HIVE-25355: - Author: ASF GitHub Bot Created on: 05/Oct/21 11:15 Start Date: 05/Oct/21 11:15 Worklog Time Spent: 10m Work Description: aasha opened a new pull request #2698: URL: https://github.com/apache/hive/pull/2698 Reverts apache/hive#2694 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 660236) Time Spent: 20m (was: 10m) > EXPLAIN statement for write transactions with hive.txn.readonly.enabled fails > - > > Key: HIVE-25355 > URL: https://issues.apache.org/jira/browse/HIVE-25355 > Project: Hive > Issue Type: Bug >Reporter: Pravin Sinha >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25011) Concurrency: Do not acquire locks for EXPLAIN and initiate implicit transaction for SHOW commands
[
https://issues.apache.org/jira/browse/HIVE-25011?focusedWorklogId=660208=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-660208
]
ASF GitHub Bot logged work on HIVE-25011:
-
Author: ASF GitHub Bot
Created on: 05/Oct/21 10:31
Start Date: 05/Oct/21 10:31
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on a change in pull request #2660:
URL: https://github.com/apache/hive/pull/2660#discussion_r722102438
##
File path: ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java
##
@@ -184,19 +183,17 @@ public boolean accept(Path path) {
public static final int MAX_STATEMENTS_PER_TXN = 1;
public static final Pattern LEGACY_BUCKET_DIGIT_PATTERN =
Pattern.compile("^[0-9]{6}");
public static final Pattern BUCKET_PATTERN =
Pattern.compile("bucket_([0-9]+)(_[0-9]+)?$");
- private static final Set READ_TXN_TOKENS = new HashSet();
+ private static final Set READ_TXN_TOKENS = new HashSet<>();
private static Cache dirCache;
private static AtomicBoolean dirCacheInited = new AtomicBoolean();
static {
READ_TXN_TOKENS.addAll(Arrays.asList(
-HiveParser.TOK_DESCDATABASE,
-HiveParser.TOK_DESCTABLE,
-HiveParser.TOK_SHOWTABLES,
-HiveParser.TOK_SHOW_TABLESTATUS,
-HiveParser.TOK_SHOW_TBLPROPERTIES,
Review comment:
Because those don't even need a txn, see a corresponding change in
`Compiler.startImplicitTxn()`. I would have removed all of the operations
(EXPLAIN, DESC), however, that cause number of old tests to fail cause they
validate txnIds. I'll handle them separately in diff JIRAs 1-by-1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 660208)
Time Spent: 1.5h (was: 1h 20m)
> Concurrency: Do not acquire locks for EXPLAIN and initiate implicit
> transaction for SHOW commands
> -
>
> Key: HIVE-25011
> URL: https://issues.apache.org/jira/browse/HIVE-25011
> Project: Hive
> Issue Type: Improvement
> Components: Locking, Transactions
>Affects Versions: 4.0.0
>Reporter: Gopal Vijayaraghavan
>Assignee: Gopal Vijayaraghavan
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-25011.1.patch, HIVE-25011.2.patch
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> {code}
> EXPLAIN UPDATE ...
> {code}
> should not be in conflict with another active ongoing UPDATE operation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25011) Concurrency: Do not acquire locks for EXPLAIN and initiate implicit transaction for SHOW commands
[
https://issues.apache.org/jira/browse/HIVE-25011?focusedWorklogId=660204=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-660204
]
ASF GitHub Bot logged work on HIVE-25011:
-
Author: ASF GitHub Bot
Created on: 05/Oct/21 10:25
Start Date: 05/Oct/21 10:25
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on a change in pull request #2660:
URL: https://github.com/apache/hive/pull/2660#discussion_r722102438
##
File path: ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java
##
@@ -184,19 +183,17 @@ public boolean accept(Path path) {
public static final int MAX_STATEMENTS_PER_TXN = 1;
public static final Pattern LEGACY_BUCKET_DIGIT_PATTERN =
Pattern.compile("^[0-9]{6}");
public static final Pattern BUCKET_PATTERN =
Pattern.compile("bucket_([0-9]+)(_[0-9]+)?$");
- private static final Set READ_TXN_TOKENS = new HashSet();
+ private static final Set READ_TXN_TOKENS = new HashSet<>();
private static Cache dirCache;
private static AtomicBoolean dirCacheInited = new AtomicBoolean();
static {
READ_TXN_TOKENS.addAll(Arrays.asList(
-HiveParser.TOK_DESCDATABASE,
-HiveParser.TOK_DESCTABLE,
-HiveParser.TOK_SHOWTABLES,
-HiveParser.TOK_SHOW_TABLESTATUS,
-HiveParser.TOK_SHOW_TBLPROPERTIES,
Review comment:
Because those don't even need a txn. I would have removed all of the
operations (EXPLAIN, DESC), however, that cause number of old tests to fail
that validate txnIds. I'll handle them separately in diff JIRAs 1-by-1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 660204)
Time Spent: 1h 20m (was: 1h 10m)
> Concurrency: Do not acquire locks for EXPLAIN and initiate implicit
> transaction for SHOW commands
> -
>
> Key: HIVE-25011
> URL: https://issues.apache.org/jira/browse/HIVE-25011
> Project: Hive
> Issue Type: Improvement
> Components: Locking, Transactions
>Affects Versions: 4.0.0
>Reporter: Gopal Vijayaraghavan
>Assignee: Gopal Vijayaraghavan
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-25011.1.patch, HIVE-25011.2.patch
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> {code}
> EXPLAIN UPDATE ...
> {code}
> should not be in conflict with another active ongoing UPDATE operation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25011) Concurrency: Do not acquire locks for EXPLAIN and initiate implicit transaction for SHOW commands
[
https://issues.apache.org/jira/browse/HIVE-25011?focusedWorklogId=660202=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-660202
]
ASF GitHub Bot logged work on HIVE-25011:
-
Author: ASF GitHub Bot
Created on: 05/Oct/21 10:25
Start Date: 05/Oct/21 10:25
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on a change in pull request #2660:
URL: https://github.com/apache/hive/pull/2660#discussion_r722102438
##
File path: ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java
##
@@ -184,19 +183,17 @@ public boolean accept(Path path) {
public static final int MAX_STATEMENTS_PER_TXN = 1;
public static final Pattern LEGACY_BUCKET_DIGIT_PATTERN =
Pattern.compile("^[0-9]{6}");
public static final Pattern BUCKET_PATTERN =
Pattern.compile("bucket_([0-9]+)(_[0-9]+)?$");
- private static final Set READ_TXN_TOKENS = new HashSet();
+ private static final Set READ_TXN_TOKENS = new HashSet<>();
private static Cache dirCache;
private static AtomicBoolean dirCacheInited = new AtomicBoolean();
static {
READ_TXN_TOKENS.addAll(Arrays.asList(
-HiveParser.TOK_DESCDATABASE,
-HiveParser.TOK_DESCTABLE,
-HiveParser.TOK_SHOWTABLES,
-HiveParser.TOK_SHOW_TABLESTATUS,
-HiveParser.TOK_SHOW_TBLPROPERTIES,
Review comment:
Because those don't even need a txn. I would have removed all of the
operations (EXPLAIN, DESC), however, that cause number of old tests to fail.
I'll handle them separately in diff JIRAs 1-by-1,.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 660202)
Time Spent: 1h (was: 50m)
> Concurrency: Do not acquire locks for EXPLAIN and initiate implicit
> transaction for SHOW commands
> -
>
> Key: HIVE-25011
> URL: https://issues.apache.org/jira/browse/HIVE-25011
> Project: Hive
> Issue Type: Improvement
> Components: Locking, Transactions
>Affects Versions: 4.0.0
>Reporter: Gopal Vijayaraghavan
>Assignee: Gopal Vijayaraghavan
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-25011.1.patch, HIVE-25011.2.patch
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> {code}
> EXPLAIN UPDATE ...
> {code}
> should not be in conflict with another active ongoing UPDATE operation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25011) Concurrency: Do not acquire locks for EXPLAIN and initiate implicit transaction for SHOW commands
[
https://issues.apache.org/jira/browse/HIVE-25011?focusedWorklogId=660203=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-660203
]
ASF GitHub Bot logged work on HIVE-25011:
-
Author: ASF GitHub Bot
Created on: 05/Oct/21 10:25
Start Date: 05/Oct/21 10:25
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on a change in pull request #2660:
URL: https://github.com/apache/hive/pull/2660#discussion_r722102438
##
File path: ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java
##
@@ -184,19 +183,17 @@ public boolean accept(Path path) {
public static final int MAX_STATEMENTS_PER_TXN = 1;
public static final Pattern LEGACY_BUCKET_DIGIT_PATTERN =
Pattern.compile("^[0-9]{6}");
public static final Pattern BUCKET_PATTERN =
Pattern.compile("bucket_([0-9]+)(_[0-9]+)?$");
- private static final Set READ_TXN_TOKENS = new HashSet();
+ private static final Set READ_TXN_TOKENS = new HashSet<>();
private static Cache dirCache;
private static AtomicBoolean dirCacheInited = new AtomicBoolean();
static {
READ_TXN_TOKENS.addAll(Arrays.asList(
-HiveParser.TOK_DESCDATABASE,
-HiveParser.TOK_DESCTABLE,
-HiveParser.TOK_SHOWTABLES,
-HiveParser.TOK_SHOW_TABLESTATUS,
-HiveParser.TOK_SHOW_TBLPROPERTIES,
Review comment:
Because those don't even need a txn. I would have removed all of the
operations (EXPLAIN, DESC), however, that cause number of old tests to fail.
I'll handle them separately in diff JIRAs 1-by-1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 660203)
Time Spent: 1h 10m (was: 1h)
> Concurrency: Do not acquire locks for EXPLAIN and initiate implicit
> transaction for SHOW commands
> -
>
> Key: HIVE-25011
> URL: https://issues.apache.org/jira/browse/HIVE-25011
> Project: Hive
> Issue Type: Improvement
> Components: Locking, Transactions
>Affects Versions: 4.0.0
>Reporter: Gopal Vijayaraghavan
>Assignee: Gopal Vijayaraghavan
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-25011.1.patch, HIVE-25011.2.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> {code}
> EXPLAIN UPDATE ...
> {code}
> should not be in conflict with another active ongoing UPDATE operation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HIVE-25587) Disable Iceberg table migration for unsupported source file formats
[ https://issues.apache.org/jira/browse/HIVE-25587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Bod resolved HIVE-25587. --- Resolution: Fixed > Disable Iceberg table migration for unsupported source file formats > --- > > Key: HIVE-25587 > URL: https://issues.apache.org/jira/browse/HIVE-25587 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > Currently, we only support migrating ORC, Parquet and Avro tables to Iceberg. > However, there is no check in the code to fail early for other formats (e.g. > text, json, rcfile), which can lead to wasted effort at best, and leaving the > source table unusable at worst. Therefore, we should check the source format > early and shortcircuit for unsupported types. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-25587) Disable Iceberg table migration for unsupported source file formats
[ https://issues.apache.org/jira/browse/HIVE-25587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17424391#comment-17424391 ] Marton Bod commented on HIVE-25587: --- Pushed to master. Thanks [~szita] and [~pvary] for the reviews. > Disable Iceberg table migration for unsupported source file formats > --- > > Key: HIVE-25587 > URL: https://issues.apache.org/jira/browse/HIVE-25587 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > Currently, we only support migrating ORC, Parquet and Avro tables to Iceberg. > However, there is no check in the code to fail early for other formats (e.g. > text, json, rcfile), which can lead to wasted effort at best, and leaving the > source table unusable at worst. Therefore, we should check the source format > early and shortcircuit for unsupported types. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25587) Disable Iceberg table migration for unsupported source file formats
[ https://issues.apache.org/jira/browse/HIVE-25587?focusedWorklogId=660185=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-660185 ] ASF GitHub Bot logged work on HIVE-25587: - Author: ASF GitHub Bot Created on: 05/Oct/21 09:38 Start Date: 05/Oct/21 09:38 Worklog Time Spent: 10m Work Description: marton-bod merged pull request #2696: URL: https://github.com/apache/hive/pull/2696 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 660185) Time Spent: 1.5h (was: 1h 20m) > Disable Iceberg table migration for unsupported source file formats > --- > > Key: HIVE-25587 > URL: https://issues.apache.org/jira/browse/HIVE-25587 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > Currently, we only support migrating ORC, Parquet and Avro tables to Iceberg. > However, there is no check in the code to fail early for other formats (e.g. > text, json, rcfile), which can lead to wasted effort at best, and leaving the > source table unusable at worst. Therefore, we should check the source format > early and shortcircuit for unsupported types. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25011) Concurrency: Do not acquire locks for EXPLAIN and initiate implicit transaction for SHOW commands
[
https://issues.apache.org/jira/browse/HIVE-25011?focusedWorklogId=660184=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-660184
]
ASF GitHub Bot logged work on HIVE-25011:
-
Author: ASF GitHub Bot
Created on: 05/Oct/21 09:34
Start Date: 05/Oct/21 09:34
Worklog Time Spent: 10m
Work Description: aasha commented on a change in pull request #2660:
URL: https://github.com/apache/hive/pull/2660#discussion_r722063151
##
File path: ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java
##
@@ -184,19 +183,17 @@ public boolean accept(Path path) {
public static final int MAX_STATEMENTS_PER_TXN = 1;
public static final Pattern LEGACY_BUCKET_DIGIT_PATTERN =
Pattern.compile("^[0-9]{6}");
public static final Pattern BUCKET_PATTERN =
Pattern.compile("bucket_([0-9]+)(_[0-9]+)?$");
- private static final Set READ_TXN_TOKENS = new HashSet();
+ private static final Set READ_TXN_TOKENS = new HashSet<>();
private static Cache dirCache;
private static AtomicBoolean dirCacheInited = new AtomicBoolean();
static {
READ_TXN_TOKENS.addAll(Arrays.asList(
-HiveParser.TOK_DESCDATABASE,
-HiveParser.TOK_DESCTABLE,
-HiveParser.TOK_SHOWTABLES,
-HiveParser.TOK_SHOW_TABLESTATUS,
-HiveParser.TOK_SHOW_TBLPROPERTIES,
Review comment:
Why did we remove show tables tok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 660184)
Time Spent: 50m (was: 40m)
> Concurrency: Do not acquire locks for EXPLAIN and initiate implicit
> transaction for SHOW commands
> -
>
> Key: HIVE-25011
> URL: https://issues.apache.org/jira/browse/HIVE-25011
> Project: Hive
> Issue Type: Improvement
> Components: Locking, Transactions
>Affects Versions: 4.0.0
>Reporter: Gopal Vijayaraghavan
>Assignee: Gopal Vijayaraghavan
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-25011.1.patch, HIVE-25011.2.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> {code}
> EXPLAIN UPDATE ...
> {code}
> should not be in conflict with another active ongoing UPDATE operation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25011) Concurrency: Do not acquire locks for EXPLAIN and initiate implicit transaction for SHOW commands
[
https://issues.apache.org/jira/browse/HIVE-25011?focusedWorklogId=660177=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-660177
]
ASF GitHub Bot logged work on HIVE-25011:
-
Author: ASF GitHub Bot
Created on: 05/Oct/21 09:25
Start Date: 05/Oct/21 09:25
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on pull request #2660:
URL: https://github.com/apache/hive/pull/2660#issuecomment-934231049
> We need to clean up the commit message / Jira title to reflect the things
we do in the change, but the change itself looks good to me.
updated commit message
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 660177)
Time Spent: 40m (was: 0.5h)
> Concurrency: Do not acquire locks for EXPLAIN and initiate implicit
> transaction for SHOW commands
> -
>
> Key: HIVE-25011
> URL: https://issues.apache.org/jira/browse/HIVE-25011
> Project: Hive
> Issue Type: Improvement
> Components: Locking, Transactions
>Affects Versions: 4.0.0
>Reporter: Gopal Vijayaraghavan
>Assignee: Gopal Vijayaraghavan
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-25011.1.patch, HIVE-25011.2.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> {code}
> EXPLAIN UPDATE ...
> {code}
> should not be in conflict with another active ongoing UPDATE operation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-25011) Concurrency: Do not acquire locks for EXPLAIN and initiate implicit transaction for SHOW commands
[
https://issues.apache.org/jira/browse/HIVE-25011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-25011:
--
Summary: Concurrency: Do not acquire locks for EXPLAIN and initiate
implicit transaction for SHOW commands (was: Concurrency: Do not acquire locks
for EXPLAIN)
> Concurrency: Do not acquire locks for EXPLAIN and initiate implicit
> transaction for SHOW commands
> -
>
> Key: HIVE-25011
> URL: https://issues.apache.org/jira/browse/HIVE-25011
> Project: Hive
> Issue Type: Improvement
> Components: Locking, Transactions
>Affects Versions: 4.0.0
>Reporter: Gopal Vijayaraghavan
>Assignee: Gopal Vijayaraghavan
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-25011.1.patch, HIVE-25011.2.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> {code}
> EXPLAIN UPDATE ...
> {code}
> should not be in conflict with another active ongoing UPDATE operation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23633) Close Metastore JDO query objects properly
[ https://issues.apache.org/jira/browse/HIVE-23633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-23633: -- Summary: Close Metastore JDO query objects properly (was: Metastore some JDO query objects do not close properly) > Close Metastore JDO query objects properly > --- > > Key: HIVE-23633 > URL: https://issues.apache.org/jira/browse/HIVE-23633 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23633.01.patch > > Time Spent: 15h 50m > Remaining Estimate: 0h > > After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895], > The metastore still has seen a memory leak on db resources: many > StatementImpls left unclosed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23633) Metastore some JDO query objects do not close properly
[ https://issues.apache.org/jira/browse/HIVE-23633?focusedWorklogId=660176=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-660176 ] ASF GitHub Bot logged work on HIVE-23633: - Author: ASF GitHub Bot Created on: 05/Oct/21 09:23 Start Date: 05/Oct/21 09:23 Worklog Time Spent: 10m Work Description: pvary merged pull request #2344: URL: https://github.com/apache/hive/pull/2344 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 660176) Time Spent: 15h 50m (was: 15h 40m) > Metastore some JDO query objects do not close properly > -- > > Key: HIVE-23633 > URL: https://issues.apache.org/jira/browse/HIVE-23633 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23633.01.patch > > Time Spent: 15h 50m > Remaining Estimate: 0h > > After patched [HIVE-10895|https://issues.apache.org/jira/browse/HIVE-10895], > The metastore still has seen a memory leak on db resources: many > StatementImpls left unclosed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-25590) Able to create views referencing temporary tables and materialized views
[
https://issues.apache.org/jira/browse/HIVE-25590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-25590:
--
Status: Patch Available (was: Open)
> Able to create views referencing temporary tables and materialized views
>
>
> Key: HIVE-25590
> URL: https://issues.apache.org/jira/browse/HIVE-25590
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Creating views/materialized views referencing temporary tables and
> materialized views are disabled in Hive. However the verification algorithm
> fails to recognize temporary tables and materialized views in subqueries.
> The verification also fails when the view definition contains joins because
> CBO transforms join branches to subqueries.
> Example1:
> {code}
> create temporary table tmp1 (c1 string, c2 string);
> create view tmp1_view as
> select subq.c1 from (select c1, c2 from tmp1) subq;
> {code}
> Example2:
> {code}
> create table t1 (a int) stored as orc tblproperties ('transactional'='true');
> create table t2 (a int) stored as orc tblproperties ('transactional'='true');
> create materialized view mv1 as
> select a from t1 where a = 10;
> create materialized view mv2 as
> select t2.a from mv1
> join t2 on (mv1.a = t2.a);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25590) Able to create views referencing temporary tables and materialized views
[
https://issues.apache.org/jira/browse/HIVE-25590?focusedWorklogId=660164=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-660164
]
ASF GitHub Bot logged work on HIVE-25590:
-
Author: ASF GitHub Bot
Created on: 05/Oct/21 08:56
Start Date: 05/Oct/21 08:56
Worklog Time Spent: 10m
Work Description: kasakrisz opened a new pull request #2697:
URL: https://github.com/apache/hive/pull/2697
### What changes were proposed in this pull request?
When validating view/materialized view instead of checking the tables in the
top QB object check all TableScan operators for temp tables and MVs
### Why are the changes needed?
Checking the top QB object is often not enough: subqueries may contain MV
and temp table scans.
### Does this PR introduce _any_ user-facing change?
Yes. At create [materialized]view if view definition has subqueries
referencing temp tables and/or materialized views an error message is printed.
### How was this patch tested?
```
mvn test -Dtest.output.overwrite -DskipSparkTests
-Dtest=TestNegativeLlapLocalCliDriver
-Dqfile=materialized_view_create_failure.q,create_view_failure11.q,create_view_failure12.q
-pl itests/qtest -Pitests
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 660164)
Remaining Estimate: 0h
Time Spent: 10m
> Able to create views referencing temporary tables and materialized views
>
>
> Key: HIVE-25590
> URL: https://issues.apache.org/jira/browse/HIVE-25590
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Creating views/materialized views referencing temporary tables and
> materialized views are disabled in Hive. However the verification algorithm
> fails to recognize temporary tables and materialized views in subqueries.
> The verification also fails when the view definition contains joins because
> CBO transforms join branches to subqueries.
> Example1:
> {code}
> create temporary table tmp1 (c1 string, c2 string);
> create view tmp1_view as
> select subq.c1 from (select c1, c2 from tmp1) subq;
> {code}
> Example2:
> {code}
> create table t1 (a int) stored as orc tblproperties ('transactional'='true');
> create table t2 (a int) stored as orc tblproperties ('transactional'='true');
> create materialized view mv1 as
> select a from t1 where a = 10;
> create materialized view mv2 as
> select t2.a from mv1
> join t2 on (mv1.a = t2.a);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-25590) Able to create views referencing temporary tables and materialized views
[
https://issues.apache.org/jira/browse/HIVE-25590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-25590:
--
Labels: pull-request-available (was: )
> Able to create views referencing temporary tables and materialized views
>
>
> Key: HIVE-25590
> URL: https://issues.apache.org/jira/browse/HIVE-25590
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Creating views/materialized views referencing temporary tables and
> materialized views are disabled in Hive. However the verification algorithm
> fails to recognize temporary tables and materialized views in subqueries.
> The verification also fails when the view definition contains joins because
> CBO transforms join branches to subqueries.
> Example1:
> {code}
> create temporary table tmp1 (c1 string, c2 string);
> create view tmp1_view as
> select subq.c1 from (select c1, c2 from tmp1) subq;
> {code}
> Example2:
> {code}
> create table t1 (a int) stored as orc tblproperties ('transactional'='true');
> create table t2 (a int) stored as orc tblproperties ('transactional'='true');
> create materialized view mv1 as
> select a from t1 where a = 10;
> create materialized view mv2 as
> select t2.a from mv1
> join t2 on (mv1.a = t2.a);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-21052) Make sure transactions get cleaned if they are aborted before addPartitions is called
[
https://issues.apache.org/jira/browse/HIVE-21052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17424366#comment-17424366
]
Denys Kuzmenko edited comment on HIVE-21052 at 10/5/21, 8:36 AM:
-
*Warning*: could lead to data-loss.
{code}
In a situation when operation involved dynamic partitioning is aborted, Cleaner
doesn't know what partition contains the aborted deltas, so it goes over all
partitions and removes aborted and `obsolete` deltas below the HighWatermark
(highest writeid that could be cleaned up). Those `obsolete` deltas could be in
reality an `active` ones (there is no easy way to identify those as
HighWatermark is defined on a table level). Current fix, in case of dynamic
partition abort operation, is to skip the check for so called `obsolete` deltas
and only handle 'aborted' ones.
{code}
Make sure to include HIVE-25502 as the fix for the described above issue.
was (Author: dkuzmenko):
*Warning*: could lead to data-loss.
{code}
In a situation when operation involved dynamic partitioning is aborted, Cleaner
doesn't know what partition contains the aborted deltas, so it goes over all
partitions and removes aborted and `obsolete` deltas below the HighWatermark
(highest writeid that could be cleaned up). Those `obsolete` deltas could be in
reality an `active` ones (there is no easy way to identify those as
HighWatermark is defined on a table level). Current fix, in case of dynamic
partition abort operation, is to skip the check for so called `obsolete` deltas
and only handle 'aborted' ones.
{code}
Make sure to include HIVE-25503 as the fix for the described above issue.
> Make sure transactions get cleaned if they are aborted before addPartitions
> is called
> -
>
> Key: HIVE-21052
> URL: https://issues.apache.org/jira/browse/HIVE-21052
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0, 3.1.1
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Critical
> Labels: pull-request-available
> Attachments: Aborted Txn w_Direct Write.pdf, HIVE-21052.1.patch,
> HIVE-21052.10.patch, HIVE-21052.11.patch, HIVE-21052.12.patch,
> HIVE-21052.2.patch, HIVE-21052.3.patch, HIVE-21052.4.patch,
> HIVE-21052.5.patch, HIVE-21052.6.patch, HIVE-21052.7.patch,
> HIVE-21052.8.patch, HIVE-21052.9.patch
>
> Time Spent: 12.5h
> Remaining Estimate: 0h
>
> If the transaction is aborted between openTxn and addPartitions and data has
> been written on the table the transaction manager will think it's an empty
> transaction and no cleaning will be done.
> This is currently an issue in the streaming API and in micromanaged tables.
> As proposed by [~ekoifman] this can be solved by:
> * Writing an entry with a special marker to TXN_COMPONENTS at openTxn and
> when addPartitions is called remove this entry from TXN_COMPONENTS and add
> the corresponding partition entry to TXN_COMPONENTS.
> * If the cleaner finds and entry with a special marker in TXN_COMPONENTS that
> specifies that a transaction was opened and it was aborted it must generate
> jobs for the worker for every possible partition available.
> cc [~ewohlstadter]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21052) Make sure transactions get cleaned if they are aborted before addPartitions is called
[
https://issues.apache.org/jira/browse/HIVE-21052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17424366#comment-17424366
]
Denys Kuzmenko commented on HIVE-21052:
---
*Warning*: could lead to data-loss.
{code}
In a situation when operation involved dynamic partitioning is aborted, Cleaner
doesn't know what partition contains the aborted deltas, so it goes over all
partitions and removes aborted and `obsolete` deltas below the HighWatermark
(highest writeid that could be cleaned up). Those `obsolete` deltas could be in
reality an `active` ones (there is no easy way to identify those as
HighWatermark is defined on a table level). Current fix, in case of dynamic
partition abort operation, is to skip the check for so called `obsolete` deltas
and only handle 'aborted' ones.
{code}
Make sure to include HIVE-25503 as the fix for the described above issue.
> Make sure transactions get cleaned if they are aborted before addPartitions
> is called
> -
>
> Key: HIVE-21052
> URL: https://issues.apache.org/jira/browse/HIVE-21052
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0, 3.1.1
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Critical
> Labels: pull-request-available
> Attachments: Aborted Txn w_Direct Write.pdf, HIVE-21052.1.patch,
> HIVE-21052.10.patch, HIVE-21052.11.patch, HIVE-21052.12.patch,
> HIVE-21052.2.patch, HIVE-21052.3.patch, HIVE-21052.4.patch,
> HIVE-21052.5.patch, HIVE-21052.6.patch, HIVE-21052.7.patch,
> HIVE-21052.8.patch, HIVE-21052.9.patch
>
> Time Spent: 12.5h
> Remaining Estimate: 0h
>
> If the transaction is aborted between openTxn and addPartitions and data has
> been written on the table the transaction manager will think it's an empty
> transaction and no cleaning will be done.
> This is currently an issue in the streaming API and in micromanaged tables.
> As proposed by [~ekoifman] this can be solved by:
> * Writing an entry with a special marker to TXN_COMPONENTS at openTxn and
> when addPartitions is called remove this entry from TXN_COMPONENTS and add
> the corresponding partition entry to TXN_COMPONENTS.
> * If the cleaner finds and entry with a special marker in TXN_COMPONENTS that
> specifies that a transaction was opened and it was aborted it must generate
> jobs for the worker for every possible partition available.
> cc [~ewohlstadter]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-25590) Able to create views referencing temporary tables and materialized views
[
https://issues.apache.org/jira/browse/HIVE-25590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa reassigned HIVE-25590:
-
> Able to create views referencing temporary tables and materialized views
>
>
> Key: HIVE-25590
> URL: https://issues.apache.org/jira/browse/HIVE-25590
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
>
> Creating views/materialized views referencing temporary tables and
> materialized views are disabled in Hive. However the verification algorithm
> fails to recognize temporary tables and materialized views in subqueries.
> The verification also fails when the view definition contains joins because
> CBO transforms join branches to subqueries.
> Example1:
> {code}
> create temporary table tmp1 (c1 string, c2 string);
> create view tmp1_view as
> select subq.c1 from (select c1, c2 from tmp1) subq;
> {code}
> Example2:
> {code}
> create table t1 (a int) stored as orc tblproperties ('transactional'='true');
> create table t2 (a int) stored as orc tblproperties ('transactional'='true');
> create materialized view mv1 as
> select a from t1 where a = 10;
> create materialized view mv2 as
> select t2.a from mv1
> join t2 on (mv1.a = t2.a);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
