[jira] [Work logged] (HIVE-26363) Time logged during repldump and replload per table is not in readable format
[ https://issues.apache.org/jira/browse/HIVE-26363?focusedWorklogId=796247&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-796247 ] ASF GitHub Bot logged work on HIVE-26363: - Author: ASF GitHub Bot Created on: 29/Jul/22 05:23 Start Date: 29/Jul/22 05:23 Worklog Time Spent: 10m Work Description: Rakshith606 closed pull request #3439: HIVE-26363: converting replLogger time format from epoch to readable UTC format URL: https://github.com/apache/hive/pull/3439 Issue Time Tracking --- Worklog Id: (was: 796247) Time Spent: 1h 10m (was: 1h) > Time logged during repldump and replload per table is not in readable format > > > Key: HIVE-26363 > URL: https://issues.apache.org/jira/browse/HIVE-26363 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, repl >Affects Versions: 4.0.0 >Reporter: Imran >Assignee: Rakshith C >Priority: Minor > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > During replDump and replLoad we capture time take for each activity in > hive.log file. This is captured in milliseconds which becomes difficult to > read during debug activity, this ticket is raised to change the time logged > in hive.log in UTC format. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-22368) Hive JDBC Storage Handler: some mysql data type can not be cast to hive data type
[
https://issues.apache.org/jira/browse/HIVE-22368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17572732#comment-17572732
]
datahu commented on HIVE-22368:
---
I encountered the same problem when converting the ResultSet obtained from JDBC
to an ORC format file, can you tell me how it was solved?
> Hive JDBC Storage Handler: some mysql data type can not be cast to hive data
> type
> -
>
> Key: HIVE-22368
> URL: https://issues.apache.org/jira/browse/HIVE-22368
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0, 3.1.1, 4.0.0
>Reporter: zhangbutao
>Assignee: zhangbutao
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22368.01.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Mysql data type（date、timestamp、decimal）can not be cast to hive data
> type（date、timestamp、decimal）。
> step to repo（take date type for example）:
> {code:java}
> //MySQL table:
> create table testdate(id date);
> //Hive table:
> CREATE EXTERNAL TABLE `hive_date`(
> id date )
> ROW FORMAT SERDE
> 'org.apache.hive.storage.jdbc.JdbcSerDe'
> STORED BY
> 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
>
> 'hive.sql.database.type'='MYSQL',
> 'hive.sql.dbcp.password'='hive',
> 'hive.sql.dbcp.username'='hive',
> 'hive.sql.jdbc.driver'='com.mysql.jdbc.Driver',
> 'hive.sql.jdbc.url'='jdbc:mysql://hadoop/test',
> 'hive.sql.table'='testdate');
> //Hive query:
> select * from hive_date;
> Error: java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: java.sql.Date cannot be cast to
> org.apache.hadoop.hive.common.type.Date (state=,code=0)
> //Error stack trace
> Caused by: java.lang.ClassCastException: java.sql.Date cannot be cast to
> org.apache.hadoop.hive.common.type.Date
> at
> org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaDateObjectInspector.getPrimitiveJavaObject(JavaDateObjectInspector.java:41)
> ~[hive-exec-3.1.0-bc3.0.1.jar:3.1.0-bc3.0.1]
> at
> org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaDateObjectInspector.getPrimitiveJavaObject(JavaDateObjectInspector.java:27)
> ~[hive-exec-3.1.0-bc3.0.1.jar:3.1.0-bc3.0.1]
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.copyToStandardObject(ObjectInspectorUtils.java:422)
> ~[hive-exec-3.1.0-bc3.0.1.jar:3.1.0-bc3.0.1]
> at
> org.apache.hadoop.hive.serde2.SerDeUtils.toThriftPayload(SerDeUtils.java:173)
> ~[hive-exec-3.1.0-bc3.0.1.jar:3.1.0-bc3.0.1]
> at
> org.apache.hadoop.hive.serde2.thrift.ThriftFormatter.convert(ThriftFormatter.java:49)
> ~[hive-exec-3.1.0-bc3.0.1.jar:3.1.0-bc3.0.1]
> at
> org.apache.hadoop.hive.ql.exec.ListSinkOperator.process(ListSinkOperator.java:94)
> ~[hive-exec-3.1.0-bc3.0.1.jar:3.1.0-bc3.0.1]
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:995)
> ~[hive-exec-3.1.0-bc3.0.1.jar:3.1.0-bc3.0.1]
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:941)
> ~[hive-exec-3.1.0-bc3.0.1.jar:3.1.0-bc3.0.1]
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:928)
> ~[hive-exec-3.1.0-bc3.0.1.jar:3.1.0-bc3.0.1]
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> ~[hive-exec-3.1.0-bc3.0.1.jar:3.1.0-bc3.0.1]
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:995)
> ~[hive-exec-3.1.0-bc3.0.1.jar:3.1.0-bc3.0.1]
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:941)
> ~[hive-exec-3.1.0-bc3.0.1.jar:3.1.0-bc3.0.1]
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125)
> ~[hive-exec-3.1.0-bc3.0.1.jar:3.1.0-bc3.0.1]
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:519)
> ~[hive-exec-3.1.0-bc3.0.1.jar:3.1.0-bc3.0.1]
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:511)
> ~[hive-exec-3.1.0-bc3.0.1.jar:3.1.0-bc3.0.1]
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:146)
> ~[hive-exec-3.1.0-bc3.0.1.jar:3.1.0-bc3.0.1]
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2706)
> ~[hive-exec-3.1.0-bc3.0.1.jar:3.1.0-bc3.0.1]
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:229)
> ~[
[jira] [Assigned] (HIVE-26396) The trunc function has a problem with precision interception and the result has many 0
[ https://issues.apache.org/jira/browse/HIVE-26396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuedong Luan reassigned HIVE-26396: --- Assignee: Xuedong Luan (was: Simhadri G) > The trunc function has a problem with precision interception and the result > has many 0 > -- > > Key: HIVE-26396 > URL: https://issues.apache.org/jira/browse/HIVE-26396 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.3 > Environment: CDP7.1.7 ,RedHat7.6 >Reporter: phZhou >Assignee: Xuedong Luan >Priority: Blocker > Labels: pull-request-available > Fix For: 3.2.0, 4.0.0, 4.0.0-alpha-2 > > Time Spent: 20m > Remaining Estimate: 0h > > The trunc function has a problem with precision interception and the result > has many 0，There is a problem with the return when the data is of decimal > type, and it is displayed normally when the data is of double type。The test > is as follows: > 1：Execute on beeline： > SELECT trunc(15.8963,3); > ++ > | _c0 | > ++ > | 15.896000 | > ++ > 1 row selected (0.074 seconds) > need return “15.896”is correct。 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-26396) The trunc function has a problem with precision interception and the result has many 0
[ https://issues.apache.org/jira/browse/HIVE-26396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuedong Luan reassigned HIVE-26396: --- Assignee: (was: Xuedong Luan) > The trunc function has a problem with precision interception and the result > has many 0 > -- > > Key: HIVE-26396 > URL: https://issues.apache.org/jira/browse/HIVE-26396 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.3 > Environment: CDP7.1.7 ,RedHat7.6 >Reporter: phZhou >Priority: Blocker > Labels: pull-request-available > Fix For: 3.2.0, 4.0.0, 4.0.0-alpha-2 > > Time Spent: 20m > Remaining Estimate: 0h > > The trunc function has a problem with precision interception and the result > has many 0，There is a problem with the return when the data is of decimal > type, and it is displayed normally when the data is of double type。The test > is as follows: > 1：Execute on beeline： > SELECT trunc(15.8963,3); > ++ > | _c0 | > ++ > | 15.896000 | > ++ > 1 row selected (0.074 seconds) > need return “15.896”is correct。 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-26437) dump unpartitioned Tables in parallel
[ https://issues.apache.org/jira/browse/HIVE-26437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17572697#comment-17572697 ] Rajesh Balamohan commented on HIVE-26437: - Can you please add more details on this? > dump unpartitioned Tables in parallel > - > > Key: HIVE-26437 > URL: https://issues.apache.org/jira/browse/HIVE-26437 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Amit Saonerkar >Assignee: Amit Saonerkar >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-24933) Replication fails for transactional tables having same name as dropped non-transactional table
[ https://issues.apache.org/jira/browse/HIVE-24933?focusedWorklogId=796203&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-796203 ] ASF GitHub Bot logged work on HIVE-24933: - Author: ASF GitHub Bot Created on: 28/Jul/22 22:41 Start Date: 28/Jul/22 22:41 Worklog Time Spent: 10m Work Description: cmunkey commented on PR #3435: URL: https://github.com/apache/hive/pull/3435#issuecomment-1198697975 Yes, I did not know this was an exiting PR that was resurrected. Yes, I thought that keeping table logic where it was was more logical, DeferredWorkHelperImpl doesn't need to know about tables, and doesn't need to be changed in the future if we find more of these issues where we look at an object in different places and get different results. Otherwise, seems fine. Issue Time Tracking --- Worklog Id: (was: 796203) Time Spent: 2.5h (was: 2h 20m) > Replication fails for transactional tables having same name as dropped > non-transactional table > -- > > Key: HIVE-24933 > URL: https://issues.apache.org/jira/browse/HIVE-24933 > Project: Hive > Issue Type: Bug >Reporter: Pratyush Madhukar >Assignee: Pratyush Madhukar >Priority: Major > Labels: pull-request-available > Time Spent: 2.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-22417) Remove stringifyException from MetaStore
[
https://issues.apache.org/jira/browse/HIVE-22417?focusedWorklogId=796188&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-796188
]
ASF GitHub Bot logged work on HIVE-22417:
-
Author: ASF GitHub Bot
Created on: 28/Jul/22 21:03
Start Date: 28/Jul/22 21:03
Worklog Time Spent: 10m
Work Description: belugabehr commented on code in PR #3478:
URL: https://github.com/apache/hive/pull/3478#discussion_r932668308
##
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java:
##
@@ -205,7 +205,7 @@ private ReplChangeManager(Configuration conf) throws
MetaException {
inited = true;
}
} catch (IOException e) {
- throw new MetaException(StringUtils.stringifyException(e));
+ throw new MetaException(e.getMessage());
Review Comment:
Also, I see I did this log-and-throw in a couple of other places. Good
catch. Thanks. Will update.
Issue Time Tracking
---
Worklog Id: (was: 796188)
Time Spent: 50m (was: 40m)
> Remove stringifyException from MetaStore
>
>
> Key: HIVE-22417
> URL: https://issues.apache.org/jira/browse/HIVE-22417
> Project: Hive
> Issue Type: Sub-task
> Components: Metastore, Standalone Metastore
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22417.1.patch, HIVE-22417.2.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-22417) Remove stringifyException from MetaStore
[
https://issues.apache.org/jira/browse/HIVE-22417?focusedWorklogId=796187&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-796187
]
ASF GitHub Bot logged work on HIVE-22417:
-
Author: ASF GitHub Bot
Created on: 28/Jul/22 20:57
Start Date: 28/Jul/22 20:57
Worklog Time Spent: 10m
Work Description: belugabehr commented on code in PR #3478:
URL: https://github.com/apache/hive/pull/3478#discussion_r932663972
##
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java:
##
@@ -378,32 +378,28 @@ static Path getCMPath(Configuration conf, String name,
String checkSum, String c
* @return Corresponding FileInfo object
*/
public static FileInfo getFileInfo(Path src, String checksumString, String
srcCMRootURI, String subDir,
- Configuration conf) throws MetaException {
-try {
- FileSystem srcFs = src.getFileSystem(conf);
- if (checksumString == null) {
-return new FileInfo(srcFs, src, subDir);
- }
+ Configuration conf) throws IOException {
+FileSystem srcFs = src.getFileSystem(conf);
+if (checksumString == null) {
+ return new FileInfo(srcFs, src, subDir);
+}
- Path cmPath = getCMPath(conf, src.getName(), checksumString,
srcCMRootURI);
- if (!srcFs.exists(src)) {
-return new FileInfo(srcFs, src, cmPath, checksumString, false, subDir);
- }
+Path cmPath = getCMPath(conf, src.getName(), checksumString, srcCMRootURI);
+if (!srcFs.exists(src)) {
+ return new FileInfo(srcFs, src, cmPath, checksumString, false, subDir);
+}
- String currentChecksumString;
- try {
-currentChecksumString = checksumFor(src, srcFs);
- } catch (IOException ex) {
-// If the file is missing or getting modified, then refer CM path
-return new FileInfo(srcFs, src, cmPath, checksumString, false, subDir);
- }
- if ((currentChecksumString == null) ||
checksumString.equals(currentChecksumString)) {
-return new FileInfo(srcFs, src, cmPath, checksumString, true, subDir);
- } else {
-return new FileInfo(srcFs, src, cmPath, checksumString, false, subDir);
- }
-} catch (IOException e) {
- throw new MetaException(StringUtils.stringifyException(e));
Review Comment:
This comes from
[HIVE-15525](https://issues.apache.org/jira/browse/HIVE-15525). I'm do not
understand why this particular decision was made.
Issue Time Tracking
---
Worklog Id: (was: 796187)
Time Spent: 40m (was: 0.5h)
> Remove stringifyException from MetaStore
>
>
> Key: HIVE-22417
> URL: https://issues.apache.org/jira/browse/HIVE-22417
> Project: Hive
> Issue Type: Sub-task
> Components: Metastore, Standalone Metastore
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22417.1.patch, HIVE-22417.2.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-22417) Remove stringifyException from MetaStore
[
https://issues.apache.org/jira/browse/HIVE-22417?focusedWorklogId=796186&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-796186
]
ASF GitHub Bot logged work on HIVE-22417:
-
Author: ASF GitHub Bot
Created on: 28/Jul/22 20:57
Start Date: 28/Jul/22 20:57
Worklog Time Spent: 10m
Work Description: belugabehr commented on code in PR #3478:
URL: https://github.com/apache/hive/pull/3478#discussion_r932663609
##
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java:
##
@@ -205,7 +205,7 @@ private ReplChangeManager(Configuration conf) throws
MetaException {
inited = true;
}
} catch (IOException e) {
- throw new MetaException(StringUtils.stringifyException(e));
+ throw new MetaException(e.getMessage());
Review Comment:
Ya, this ideally should just throw something other than a MetaException, but
it wasn't easy to fix like the other location. Log-and-Throw is a bad
practice; the code should do one or the other, not both. However, if you would
like, I can add a log along side the throw. Log-and-throw is probably a better
compromise.
The idea in general though is that MetaException is an Exception that gets
passed back to the client via the Thrift API. It should not be part of any of
the core Hive code,... Hive should simply log the exception (`error`) at the
Thrift service shim and throw MetaException back to the client.
Issue Time Tracking
---
Worklog Id: (was: 796186)
Time Spent: 0.5h (was: 20m)
> Remove stringifyException from MetaStore
>
>
> Key: HIVE-22417
> URL: https://issues.apache.org/jira/browse/HIVE-22417
> Project: Hive
> Issue Type: Sub-task
> Components: Metastore, Standalone Metastore
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22417.1.patch, HIVE-22417.2.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-22417) Remove stringifyException from MetaStore
[
https://issues.apache.org/jira/browse/HIVE-22417?focusedWorklogId=796139&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-796139
]
ASF GitHub Bot logged work on HIVE-22417:
-
Author: ASF GitHub Bot
Created on: 28/Jul/22 17:10
Start Date: 28/Jul/22 17:10
Worklog Time Spent: 10m
Work Description: zabetak commented on code in PR #3478:
URL: https://github.com/apache/hive/pull/3478#discussion_r932413646
##
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java:
##
@@ -205,7 +205,7 @@ private ReplChangeManager(Configuration conf) throws
MetaException {
inited = true;
}
} catch (IOException e) {
- throw new MetaException(StringUtils.stringifyException(e));
+ throw new MetaException(e.getMessage());
Review Comment:
Should we `LOG.error("..", e)` before throwing to avoid losing the complete
stacktrace?
##
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java:
##
@@ -378,32 +378,28 @@ static Path getCMPath(Configuration conf, String name,
String checkSum, String c
* @return Corresponding FileInfo object
*/
public static FileInfo getFileInfo(Path src, String checksumString, String
srcCMRootURI, String subDir,
- Configuration conf) throws MetaException {
-try {
- FileSystem srcFs = src.getFileSystem(conf);
- if (checksumString == null) {
-return new FileInfo(srcFs, src, subDir);
- }
+ Configuration conf) throws IOException {
+FileSystem srcFs = src.getFileSystem(conf);
+if (checksumString == null) {
+ return new FileInfo(srcFs, src, subDir);
+}
- Path cmPath = getCMPath(conf, src.getName(), checksumString,
srcCMRootURI);
- if (!srcFs.exists(src)) {
-return new FileInfo(srcFs, src, cmPath, checksumString, false, subDir);
- }
+Path cmPath = getCMPath(conf, src.getName(), checksumString, srcCMRootURI);
+if (!srcFs.exists(src)) {
+ return new FileInfo(srcFs, src, cmPath, checksumString, false, subDir);
+}
- String currentChecksumString;
- try {
-currentChecksumString = checksumFor(src, srcFs);
- } catch (IOException ex) {
-// If the file is missing or getting modified, then refer CM path
-return new FileInfo(srcFs, src, cmPath, checksumString, false, subDir);
- }
- if ((currentChecksumString == null) ||
checksumString.equals(currentChecksumString)) {
-return new FileInfo(srcFs, src, cmPath, checksumString, true, subDir);
- } else {
-return new FileInfo(srcFs, src, cmPath, checksumString, false, subDir);
- }
-} catch (IOException e) {
- throw new MetaException(StringUtils.stringifyException(e));
Review Comment:
Dropping the `MetaException` from this method seems like a good idea to me
but do we understand why it was added in the first place?
Issue Time Tracking
---
Worklog Id: (was: 796139)
Time Spent: 20m (was: 10m)
> Remove stringifyException from MetaStore
>
>
> Key: HIVE-22417
> URL: https://issues.apache.org/jira/browse/HIVE-22417
> Project: Hive
> Issue Type: Sub-task
> Components: Metastore, Standalone Metastore
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22417.1.patch, HIVE-22417.2.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26433) StorageHandler authorization doesn't work with "METASTORE" db type
[
https://issues.apache.org/jira/browse/HIVE-26433?focusedWorklogId=796135&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-796135
]
ASF GitHub Bot logged work on HIVE-26433:
-
Author: ASF GitHub Bot
Created on: 28/Jul/22 16:46
Start Date: 28/Jul/22 16:46
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3482:
URL: https://github.com/apache/hive/pull/3482#discussion_r932466604
##
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/JdbcStorageHandler.java:
##
@@ -99,7 +101,10 @@ public void configureInputJobProperties(TableDesc
tableDesc, Map
@Override
public URI getURIForAuth(Table table) throws URISyntaxException {
Map tableProperties =
HiveCustomStorageHandlerUtils.getTableProperties(table);
-String host_url = tableProperties.get(Constants.JDBC_URL);
+DatabaseType dbType = DatabaseType.valueOf(
+ tableProperties.get(JdbcStorageConfig.DATABASE_TYPE.getPropertyName()));
+String host_url = DatabaseType.METASTORE == dbType ?
+ "internal:metastore://" : tableProperties.get(Constants.JDBC_URL);
Review Comment:
should it be `jdbc:metastore://`?
Issue Time Tracking
---
Worklog Id: (was: 796135)
Time Spent: 40m (was: 0.5h)
> StorageHandler authorization doesn't work with "METASTORE" db type
> --
>
> Key: HIVE-26433
> URL: https://issues.apache.org/jira/browse/HIVE-26433
> Project: Hive
> Issue Type: Task
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> StorageHandler Authorization doesn't work with the "METASTORE" db type:
> {code}
> CREATE EXTERNAL TABLE IF NOT EXISTS `SYS`.`HIVE_LOCKS` (
> `HL_LOCK_EXT_ID` bigint,
> `HL_LOCK_INT_ID` bigint,
> `HL_TXNID` bigint,
> `HL_DB` string,
> `HL_TABLE` string,
> `HL_PARTITION` string,
> `HL_LOCK_STATE` string,
> `HL_LOCK_TYPE` string,
> `HL_LAST_HEARTBEAT` bigint,
> `HL_ACQUIRED_AT` bigint,
> `HL_USER` string,
> `HL_HOST` string,
> `HL_HEARTBEAT_COUNT` int,
> `HL_AGENT_INFO` string,
> `HL_BLOCKEDBY_EXT_ID` bigint,
> `HL_BLOCKEDBY_INT_ID` bigint
> )
> STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
> "hive.sql.database.type" = "METASTORE",
> "hive.sql.query" =
> "SELECT
> \"HL_LOCK_EXT_ID\",
> \"HL_LOCK_INT_ID\",
> \"HL_TXNID\",
> \"HL_DB\",
> \"HL_TABLE\",
> \"HL_PARTITION\",
> \"HL_LOCK_STATE\",
> \"HL_LOCK_TYPE\",
> \"HL_LAST_HEARTBEAT\",
> \"HL_ACQUIRED_AT\",
> \"HL_USER\",
> \"HL_HOST\",
> \"HL_HEARTBEAT_COUNT\",
> \"HL_AGENT_INFO\",
> \"HL_BLOCKEDBY_EXT_ID\",
> \"HL_BLOCKEDBY_INT_ID\"
> FROM \"HIVE_LOCKS\""
> );
> {code}
> Exception thrown:
> {code}
> Error: Error while compiling statement: FAILED: HiveAccessControlException
> Permission denied: user [hive] does not have [RWSTORAGE] privilege on
> [null/null] (state=42000,code=4)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26433) StorageHandler authorization doesn't work with "METASTORE" db type
[
https://issues.apache.org/jira/browse/HIVE-26433?focusedWorklogId=796132&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-796132
]
ASF GitHub Bot logged work on HIVE-26433:
-
Author: ASF GitHub Bot
Created on: 28/Jul/22 16:38
Start Date: 28/Jul/22 16:38
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3482:
URL: https://github.com/apache/hive/pull/3482#discussion_r932459551
##
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/JdbcStorageHandler.java:
##
@@ -99,7 +101,10 @@ public void configureInputJobProperties(TableDesc
tableDesc, Map
@Override
public URI getURIForAuth(Table table) throws URISyntaxException {
Map tableProperties =
HiveCustomStorageHandlerUtils.getTableProperties(table);
-String host_url = tableProperties.get(Constants.JDBC_URL);
+DatabaseType dbType = DatabaseType.valueOf(
+ tableProperties.get(JdbcStorageConfig.DATABASE_TYPE.getPropertyName()));
+String host_url = DatabaseType.METASTORE == dbType ?
+ "internal:metastore://" : tableProperties.get(Constants.JDBC_URL);
Review Comment:
With the current setup
'HivePrivilegeObject':{'type':STORAGEHANDLER_URI, 'dbName':null,
'objectType':STORAGEHANDLER_URI, 'objectName':internal:metastore:///null,
'columns':[], 'partKeys':[], 'commandParams':[], 'actionType':OTHER,
'owner':hive}
elements={storage-type=internal:metastore; storage-url=/null; }
Why do we need to remove extra "/" after "metastore?
Issue Time Tracking
---
Worklog Id: (was: 796132)
Time Spent: 0.5h (was: 20m)
> StorageHandler authorization doesn't work with "METASTORE" db type
> --
>
> Key: HIVE-26433
> URL: https://issues.apache.org/jira/browse/HIVE-26433
> Project: Hive
> Issue Type: Task
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> StorageHandler Authorization doesn't work with the "METASTORE" db type:
> {code}
> CREATE EXTERNAL TABLE IF NOT EXISTS `SYS`.`HIVE_LOCKS` (
> `HL_LOCK_EXT_ID` bigint,
> `HL_LOCK_INT_ID` bigint,
> `HL_TXNID` bigint,
> `HL_DB` string,
> `HL_TABLE` string,
> `HL_PARTITION` string,
> `HL_LOCK_STATE` string,
> `HL_LOCK_TYPE` string,
> `HL_LAST_HEARTBEAT` bigint,
> `HL_ACQUIRED_AT` bigint,
> `HL_USER` string,
> `HL_HOST` string,
> `HL_HEARTBEAT_COUNT` int,
> `HL_AGENT_INFO` string,
> `HL_BLOCKEDBY_EXT_ID` bigint,
> `HL_BLOCKEDBY_INT_ID` bigint
> )
> STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
> "hive.sql.database.type" = "METASTORE",
> "hive.sql.query" =
> "SELECT
> \"HL_LOCK_EXT_ID\",
> \"HL_LOCK_INT_ID\",
> \"HL_TXNID\",
> \"HL_DB\",
> \"HL_TABLE\",
> \"HL_PARTITION\",
> \"HL_LOCK_STATE\",
> \"HL_LOCK_TYPE\",
> \"HL_LAST_HEARTBEAT\",
> \"HL_ACQUIRED_AT\",
> \"HL_USER\",
> \"HL_HOST\",
> \"HL_HEARTBEAT_COUNT\",
> \"HL_AGENT_INFO\",
> \"HL_BLOCKEDBY_EXT_ID\",
> \"HL_BLOCKEDBY_INT_ID\"
> FROM \"HIVE_LOCKS\""
> );
> {code}
> Exception thrown:
> {code}
> Error: Error while compiling statement: FAILED: HiveAccessControlException
> Permission denied: user [hive] does not have [RWSTORAGE] privilege on
> [null/null] (state=42000,code=4)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26426) Avoid StringIndexOutOfBoundsException in canCBOHandleAst() method.
[
https://issues.apache.org/jira/browse/HIVE-26426?focusedWorklogId=796125&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-796125
]
ASF GitHub Bot logged work on HIVE-26426:
-
Author: ASF GitHub Bot
Created on: 28/Jul/22 16:05
Start Date: 28/Jul/22 16:05
Worklog Time Spent: 10m
Work Description: achennagiri commented on code in PR #3474:
URL: https://github.com/apache/hive/pull/3474#discussion_r932428871
##
ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java:
##
@@ -947,7 +947,7 @@ Pair canCBOHandleAst(ASTNode ast, QB qb,
PreCboCtx cboCtx) {
// Now check QB in more detail. canHandleQbForCbo returns null if query can
// be handled.
msg = CalcitePlanner.canHandleQbForCbo(queryProperties, conf, true,
needToLogMessage);
-if (msg == null) {
+if (msg == null || msg.isEmpty()) {
return Pair.of(true, msg);
}
msg = msg.substring(0, msg.length() - 2);
Review Comment:
Great. Thanks for the review Stamatis!
Issue Time Tracking
---
Worklog Id: (was: 796125)
Time Spent: 1h 50m (was: 1h 40m)
> Avoid StringIndexOutOfBoundsException in canCBOHandleAst() method.
> --
>
> Key: HIVE-26426
> URL: https://issues.apache.org/jira/browse/HIVE-26426
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Abhay
>Assignee: Abhay
>Priority: Minor
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-2
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> The call to canHandleQbForCbo() can result in an
> StringIndexOutOfBoundsException. The assumption in the code is that the msg
> can only be null and we handle that but the msg can also be an empty string
> if the *verbose* is set to false. This can happen if INFO Logging is not
> enabled. We need to handle that case.
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java#L913]
> Here is the stack trace for reference:
> {noformat}
> FAILED: StringIndexOutOfBoundsException String index out of range: -2
> 15:10:24.192 [HiveServer2-Background-Pool: Thread-305] ERROR
> org.apache.hadoop.hive.ql.Driver - FAILED: StringIndexOutOfBoundsException
> String index out of range: -2
> java.lang.StringIndexOutOfBoundsException: String index out of range: -2
> at java.lang.String.substring(String.java:1967)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.canCBOHandleAst(CalcitePlanner.java:996)
>
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:572)
>
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:13063)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:472)
>
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:314)
> at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:223)
> at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:105)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:201)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:650)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:596)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:590)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:127)
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:206)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:336)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:358)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750){noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26436) Hive on MR NullPointerException
[ https://issues.apache.org/jira/browse/HIVE-26436?focusedWorklogId=796095&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-796095 ] ASF GitHub Bot logged work on HIVE-26436: - Author: ASF GitHub Bot Created on: 28/Jul/22 15:14 Start Date: 28/Jul/22 15:14 Worklog Time Spent: 10m Work Description: ganlute opened a new pull request, #3485: URL: https://github.com/apache/hive/pull/3485 … been called and close called. ### What changes were proposed in this pull request? When the operator has not been called to initialize, and it is called to close, exception will happen. I fix it by check conf is null so that the close could be finished. It work well in my cluster. However, I am not sure, it may be a better way to skip close if the operator has not been initialized ? ### Why are the changes needed? I think it may be a bug. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? 1. unit test passed. 2. apply the patch to cluster and task passed. Issue Time Tracking --- Worklog Id: (was: 796095) Remaining Estimate: 0h Time Spent: 10m > Hive on MR NullPointerException > --- > > Key: HIVE-26436 > URL: https://issues.apache.org/jira/browse/HIVE-26436 > Project: Hive > Issue Type: Bug > Components: Query Planning >Affects Versions: 3.1.2 >Reporter: TE >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > 2022-07-27 11:23:26,337 INFO [AsyncDispatcher event handler] > org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics > report from attempt_1658508427196_17228_m_00_3: Error: > java.lang.RuntimeException: Hive Runtime Error while closing operators > at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:211) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61) > at > org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.exec.Operator.getCounterName(Operator.java:502) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:742) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:757) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:757) > at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:193) > ... 9 more > > Hive on MR run failed and the error details is above. > Hive version: 3.1.2 > > When initializeOp has not been called and close called, this exception will > happen. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26436) Hive on MR NullPointerException
[ https://issues.apache.org/jira/browse/HIVE-26436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-26436: -- Labels: pull-request-available (was: ) > Hive on MR NullPointerException > --- > > Key: HIVE-26436 > URL: https://issues.apache.org/jira/browse/HIVE-26436 > Project: Hive > Issue Type: Bug > Components: Query Planning >Affects Versions: 3.1.2 >Reporter: TE >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > 2022-07-27 11:23:26,337 INFO [AsyncDispatcher event handler] > org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics > report from attempt_1658508427196_17228_m_00_3: Error: > java.lang.RuntimeException: Hive Runtime Error while closing operators > at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:211) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61) > at > org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.exec.Operator.getCounterName(Operator.java:502) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:742) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:757) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:757) > at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:193) > ... 9 more > > Hive on MR run failed and the error details is above. > Hive version: 3.1.2 > > When initializeOp has not been called and close called, this exception will > happen. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26436) Hive on MR NullPointerException
[ https://issues.apache.org/jira/browse/HIVE-26436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] TE updated HIVE-26436: -- Description: 2022-07-27 11:23:26,337 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1658508427196_17228_m_00_3: Error: java.lang.RuntimeException: Hive Runtime Error while closing operators at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:211) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61) at org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) Caused by: java.lang.NullPointerException at org.apache.hadoop.hive.ql.exec.Operator.getCounterName(Operator.java:502) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:742) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:757) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:757) at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:193) ... 9 more Hive on MR run failed and the error details is above. Hive version: 3.1.2 When initializeOp has not been called and close called, this exception will happen. was: 2022-07-27 11:23:26,337 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1658508427196_17228_m_00_3: Error: java.lang.RuntimeException: Hive Runtime Error while closing operators at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:211) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61) at org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) Caused by: java.lang.NullPointerException at org.apache.hadoop.hive.ql.exec.Operator.getCounterName(Operator.java:502) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:742) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:757) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:757) at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:193) ... 9 more Hive on MR run failed and the error details is above. Hive version: 3.1.2 > Hive on MR NullPointerException > --- > > Key: HIVE-26436 > URL: https://issues.apache.org/jira/browse/HIVE-26436 > Project: Hive > Issue Type: Bug > Components: Query Planning >Affects Versions: 3.1.2 >Reporter: TE >Priority: Minor > > 2022-07-27 11:23:26,337 INFO [AsyncDispatcher event handler] > org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics > report from attempt_1658508427196_17228_m_00_3: Error: > java.lang.RuntimeException: Hive Runtime Error while closing operators > at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:211) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61) > at > org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.exec.Operator.getCounterName(Operator.java:502) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:742) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:757) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:757) >
[jira] [Assigned] (HIVE-26437) dump unpartitioned Tables in parallel
[ https://issues.apache.org/jira/browse/HIVE-26437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Saonerkar reassigned HIVE-26437: - > dump unpartitioned Tables in parallel > - > > Key: HIVE-26437 > URL: https://issues.apache.org/jira/browse/HIVE-26437 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Amit Saonerkar >Assignee: Amit Saonerkar >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-26436) Hive on MR NullPointerException
[ https://issues.apache.org/jira/browse/HIVE-26436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] TE reassigned HIVE-26436: - Assignee: (was: TE) > Hive on MR NullPointerException > --- > > Key: HIVE-26436 > URL: https://issues.apache.org/jira/browse/HIVE-26436 > Project: Hive > Issue Type: Bug > Components: Query Planning >Affects Versions: 3.1.2 >Reporter: TE >Priority: Minor > > 2022-07-27 11:23:26,337 INFO [AsyncDispatcher event handler] > org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics > report from attempt_1658508427196_17228_m_00_3: Error: > java.lang.RuntimeException: Hive Runtime Error while closing operators > at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:211) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61) > at > org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.exec.Operator.getCounterName(Operator.java:502) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:742) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:757) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:757) > at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:193) > ... 9 more > > Hive on MR run failed and the error details is above. > Hive version: 3.1.2 > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26436) Hive on MR NullPointerException
[ https://issues.apache.org/jira/browse/HIVE-26436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] TE updated HIVE-26436: -- Description: 2022-07-27 11:23:26,337 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1658508427196_17228_m_00_3: Error: java.lang.RuntimeException: Hive Runtime Error while closing operators at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:211) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61) at org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) Caused by: java.lang.NullPointerException at org.apache.hadoop.hive.ql.exec.Operator.getCounterName(Operator.java:502) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:742) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:757) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:757) at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:193) ... 9 more Hive on MR run failed and the error details is above. Hive version: 3.1.2 was: 2022-07-27 11:23:26,337 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1658508427196_17228_m_00_3: Error: java.lang.RuntimeException: Hive Runtime Error while closing operators at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:211) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61) at org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) Caused by: java.lang.NullPointerException at org.apache.hadoop.hive.ql.exec.Operator.getCounterName(Operator.java:502) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:742) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:757) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:757) at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:193) ... 9 more Hive on MR run failed and the error info is above. Hive version: 3.1.2 > Hive on MR NullPointerException > --- > > Key: HIVE-26436 > URL: https://issues.apache.org/jira/browse/HIVE-26436 > Project: Hive > Issue Type: Bug > Components: Query Planning >Affects Versions: 3.1.2 >Reporter: TE >Assignee: TE >Priority: Minor > > 2022-07-27 11:23:26,337 INFO [AsyncDispatcher event handler] > org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics > report from attempt_1658508427196_17228_m_00_3: Error: > java.lang.RuntimeException: Hive Runtime Error while closing operators > at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:211) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61) > at > org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.exec.Operator.getCounterName(Operator.java:502) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:742) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:757) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:757) > at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMap
[jira] [Updated] (HIVE-26436) Hive on MR NullPointerException
[ https://issues.apache.org/jira/browse/HIVE-26436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] TE updated HIVE-26436: -- Description: 2022-07-27 11:23:26,337 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1658508427196_17228_m_00_3: Error: java.lang.RuntimeException: Hive Runtime Error while closing operators at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:211) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61) at org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) Caused by: java.lang.NullPointerException at org.apache.hadoop.hive.ql.exec.Operator.getCounterName(Operator.java:502) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:742) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:757) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:757) at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:193) ... 9 more Hive on MR run failed and the error info is above. Hive version: 3.1.2 Environment: (was: 2022-07-27 11:23:26,337 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1658508427196_17228_m_00_3: Error: java.lang.RuntimeException: Hive Runtime Error while closing operators at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:211) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61) at org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) Caused by: java.lang.NullPointerException at org.apache.hadoop.hive.ql.exec.Operator.getCounterName(Operator.java:502) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:742) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:757) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:757) at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:193) ... 9 more) Summary: Hive on MR NullPointerException (was: test) > Hive on MR NullPointerException > --- > > Key: HIVE-26436 > URL: https://issues.apache.org/jira/browse/HIVE-26436 > Project: Hive > Issue Type: Bug > Components: Query Planning >Affects Versions: 3.1.2 >Reporter: TE >Assignee: TE >Priority: Minor > > 2022-07-27 11:23:26,337 INFO [AsyncDispatcher event handler] > org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics > report from attempt_1658508427196_17228_m_00_3: Error: > java.lang.RuntimeException: Hive Runtime Error while closing operators > at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:211) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61) > at > org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.exec.Operator.getCounterName(Operator.java:502) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:742) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:757) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:757) > at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapp
[jira] [Updated] (HIVE-26436) test
[ https://issues.apache.org/jira/browse/HIVE-26436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] TE updated HIVE-26436: -- Environment: 2022-07-27 11:23:26,337 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1658508427196_17228_m_00_3: Error: java.lang.RuntimeException: Hive Runtime Error while closing operators at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:211) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61) at org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) Caused by: java.lang.NullPointerException at org.apache.hadoop.hive.ql.exec.Operator.getCounterName(Operator.java:502) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:742) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:757) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:757) at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:193) ... 9 more > test > > > Key: HIVE-26436 > URL: https://issues.apache.org/jira/browse/HIVE-26436 > Project: Hive > Issue Type: Bug > Components: Query Planning >Affects Versions: 3.1.2 > Environment: 2022-07-27 11:23:26,337 INFO [AsyncDispatcher event > handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: > Diagnostics report from attempt_1658508427196_17228_m_00_3: Error: > java.lang.RuntimeException: Hive Runtime Error while closing operators > at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:211) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61) > at > org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.exec.Operator.getCounterName(Operator.java:502) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:742) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:757) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:757) > at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:193) > ... 9 more >Reporter: TE >Assignee: TE >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-26436) test
[ https://issues.apache.org/jira/browse/HIVE-26436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] TE reassigned HIVE-26436: - > test > > > Key: HIVE-26436 > URL: https://issues.apache.org/jira/browse/HIVE-26436 > Project: Hive > Issue Type: Bug > Components: Query Planning >Affects Versions: 3.1.2 >Reporter: TE >Assignee: TE >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26433) StorageHandler authorization doesn't work with "METASTORE" db type
[
https://issues.apache.org/jira/browse/HIVE-26433?focusedWorklogId=796063&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-796063
]
ASF GitHub Bot logged work on HIVE-26433:
-
Author: ASF GitHub Bot
Created on: 28/Jul/22 13:11
Start Date: 28/Jul/22 13:11
Worklog Time Spent: 10m
Work Description: saihemanth-cloudera commented on code in PR #3482:
URL: https://github.com/apache/hive/pull/3482#discussion_r932198572
##
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/JdbcStorageHandler.java:
##
@@ -99,7 +101,10 @@ public void configureInputJobProperties(TableDesc
tableDesc, Map
@Override
public URI getURIForAuth(Table table) throws URISyntaxException {
Map tableProperties =
HiveCustomStorageHandlerUtils.getTableProperties(table);
-String host_url = tableProperties.get(Constants.JDBC_URL);
+DatabaseType dbType = DatabaseType.valueOf(
+ tableProperties.get(JdbcStorageConfig.DATABASE_TYPE.getPropertyName()));
+String host_url = DatabaseType.METASTORE == dbType ?
+ "internal:metastore://" : tableProperties.get(Constants.JDBC_URL);
Review Comment:
All the storage handlers will have prefix in the Storage URL including jdbc
storage handler. Can we make this hardcoded uri also to be in the same format?
Essentially you would need to add "JDBC://" before the string
"internal:metastore:/" (please take out extra "/" after "metastore:/")?
Issue Time Tracking
---
Worklog Id: (was: 796063)
Time Spent: 20m (was: 10m)
> StorageHandler authorization doesn't work with "METASTORE" db type
> --
>
> Key: HIVE-26433
> URL: https://issues.apache.org/jira/browse/HIVE-26433
> Project: Hive
> Issue Type: Task
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> StorageHandler Authorization doesn't work with the "METASTORE" db type:
> {code}
> CREATE EXTERNAL TABLE IF NOT EXISTS `SYS`.`HIVE_LOCKS` (
> `HL_LOCK_EXT_ID` bigint,
> `HL_LOCK_INT_ID` bigint,
> `HL_TXNID` bigint,
> `HL_DB` string,
> `HL_TABLE` string,
> `HL_PARTITION` string,
> `HL_LOCK_STATE` string,
> `HL_LOCK_TYPE` string,
> `HL_LAST_HEARTBEAT` bigint,
> `HL_ACQUIRED_AT` bigint,
> `HL_USER` string,
> `HL_HOST` string,
> `HL_HEARTBEAT_COUNT` int,
> `HL_AGENT_INFO` string,
> `HL_BLOCKEDBY_EXT_ID` bigint,
> `HL_BLOCKEDBY_INT_ID` bigint
> )
> STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
> "hive.sql.database.type" = "METASTORE",
> "hive.sql.query" =
> "SELECT
> \"HL_LOCK_EXT_ID\",
> \"HL_LOCK_INT_ID\",
> \"HL_TXNID\",
> \"HL_DB\",
> \"HL_TABLE\",
> \"HL_PARTITION\",
> \"HL_LOCK_STATE\",
> \"HL_LOCK_TYPE\",
> \"HL_LAST_HEARTBEAT\",
> \"HL_ACQUIRED_AT\",
> \"HL_USER\",
> \"HL_HOST\",
> \"HL_HEARTBEAT_COUNT\",
> \"HL_AGENT_INFO\",
> \"HL_BLOCKEDBY_EXT_ID\",
> \"HL_BLOCKEDBY_INT_ID\"
> FROM \"HIVE_LOCKS\""
> );
> {code}
> Exception thrown:
> {code}
> Error: Error while compiling statement: FAILED: HiveAccessControlException
> Permission denied: user [hive] does not have [RWSTORAGE] privilege on
> [null/null] (state=42000,code=4)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HIVE-26426) Avoid StringIndexOutOfBoundsException in canCBOHandleAst() method.
[
https://issues.apache.org/jira/browse/HIVE-26426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stamatis Zampetakis resolved HIVE-26426.
Fix Version/s: 4.0.0-alpha-2
Resolution: Fixed
Fixed in
https://github.com/apache/hive/commit/186fb0d85af63b61bc10ba5372e35895754b1a6a.
Thanks for the PR [~achennagiri]!
> Avoid StringIndexOutOfBoundsException in canCBOHandleAst() method.
> --
>
> Key: HIVE-26426
> URL: https://issues.apache.org/jira/browse/HIVE-26426
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Abhay
>Assignee: Abhay
>Priority: Minor
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-2
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> The call to canHandleQbForCbo() can result in an
> StringIndexOutOfBoundsException. The assumption in the code is that the msg
> can only be null and we handle that but the msg can also be an empty string
> if the *verbose* is set to false. This can happen if INFO Logging is not
> enabled. We need to handle that case.
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java#L913]
> Here is the stack trace for reference:
> {noformat}
> FAILED: StringIndexOutOfBoundsException String index out of range: -2
> 15:10:24.192 [HiveServer2-Background-Pool: Thread-305] ERROR
> org.apache.hadoop.hive.ql.Driver - FAILED: StringIndexOutOfBoundsException
> String index out of range: -2
> java.lang.StringIndexOutOfBoundsException: String index out of range: -2
> at java.lang.String.substring(String.java:1967)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.canCBOHandleAst(CalcitePlanner.java:996)
>
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:572)
>
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:13063)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:472)
>
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:314)
> at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:223)
> at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:105)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:201)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:650)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:596)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:590)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:127)
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:206)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:336)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:358)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750){noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26426) Avoid StringIndexOutOfBoundsException in canCBOHandleAst() method.
[
https://issues.apache.org/jira/browse/HIVE-26426?focusedWorklogId=796062&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-796062
]
ASF GitHub Bot logged work on HIVE-26426:
-
Author: ASF GitHub Bot
Created on: 28/Jul/22 12:56
Start Date: 28/Jul/22 12:56
Worklog Time Spent: 10m
Work Description: zabetak closed pull request #3474: HIVE-26426: Avoid
StringIndexOutOfBoundsException in canCBOHandleAst() method
URL: https://github.com/apache/hive/pull/3474
Issue Time Tracking
---
Worklog Id: (was: 796062)
Time Spent: 1h 40m (was: 1.5h)
> Avoid StringIndexOutOfBoundsException in canCBOHandleAst() method.
> --
>
> Key: HIVE-26426
> URL: https://issues.apache.org/jira/browse/HIVE-26426
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Abhay
>Assignee: Abhay
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> The call to canHandleQbForCbo() can result in an
> StringIndexOutOfBoundsException. The assumption in the code is that the msg
> can only be null and we handle that but the msg can also be an empty string
> if the *verbose* is set to false. This can happen if INFO Logging is not
> enabled. We need to handle that case.
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java#L913]
> Here is the stack trace for reference:
> {noformat}
> FAILED: StringIndexOutOfBoundsException String index out of range: -2
> 15:10:24.192 [HiveServer2-Background-Pool: Thread-305] ERROR
> org.apache.hadoop.hive.ql.Driver - FAILED: StringIndexOutOfBoundsException
> String index out of range: -2
> java.lang.StringIndexOutOfBoundsException: String index out of range: -2
> at java.lang.String.substring(String.java:1967)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.canCBOHandleAst(CalcitePlanner.java:996)
>
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:572)
>
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:13063)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:472)
>
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:314)
> at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:223)
> at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:105)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:201)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:650)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:596)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:590)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:127)
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:206)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:336)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:358)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750){noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26426) Avoid StringIndexOutOfBoundsException in canCBOHandleAst() method.
[
https://issues.apache.org/jira/browse/HIVE-26426?focusedWorklogId=796060&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-796060
]
ASF GitHub Bot logged work on HIVE-26426:
-
Author: ASF GitHub Bot
Created on: 28/Jul/22 12:45
Start Date: 28/Jul/22 12:45
Worklog Time Spent: 10m
Work Description: zabetak commented on code in PR #3474:
URL: https://github.com/apache/hive/pull/3474#discussion_r932172826
##
ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java:
##
@@ -947,7 +947,7 @@ Pair canCBOHandleAst(ASTNode ast, QB qb,
PreCboCtx cboCtx) {
// Now check QB in more detail. canHandleQbForCbo returns null if query can
// be handled.
msg = CalcitePlanner.canHandleQbForCbo(queryProperties, conf, true,
needToLogMessage);
-if (msg == null) {
+if (msg == null || msg.isEmpty()) {
return Pair.of(true, msg);
}
msg = msg.substring(0, msg.length() - 2);
Review Comment:
The current changes in the PR solve the IOBE, tests are green, and there is
no change in behavior so I plan to merge this now.
The rest of the changes proposed by @jfsii are an improvement worth having
and are inline with what I briefly mentioned previously:
> allowing `CalcitePlanner.canHandleQbForCbo(queryProperties, conf, true,
needToLogMessage)` to return an empty `String` is not good
I would like to get these additional changes merged as well so let's log a
new JIRA and I will review ASAP. If nobody takes it in the next few days I can
also work on it.
Issue Time Tracking
---
Worklog Id: (was: 796060)
Time Spent: 1.5h (was: 1h 20m)
> Avoid StringIndexOutOfBoundsException in canCBOHandleAst() method.
> --
>
> Key: HIVE-26426
> URL: https://issues.apache.org/jira/browse/HIVE-26426
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Abhay
>Assignee: Abhay
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> The call to canHandleQbForCbo() can result in an
> StringIndexOutOfBoundsException. The assumption in the code is that the msg
> can only be null and we handle that but the msg can also be an empty string
> if the *verbose* is set to false. This can happen if INFO Logging is not
> enabled. We need to handle that case.
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java#L913]
> Here is the stack trace for reference:
> {noformat}
> FAILED: StringIndexOutOfBoundsException String index out of range: -2
> 15:10:24.192 [HiveServer2-Background-Pool: Thread-305] ERROR
> org.apache.hadoop.hive.ql.Driver - FAILED: StringIndexOutOfBoundsException
> String index out of range: -2
> java.lang.StringIndexOutOfBoundsException: String index out of range: -2
> at java.lang.String.substring(String.java:1967)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.canCBOHandleAst(CalcitePlanner.java:996)
>
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:572)
>
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:13063)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:472)
>
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:314)
> at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:223)
> at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:105)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:201)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:650)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:596)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:590)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:127)
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:206)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:336)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:358)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Thr
[jira] [Commented] (HIVE-21650) QOutProcessor should provide configurable partial masks for qtests
[
https://issues.apache.org/jira/browse/HIVE-21650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17572407#comment-17572407
]
Ayush Saxena commented on HIVE-21650:
-
The test introduced seems to causing some issues, Recursive listing post
hadoop-3.3 doesn't seems to be guaranteed to be sorted for all FileSystems. The
config introduced doesn't seems to be used anywhere as well.
>From Zoltan:- I think {{qt:replace}} could do the same..
Planning to revert this by EOW, Doesn't seem to be a functional bit of code
either. If required we can rework it here.
> QOutProcessor should provide configurable partial masks for qtests
> --
>
> Key: HIVE-21650
> URL: https://issues.apache.org/jira/browse/HIVE-21650
> Project: Hive
> Issue Type: Improvement
> Components: Test, Testing Infrastructure
>Reporter: Aditya Shah
>Assignee: Aditya Shah
>Priority: Major
> Fix For: 4.0.0, 4.0.0-alpha-1
>
> Attachments: HIVE-21650-1.patch, HIVE-21650.1.patch, HIVE-21650.patch
>
>
> QOutProcessor would mask a whole bunch of outputs in q.out files if it sees
> any of the target mask patterns. This restricts us from testing a whole bunch
> of tests like for example testing directories being formed for an acid table.
> Thus, internal configurations where we can provide additional partial masks
> for us to cover such similar case would help us make our tests better.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-24933) Replication fails for transactional tables having same name as dropped non-transactional table
[
https://issues.apache.org/jira/browse/HIVE-24933?focusedWorklogId=796037&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-796037
]
ASF GitHub Bot logged work on HIVE-24933:
-
Author: ASF GitHub Bot
Created on: 28/Jul/22 10:55
Start Date: 28/Jul/22 10:55
Worklog Time Spent: 10m
Work Description: jhungund commented on PR #3435:
URL: https://github.com/apache/hive/pull/3435#issuecomment-1197987231
Hi Francis,
Thank you for taking a look at this change.
Below are my replies to your comments (starting with **[JH]**) below
> Can you add an analysis? Like the reason for the bug and how you are
fixing it?
>
**[JH]**I have added a detailed comment about the root cause analysis.
Please take a look.
> From the changes, I can infer that the access is "deferred", moved from
occurring semantic analysis "load table" time, to later when the task actually
runs and does the copy.
**[JH]** Yes, this is right.
>
> But what necessitates needing this? Did some state change between the two
points?
**[JH]** Please check the detailed comment added earlier.
>
> Is this the significant change?
>
> > table = ImportSemanticAnalyzer.tableIfExists(tblDesc, hive);
> > if (table == null) {
> > table = ImportSemanticAnalyzer.createNewTableMetadataObject(tblDesc,
true);
> > }
>
**[JH]** Yes, the access and the subsequent check:
if (AcidUtils.isTransactionalTable(table)) {
returns different results at task creation time and task execution time.
> So at deferred time, the copy/move/etc is only done if the table exists?
>
> Just wondering if you can pass runnable/callable/function from
ImportSemanticAnaylzer, so to keep all that current logic in the same place and
not moved to another file. Then the DeferredWorkHelperImpl just needs to
execute it, and logic can be removed from there.
**[JH]** The current logic of moving code in deferred implementation
simplifies the callers which only need to set the deferred implementation
class. This keeps the logic at the same place as the execution. Hence, it looks
OK to me.
Do let me know, if it looks ok. If not, I can try to change it as per your
comment.
Thanks,
Janardhan
Issue Time Tracking
---
Worklog Id: (was: 796037)
Time Spent: 2h 20m (was: 2h 10m)
> Replication fails for transactional tables having same name as dropped
> non-transactional table
> --
>
> Key: HIVE-24933
> URL: https://issues.apache.org/jira/browse/HIVE-24933
> Project: Hive
> Issue Type: Bug
>Reporter: Pratyush Madhukar
>Assignee: Pratyush Madhukar
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-26433) StorageHandler authorization doesn't work with "METASTORE" db type
[
https://issues.apache.org/jira/browse/HIVE-26433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko reassigned HIVE-26433:
-
Assignee: Denys Kuzmenko
> StorageHandler authorization doesn't work with "METASTORE" db type
> --
>
> Key: HIVE-26433
> URL: https://issues.apache.org/jira/browse/HIVE-26433
> Project: Hive
> Issue Type: Task
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> StorageHandler Authorization doesn't work with the "METASTORE" db type:
> {code}
> CREATE EXTERNAL TABLE IF NOT EXISTS `SYS`.`HIVE_LOCKS` (
> `HL_LOCK_EXT_ID` bigint,
> `HL_LOCK_INT_ID` bigint,
> `HL_TXNID` bigint,
> `HL_DB` string,
> `HL_TABLE` string,
> `HL_PARTITION` string,
> `HL_LOCK_STATE` string,
> `HL_LOCK_TYPE` string,
> `HL_LAST_HEARTBEAT` bigint,
> `HL_ACQUIRED_AT` bigint,
> `HL_USER` string,
> `HL_HOST` string,
> `HL_HEARTBEAT_COUNT` int,
> `HL_AGENT_INFO` string,
> `HL_BLOCKEDBY_EXT_ID` bigint,
> `HL_BLOCKEDBY_INT_ID` bigint
> )
> STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
> "hive.sql.database.type" = "METASTORE",
> "hive.sql.query" =
> "SELECT
> \"HL_LOCK_EXT_ID\",
> \"HL_LOCK_INT_ID\",
> \"HL_TXNID\",
> \"HL_DB\",
> \"HL_TABLE\",
> \"HL_PARTITION\",
> \"HL_LOCK_STATE\",
> \"HL_LOCK_TYPE\",
> \"HL_LAST_HEARTBEAT\",
> \"HL_ACQUIRED_AT\",
> \"HL_USER\",
> \"HL_HOST\",
> \"HL_HEARTBEAT_COUNT\",
> \"HL_AGENT_INFO\",
> \"HL_BLOCKEDBY_EXT_ID\",
> \"HL_BLOCKEDBY_INT_ID\"
> FROM \"HIVE_LOCKS\""
> );
> {code}
> Exception thrown:
> {code}
> Error: Error while compiling statement: FAILED: HiveAccessControlException
> Permission denied: user [hive] does not have [RWSTORAGE] privilege on
> [null/null] (state=42000,code=4)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-26433) StorageHandler authorization doesn't work with "METASTORE" db type
[
https://issues.apache.org/jira/browse/HIVE-26433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-26433:
--
Description:
StorageHandler Authorization doesn't work with the "METASTORE" db type:
{code}
CREATE EXTERNAL TABLE IF NOT EXISTS `SYS`.`HIVE_LOCKS` (
`HL_LOCK_EXT_ID` bigint,
`HL_LOCK_INT_ID` bigint,
`HL_TXNID` bigint,
`HL_DB` string,
`HL_TABLE` string,
`HL_PARTITION` string,
`HL_LOCK_STATE` string,
`HL_LOCK_TYPE` string,
`HL_LAST_HEARTBEAT` bigint,
`HL_ACQUIRED_AT` bigint,
`HL_USER` string,
`HL_HOST` string,
`HL_HEARTBEAT_COUNT` int,
`HL_AGENT_INFO` string,
`HL_BLOCKEDBY_EXT_ID` bigint,
`HL_BLOCKEDBY_INT_ID` bigint
)
STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
TBLPROPERTIES (
"hive.sql.database.type" = "METASTORE",
"hive.sql.query" =
"SELECT
\"HL_LOCK_EXT_ID\",
\"HL_LOCK_INT_ID\",
\"HL_TXNID\",
\"HL_DB\",
\"HL_TABLE\",
\"HL_PARTITION\",
\"HL_LOCK_STATE\",
\"HL_LOCK_TYPE\",
\"HL_LAST_HEARTBEAT\",
\"HL_ACQUIRED_AT\",
\"HL_USER\",
\"HL_HOST\",
\"HL_HEARTBEAT_COUNT\",
\"HL_AGENT_INFO\",
\"HL_BLOCKEDBY_EXT_ID\",
\"HL_BLOCKEDBY_INT_ID\"
FROM \"HIVE_LOCKS\""
);
{code}
Exception thrown:
{code}
Error: Error while compiling statement: FAILED: HiveAccessControlException
Permission denied: user [hive] does not have [RWSTORAGE] privilege on
[null/null] (state=42000,code=4)
{code}
was:
StorageHandler Authorization doesn't with "METASTORE" db type:
{code}
CREATE EXTERNAL TABLE IF NOT EXISTS `SYS`.`HIVE_LOCKS` (
`HL_LOCK_EXT_ID` bigint,
`HL_LOCK_INT_ID` bigint,
`HL_TXNID` bigint,
`HL_DB` string,
`HL_TABLE` string,
`HL_PARTITION` string,
`HL_LOCK_STATE` string,
`HL_LOCK_TYPE` string,
`HL_LAST_HEARTBEAT` bigint,
`HL_ACQUIRED_AT` bigint,
`HL_USER` string,
`HL_HOST` string,
`HL_HEARTBEAT_COUNT` int,
`HL_AGENT_INFO` string,
`HL_BLOCKEDBY_EXT_ID` bigint,
`HL_BLOCKEDBY_INT_ID` bigint
)
STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
TBLPROPERTIES (
"hive.sql.database.type" = "METASTORE",
"hive.sql.query" =
"SELECT
\"HL_LOCK_EXT_ID\",
\"HL_LOCK_INT_ID\",
\"HL_TXNID\",
\"HL_DB\",
\"HL_TABLE\",
\"HL_PARTITION\",
\"HL_LOCK_STATE\",
\"HL_LOCK_TYPE\",
\"HL_LAST_HEARTBEAT\",
\"HL_ACQUIRED_AT\",
\"HL_USER\",
\"HL_HOST\",
\"HL_HEARTBEAT_COUNT\",
\"HL_AGENT_INFO\",
\"HL_BLOCKEDBY_EXT_ID\",
\"HL_BLOCKEDBY_INT_ID\"
FROM \"HIVE_LOCKS\""
);
{code}
Exception thrown:
{code}
Error: Error while compiling statement: FAILED: HiveAccessControlException
Permission denied: user [hive] does not have [RWSTORAGE] privilege on
[null/null] (state=42000,code=4)
{code}
> StorageHandler authorization doesn't work with "METASTORE" db type
> --
>
> Key: HIVE-26433
> URL: https://issues.apache.org/jira/browse/HIVE-26433
> Project: Hive
> Issue Type: Task
>Reporter: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> StorageHandler Authorization doesn't work with the "METASTORE" db type:
> {code}
> CREATE EXTERNAL TABLE IF NOT EXISTS `SYS`.`HIVE_LOCKS` (
> `HL_LOCK_EXT_ID` bigint,
> `HL_LOCK_INT_ID` bigint,
> `HL_TXNID` bigint,
> `HL_DB` string,
> `HL_TABLE` string,
> `HL_PARTITION` string,
> `HL_LOCK_STATE` string,
> `HL_LOCK_TYPE` string,
> `HL_LAST_HEARTBEAT` bigint,
> `HL_ACQUIRED_AT` bigint,
> `HL_USER` string,
> `HL_HOST` string,
> `HL_HEARTBEAT_COUNT` int,
> `HL_AGENT_INFO` string,
> `HL_BLOCKEDBY_EXT_ID` bigint,
> `HL_BLOCKEDBY_INT_ID` bigint
> )
> STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
> "hive.sql.database.type" = "METASTORE",
> "hive.sql.query" =
> "SELECT
> \"HL_LOCK_EXT_ID\",
> \"HL_LOCK_INT_ID\",
> \"HL_TXNID\",
> \"HL_DB\",
> \"HL_TABLE\",
> \"HL_PARTITION\",
> \"HL_LOCK_STATE\",
> \"HL_LOCK_TYPE\",
> \"HL_LAST_HEARTBEAT\",
> \"HL_ACQUIRED_AT\",
> \"HL_USER\",
> \"HL_HOST\",
> \"HL_HEARTBEAT_COUNT\",
> \"HL_AGENT_INFO\",
> \"HL_BLOCKEDBY_EXT_ID\",
> \"HL_BLOCKEDBY_INT_ID\"
> FROM \"HIVE_LOCKS\""
> );
> {code}
> Exception thrown:
> {code}
> Error: Error while compiling statement: FAILED: HiveAccessControlException
> Permission denied: user [hive] does not have [RWSTORAGE] privilege on
> [null/null] (state=42000,code=4)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work started] (HIVE-26433) StorageHandler authorization doesn't work with "METASTORE" db type
[
https://issues.apache.org/jira/browse/HIVE-26433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-26433 started by Denys Kuzmenko.
-
> StorageHandler authorization doesn't work with "METASTORE" db type
> --
>
> Key: HIVE-26433
> URL: https://issues.apache.org/jira/browse/HIVE-26433
> Project: Hive
> Issue Type: Task
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> StorageHandler Authorization doesn't work with the "METASTORE" db type:
> {code}
> CREATE EXTERNAL TABLE IF NOT EXISTS `SYS`.`HIVE_LOCKS` (
> `HL_LOCK_EXT_ID` bigint,
> `HL_LOCK_INT_ID` bigint,
> `HL_TXNID` bigint,
> `HL_DB` string,
> `HL_TABLE` string,
> `HL_PARTITION` string,
> `HL_LOCK_STATE` string,
> `HL_LOCK_TYPE` string,
> `HL_LAST_HEARTBEAT` bigint,
> `HL_ACQUIRED_AT` bigint,
> `HL_USER` string,
> `HL_HOST` string,
> `HL_HEARTBEAT_COUNT` int,
> `HL_AGENT_INFO` string,
> `HL_BLOCKEDBY_EXT_ID` bigint,
> `HL_BLOCKEDBY_INT_ID` bigint
> )
> STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
> "hive.sql.database.type" = "METASTORE",
> "hive.sql.query" =
> "SELECT
> \"HL_LOCK_EXT_ID\",
> \"HL_LOCK_INT_ID\",
> \"HL_TXNID\",
> \"HL_DB\",
> \"HL_TABLE\",
> \"HL_PARTITION\",
> \"HL_LOCK_STATE\",
> \"HL_LOCK_TYPE\",
> \"HL_LAST_HEARTBEAT\",
> \"HL_ACQUIRED_AT\",
> \"HL_USER\",
> \"HL_HOST\",
> \"HL_HEARTBEAT_COUNT\",
> \"HL_AGENT_INFO\",
> \"HL_BLOCKEDBY_EXT_ID\",
> \"HL_BLOCKEDBY_INT_ID\"
> FROM \"HIVE_LOCKS\""
> );
> {code}
> Exception thrown:
> {code}
> Error: Error while compiling statement: FAILED: HiveAccessControlException
> Permission denied: user [hive] does not have [RWSTORAGE] privilege on
> [null/null] (state=42000,code=4)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-26433) StorageHandler authorization doesn't work with "METASTORE" db type
[
https://issues.apache.org/jira/browse/HIVE-26433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-26433:
--
Summary: StorageHandler authorization doesn't work with "METASTORE" db type
(was: HivePrivilegeObject's objectName is NULL when JdbcStorageHandler is used
with METASTORE type)
> StorageHandler authorization doesn't work with "METASTORE" db type
> --
>
> Key: HIVE-26433
> URL: https://issues.apache.org/jira/browse/HIVE-26433
> Project: Hive
> Issue Type: Task
>Reporter: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> StorageHandler Authorization doesn't with "METASTORE" db type:
> {code}
> CREATE EXTERNAL TABLE IF NOT EXISTS `SYS`.`HIVE_LOCKS` (
> `HL_LOCK_EXT_ID` bigint,
> `HL_LOCK_INT_ID` bigint,
> `HL_TXNID` bigint,
> `HL_DB` string,
> `HL_TABLE` string,
> `HL_PARTITION` string,
> `HL_LOCK_STATE` string,
> `HL_LOCK_TYPE` string,
> `HL_LAST_HEARTBEAT` bigint,
> `HL_ACQUIRED_AT` bigint,
> `HL_USER` string,
> `HL_HOST` string,
> `HL_HEARTBEAT_COUNT` int,
> `HL_AGENT_INFO` string,
> `HL_BLOCKEDBY_EXT_ID` bigint,
> `HL_BLOCKEDBY_INT_ID` bigint
> )
> STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
> "hive.sql.database.type" = "METASTORE",
> "hive.sql.query" =
> "SELECT
> \"HL_LOCK_EXT_ID\",
> \"HL_LOCK_INT_ID\",
> \"HL_TXNID\",
> \"HL_DB\",
> \"HL_TABLE\",
> \"HL_PARTITION\",
> \"HL_LOCK_STATE\",
> \"HL_LOCK_TYPE\",
> \"HL_LAST_HEARTBEAT\",
> \"HL_ACQUIRED_AT\",
> \"HL_USER\",
> \"HL_HOST\",
> \"HL_HEARTBEAT_COUNT\",
> \"HL_AGENT_INFO\",
> \"HL_BLOCKEDBY_EXT_ID\",
> \"HL_BLOCKEDBY_INT_ID\"
> FROM \"HIVE_LOCKS\""
> );
> {code}
> Exception thrown:
> {code}
> Error: Error while compiling statement: FAILED: HiveAccessControlException
> Permission denied: user [hive] does not have [RWSTORAGE] privilege on
> [null/null] (state=42000,code=4)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-26433) HivePrivilegeObject's objectName is NULL when JdbcStorageHandler is used with METASTORE type
[
https://issues.apache.org/jira/browse/HIVE-26433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-26433:
--
Description:
StorageHandler Authorization doesn't with "METASTORE" db type:
{code}
CREATE EXTERNAL TABLE IF NOT EXISTS `SYS`.`HIVE_LOCKS` (
`HL_LOCK_EXT_ID` bigint,
`HL_LOCK_INT_ID` bigint,
`HL_TXNID` bigint,
`HL_DB` string,
`HL_TABLE` string,
`HL_PARTITION` string,
`HL_LOCK_STATE` string,
`HL_LOCK_TYPE` string,
`HL_LAST_HEARTBEAT` bigint,
`HL_ACQUIRED_AT` bigint,
`HL_USER` string,
`HL_HOST` string,
`HL_HEARTBEAT_COUNT` int,
`HL_AGENT_INFO` string,
`HL_BLOCKEDBY_EXT_ID` bigint,
`HL_BLOCKEDBY_INT_ID` bigint
)
STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
TBLPROPERTIES (
"hive.sql.database.type" = "METASTORE",
"hive.sql.query" =
"SELECT
\"HL_LOCK_EXT_ID\",
\"HL_LOCK_INT_ID\",
\"HL_TXNID\",
\"HL_DB\",
\"HL_TABLE\",
\"HL_PARTITION\",
\"HL_LOCK_STATE\",
\"HL_LOCK_TYPE\",
\"HL_LAST_HEARTBEAT\",
\"HL_ACQUIRED_AT\",
\"HL_USER\",
\"HL_HOST\",
\"HL_HEARTBEAT_COUNT\",
\"HL_AGENT_INFO\",
\"HL_BLOCKEDBY_EXT_ID\",
\"HL_BLOCKEDBY_INT_ID\"
FROM \"HIVE_LOCKS\""
);
{code}
Exception thrown:
{code}
Error: Error while compiling statement: FAILED: HiveAccessControlException
Permission denied: user [hive] does not have [RWSTORAGE] privilege on
[null/null] (state=42000,code=4)
{code}
was:
Create table with JdbcStorageHandler and "METASTORE" db type fails during
authorization
{code}
CREATE EXTERNAL TABLE IF NOT EXISTS `SYS`.`HIVE_LOCKS` (
`HL_LOCK_EXT_ID` bigint,
`HL_LOCK_INT_ID` bigint,
`HL_TXNID` bigint,
`HL_DB` string,
`HL_TABLE` string,
`HL_PARTITION` string,
`HL_LOCK_STATE` string,
`HL_LOCK_TYPE` string,
`HL_LAST_HEARTBEAT` bigint,
`HL_ACQUIRED_AT` bigint,
`HL_USER` string,
`HL_HOST` string,
`HL_HEARTBEAT_COUNT` int,
`HL_AGENT_INFO` string,
`HL_BLOCKEDBY_EXT_ID` bigint,
`HL_BLOCKEDBY_INT_ID` bigint
)
STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
TBLPROPERTIES (
"hive.sql.database.type" = "METASTORE",
"hive.sql.query" =
"SELECT
\"HL_LOCK_EXT_ID\",
\"HL_LOCK_INT_ID\",
\"HL_TXNID\",
\"HL_DB\",
\"HL_TABLE\",
\"HL_PARTITION\",
\"HL_LOCK_STATE\",
\"HL_LOCK_TYPE\",
\"HL_LAST_HEARTBEAT\",
\"HL_ACQUIRED_AT\",
\"HL_USER\",
\"HL_HOST\",
\"HL_HEARTBEAT_COUNT\",
\"HL_AGENT_INFO\",
\"HL_BLOCKEDBY_EXT_ID\",
\"HL_BLOCKEDBY_INT_ID\"
FROM \"HIVE_LOCKS\""
);
{code}
Exception thrown:
{code}
Error: Error while compiling statement: FAILED: HiveAccessControlException
Permission denied: user [hive] does not have [RWSTORAGE] privilege on
[null/null] (state=42000,code=4)
{code}
> HivePrivilegeObject's objectName is NULL when JdbcStorageHandler is used with
> METASTORE type
>
>
> Key: HIVE-26433
> URL: https://issues.apache.org/jira/browse/HIVE-26433
> Project: Hive
> Issue Type: Task
>Reporter: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> StorageHandler Authorization doesn't with "METASTORE" db type:
> {code}
> CREATE EXTERNAL TABLE IF NOT EXISTS `SYS`.`HIVE_LOCKS` (
> `HL_LOCK_EXT_ID` bigint,
> `HL_LOCK_INT_ID` bigint,
> `HL_TXNID` bigint,
> `HL_DB` string,
> `HL_TABLE` string,
> `HL_PARTITION` string,
> `HL_LOCK_STATE` string,
> `HL_LOCK_TYPE` string,
> `HL_LAST_HEARTBEAT` bigint,
> `HL_ACQUIRED_AT` bigint,
> `HL_USER` string,
> `HL_HOST` string,
> `HL_HEARTBEAT_COUNT` int,
> `HL_AGENT_INFO` string,
> `HL_BLOCKEDBY_EXT_ID` bigint,
> `HL_BLOCKEDBY_INT_ID` bigint
> )
> STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
> "hive.sql.database.type" = "METASTORE",
> "hive.sql.query" =
> "SELECT
> \"HL_LOCK_EXT_ID\",
> \"HL_LOCK_INT_ID\",
> \"HL_TXNID\",
> \"HL_DB\",
> \"HL_TABLE\",
> \"HL_PARTITION\",
> \"HL_LOCK_STATE\",
> \"HL_LOCK_TYPE\",
> \"HL_LAST_HEARTBEAT\",
> \"HL_ACQUIRED_AT\",
> \"HL_USER\",
> \"HL_HOST\",
> \"HL_HEARTBEAT_COUNT\",
> \"HL_AGENT_INFO\",
> \"HL_BLOCKEDBY_EXT_ID\",
> \"HL_BLOCKEDBY_INT_ID\"
> FROM \"HIVE_LOCKS\""
> );
> {code}
> Exception thrown:
> {code}
> Error: Error while compiling statement: FAILED: HiveAccessControlException
> Permission denied: user [hive] does not have [RWSTORAGE] privilege on
> [null/null] (state=42000,code=4)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-26433) HivePrivilegeObject's objectName is NULL when JdbcStorageHandler is used with METASTORE type
[
https://issues.apache.org/jira/browse/HIVE-26433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-26433:
--
Description:
Create table with JdbcStorageHandler and "METASTORE" db type fails during
authorization
{code}
CREATE EXTERNAL TABLE IF NOT EXISTS `SYS`.`HIVE_LOCKS` (
`HL_LOCK_EXT_ID` bigint,
`HL_LOCK_INT_ID` bigint,
`HL_TXNID` bigint,
`HL_DB` string,
`HL_TABLE` string,
`HL_PARTITION` string,
`HL_LOCK_STATE` string,
`HL_LOCK_TYPE` string,
`HL_LAST_HEARTBEAT` bigint,
`HL_ACQUIRED_AT` bigint,
`HL_USER` string,
`HL_HOST` string,
`HL_HEARTBEAT_COUNT` int,
`HL_AGENT_INFO` string,
`HL_BLOCKEDBY_EXT_ID` bigint,
`HL_BLOCKEDBY_INT_ID` bigint
)
STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
TBLPROPERTIES (
"hive.sql.database.type" = "METASTORE",
"hive.sql.query" =
"SELECT
\"HL_LOCK_EXT_ID\",
\"HL_LOCK_INT_ID\",
\"HL_TXNID\",
\"HL_DB\",
\"HL_TABLE\",
\"HL_PARTITION\",
\"HL_LOCK_STATE\",
\"HL_LOCK_TYPE\",
\"HL_LAST_HEARTBEAT\",
\"HL_ACQUIRED_AT\",
\"HL_USER\",
\"HL_HOST\",
\"HL_HEARTBEAT_COUNT\",
\"HL_AGENT_INFO\",
\"HL_BLOCKEDBY_EXT_ID\",
\"HL_BLOCKEDBY_INT_ID\"
FROM \"HIVE_LOCKS\""
);
{code}
Exception thrown:
{code}
Error: Error while compiling statement: FAILED: HiveAccessControlException
Permission denied: user [hive] does not have [RWSTORAGE] privilege on
[null/null] (state=42000,code=4)
{code}
> HivePrivilegeObject's objectName is NULL when JdbcStorageHandler is used with
> METASTORE type
>
>
> Key: HIVE-26433
> URL: https://issues.apache.org/jira/browse/HIVE-26433
> Project: Hive
> Issue Type: Task
>Reporter: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Create table with JdbcStorageHandler and "METASTORE" db type fails during
> authorization
> {code}
> CREATE EXTERNAL TABLE IF NOT EXISTS `SYS`.`HIVE_LOCKS` (
> `HL_LOCK_EXT_ID` bigint,
> `HL_LOCK_INT_ID` bigint,
> `HL_TXNID` bigint,
> `HL_DB` string,
> `HL_TABLE` string,
> `HL_PARTITION` string,
> `HL_LOCK_STATE` string,
> `HL_LOCK_TYPE` string,
> `HL_LAST_HEARTBEAT` bigint,
> `HL_ACQUIRED_AT` bigint,
> `HL_USER` string,
> `HL_HOST` string,
> `HL_HEARTBEAT_COUNT` int,
> `HL_AGENT_INFO` string,
> `HL_BLOCKEDBY_EXT_ID` bigint,
> `HL_BLOCKEDBY_INT_ID` bigint
> )
> STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
> "hive.sql.database.type" = "METASTORE",
> "hive.sql.query" =
> "SELECT
> \"HL_LOCK_EXT_ID\",
> \"HL_LOCK_INT_ID\",
> \"HL_TXNID\",
> \"HL_DB\",
> \"HL_TABLE\",
> \"HL_PARTITION\",
> \"HL_LOCK_STATE\",
> \"HL_LOCK_TYPE\",
> \"HL_LAST_HEARTBEAT\",
> \"HL_ACQUIRED_AT\",
> \"HL_USER\",
> \"HL_HOST\",
> \"HL_HEARTBEAT_COUNT\",
> \"HL_AGENT_INFO\",
> \"HL_BLOCKEDBY_EXT_ID\",
> \"HL_BLOCKEDBY_INT_ID\"
> FROM \"HIVE_LOCKS\""
> );
> {code}
> Exception thrown:
> {code}
> Error: Error while compiling statement: FAILED: HiveAccessControlException
> Permission denied: user [hive] does not have [RWSTORAGE] privilege on
> [null/null] (state=42000,code=4)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HIVE-26394) Query based compaction fails for table with more than 6 columns
[
https://issues.apache.org/jira/browse/HIVE-26394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mahesh kumar behera resolved HIVE-26394.
Resolution: Fixed
> Query based compaction fails for table with more than 6 columns
> ---
>
> Key: HIVE-26394
> URL: https://issues.apache.org/jira/browse/HIVE-26394
> Project: Hive
> Issue Type: Bug
> Components: Hive, HiveServer2
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Query based compaction creates a temp external table with location pointing
> to the location of the table being compacted. So this external table has file
> of ACID type. When query is done on this table, the table type is decided by
> reading the files present at the table location. As the table location has
> files compatible to ACID format, it is assuming it to be ACID table. This is
> causing issue while generating the SARG columns as the column number does not
> match with the schema.
>
> {code:java}
> Error doing query based minor compaction
> org.apache.hadoop.hive.ql.metadata.HiveException: Failed to run INSERT into
> table delta_cara_pn_tmp_compactor_clean_1656061070392_result select
> `operation`, `originalTransaction`, `bucket`, `rowId`, `currentTransaction`,
> `row` from delta_clean_1656061070392 where `originalTransaction` not in
> (749,750,766,768,779,783,796,799,818,1145,1149,1150,1158,1159,1160,1165,1166,1169,1173,1175,1176,1871,9631)
> at
> org.apache.hadoop.hive.ql.DriverUtils.runOnDriver(DriverUtils.java:73)
> at
> org.apache.hadoop.hive.ql.txn.compactor.QueryCompactor.runCompactionQueries(QueryCompactor.java:138)
> at
> org.apache.hadoop.hive.ql.txn.compactor.MinorQueryCompactor.runCompaction(MinorQueryCompactor.java:70)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Worker.findNextCompactionAndExecute(Worker.java:498)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Worker.lambda$run$0(Worker.java:120)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750)
> Caused by: (responseCode = 2, errorMessage = FAILED: Execution Error, return
> code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed,
> vertexName=Map 1, vertexId=vertex_1656061159324__1_00, diagnostics=[Task
> failed, taskId=task_1656061159324__1_00_00, diagnostics=[TaskAttempt
> 0 failed, info=[Error: Error while running task ( failure ) :
> attempt_1656061159324__1_00_00_0:java.lang.RuntimeException:
> java.lang.RuntimeException: java.io.IOException:
> java.lang.ArrayIndexOutOfBoundsException: 6
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:348)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:277)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:75)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:62)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:62)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:38)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:118)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750)
> Caused by: java.lang.RuntimeException: java.io.IOException:
> java.lang.ArrayIndexOutOfBoundsException: 6
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:206)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.(TezGroupedSplitsInputFormat.
[jira] [Work logged] (HIVE-24933) Replication fails for transactional tables having same name as dropped non-transactional table
[ https://issues.apache.org/jira/browse/HIVE-24933?focusedWorklogId=796014&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-796014 ] ASF GitHub Bot logged work on HIVE-24933: - Author: ASF GitHub Bot Created on: 28/Jul/22 10:07 Start Date: 28/Jul/22 10:07 Worklog Time Spent: 10m Work Description: jhungund commented on PR #3435: URL: https://github.com/apache/hive/pull/3435#issuecomment-1197938266 Summary of the change: While setting up the tasks during the repl-load phase of the replication, the delay the access to the metadata until the task execution. This will avoid inconsistent metadata state by the tasks. **Root Cause Analysis** Background: During the incremental load phase of replication, all event logs are processed sequentially. One or more tasks are additionally spawned/created during the processing of each event. All the spawned tasks are also, subsequently, executed sequentially. **Scenario of the issue:** The issue is seen in the following scenario: 1. An external table(Eg. T1) is already replication to target cluster from source cluster during earlier replication cycles. 2. This external table is dropped. 3. A new managed table with the same name (T1) is recreated. **Root cause:** 1. The above mentioned operations (table drop and recreation) are propagated to the target cluster vis event logs during the subsequent incremental phase of replication. 2. We create tasks to drop the old external tables for drop table event. 3. We also create new tasks to create and load the table for the new table. 4. Additionally, some additional events are logged which create tasks to load the table. 5. During the creation of these load-table tasks, we try to access the metadata corresponding to the new table from the metadata store. In normal scenario of a fresh table creation, the metadata store will not have data corresponding to the new table (yet to be created). However, in this scenario, the old table still exists and hence, we end up using the metadata corresponding to old table. We try to use this metadata to create the load tasks for the new table. During the execution of these load tasks, which subsequently execute after the drop and recreate tasks, we find that the metadata set in the task context is stale and is inconsistent with the newly created table. Hence, the error. **Fix:** Do not access the metadata during the task creation for load table. Instead, access the metadata during the task execution. By that time, the metadata is updated to the latest state with the previously executed tasks. Issue Time Tracking --- Worklog Id: (was: 796014) Time Spent: 2h 10m (was: 2h) > Replication fails for transactional tables having same name as dropped > non-transactional table > -- > > Key: HIVE-24933 > URL: https://issues.apache.org/jira/browse/HIVE-24933 > Project: Hive > Issue Type: Bug >Reporter: Pratyush Madhukar >Assignee: Pratyush Madhukar >Priority: Major > Labels: pull-request-available > Time Spent: 2h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26431) Use correct schema for iceberg time travel queries
[
https://issues.apache.org/jira/browse/HIVE-26431?focusedWorklogId=795990&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-795990
]
ASF GitHub Bot logged work on HIVE-26431:
-
Author: ASF GitHub Bot
Created on: 28/Jul/22 08:37
Start Date: 28/Jul/22 08:37
Worklog Time Spent: 10m
Work Description: szlta commented on code in PR #3479:
URL: https://github.com/apache/hive/pull/3479#discussion_r931934208
##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergSerDe.java:
##
@@ -105,8 +106,18 @@ public void initialize(@Nullable Configuration
configuration, Properties serDePr
} else {
try {
Table table = IcebergTableUtil.getTable(configuration,
serDeProperties);
-// always prefer the original table schema if there is one
-this.tableSchema = table.schema();
+if (serDeProperties.containsKey(serdeConstants.AS_OF_VERSION)) {
+ this.tableSchema = SnapshotUtil.schemaFor(table,
+
Long.valueOf(serDeProperties.getProperty(serdeConstants.AS_OF_VERSION)));
+ serDeProperties.remove(serdeConstants.AS_OF_VERSION);
Review Comment:
Let's explain in a comment why we need to remove this from serdeProperties,
and perhaps also why we need serdeProperties to carry this information nstead
of job configuration in the first place.
##
ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java:
##
@@ -347,7 +351,15 @@ final public Deserializer getDeserializer(boolean
skipConfError) {
final public Deserializer getDeserializerFromMetaStore(boolean
skipConfError) {
try {
- return HiveMetaStoreUtils.getDeserializer(SessionState.getSessionConf(),
tTable, metaTable, skipConfError);
+ String asOfTimestamp = null;
+ if (getAsOfTimestamp() != null) {
+ZoneId zoneId = SessionState.get() == null ? new
HiveConf().getLocalTimeZone() :
+SessionState.get().getConf().getLocalTimeZone();
+TimestampTZ time =
TimestampTZUtil.parse(PlanUtils.stripQuotes(getAsOfTimestamp()), zoneId);
+asOfTimestamp = Long.toString(time.toEpochMilli());
Review Comment:
This calculation of Iceberg As Of information from string to long is found
in multiple places, I think we should refactor this logic into a utility class
of ql perhaps.
Issue Time Tracking
---
Worklog Id: (was: 795990)
Time Spent: 20m (was: 10m)
> Use correct schema for iceberg time travel queries
> --
>
> Key: HIVE-26431
> URL: https://issues.apache.org/jira/browse/HIVE-26431
> Project: Hive
> Issue Type: Improvement
>Reporter: László Pintér
>Assignee: László Pintér
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> For time travel queries we currently always use the latest schema in
> IcebergSerDe, however since schemas are now versioned, we could use the
> schema which was active at the time in the past.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26421) HmsClient atler_table_req() is called twice when running an alter operation on iceberg table
[
https://issues.apache.org/jira/browse/HIVE-26421?focusedWorklogId=795982&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-795982
]
ASF GitHub Bot logged work on HIVE-26421:
-
Author: ASF GitHub Bot
Created on: 28/Jul/22 08:18
Start Date: 28/Jul/22 08:18
Worklog Time Spent: 10m
Work Description: szlta commented on code in PR #3469:
URL: https://github.com/apache/hive/pull/3469#discussion_r931919330
##
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaHook.java:
##
@@ -52,6 +52,7 @@ public interface HiveMetaHook {
String PROPERTIES_SEPARATOR = "'";
String MIGRATE_HIVE_TO_ICEBERG = "migrate_hive_to_iceberg";
String INITIALIZE_ROLLBACK_MIGRATION = "initialize_rollback_migration";
+ String SKIP_METASTORE_ALTER = "skip_metastore_alter";
Review Comment:
Can we have short description for what would happen if this property is
placed into the environment context of HMS calls?
Issue Time Tracking
---
Worklog Id: (was: 795982)
Remaining Estimate: 0h
Time Spent: 10m
> HmsClient atler_table_req() is called twice when running an alter operation
> on iceberg table
> -
>
> Key: HIVE-26421
> URL: https://issues.apache.org/jira/browse/HIVE-26421
> Project: Hive
> Issue Type: Bug
>Reporter: László Pintér
>Assignee: László Pintér
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> HMSClient alter_table_req() is triggered twice when altering an iceberg
> table. The first occurrence is right after returning from the
> HiveIcebergMetaHook#preAlterTable() and the second one is from
> HiveTableOperations#doCommit(). We should find a way to skip the first call.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-26421) HmsClient atler_table_req() is called twice when running an alter operation on iceberg table
[ https://issues.apache.org/jira/browse/HIVE-26421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-26421: -- Labels: pull-request-available (was: ) > HmsClient atler_table_req() is called twice when running an alter operation > on iceberg table > - > > Key: HIVE-26421 > URL: https://issues.apache.org/jira/browse/HIVE-26421 > Project: Hive > Issue Type: Bug >Reporter: László Pintér >Assignee: László Pintér >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > HMSClient alter_table_req() is triggered twice when altering an iceberg > table. The first occurrence is right after returning from the > HiveIcebergMetaHook#preAlterTable() and the second one is from > HiveTableOperations#doCommit(). We should find a way to skip the first call. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26012) HMS APIs to be enhanced for metadata replication
[ https://issues.apache.org/jira/browse/HIVE-26012?focusedWorklogId=795974&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-795974 ] ASF GitHub Bot logged work on HIVE-26012: - Author: ASF GitHub Bot Created on: 28/Jul/22 07:54 Start Date: 28/Jul/22 07:54 Worklog Time Spent: 10m Work Description: saihemanth-cloudera commented on code in PR #3477: URL: https://github.com/apache/hive/pull/3477#discussion_r931896175 ## standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java: ## @@ -1399,6 +1403,42 @@ public void create_database(final Database db) } } + public void create_database_req(final CreateDatabaseRequest req) Review Comment: Shouldn't this be an overridden method? Issue Time Tracking --- Worklog Id: (was: 795974) Time Spent: 40m (was: 0.5h) > HMS APIs to be enhanced for metadata replication > > > Key: HIVE-26012 > URL: https://issues.apache.org/jira/browse/HIVE-26012 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 3.1.0 >Reporter: Naveen Gangam >Assignee: Hongdan Zhu >Priority: Major > Labels: pull-request-available > Attachments: HMS APIs to be enhanced for metadata replication.docx > > Time Spent: 40m > Remaining Estimate: 0h > > HMS currently has APIs like these that automatically create/delete the > directories on the associated DFS. > [create/drop]_database > [create/drop]_table* > [add/append/drop]_partition* > This is expected and should be this way when query processors use this APIs. > However, when tools that replicate hive metadata use this APIs on the target > cluster, creating these dirs on target side which cause the replication of > DFS-snapshots to fail. > So we if provide an option to bypass this creation of dirs, dfs replications > will be smoother. In the future we will need to restrict users that can use > these APIs. So we will have some sort of an authorization policy. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26012) HMS APIs to be enhanced for metadata replication
[
https://issues.apache.org/jira/browse/HIVE-26012?focusedWorklogId=795971&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-795971
]
ASF GitHub Bot logged work on HIVE-26012:
-
Author: ASF GitHub Bot
Created on: 28/Jul/22 07:48
Start Date: 28/Jul/22 07:48
Worklog Time Spent: 10m
Work Description: saihemanth-cloudera commented on code in PR #3477:
URL: https://github.com/apache/hive/pull/3477#discussion_r931890038
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java:
##
@@ -2361,32 +2402,36 @@ private void create_table_core(final RawStore ms, final
CreateTableRequest req)
firePreEvent(new PreCreateTableEvent(tbl, db, this));
- if (!TableType.VIRTUAL_VIEW.toString().equals(tbl.getTableType())) {
-if (tbl.getSd().getLocation() == null
-|| tbl.getSd().getLocation().isEmpty()) {
- tblPath = wh.getDefaultTablePath(db, tbl.getTableName() +
getTableSuffix(tbl), isExternal(tbl));
-} else {
- if (!isExternal(tbl) && !MetaStoreUtils.isNonNativeTable(tbl)) {
-LOG.warn("Location: " + tbl.getSd().getLocation()
-+ " specified for non-external table:" + tbl.getTableName());
- }
- tblPath = wh.getDnsPath(new Path(tbl.getSd().getLocation()));
- // ignore suffix if it's already there (direct-write CTAS)
- if (!tblPath.getName().matches("(.*)" + SOFT_DELETE_TABLE_PATTERN)) {
-tblPath = new Path(tblPath + getTableSuffix(tbl));
+ if (!skipFSWrites) {
+if (!TableType.VIRTUAL_VIEW.toString().equals(tbl.getTableType())) {
+ if (tbl.getSd().getLocation() == null
+ || tbl.getSd().getLocation().isEmpty()) {
+tblPath = wh.getDefaultTablePath(db, tbl.getTableName() +
getTableSuffix(tbl), isExternal(tbl));
+ } else {
+if (!isExternal(tbl) && !MetaStoreUtils.isNonNativeTable(tbl)) {
+ LOG.warn("Location: " + tbl.getSd().getLocation()
+ + " specified for non-external table:" +
tbl.getTableName());
+}
+tblPath = wh.getDnsPath(new Path(tbl.getSd().getLocation()));
+// ignore suffix if it's already there (direct-write CTAS)
+if (!tblPath.getName().matches("(.*)" +
SOFT_DELETE_TABLE_PATTERN)) {
+ tblPath = new Path(tblPath + getTableSuffix(tbl));
+}
}
+ tbl.getSd().setLocation(tblPath.toString());
}
-tbl.getSd().setLocation(tblPath.toString());
- }
- if (tblPath != null) {
-if (!wh.isDir(tblPath)) {
- if (!wh.mkdirs(tblPath)) {
-throw new MetaException(tblPath
-+ " is not a directory or unable to create one");
+if (tblPath != null) {
+ if (!wh.isDir(tblPath)) {
+if (!wh.mkdirs(tblPath)) {
+ throw new MetaException(tblPath
+ + " is not a directory or unable to create one");
+}
+madeDir = true;
}
- madeDir = true;
}
+ } else {
+LOG.warn("Because skipFSWrites is true, skip the creation of
directories for tables.");
Review Comment:
Why do you want to set this log to warn? Can we leave this as Log.info?
Issue Time Tracking
---
Worklog Id: (was: 795971)
Time Spent: 0.5h (was: 20m)
> HMS APIs to be enhanced for metadata replication
>
>
> Key: HIVE-26012
> URL: https://issues.apache.org/jira/browse/HIVE-26012
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 3.1.0
>Reporter: Naveen Gangam
>Assignee: Hongdan Zhu
>Priority: Major
> Labels: pull-request-available
> Attachments: HMS APIs to be enhanced for metadata replication.docx
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> HMS currently has APIs like these that automatically create/delete the
> directories on the associated DFS.
> [create/drop]_database
> [create/drop]_table*
> [add/append/drop]_partition*
> This is expected and should be this way when query processors use this APIs.
> However, when tools that replicate hive metadata use this APIs on the target
> cluster, creating these dirs on target side which cause the replication of
> DFS-snapshots to fail.
> So we if provide an option to bypass this creation of dirs, dfs replications
> will be smoother. In the future we will need to restrict users that can use
> these APIs. So we will have some sort of an authorization policy.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26012) HMS APIs to be enhanced for metadata replication
[
https://issues.apache.org/jira/browse/HIVE-26012?focusedWorklogId=795970&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-795970
]
ASF GitHub Bot logged work on HIVE-26012:
-
Author: ASF GitHub Bot
Created on: 28/Jul/22 07:46
Start Date: 28/Jul/22 07:46
Worklog Time Spent: 10m
Work Description: saihemanth-cloudera commented on code in PR #3477:
URL: https://github.com/apache/hive/pull/3477#discussion_r931888382
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java:
##
@@ -3958,12 +4003,16 @@ private Partition append_partition_common(RawStore ms,
String catName, String db
throw new AlreadyExistsException("Partition already exists:" + part);
}
- if (!wh.isDir(partLocation)) {
-if (!wh.mkdirs(partLocation)) {
- throw new MetaException(partLocation
- + " is not a directory or unable to create one");
+ if (!skipFSWrites) {
+if (!wh.isDir(partLocation)) {
+ if (!wh.mkdirs(partLocation)) {
+throw new MetaException(partLocation
++ " is not a directory or unable to create one");
+ }
+ madeDir = true;
}
-madeDir = true;
+ } else {
+LOG.warn("Because skipFSWrites is true, skip creating directories for
partitions.");
Review Comment:
Why do you want to set this log to warn? Can we leave this as Log.info?
Issue Time Tracking
---
Worklog Id: (was: 795970)
Time Spent: 20m (was: 10m)
> HMS APIs to be enhanced for metadata replication
>
>
> Key: HIVE-26012
> URL: https://issues.apache.org/jira/browse/HIVE-26012
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 3.1.0
>Reporter: Naveen Gangam
>Assignee: Hongdan Zhu
>Priority: Major
> Labels: pull-request-available
> Attachments: HMS APIs to be enhanced for metadata replication.docx
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> HMS currently has APIs like these that automatically create/delete the
> directories on the associated DFS.
> [create/drop]_database
> [create/drop]_table*
> [add/append/drop]_partition*
> This is expected and should be this way when query processors use this APIs.
> However, when tools that replicate hive metadata use this APIs on the target
> cluster, creating these dirs on target side which cause the replication of
> DFS-snapshots to fail.
> So we if provide an option to bypass this creation of dirs, dfs replications
> will be smoother. In the future we will need to restrict users that can use
> these APIs. So we will have some sort of an authorization policy.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-26012) HMS APIs to be enhanced for metadata replication
[ https://issues.apache.org/jira/browse/HIVE-26012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-26012: -- Labels: pull-request-available (was: ) > HMS APIs to be enhanced for metadata replication > > > Key: HIVE-26012 > URL: https://issues.apache.org/jira/browse/HIVE-26012 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 3.1.0 >Reporter: Naveen Gangam >Assignee: Hongdan Zhu >Priority: Major > Labels: pull-request-available > Attachments: HMS APIs to be enhanced for metadata replication.docx > > Time Spent: 10m > Remaining Estimate: 0h > > HMS currently has APIs like these that automatically create/delete the > directories on the associated DFS. > [create/drop]_database > [create/drop]_table* > [add/append/drop]_partition* > This is expected and should be this way when query processors use this APIs. > However, when tools that replicate hive metadata use this APIs on the target > cluster, creating these dirs on target side which cause the replication of > DFS-snapshots to fail. > So we if provide an option to bypass this creation of dirs, dfs replications > will be smoother. In the future we will need to restrict users that can use > these APIs. So we will have some sort of an authorization policy. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26012) HMS APIs to be enhanced for metadata replication
[
https://issues.apache.org/jira/browse/HIVE-26012?focusedWorklogId=795950&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-795950
]
ASF GitHub Bot logged work on HIVE-26012:
-
Author: ASF GitHub Bot
Created on: 28/Jul/22 07:09
Start Date: 28/Jul/22 07:09
Worklog Time Spent: 10m
Work Description: saihemanth-cloudera commented on code in PR #3477:
URL: https://github.com/apache/hive/pull/3477#discussion_r931856910
##
standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift:
##
@@ -2025,7 +2027,9 @@ struct CreateTableRequest {
7: optional list defaultConstraints,
8: optional list checkConstraints,
9: optional list processorCapabilities,
- 10: optional string processorIdentifier
+ 10: optional string processorIdentifier,
+ // use boolean skipFSWrites to decide whether create directory in file
system or not
Review Comment:
Add "HIVE-26012" in the comment so that if anyone needs more details on
this, they can review the jira.
Issue Time Tracking
---
Worklog Id: (was: 795950)
Remaining Estimate: 0h
Time Spent: 10m
> HMS APIs to be enhanced for metadata replication
>
>
> Key: HIVE-26012
> URL: https://issues.apache.org/jira/browse/HIVE-26012
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 3.1.0
>Reporter: Naveen Gangam
>Assignee: Hongdan Zhu
>Priority: Major
> Attachments: HMS APIs to be enhanced for metadata replication.docx
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> HMS currently has APIs like these that automatically create/delete the
> directories on the associated DFS.
> [create/drop]_database
> [create/drop]_table*
> [add/append/drop]_partition*
> This is expected and should be this way when query processors use this APIs.
> However, when tools that replicate hive metadata use this APIs on the target
> cluster, creating these dirs on target side which cause the replication of
> DFS-snapshots to fail.
> So we if provide an option to bypass this creation of dirs, dfs replications
> will be smoother. In the future we will need to restrict users that can use
> these APIs. So we will have some sort of an authorization policy.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
