[jira] [Comment Edited] (HIVE-16398) [Hive on Tez] Jar Localization issues
[
https://issues.apache.org/jira/browse/HIVE-16398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17179318#comment-17179318
]
tanghui edited comment on HIVE-16398 at 8/18/20, 3:06 AM:
--
May be hive-env.sh or hive-site.xml missing HIVE_AUX_JARS_PATH, you can set
HIVE_AUX_JARS_PATH , add your jar to the path like /usr/lib/hivejar on
HiveServer2 and Hive Metastore Server host。chown hive:hive /usr/lib/hivejar
was (Author: sanguines):
May be hive-env.sh or hive-site.xml missing HIVE_AUX_JARS_PATH, you can set
HIVE_AUX_JARS_PATH , add your jar to the path like /usr/lib/hivejar on
HiveServer2 and HiveMetastore host。chown hive:hive /usr/lib/hivejar
> [Hive on Tez] Jar Localization issues
> -
>
> Key: HIVE-16398
> URL: https://issues.apache.org/jira/browse/HIVE-16398
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Affects Versions: 2.1.0
>Reporter: Ravi Teja Chilukuri
>Priority: Minor
>
> The jar localization fails with the following error in these cases
> 1) If a jar is already present in the classpath and is added again via
> aux.jars or using add jar command by the user.
> 2) If the jar is missing- Ideally it should ignore or either throw a
> error/warning stating jar not found.
> 3) When a "*" is present in some of the classpath entries, it will try to
> replicate the same * path on hdfs and fails.
> Error:
> {noformat}
> Total jobs = 1
> Launching Job 1 out of 1
> Exception in thread "Thread-17" java.lang.RuntimeException:
> java.io.IOException: Previous writer likely failed to write
> hdfs://clstr1/tmp/hive/adhoc-hive-user/_tez_session_dir/edd821a7-d4f5-4980-b693-6b9c17ac4257/hadoop-4mc-1.1.0.jar.
> Failing because I am unlikely to write too.
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:617)
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:526)
> at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:78)
> Caused by: java.io.IOException: Previous writer likely failed to write
> hdfs://clstr1/tmp/hive/adhoc-hive-user/_tez_session_dir/edd821a7-d4f5-4980-b693-6b9c17ac4257/hadoop-4mc-1.1.0.jar.
> Failing because I am unlikely to write too.
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeResource(DagUtils.java:1020)
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.addTempResources(DagUtils.java:900)
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeTempFilesFromConf(DagUtils.java:843)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.refreshLocalResourcesFromConf(TezSessionState.java:451)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.openInternal(TezSessionState.java:247)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:222)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:189)
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:612)
> ... 2 more
> FAILED: Execution Error, return code -1 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-16398) [Hive on Tez] Jar Localization issues
[
https://issues.apache.org/jira/browse/HIVE-16398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17179318#comment-17179318
]
tanghui commented on HIVE-16398:
May be hive-env.sh or hive-site.xml missing HIVE_AUX_JARS_PATH, you can set
HIVE_AUX_JARS_PATH , add your jar to the path like /usr/lib/hivejar on
HiveServer2 and HiveMetastore host。chown hive:hive /usr/lib/hivejar
> [Hive on Tez] Jar Localization issues
> -
>
> Key: HIVE-16398
> URL: https://issues.apache.org/jira/browse/HIVE-16398
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Affects Versions: 2.1.0
>Reporter: Ravi Teja Chilukuri
>Priority: Minor
>
> The jar localization fails with the following error in these cases
> 1) If a jar is already present in the classpath and is added again via
> aux.jars or using add jar command by the user.
> 2) If the jar is missing- Ideally it should ignore or either throw a
> error/warning stating jar not found.
> 3) When a "*" is present in some of the classpath entries, it will try to
> replicate the same * path on hdfs and fails.
> Error:
> {noformat}
> Total jobs = 1
> Launching Job 1 out of 1
> Exception in thread "Thread-17" java.lang.RuntimeException:
> java.io.IOException: Previous writer likely failed to write
> hdfs://clstr1/tmp/hive/adhoc-hive-user/_tez_session_dir/edd821a7-d4f5-4980-b693-6b9c17ac4257/hadoop-4mc-1.1.0.jar.
> Failing because I am unlikely to write too.
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:617)
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:526)
> at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:78)
> Caused by: java.io.IOException: Previous writer likely failed to write
> hdfs://clstr1/tmp/hive/adhoc-hive-user/_tez_session_dir/edd821a7-d4f5-4980-b693-6b9c17ac4257/hadoop-4mc-1.1.0.jar.
> Failing because I am unlikely to write too.
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeResource(DagUtils.java:1020)
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.addTempResources(DagUtils.java:900)
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeTempFilesFromConf(DagUtils.java:843)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.refreshLocalResourcesFromConf(TezSessionState.java:451)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.openInternal(TezSessionState.java:247)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:222)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:189)
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:612)
> ... 2 more
> FAILED: Execution Error, return code -1 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23676) Test to cover wildcard partVals in listPartitionNames
[
https://issues.apache.org/jira/browse/HIVE-23676?focusedWorklogId=471754=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-471754
]
ASF GitHub Bot logged work on HIVE-23676:
-
Author: ASF GitHub Bot
Created on: 18/Aug/20 00:39
Start Date: 18/Aug/20 00:39
Worklog Time Spent: 10m
Work Description: github-actions[bot] closed pull request #1093:
URL: https://github.com/apache/hive/pull/1093
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 471754)
Time Spent: 0.5h (was: 20m)
> Test to cover wildcard partVals in listPartitionNames
> -
>
> Key: HIVE-23676
> URL: https://issues.apache.org/jira/browse/HIVE-23676
> Project: Hive
> Issue Type: Bug
> Components: Metastore, Test
>Reporter: anton lin
>Assignee: anton lin
>Priority: Minor
> Labels: pull-request-available
> Attachments: HIVE-23676.branch-3.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Incorrect documentation for MetaStoreClient method
> {code:java}
> List listPartitionNames(String db_name, String tbl_name,
> List part_vals, short max_parts)
> {code}
> Saying _"...If you wish to accept any value for a particular key you can pass
> ".*" for that value in this list..."_
> Any value wildcard behaviour is achieved with empty string _""_.
> Documentation and tests should reflect this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23980) Shade guava from existing Hive versions
[
https://issues.apache.org/jira/browse/HIVE-23980?focusedWorklogId=471748=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-471748
]
ASF GitHub Bot logged work on HIVE-23980:
-
Author: ASF GitHub Bot
Created on: 18/Aug/20 00:26
Start Date: 18/Aug/20 00:26
Worklog Time Spent: 10m
Work Description: sunchao merged pull request #1397:
URL: https://github.com/apache/hive/pull/1397
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 471748)
Time Spent: 2h 50m (was: 2h 40m)
> Shade guava from existing Hive versions
> ---
>
> Key: HIVE-23980
> URL: https://issues.apache.org/jira/browse/HIVE-23980
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.3.7
>Reporter: L. C. Hsieh
>Assignee: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23980.01.branch-2.3.patch
>
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> I'm trying to upgrade Guava version in Spark. The JIRA ticket is SPARK-32502.
> Running test hits an error:
> {code}
> sbt.ForkMain$ForkError: sbt.ForkMain$ForkError: java.lang.IllegalAccessError:
> tried to access method
> com.google.common.collect.Iterators.emptyIterator()Lcom/google/common/collect/UnmodifiableIterator;
> from class org.apache.hadoop.hive.ql.exec.FetchOperator
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.(FetchOperator.java:108)
> at
> org.apache.hadoop.hive.ql.exec.FetchTask.initialize(FetchTask.java:87)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:541)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1457)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)
> {code}
> I know that hive-exec doesn't shade Guava until HIVE-22126 but that work
> targets 4.0.0. I'm wondering if there is a solution for current Hive
> versions, e.g. Hive 2.3.7? Any ideas?
> Thanks.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-24045) No logging related to when default database is created
[ https://issues.apache.org/jira/browse/HIVE-24045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Hemanth Gantasala reassigned HIVE-24045: > No logging related to when default database is created > -- > > Key: HIVE-24045 > URL: https://issues.apache.org/jira/browse/HIVE-24045 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Sai Hemanth Gantasala >Assignee: Sai Hemanth Gantasala >Priority: Major > > There does not appear to be any HMS logs related to when the "default" > database is first created in Hive. This would be useful for troubleshooting. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23887) Reset table level basic/column stats during import.
[ https://issues.apache.org/jira/browse/HIVE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-23887: Summary: Reset table level basic/column stats during import. (was: Reset basic/column stats during import.) > Reset table level basic/column stats during import. > --- > > Key: HIVE-23887 > URL: https://issues.apache.org/jira/browse/HIVE-23887 > Project: Hive > Issue Type: Bug > Components: Import/Export, Statistics >Reporter: Ashish Sharma >Assignee: Ashish Sharma >Priority: Minor > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > While doing "export table db.table to '/import/table' " column stats are not > dumped but import doesn't reset the flag which leads to incorrect stats. > Reset columns stats while import to force Imported to recalculate the Columns > stats -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23887) Reset basic/column stats in during import.
[
https://issues.apache.org/jira/browse/HIVE-23887?focusedWorklogId=471521=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-471521
]
ASF GitHub Bot logged work on HIVE-23887:
-
Author: ASF GitHub Bot
Created on: 17/Aug/20 14:59
Start Date: 17/Aug/20 14:59
Worklog Time Spent: 10m
Work Description: sankarh commented on a change in pull request #1370:
URL: https://github.com/apache/hive/pull/1370#discussion_r471526397
##
File path: ql/src/test/org/apache/hadoop/hive/ql/TestTxnExIm.java
##
@@ -566,4 +581,33 @@ public void testMMExportAborted() throws Exception {
TestTxnCommands2.stringifyValues(data), rs);
}
-}
+
+
+ @Test public void testImportOrc() throws Exception {
Review comment:
Can we also check if import sets stats to true in case of
autostats_gather is set to true? For both partitioned and non-partitioned
tables. Also check the values of stats listed in
StatsSetupConst.SUPPORTED_STATS (atleast ROWNUM should match) in both source
and target tables and they should match.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 471521)
Time Spent: 1h 10m (was: 1h)
> Reset basic/column stats in during import.
> --
>
> Key: HIVE-23887
> URL: https://issues.apache.org/jira/browse/HIVE-23887
> Project: Hive
> Issue Type: Bug
> Components: Import/Export, Statistics
>Reporter: Ashish Sharma
>Assignee: Ashish Sharma
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> While doing "export table db.table to '/import/table' " column stats are not
> dumped but import doesn't reset the flag which leads to incorrect stats.
> Reset columns stats while import to force Imported to recalculate the Columns
> stats
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23887) Reset basic/column stats during import.
[ https://issues.apache.org/jira/browse/HIVE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-23887: Summary: Reset basic/column stats during import. (was: Reset basic/column stats in during import.) > Reset basic/column stats during import. > --- > > Key: HIVE-23887 > URL: https://issues.apache.org/jira/browse/HIVE-23887 > Project: Hive > Issue Type: Bug > Components: Import/Export, Statistics >Reporter: Ashish Sharma >Assignee: Ashish Sharma >Priority: Minor > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > While doing "export table db.table to '/import/table' " column stats are not > dumped but import doesn't reset the flag which leads to incorrect stats. > Reset columns stats while import to force Imported to recalculate the Columns > stats -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23887) Reset basic/column stats in during import.
[
https://issues.apache.org/jira/browse/HIVE-23887?focusedWorklogId=471520=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-471520
]
ASF GitHub Bot logged work on HIVE-23887:
-
Author: ASF GitHub Bot
Created on: 17/Aug/20 14:59
Start Date: 17/Aug/20 14:59
Worklog Time Spent: 10m
Work Description: sankarh commented on a change in pull request #1370:
URL: https://github.com/apache/hive/pull/1370#discussion_r471526397
##
File path: ql/src/test/org/apache/hadoop/hive/ql/TestTxnExIm.java
##
@@ -566,4 +581,33 @@ public void testMMExportAborted() throws Exception {
TestTxnCommands2.stringifyValues(data), rs);
}
-}
+
+
+ @Test public void testImportOrc() throws Exception {
Review comment:
Can we also check if import sets stats to true in case of
autostats_gather is set to true? For both partitioned and non-partitioned
tables. Also check the values of stats listed in
StatsSetupConst.SUPPORTED_STATS in both source and target tables and they
should match.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 471520)
Time Spent: 1h (was: 50m)
> Reset basic/column stats in during import.
> --
>
> Key: HIVE-23887
> URL: https://issues.apache.org/jira/browse/HIVE-23887
> Project: Hive
> Issue Type: Bug
> Components: Import/Export, Statistics
>Reporter: Ashish Sharma
>Assignee: Ashish Sharma
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> While doing "export table db.table to '/import/table' " column stats are not
> dumped but import doesn't reset the flag which leads to incorrect stats.
> Reset columns stats while import to force Imported to recalculate the Columns
> stats
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23887) Reset basic/column stats in during import.
[ https://issues.apache.org/jira/browse/HIVE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-23887: Description: While doing "export table db.table to '/import/table' " column stats are not dumped but import doesn't reset the flag which leads to incorrect stats. Reset columns stats while import to force Imported to recalculate the Columns stats was: While doing "export table db.table to '/import/table' " entire column stat is getting exported which lead to columns stats corruption in import table. Reset columns stats while export to force Imported to recalculate the Columns stats > Reset basic/column stats in during import. > -- > > Key: HIVE-23887 > URL: https://issues.apache.org/jira/browse/HIVE-23887 > Project: Hive > Issue Type: Bug > Components: Import/Export, Statistics >Reporter: Ashish Sharma >Assignee: Ashish Sharma >Priority: Minor > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > While doing "export table db.table to '/import/table' " column stats are not > dumped but import doesn't reset the flag which leads to incorrect stats. > Reset columns stats while import to force Imported to recalculate the Columns > stats -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23887) Reset basic/column stats in during import.
[ https://issues.apache.org/jira/browse/HIVE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-23887: Summary: Reset basic/column stats in during import. (was: Reset Columns stats in Export Statement) > Reset basic/column stats in during import. > -- > > Key: HIVE-23887 > URL: https://issues.apache.org/jira/browse/HIVE-23887 > Project: Hive > Issue Type: Bug > Components: Import/Export, Statistics >Reporter: Ashish Sharma >Assignee: Ashish Sharma >Priority: Minor > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > While doing "export table db.table to '/import/table' " entire column stat is > getting exported which lead to columns stats corruption in import table. > Reset columns stats while export to force Imported to recalculate the Columns > stats -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23887) Reset Columns stats in Export Statement
[
https://issues.apache.org/jira/browse/HIVE-23887?focusedWorklogId=471515=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-471515
]
ASF GitHub Bot logged work on HIVE-23887:
-
Author: ASF GitHub Bot
Created on: 17/Aug/20 14:41
Start Date: 17/Aug/20 14:41
Worklog Time Spent: 10m
Work Description: sankarh commented on a change in pull request #1370:
URL: https://github.com/apache/hive/pull/1370#discussion_r471511349
##
File path: ql/src/test/org/apache/hadoop/hive/ql/TestTxnExIm.java
##
@@ -566,4 +581,33 @@ public void testMMExportAborted() throws Exception {
TestTxnCommands2.stringifyValues(data), rs);
}
-}
+
+
Review comment:
Nit: Remove extra blank line.
##
File path: ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
##
@@ -3185,7 +3185,7 @@ public void loadTable(Path loadPath, String tableName,
LoadFileType loadFileType
//column stats will be inaccurate
if (resetStatistics) {
LOG.debug("Clearing table statistics for " + tbl.getDbName() + "." +
tbl.getTableName());
- StatsSetupConst.clearColumnStatsState(tbl.getParameters());
+
StatsSetupConst.setBasicStatsState(tbl.getParameters(),StatsSetupConst.FALSE);
Review comment:
Even during loadPartition, we need to reset stats in table properties as
well which is missing now.
##
File path: ql/src/test/org/apache/hadoop/hive/ql/TestTxnExIm.java
##
@@ -566,4 +581,33 @@ public void testMMExportAborted() throws Exception {
TestTxnCommands2.stringifyValues(data), rs);
}
-}
+
+
+ @Test public void testImportOrc() throws Exception {
+
+runStatementOnDriver("drop table if exists T");
+runStatementOnDriver("drop table if exists Tstage");
+runStatementOnDriver("create table T (a int, b int) stored"
++ " as orc tblproperties('transactional'='true')");
+//Tstage is the target table
Review comment:
Nit: Add single blank line before a comment line. Check below places too.
##
File path: ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
##
@@ -3185,7 +3185,7 @@ public void loadTable(Path loadPath, String tableName,
LoadFileType loadFileType
//column stats will be inaccurate
if (resetStatistics) {
LOG.debug("Clearing table statistics for " + tbl.getDbName() + "." +
tbl.getTableName());
- StatsSetupConst.clearColumnStatsState(tbl.getParameters());
+
StatsSetupConst.setBasicStatsState(tbl.getParameters(),StatsSetupConst.FALSE);
Review comment:
Can be combined with previous if block which also does the same thing.
Also, the comment says "column stats will be inaccurate" but we are resetting
both basic and column stats. Correct the log message and comment accordingly.
##
File path: ql/src/test/org/apache/hadoop/hive/ql/TestTxnExIm.java
##
@@ -566,4 +581,33 @@ public void testMMExportAborted() throws Exception {
TestTxnCommands2.stringifyValues(data), rs);
}
-}
+
+
+ @Test public void testImportOrc() throws Exception {
+
+runStatementOnDriver("drop table if exists T");
+runStatementOnDriver("drop table if exists Tstage");
+runStatementOnDriver("create table T (a int, b int) stored"
++ " as orc tblproperties('transactional'='true')");
+//Tstage is the target table
+runStatementOnDriver("create table Tstage (a int, b int) stored"
++ " as orc tblproperties('transactional'='true')");
+//this creates an ORC data file with correct schema under table root
+runStatementOnDriver("insert into Tstage values(1,2),(3,4),(5,6)");
+final int[][] rows = { { 3 } };
Review comment:
Can we check if stats are set in source table Tstage?
##
File path: ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
##
@@ -3185,7 +3185,7 @@ public void loadTable(Path loadPath, String tableName,
LoadFileType loadFileType
//column stats will be inaccurate
if (resetStatistics) {
LOG.debug("Clearing table statistics for " + tbl.getDbName() + "." +
tbl.getTableName());
- StatsSetupConst.clearColumnStatsState(tbl.getParameters());
+
StatsSetupConst.setBasicStatsState(tbl.getParameters(),StatsSetupConst.FALSE);
Review comment:
Nit: Need a space after comma ",".
##
File path: ql/src/test/org/apache/hadoop/hive/ql/TestTxnExIm.java
##
@@ -404,6 +404,21 @@ public void testImportPartitionedOrc() throws Exception {
//load T
runStatementOnDriver("import table T from '" + getWarehouseDir() + "/1'");
+
+//check basic stats in tblproperties
+List rsProperties = runStatementOnDriver("show tblproperties T");
+Assert
+.assertEquals("COLUMN_STATS_ACCURATE of imported table",
rsProperties.contains("COLUMN_STATS_ACCURATE"), false);
+
+//check basic stats in partition properties
+List
[jira] [Resolved] (HIVE-23358) MSCK REPAIR should remove all insignificant zeroes from numeric partition values
[
https://issues.apache.org/jira/browse/HIVE-23358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan resolved HIVE-23358.
-
Fix Version/s: 4.0.0
Resolution: Fixed
Merged to master.
Thanks [~adeshrao] for the patch!
> MSCK REPAIR should remove all insignificant zeroes from numeric partition
> values
>
>
> Key: HIVE-23358
> URL: https://issues.apache.org/jira/browse/HIVE-23358
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Adesh Kumar Rao
>Assignee: Adesh Kumar Rao
>Priority: Minor
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> For the following scenario
> 1. Have partitioned data path as follows.
> hdfs://mycluster/datapath/t1/year=2020/month=03/day=10
> hdfs://mycluster/datapath/t1/year=2020/month=03/day=11
> 2. create external table t1 (key int, value string) partitioned by (Year int,
> Month int, Day int) stored as orc location hdfs://mycluster/datapath/t1'';
> 3. msck repair table t1;
> 4. show partitions t1;
> {noformat}
> ++
> | partition |
> ++
> | year=2020/month=03/day=10 |
> | year=2020/month=03/day=11 |
> ++
> {noformat}
> 5.show table extended like 't1' partition (Year=2020, Month=03, Day=11);
> will throw an error:
> {noformat}
> Error: Error while compiling statement: FAILED: SemanticException [Error
> 10006]: Partition not found {year=2020, month=3, day=11}
> (state=42000,code=10006)
> {noformat}
> When the partition directory are created without the extra zeroes, this works
> fine.
> {noformat}
> hdfs://mycluster/datapath/t1/year=2020/month=3/day=10
> hdfs://mycluster/datapath/t1/year=2020/month=3/day=11
> {noformat}
> This happens because while searching for partitions, hive strips the extra
> "0" in month key and then queries the metastore
> (partSpec="year=2020/month=3/day=10") which returns no rows.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23358) MSCK REPAIR should remove all insignificant zeroes from numeric partition values
[
https://issues.apache.org/jira/browse/HIVE-23358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-23358:
Summary: MSCK REPAIR should remove all insignificant zeroes from numeric
partition values (was: MSCK REPAIR should remove all insignificant zeroes from
partition values (for numeric datatypes) before creating the partitions)
> MSCK REPAIR should remove all insignificant zeroes from numeric partition
> values
>
>
> Key: HIVE-23358
> URL: https://issues.apache.org/jira/browse/HIVE-23358
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Adesh Kumar Rao
>Assignee: Adesh Kumar Rao
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> For the following scenario
> 1. Have partitioned data path as follows.
> hdfs://mycluster/datapath/t1/year=2020/month=03/day=10
> hdfs://mycluster/datapath/t1/year=2020/month=03/day=11
> 2. create external table t1 (key int, value string) partitioned by (Year int,
> Month int, Day int) stored as orc location hdfs://mycluster/datapath/t1'';
> 3. msck repair table t1;
> 4. show partitions t1;
> {noformat}
> ++
> | partition |
> ++
> | year=2020/month=03/day=10 |
> | year=2020/month=03/day=11 |
> ++
> {noformat}
> 5.show table extended like 't1' partition (Year=2020, Month=03, Day=11);
> will throw an error:
> {noformat}
> Error: Error while compiling statement: FAILED: SemanticException [Error
> 10006]: Partition not found {year=2020, month=3, day=11}
> (state=42000,code=10006)
> {noformat}
> When the partition directory are created without the extra zeroes, this works
> fine.
> {noformat}
> hdfs://mycluster/datapath/t1/year=2020/month=3/day=10
> hdfs://mycluster/datapath/t1/year=2020/month=3/day=11
> {noformat}
> This happens because while searching for partitions, hive strips the extra
> "0" in month key and then queries the metastore
> (partSpec="year=2020/month=3/day=10") which returns no rows.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23358) MSCK REPAIR should remove all insignificant zeroes from partition values (for numeric datatypes) before creating the partitions
[
https://issues.apache.org/jira/browse/HIVE-23358?focusedWorklogId=471492=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-471492
]
ASF GitHub Bot logged work on HIVE-23358:
-
Author: ASF GitHub Bot
Created on: 17/Aug/20 13:42
Start Date: 17/Aug/20 13:42
Worklog Time Spent: 10m
Work Description: sankarh merged pull request #1050:
URL: https://github.com/apache/hive/pull/1050
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 471492)
Time Spent: 1.5h (was: 1h 20m)
> MSCK REPAIR should remove all insignificant zeroes from partition values (for
> numeric datatypes) before creating the partitions
> ---
>
> Key: HIVE-23358
> URL: https://issues.apache.org/jira/browse/HIVE-23358
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Adesh Kumar Rao
>Assignee: Adesh Kumar Rao
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> For the following scenario
> 1. Have partitioned data path as follows.
> hdfs://mycluster/datapath/t1/year=2020/month=03/day=10
> hdfs://mycluster/datapath/t1/year=2020/month=03/day=11
> 2. create external table t1 (key int, value string) partitioned by (Year int,
> Month int, Day int) stored as orc location hdfs://mycluster/datapath/t1'';
> 3. msck repair table t1;
> 4. show partitions t1;
> {noformat}
> ++
> | partition |
> ++
> | year=2020/month=03/day=10 |
> | year=2020/month=03/day=11 |
> ++
> {noformat}
> 5.show table extended like 't1' partition (Year=2020, Month=03, Day=11);
> will throw an error:
> {noformat}
> Error: Error while compiling statement: FAILED: SemanticException [Error
> 10006]: Partition not found {year=2020, month=3, day=11}
> (state=42000,code=10006)
> {noformat}
> When the partition directory are created without the extra zeroes, this works
> fine.
> {noformat}
> hdfs://mycluster/datapath/t1/year=2020/month=3/day=10
> hdfs://mycluster/datapath/t1/year=2020/month=3/day=11
> {noformat}
> This happens because while searching for partitions, hive strips the extra
> "0" in month key and then queries the metastore
> (partSpec="year=2020/month=3/day=10") which returns no rows.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23358) MSCK REPAIR should remove all insignificant zeroes from partition values (for numeric datatypes) before creating the partitions
[
https://issues.apache.org/jira/browse/HIVE-23358?focusedWorklogId=471435=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-471435
]
ASF GitHub Bot logged work on HIVE-23358:
-
Author: ASF GitHub Bot
Created on: 17/Aug/20 10:03
Start Date: 17/Aug/20 10:03
Worklog Time Spent: 10m
Work Description: adesh-rao opened a new pull request #1050:
URL: https://github.com/apache/hive/pull/1050
Reference for converting hive types to java
Hive datatypes:
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Types
Java Datatypes:
https://docs.oracle.com/javase/tutorial/java/nutsandbolts/datatypes.html
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 471435)
Time Spent: 1h 20m (was: 1h 10m)
> MSCK REPAIR should remove all insignificant zeroes from partition values (for
> numeric datatypes) before creating the partitions
> ---
>
> Key: HIVE-23358
> URL: https://issues.apache.org/jira/browse/HIVE-23358
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Adesh Kumar Rao
>Assignee: Adesh Kumar Rao
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> For the following scenario
> 1. Have partitioned data path as follows.
> hdfs://mycluster/datapath/t1/year=2020/month=03/day=10
> hdfs://mycluster/datapath/t1/year=2020/month=03/day=11
> 2. create external table t1 (key int, value string) partitioned by (Year int,
> Month int, Day int) stored as orc location hdfs://mycluster/datapath/t1'';
> 3. msck repair table t1;
> 4. show partitions t1;
> {noformat}
> ++
> | partition |
> ++
> | year=2020/month=03/day=10 |
> | year=2020/month=03/day=11 |
> ++
> {noformat}
> 5.show table extended like 't1' partition (Year=2020, Month=03, Day=11);
> will throw an error:
> {noformat}
> Error: Error while compiling statement: FAILED: SemanticException [Error
> 10006]: Partition not found {year=2020, month=3, day=11}
> (state=42000,code=10006)
> {noformat}
> When the partition directory are created without the extra zeroes, this works
> fine.
> {noformat}
> hdfs://mycluster/datapath/t1/year=2020/month=3/day=10
> hdfs://mycluster/datapath/t1/year=2020/month=3/day=11
> {noformat}
> This happens because while searching for partitions, hive strips the extra
> "0" in month key and then queries the metastore
> (partSpec="year=2020/month=3/day=10") which returns no rows.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23851) MSCK REPAIR Command With Partition Filtering Fails While Dropping Partitions
[
https://issues.apache.org/jira/browse/HIVE-23851?focusedWorklogId=471398=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-471398
]
ASF GitHub Bot logged work on HIVE-23851:
-
Author: ASF GitHub Bot
Created on: 17/Aug/20 09:05
Start Date: 17/Aug/20 09:05
Worklog Time Spent: 10m
Work Description: shameersss1 commented on pull request #1271:
URL: https://github.com/apache/hive/pull/1271#issuecomment-674757864
@kgyrtkirk @jcamachor Ping for review request!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 471398)
Time Spent: 2h 10m (was: 2h)
> MSCK REPAIR Command With Partition Filtering Fails While Dropping Partitions
>
>
> Key: HIVE-23851
> URL: https://issues.apache.org/jira/browse/HIVE-23851
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Syed Shameerur Rahman
>Assignee: Syed Shameerur Rahman
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> *Steps to reproduce:*
> # Create external table
> # Run msck command to sync all the partitions with metastore
> # Remove one of the partition path
> # Run msck repair with partition filtering
> *Stack Trace:*
> {code:java}
> 2020-07-15T02:10:29,045 ERROR [4dad298b-28b1-4e6b-94b6-aa785b60c576 main]
> ppr.PartitionExpressionForMetastore: Failed to deserialize the expression
> java.lang.IndexOutOfBoundsException: Index: 110, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:657) ~[?:1.8.0_192]
> at java.util.ArrayList.get(ArrayList.java:433) ~[?:1.8.0_192]
> at
> org.apache.hive.com.esotericsoftware.kryo.util.MapReferenceResolver.getReadObject(MapReferenceResolver.java:60)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readReferenceOrNull(Kryo.java:857)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:707)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:211)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities.deserializeObjectFromKryo(SerializationUtilities.java:806)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities.deserializeExpressionFromKryo(SerializationUtilities.java:775)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.optimizer.ppr.PartitionExpressionForMetastore.deserializeExpr(PartitionExpressionForMetastore.java:96)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.optimizer.ppr.PartitionExpressionForMetastore.convertExprToFilter(PartitionExpressionForMetastore.java:52)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.PartFilterExprUtil.makeExpressionTree(PartFilterExprUtil.java:48)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByExprInternal(ObjectStore.java:3593)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.VerifyingObjectStore.getPartitionsByExpr(VerifyingObjectStore.java:80)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT-tests.jar:4.0.0-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_192]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_192]
> {code}
> *Cause:*
> In case of msck repair with partition filtering we expect expression proxy
> class to be set as PartitionExpressionForMetastore (
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/ddl/misc/msck/MsckAnalyzer.java#L78
> ), While dropping partition we serialize the drop partition filter
> expression as (
> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java#L589
> ) which is incompatible during deserializtion happening in
> PartitionExpressionForMetastore (
>
[jira] [Work logged] (HIVE-23980) Shade guava from existing Hive versions
[
https://issues.apache.org/jira/browse/HIVE-23980?focusedWorklogId=471368=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-471368
]
ASF GitHub Bot logged work on HIVE-23980:
-
Author: ASF GitHub Bot
Created on: 17/Aug/20 07:05
Start Date: 17/Aug/20 07:05
Worklog Time Spent: 10m
Work Description: sunchao commented on pull request #1397:
URL: https://github.com/apache/hive/pull/1397#issuecomment-674701692
Yeah, however I think this PR is trivial enough and I don't see the tests
are related.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 471368)
Time Spent: 2h 40m (was: 2.5h)
> Shade guava from existing Hive versions
> ---
>
> Key: HIVE-23980
> URL: https://issues.apache.org/jira/browse/HIVE-23980
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.3.7
>Reporter: L. C. Hsieh
>Assignee: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23980.01.branch-2.3.patch
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> I'm trying to upgrade Guava version in Spark. The JIRA ticket is SPARK-32502.
> Running test hits an error:
> {code}
> sbt.ForkMain$ForkError: sbt.ForkMain$ForkError: java.lang.IllegalAccessError:
> tried to access method
> com.google.common.collect.Iterators.emptyIterator()Lcom/google/common/collect/UnmodifiableIterator;
> from class org.apache.hadoop.hive.ql.exec.FetchOperator
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.(FetchOperator.java:108)
> at
> org.apache.hadoop.hive.ql.exec.FetchTask.initialize(FetchTask.java:87)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:541)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1457)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)
> {code}
> I know that hive-exec doesn't shade Guava until HIVE-22126 but that work
> targets 4.0.0. I'm wondering if there is a solution for current Hive
> versions, e.g. Hive 2.3.7? Any ideas?
> Thanks.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23980) Shade guava from existing Hive versions
[
https://issues.apache.org/jira/browse/HIVE-23980?focusedWorklogId=471361=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-471361
]
ASF GitHub Bot logged work on HIVE-23980:
-
Author: ASF GitHub Bot
Created on: 17/Aug/20 06:15
Start Date: 17/Aug/20 06:15
Worklog Time Spent: 10m
Work Description: viirya commented on pull request #1397:
URL: https://github.com/apache/hive/pull/1397#issuecomment-674680831
> > @sunchao Can you reproduce the test failure locally?
>
> NO I took two failed tests and can't reproduce them locally, with and
without your PR.
Hm, it makes harder to debug...
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 471361)
Time Spent: 2.5h (was: 2h 20m)
> Shade guava from existing Hive versions
> ---
>
> Key: HIVE-23980
> URL: https://issues.apache.org/jira/browse/HIVE-23980
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.3.7
>Reporter: L. C. Hsieh
>Assignee: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23980.01.branch-2.3.patch
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> I'm trying to upgrade Guava version in Spark. The JIRA ticket is SPARK-32502.
> Running test hits an error:
> {code}
> sbt.ForkMain$ForkError: sbt.ForkMain$ForkError: java.lang.IllegalAccessError:
> tried to access method
> com.google.common.collect.Iterators.emptyIterator()Lcom/google/common/collect/UnmodifiableIterator;
> from class org.apache.hadoop.hive.ql.exec.FetchOperator
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.(FetchOperator.java:108)
> at
> org.apache.hadoop.hive.ql.exec.FetchTask.initialize(FetchTask.java:87)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:541)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1457)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)
> {code}
> I know that hive-exec doesn't shade Guava until HIVE-22126 but that work
> targets 4.0.0. I'm wondering if there is a solution for current Hive
> versions, e.g. Hive 2.3.7? Any ideas?
> Thanks.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23980) Shade guava from existing Hive versions
[
https://issues.apache.org/jira/browse/HIVE-23980?focusedWorklogId=471357=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-471357
]
ASF GitHub Bot logged work on HIVE-23980:
-
Author: ASF GitHub Bot
Created on: 17/Aug/20 06:07
Start Date: 17/Aug/20 06:07
Worklog Time Spent: 10m
Work Description: sunchao commented on pull request #1397:
URL: https://github.com/apache/hive/pull/1397#issuecomment-674677369
> @sunchao Can you reproduce the test failure locally?
NO I took two failed tests and can't reproduce them locally, with and
without your PR.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 471357)
Time Spent: 2h 20m (was: 2h 10m)
> Shade guava from existing Hive versions
> ---
>
> Key: HIVE-23980
> URL: https://issues.apache.org/jira/browse/HIVE-23980
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.3.7
>Reporter: L. C. Hsieh
>Assignee: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23980.01.branch-2.3.patch
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> I'm trying to upgrade Guava version in Spark. The JIRA ticket is SPARK-32502.
> Running test hits an error:
> {code}
> sbt.ForkMain$ForkError: sbt.ForkMain$ForkError: java.lang.IllegalAccessError:
> tried to access method
> com.google.common.collect.Iterators.emptyIterator()Lcom/google/common/collect/UnmodifiableIterator;
> from class org.apache.hadoop.hive.ql.exec.FetchOperator
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.(FetchOperator.java:108)
> at
> org.apache.hadoop.hive.ql.exec.FetchTask.initialize(FetchTask.java:87)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:541)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1457)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)
> {code}
> I know that hive-exec doesn't shade Guava until HIVE-22126 but that work
> targets 4.0.0. I'm wondering if there is a solution for current Hive
> versions, e.g. Hive 2.3.7? Any ideas?
> Thanks.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
