[jira] [Work logged] (HIVE-26599) Fix NPE encountered in second dump cycle of optimised bootstrap
[
https://issues.apache.org/jira/browse/HIVE-26599?focusedWorklogId=840170=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-840170
]
ASF GitHub Bot logged work on HIVE-26599:
-
Author: ASF GitHub Bot
Created on: 19/Jan/23 07:52
Start Date: 19/Jan/23 07:52
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3963:
URL: https://github.com/apache/hive/pull/3963#issuecomment-1396566117
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3963)
[](https://sonarcloud.io/project/issues?id=apache_hive=3963=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3963=false=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3963=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3963=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3963=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3963=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3963=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3963=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3963=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3963=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3963=false=CODE_SMELL)
[0 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3963=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3963=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3963=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 840170)
Time Spent: 0.5h (was: 20m)
> Fix NPE encountered in second dump cycle of optimised bootstrap
> ---
>
> Key: HIVE-26599
> URL: https://issues.apache.org/jira/browse/HIVE-26599
> Project: Hive
> Issue Type: Bug
>Reporter: Teddy Choi
>Assignee: Vinit Patni
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> After creating reverse replication policy after failover is completed from
> Primary to DR cluster and DR takes over. First dump and load cycle of
> optimised bootstrap is completing successfully, But We are encountering Null
> pointer exception in the second dump cycle which is halting this reverse
> replication and major blocker to test complete cycle of replication.
> {code:java}
> Scheduled Query Executor(schedule:repl_reverse, execution_id:14)]: FAILED:
> Execution Error, return code -101 from
> org.apache.hadoop.hive.ql.exec.repl.ReplDumpTask.
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.parse.repl.metric.ReplicationMetricCollector.reportStageProgress(ReplicationMetricCollector.java:192)
> at
> org.apache.hadoop.hive.ql.exec.repl.ReplDumpTask.dumpTable(ReplDumpTask.java:1458)
> at
> org.apache.hadoop.hive.ql.exec.repl.ReplDumpTask.incrementalDump(ReplDumpTask.java:961)
> at
>
[jira] [Work logged] (HIVE-26950) (CTLT) Create external table like V2 table is not preserving table properties
[ https://issues.apache.org/jira/browse/HIVE-26950?focusedWorklogId=840161=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-840161 ] ASF GitHub Bot logged work on HIVE-26950: - Author: ASF GitHub Bot Created on: 19/Jan/23 07:15 Start Date: 19/Jan/23 07:15 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3965: URL: https://github.com/apache/hive/pull/3965#issuecomment-1396537991 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3965) [](https://sonarcloud.io/project/issues?id=apache_hive=3965=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3965=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3965=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3965=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3965=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3965=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3965=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3965=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3965=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3965=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3965=false=CODE_SMELL) [1 Code Smell](https://sonarcloud.io/project/issues?id=apache_hive=3965=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3965=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3965=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 840161) Time Spent: 20m (was: 10m) > (CTLT) Create external table like V2 table is not preserving table properties > - > > Key: HIVE-26950 > URL: https://issues.apache.org/jira/browse/HIVE-26950 > Project: Hive > Issue Type: Improvement > Components: Iceberg integration >Reporter: Rajesh Balamohan >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > # Create an external iceberg V2 table. e.g t1 > # "create external table t2 like t1" <--- This ends up creating V1 table and > "format-version=2" is not retained and "'format'='iceberg/parquet'" is also > not retained. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26400) Provide docker images for Hive
[ https://issues.apache.org/jira/browse/HIVE-26400?focusedWorklogId=840157=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-840157 ] ASF GitHub Bot logged work on HIVE-26400: - Author: ASF GitHub Bot Created on: 19/Jan/23 06:49 Start Date: 19/Jan/23 06:49 Worklog Time Spent: 10m Work Description: dengzhhu653 commented on PR #3448: URL: https://github.com/apache/hive/pull/3448#issuecomment-1396511930 > I had some comments earlier, if you can confirm you addressed those @dengzhhu653 , I'll take a second look and approve, it's time to merge this I believe please create an umbrella ticket for hive docker improvements and add this one as the first sub-jira, let's track further improvements there I've merged the two images into only one(apache/hive), and create a parent jira to track the improvements, please take a look if have time, thank you! Issue Time Tracking --- Worklog Id: (was: 840157) Time Spent: 6h 50m (was: 6h 40m) > Provide docker images for Hive > -- > > Key: HIVE-26400 > URL: https://issues.apache.org/jira/browse/HIVE-26400 > Project: Hive > Issue Type: Sub-task > Components: Build Infrastructure >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Blocker > Labels: hive-4.0.0-must, pull-request-available > Time Spent: 6h 50m > Remaining Estimate: 0h > > Make Apache Hive be able to run inside docker container in pseudo-distributed > mode, with MySQL/Derby as its back database, provide the following: > * Quick-start/Debugging/Prepare a test env for Hive; > * Tools to build target image with specified version of Hive and its > dependencies; > * Images can be used as the basis for the Kubernetes operator. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26400) Provide docker images for Hive
[ https://issues.apache.org/jira/browse/HIVE-26400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated HIVE-26400: --- Parent: HIVE-26965 Issue Type: Sub-task (was: Improvement) > Provide docker images for Hive > -- > > Key: HIVE-26400 > URL: https://issues.apache.org/jira/browse/HIVE-26400 > Project: Hive > Issue Type: Sub-task > Components: Build Infrastructure >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Blocker > Labels: hive-4.0.0-must, pull-request-available > Time Spent: 6h 40m > Remaining Estimate: 0h > > Make Apache Hive be able to run inside docker container in pseudo-distributed > mode, with MySQL/Derby as its back database, provide the following: > * Quick-start/Debugging/Prepare a test env for Hive; > * Tools to build target image with specified version of Hive and its > dependencies; > * Images can be used as the basis for the Kubernetes operator. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26952) set the value of metastore.storage.schema.reader.impl to org.apache.hadoop.hive.metastore.SerDeStorageSchemaReader as default
[
https://issues.apache.org/jira/browse/HIVE-26952?focusedWorklogId=840154=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-840154
]
ASF GitHub Bot logged work on HIVE-26952:
-
Author: ASF GitHub Bot
Created on: 19/Jan/23 06:28
Start Date: 19/Jan/23 06:28
Worklog Time Spent: 10m
Work Description: tarak271 commented on code in PR #3959:
URL: https://github.com/apache/hive/pull/3959#discussion_r1080851725
##
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
##
@@ -67,6 +67,9 @@ public class MetastoreConf {
static final String DEFAULT_STORAGE_SCHEMA_READER_CLASS =
"org.apache.hadoop.hive.metastore.DefaultStorageSchemaReader";
@VisibleForTesting
+ static final String SERDE_STORAGE_SCHEMA_READER_CLASS =
+ "org.apache.hadoop.hive.metastore.SerDeStorageSchemaReader";
Review Comment:
Added assertion about new class in the test case
Issue Time Tracking
---
Worklog Id: (was: 840154)
Time Spent: 50m (was: 40m)
> set the value of metastore.storage.schema.reader.impl
to
> org.apache.hadoop.hive.metastore.SerDeStorageSchemaReader as default
> --
>
> Key: HIVE-26952
> URL: https://issues.apache.org/jira/browse/HIVE-26952
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Reporter: Taraka Rama Rao Lethavadla
>Assignee: Taraka Rama Rao Lethavadla
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> With the default value of
>
> {code:java}
> DefaultStorageSchemaReader.class.getName(){code}
>
> in the Metastore Config, *metastore.storage.schema.reader.impl*
> below exception is thrown when trying to read Avro schema
> {noformat}
> Caused by: org.apache.hive.service.cli.HiveSQLException: MetaException
> (message:java.lang.UnsupportedOperationException: Storage schema reading not
> supported)
> at
> org.apache.hive.service.cli.operation.GetColumnsOperation.runInternal(GetColumnsOperation.java:213)
> at org.apache.hive.service.cli.operation.Operation.run(Operation.java:247)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.getColumns(HiveSessionImpl.java:729)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.access-zsh(HiveSessionProxy.java:36)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.run(HiveSessionProxy.java:63)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
> at com.sun.proxy..getColumns(Unknown Source)
> at
> org.apache.hive.service.cli.CLIService.getColumns(CLIService.java:390){noformat}
> setting the above config with
> *org.apache.hadoop.hive.metastore.SerDeStorageSchemaReader* resolves issue
> Proposing to make this value as default in code base, so that in upcoming
> versions we don't have to set this value manually
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26950) (CTLT) Create external table like V2 table is not preserving table properties
[ https://issues.apache.org/jira/browse/HIVE-26950?focusedWorklogId=840152=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-840152 ] ASF GitHub Bot logged work on HIVE-26950: - Author: ASF GitHub Bot Created on: 19/Jan/23 05:52 Start Date: 19/Jan/23 05:52 Worklog Time Spent: 10m Work Description: ayushtkn opened a new pull request, #3965: URL: https://github.com/apache/hive/pull/3965 ### What changes were proposed in this pull request? Two changes: - Preserve the format version for the iceberg table - Add auto translation logic for iceberg CTLT tables as well(if table created without external keyword, allow that by auto translating to external with purge true) ### Why are the changes needed? Better usability of iceberg tables with CTLT ### Does this PR introduce _any_ user-facing change? Yes, format version preserved for Iceberg CTLT tables & auto translation to external with purge true like other commands for iceberg tables. ### How was this patch tested? UT Issue Time Tracking --- Worklog Id: (was: 840152) Remaining Estimate: 0h Time Spent: 10m > (CTLT) Create external table like V2 table is not preserving table properties > - > > Key: HIVE-26950 > URL: https://issues.apache.org/jira/browse/HIVE-26950 > Project: Hive > Issue Type: Improvement > Components: Iceberg integration >Reporter: Rajesh Balamohan >Assignee: Ayush Saxena >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > # Create an external iceberg V2 table. e.g t1 > # "create external table t2 like t1" <--- This ends up creating V1 table and > "format-version=2" is not retained and "'format'='iceberg/parquet'" is also > not retained. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26950) (CTLT) Create external table like V2 table is not preserving table properties
[ https://issues.apache.org/jira/browse/HIVE-26950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-26950: -- Labels: pull-request-available (was: ) > (CTLT) Create external table like V2 table is not preserving table properties > - > > Key: HIVE-26950 > URL: https://issues.apache.org/jira/browse/HIVE-26950 > Project: Hive > Issue Type: Improvement > Components: Iceberg integration >Reporter: Rajesh Balamohan >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > # Create an external iceberg V2 table. e.g t1 > # "create external table t2 like t1" <--- This ends up creating V1 table and > "format-version=2" is not retained and "'format'='iceberg/parquet'" is also > not retained. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-26711) The very first REPL Load should make the Target Database read-only
[ https://issues.apache.org/jira/browse/HIVE-26711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shreenidhi resolved HIVE-26711. --- Resolution: Fixed > The very first REPL Load should make the Target Database read-only > -- > > Key: HIVE-26711 > URL: https://issues.apache.org/jira/browse/HIVE-26711 > Project: Hive > Issue Type: Task >Reporter: Shreenidhi >Assignee: Shreenidhi >Priority: Major > Labels: pull-request-available > Time Spent: 2h 10m > Remaining Estimate: 0h > > Use EnforceReadOnly hook to set TARGET database read only during BootStrap > load. > Also ensure backward compatibility. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-26598) Fix unsetting of db params for optimized bootstrap when repl dump initiates data copy
[ https://issues.apache.org/jira/browse/HIVE-26598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rakshith C resolved HIVE-26598. --- Resolution: Fixed > Fix unsetting of db params for optimized bootstrap when repl dump initiates > data copy > - > > Key: HIVE-26598 > URL: https://issues.apache.org/jira/browse/HIVE-26598 > Project: Hive > Issue Type: Bug >Reporter: Teddy Choi >Assignee: Rakshith C >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > when hive.repl.run.data.copy.tasks.on.target is set to false, repl dump task > will initiate the copy task from source cluster to staging directory. > In current code flow repl dump task dumps the metadata and then creates > another repl dump task with datacopyIterators initialized. > when the second dump cycle executes, it directly begins data copy tasks. > Because of this we don't enter second reverse dump flow and > unsetDbPropertiesForOptimisedBootstrap is never set to true again. > this results in db params (repl.target.for, repl.background.threads, etc) not > being unset. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26942) Prevent updation of "repl.target.last.id" when RESUME action initiated
[ https://issues.apache.org/jira/browse/HIVE-26942?focusedWorklogId=840151=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-840151 ] ASF GitHub Bot logged work on HIVE-26942: - Author: ASF GitHub Bot Created on: 19/Jan/23 05:17 Start Date: 19/Jan/23 05:17 Worklog Time Spent: 10m Work Description: pudidic commented on PR #3951: URL: https://github.com/apache/hive/pull/3951#issuecomment-1396452258 There are several tests failing. Please check them. Issue Time Tracking --- Worklog Id: (was: 840151) Time Spent: 50m (was: 40m) > Prevent updation of "repl.target.last.id" when RESUME action initiated > -- > > Key: HIVE-26942 > URL: https://issues.apache.org/jira/browse/HIVE-26942 > Project: Hive > Issue Type: Task >Reporter: Shreenidhi >Assignee: Shreenidhi >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > *Problem :* > Currently when "repl.last.id" get's changed/updated this updates the > corresponding "repl.target.last.id" too. This is problematic incase of resume > because we are updating both the properties at source. > > *Solution :* > To prevent the updation of "repl.target.last.id" whenever "repl.last.id" > changed, we can introduce new db param called "repl.resume.started". This new > param will be set to "true" by orchestrator when RESUME is initiated. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26598) Fix unsetting of db params for optimized bootstrap when repl dump initiates data copy
[ https://issues.apache.org/jira/browse/HIVE-26598?focusedWorklogId=840150=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-840150 ] ASF GitHub Bot logged work on HIVE-26598: - Author: ASF GitHub Bot Created on: 19/Jan/23 05:16 Start Date: 19/Jan/23 05:16 Worklog Time Spent: 10m Work Description: pudidic merged PR #3780: URL: https://github.com/apache/hive/pull/3780 Issue Time Tracking --- Worklog Id: (was: 840150) Time Spent: 1h 10m (was: 1h) > Fix unsetting of db params for optimized bootstrap when repl dump initiates > data copy > - > > Key: HIVE-26598 > URL: https://issues.apache.org/jira/browse/HIVE-26598 > Project: Hive > Issue Type: Bug >Reporter: Teddy Choi >Assignee: Rakshith C >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > when hive.repl.run.data.copy.tasks.on.target is set to false, repl dump task > will initiate the copy task from source cluster to staging directory. > In current code flow repl dump task dumps the metadata and then creates > another repl dump task with datacopyIterators initialized. > when the second dump cycle executes, it directly begins data copy tasks. > Because of this we don't enter second reverse dump flow and > unsetDbPropertiesForOptimisedBootstrap is never set to true again. > this results in db params (repl.target.for, repl.background.threads, etc) not > being unset. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26598) Fix unsetting of db params for optimized bootstrap when repl dump initiates data copy
[ https://issues.apache.org/jira/browse/HIVE-26598?focusedWorklogId=840149=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-840149 ] ASF GitHub Bot logged work on HIVE-26598: - Author: ASF GitHub Bot Created on: 19/Jan/23 05:15 Start Date: 19/Jan/23 05:15 Worklog Time Spent: 10m Work Description: pudidic commented on PR #3780: URL: https://github.com/apache/hive/pull/3780#issuecomment-1396451284 LGTM +1. I'll merge it. Issue Time Tracking --- Worklog Id: (was: 840149) Time Spent: 1h (was: 50m) > Fix unsetting of db params for optimized bootstrap when repl dump initiates > data copy > - > > Key: HIVE-26598 > URL: https://issues.apache.org/jira/browse/HIVE-26598 > Project: Hive > Issue Type: Bug >Reporter: Teddy Choi >Assignee: Rakshith C >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > when hive.repl.run.data.copy.tasks.on.target is set to false, repl dump task > will initiate the copy task from source cluster to staging directory. > In current code flow repl dump task dumps the metadata and then creates > another repl dump task with datacopyIterators initialized. > when the second dump cycle executes, it directly begins data copy tasks. > Because of this we don't enter second reverse dump flow and > unsetDbPropertiesForOptimisedBootstrap is never set to true again. > this results in db params (repl.target.for, repl.background.threads, etc) not > being unset. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26711) The very first REPL Load should make the Target Database read-only
[ https://issues.apache.org/jira/browse/HIVE-26711?focusedWorklogId=840148=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-840148 ] ASF GitHub Bot logged work on HIVE-26711: - Author: ASF GitHub Bot Created on: 19/Jan/23 05:15 Start Date: 19/Jan/23 05:15 Worklog Time Spent: 10m Work Description: pudidic merged PR #3736: URL: https://github.com/apache/hive/pull/3736 Issue Time Tracking --- Worklog Id: (was: 840148) Time Spent: 2h 10m (was: 2h) > The very first REPL Load should make the Target Database read-only > -- > > Key: HIVE-26711 > URL: https://issues.apache.org/jira/browse/HIVE-26711 > Project: Hive > Issue Type: Task >Reporter: Shreenidhi >Assignee: Shreenidhi >Priority: Major > Labels: pull-request-available > Time Spent: 2h 10m > Remaining Estimate: 0h > > Use EnforceReadOnly hook to set TARGET database read only during BootStrap > load. > Also ensure backward compatibility. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26711) The very first REPL Load should make the Target Database read-only
[ https://issues.apache.org/jira/browse/HIVE-26711?focusedWorklogId=840147=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-840147 ] ASF GitHub Bot logged work on HIVE-26711: - Author: ASF GitHub Bot Created on: 19/Jan/23 05:14 Start Date: 19/Jan/23 05:14 Worklog Time Spent: 10m Work Description: pudidic commented on PR #3736: URL: https://github.com/apache/hive/pull/3736#issuecomment-1396450347 LGTM +1. I'll merge it. Issue Time Tracking --- Worklog Id: (was: 840147) Time Spent: 2h (was: 1h 50m) > The very first REPL Load should make the Target Database read-only > -- > > Key: HIVE-26711 > URL: https://issues.apache.org/jira/browse/HIVE-26711 > Project: Hive > Issue Type: Task >Reporter: Shreenidhi >Assignee: Shreenidhi >Priority: Major > Labels: pull-request-available > Time Spent: 2h > Remaining Estimate: 0h > > Use EnforceReadOnly hook to set TARGET database read only during BootStrap > load. > Also ensure backward compatibility. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-26922) Deadlock when rebuilding Materialized view stored by Iceberg
[
https://issues.apache.org/jira/browse/HIVE-26922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa resolved HIVE-26922.
---
Resolution: Fixed
Merged to master. Thanks [~amansinha100], [~zabetak], [~sjc362000] for review.
> Deadlock when rebuilding Materialized view stored by Iceberg
>
>
> Key: HIVE-26922
> URL: https://issues.apache.org/jira/browse/HIVE-26922
> Project: Hive
> Issue Type: Bug
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> {code}
> create table tbl_ice(a int, b string, c int) stored by iceberg stored as orc
> tblproperties ('format-version'='1');
> insert into tbl_ice values (1, 'one', 50), (2, 'two', 51), (3, 'three', 52),
> (4, 'four', 53), (5, 'five', 54);
> create materialized view mat1 stored by iceberg stored as orc tblproperties
> ('format-version'='1') as
> select tbl_ice.b, tbl_ice.c from tbl_ice where tbl_ice.c > 52;
> insert into tbl_ice values (10, 'ten', 60);
> alter materialized view mat1 rebuild;
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26922) Deadlock when rebuilding Materialized view stored by Iceberg
[
https://issues.apache.org/jira/browse/HIVE-26922?focusedWorklogId=840146=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-840146
]
ASF GitHub Bot logged work on HIVE-26922:
-
Author: ASF GitHub Bot
Created on: 19/Jan/23 05:08
Start Date: 19/Jan/23 05:08
Worklog Time Spent: 10m
Work Description: kasakrisz merged PR #3934:
URL: https://github.com/apache/hive/pull/3934
Issue Time Tracking
---
Worklog Id: (was: 840146)
Time Spent: 2.5h (was: 2h 20m)
> Deadlock when rebuilding Materialized view stored by Iceberg
>
>
> Key: HIVE-26922
> URL: https://issues.apache.org/jira/browse/HIVE-26922
> Project: Hive
> Issue Type: Bug
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> {code}
> create table tbl_ice(a int, b string, c int) stored by iceberg stored as orc
> tblproperties ('format-version'='1');
> insert into tbl_ice values (1, 'one', 50), (2, 'two', 51), (3, 'three', 52),
> (4, 'four', 53), (5, 'five', 54);
> create materialized view mat1 stored by iceberg stored as orc tblproperties
> ('format-version'='1') as
> select tbl_ice.b, tbl_ice.c from tbl_ice where tbl_ice.c > 52;
> insert into tbl_ice values (10, 'ten', 60);
> alter materialized view mat1 rebuild;
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26808) Port Iceberg catalog changes
[ https://issues.apache.org/jira/browse/HIVE-26808?focusedWorklogId=840141=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-840141 ] ASF GitHub Bot logged work on HIVE-26808: - Author: ASF GitHub Bot Created on: 19/Jan/23 02:57 Start Date: 19/Jan/23 02:57 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3907: URL: https://github.com/apache/hive/pull/3907#issuecomment-1396377426 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3907) [](https://sonarcloud.io/project/issues?id=apache_hive=3907=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3907=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3907=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3907=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3907=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3907=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3907=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3907=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3907=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3907=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3907=false=CODE_SMELL) [2 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3907=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3907=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3907=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 840141) Time Spent: 1h 40m (was: 1.5h) > Port Iceberg catalog changes > > > Key: HIVE-26808 > URL: https://issues.apache.org/jira/browse/HIVE-26808 > Project: Hive > Issue Type: Improvement > Components: Iceberg integration >Reporter: Zsolt Miskolczi >Assignee: Zsolt Miskolczi >Priority: Major > Labels: pull-request-available > Time Spent: 1h 40m > Remaining Estimate: 0h > > The last round of porting happened in 2022 april, there were a couple of > changes especially in HiveTableOperations worth porting into iceberg-catalog. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26928) LlapIoImpl::getParquetFooterBuffersFromCache throws exception when metadata cache is disabled
[
https://issues.apache.org/jira/browse/HIVE-26928?focusedWorklogId=840138=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-840138
]
ASF GitHub Bot logged work on HIVE-26928:
-
Author: ASF GitHub Bot
Created on: 19/Jan/23 01:47
Start Date: 19/Jan/23 01:47
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3962:
URL: https://github.com/apache/hive/pull/3962#issuecomment-1396336653
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3962)
[](https://sonarcloud.io/project/issues?id=apache_hive=3962=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3962=false=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3962=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3962=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3962=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3962=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3962=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3962=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3962=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3962=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3962=false=CODE_SMELL)
[0 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3962=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3962=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3962=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 840138)
Time Spent: 50m (was: 40m)
> LlapIoImpl::getParquetFooterBuffersFromCache throws exception when metadata
> cache is disabled
> -
>
> Key: HIVE-26928
> URL: https://issues.apache.org/jira/browse/HIVE-26928
> Project: Hive
> Issue Type: Improvement
> Components: Iceberg integration
>Reporter: Rajesh Balamohan
>Assignee: Simhadri Govindappa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> When metadata / LLAP cache is disabled, "iceberg + parquet" throws the
> following error. "{color:#5a656d}hive.llap.io.memory.mode=none"{color}
> It should check for "metadatacache" correctly or fix it in LlapIoImpl.
>
> {noformat}
> Caused by: java.lang.NullPointerException: Metadata cache must not be null
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:897)
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapIoImpl.getParquetFooterBuffersFromCache(LlapIoImpl.java:467)
> at
> org.apache.iceberg.mr.hive.vector.HiveVectorizedReader.parquetRecordReader(HiveVectorizedReader.java:227)
> at
> org.apache.iceberg.mr.hive.vector.HiveVectorizedReader.reader(HiveVectorizedReader.java:162)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
>
[jira] [Work logged] (HIVE-26947) Hive compactor.Worker can respawn connections to HMS at extremely high frequency
[ https://issues.apache.org/jira/browse/HIVE-26947?focusedWorklogId=840134=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-840134 ] ASF GitHub Bot logged work on HIVE-26947: - Author: ASF GitHub Bot Created on: 19/Jan/23 00:57 Start Date: 19/Jan/23 00:57 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3955: URL: https://github.com/apache/hive/pull/3955#issuecomment-1396304048 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3955) [](https://sonarcloud.io/project/issues?id=apache_hive=3955=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3955=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3955=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3955=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3955=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3955=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3955=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3955=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3955=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3955=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3955=false=CODE_SMELL) [10 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3955=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3955=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3955=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 840134) Time Spent: 1h 20m (was: 1h 10m) > Hive compactor.Worker can respawn connections to HMS at extremely high > frequency > > > Key: HIVE-26947 > URL: https://issues.apache.org/jira/browse/HIVE-26947 > Project: Hive > Issue Type: Bug >Reporter: Akshat Mathur >Assignee: Akshat Mathur >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > After catching the exception generated by the findNextCompactionAndExecute() > task, HS2 appears to immediately rerun the task with no delay or backoff. As > a result there are ~3500 connection attempts from HS2 to HMS over just a 5 > second period in the HS2 log > The compactor.Worker should wait between failed attempts and maybe do an > exponential backoff. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26922) Deadlock when rebuilding Materialized view stored by Iceberg
[
https://issues.apache.org/jira/browse/HIVE-26922?focusedWorklogId=840117=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-840117
]
ASF GitHub Bot logged work on HIVE-26922:
-
Author: ASF GitHub Bot
Created on: 18/Jan/23 23:39
Start Date: 18/Jan/23 23:39
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3934:
URL: https://github.com/apache/hive/pull/3934#issuecomment-1396235662
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3934)
[](https://sonarcloud.io/project/issues?id=apache_hive=3934=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3934=false=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3934=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3934=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3934=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3934=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3934=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3934=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3934=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3934=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3934=false=CODE_SMELL)
[0 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3934=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3934=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3934=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 840117)
Time Spent: 2h 20m (was: 2h 10m)
> Deadlock when rebuilding Materialized view stored by Iceberg
>
>
> Key: HIVE-26922
> URL: https://issues.apache.org/jira/browse/HIVE-26922
> Project: Hive
> Issue Type: Bug
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> {code}
> create table tbl_ice(a int, b string, c int) stored by iceberg stored as orc
> tblproperties ('format-version'='1');
> insert into tbl_ice values (1, 'one', 50), (2, 'two', 51), (3, 'three', 52),
> (4, 'four', 53), (5, 'five', 54);
> create materialized view mat1 stored by iceberg stored as orc tblproperties
> ('format-version'='1') as
> select tbl_ice.b, tbl_ice.c from tbl_ice where tbl_ice.c > 52;
> insert into tbl_ice values (10, 'ten', 60);
> alter materialized view mat1 rebuild;
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-22977) Merge delta files instead of running a query in major/minor compaction
[ https://issues.apache.org/jira/browse/HIVE-22977?focusedWorklogId=840104=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-840104 ] ASF GitHub Bot logged work on HIVE-22977: - Author: ASF GitHub Bot Created on: 18/Jan/23 22:06 Start Date: 18/Jan/23 22:06 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3801: URL: https://github.com/apache/hive/pull/3801#issuecomment-1396155011 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3801) [](https://sonarcloud.io/project/issues?id=apache_hive=3801=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3801=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3801=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3801=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3801=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3801=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3801=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3801=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3801=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3801=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3801=false=CODE_SMELL) [2 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3801=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3801=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3801=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 840104) Time Spent: 5h (was: 4h 50m) > Merge delta files instead of running a query in major/minor compaction > -- > > Key: HIVE-22977 > URL: https://issues.apache.org/jira/browse/HIVE-22977 > Project: Hive > Issue Type: Improvement >Reporter: László Pintér >Assignee: Sourabh Badhya >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22977.01.patch, HIVE-22977.02.patch > > Time Spent: 5h > Remaining Estimate: 0h > > [Compaction Optimiziation] > We should analyse the possibility to move a delta file instead of running a > major/minor compaction query. > Please consider the following use cases: > - full acid table but only insert queries were run. This means that no > delete delta directories were created. Is it possible to merge the delta > directory contents without running a compaction query? > - full acid table, initiating queries through the streaming API. If there > are no abort transactions during the streaming, is it possible to merge the > delta directory contents without running a compaction query? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26808) Port Iceberg catalog changes
[ https://issues.apache.org/jira/browse/HIVE-26808?focusedWorklogId=840091=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-840091 ] ASF GitHub Bot logged work on HIVE-26808: - Author: ASF GitHub Bot Created on: 18/Jan/23 21:20 Start Date: 18/Jan/23 21:20 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3907: URL: https://github.com/apache/hive/pull/3907#issuecomment-1396102462 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3907) [](https://sonarcloud.io/project/issues?id=apache_hive=3907=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3907=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3907=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3907=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3907=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3907=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3907=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3907=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3907=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3907=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3907=false=CODE_SMELL) [2 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3907=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3907=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3907=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 840091) Time Spent: 1.5h (was: 1h 20m) > Port Iceberg catalog changes > > > Key: HIVE-26808 > URL: https://issues.apache.org/jira/browse/HIVE-26808 > Project: Hive > Issue Type: Improvement > Components: Iceberg integration >Reporter: Zsolt Miskolczi >Assignee: Zsolt Miskolczi >Priority: Major > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > The last round of porting happened in 2022 april, there were a couple of > changes especially in HiveTableOperations worth porting into iceberg-catalog. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26598) Fix unsetting of db params for optimized bootstrap when repl dump initiates data copy
[ https://issues.apache.org/jira/browse/HIVE-26598?focusedWorklogId=840075=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-840075 ] ASF GitHub Bot logged work on HIVE-26598: - Author: ASF GitHub Bot Created on: 18/Jan/23 20:03 Start Date: 18/Jan/23 20:03 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3780: URL: https://github.com/apache/hive/pull/3780#issuecomment-1387703835 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3780) [](https://sonarcloud.io/project/issues?id=apache_hive=3780=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3780=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3780=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3780=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3780=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3780=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3780=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3780=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3780=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3780=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3780=false=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3780=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3780=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3780=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 840075) Time Spent: 50m (was: 40m) > Fix unsetting of db params for optimized bootstrap when repl dump initiates > data copy > - > > Key: HIVE-26598 > URL: https://issues.apache.org/jira/browse/HIVE-26598 > Project: Hive > Issue Type: Bug >Reporter: Teddy Choi >Assignee: Rakshith C >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > when hive.repl.run.data.copy.tasks.on.target is set to false, repl dump task > will initiate the copy task from source cluster to staging directory. > In current code flow repl dump task dumps the metadata and then creates > another repl dump task with datacopyIterators initialized. > when the second dump cycle executes, it directly begins data copy tasks. > Because of this we don't enter second reverse dump flow and > unsetDbPropertiesForOptimisedBootstrap is never set to true again. > this results in db params (repl.target.for, repl.background.threads, etc) not > being unset. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26945) Test fixes for query*.q files
[ https://issues.apache.org/jira/browse/HIVE-26945?focusedWorklogId=840053=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-840053 ] ASF GitHub Bot logged work on HIVE-26945: - Author: ASF GitHub Bot Created on: 18/Jan/23 18:58 Start Date: 18/Jan/23 18:58 Worklog Time Spent: 10m Work Description: amanraj2520 commented on PR #3954: URL: https://github.com/apache/hive/pull/3954#issuecomment-1387598230 Hi @abstractdog these tests failed due to HIVE-20423. Link : https://issues.apache.org/jira/browse/HIVE-20423 Issue Time Tracking --- Worklog Id: (was: 840053) Time Spent: 40m (was: 0.5h) > Test fixes for query*.q files > - > > Key: HIVE-26945 > URL: https://issues.apache.org/jira/browse/HIVE-26945 > Project: Hive > Issue Type: Sub-task >Reporter: Aman Raj >Assignee: Aman Raj >Priority: Critical > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > The tests has outdated q.out files which need to be updated. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26955) Select query fails when decimal column data type is changed to string/char/varchar in Parquet
[
https://issues.apache.org/jira/browse/HIVE-26955?focusedWorklogId=840048=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-840048
]
ASF GitHub Bot logged work on HIVE-26955:
-
Author: ASF GitHub Bot
Created on: 18/Jan/23 18:51
Start Date: 18/Jan/23 18:51
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3964:
URL: https://github.com/apache/hive/pull/3964#issuecomment-1387581260
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3964)
[](https://sonarcloud.io/project/issues?id=apache_hive=3964=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3964=false=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3964=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3964=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3964=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3964=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3964=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3964=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3964=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3964=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3964=false=CODE_SMELL)
[1 Code
Smell](https://sonarcloud.io/project/issues?id=apache_hive=3964=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3964=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3964=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 840048)
Time Spent: 50m (was: 40m)
> Select query fails when decimal column data type is changed to
> string/char/varchar in Parquet
> -
>
> Key: HIVE-26955
> URL: https://issues.apache.org/jira/browse/HIVE-26955
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Taraka Rama Rao Lethavadla
>Assignee: Sourabh Badhya
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Steps to reproduce
> {noformat}
> create table test_parquet (id decimal) stored as parquet;
> insert into test_parquet values(238);
> alter table test_parquet change id id string;
> select * from test_parquet;
> Error: java.io.IOException: org.apache.parquet.io.ParquetDecodingException:
> Can not read value at 1 in block 0 in file
> hdfs:/namenode:8020/warehouse/tablespace/managed/hive/test_parquet/delta_001_001_/00_0
> (state=,code=0)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:624)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:531)
> at
> org.apache.hadoop.hive.ql.exec.FetchTask.executeInner(FetchTask.java:194)
> ... 55 more
> Caused by: org.apache.parquet.io.ParquetDecodingException: Can not read value
> at 1 in block 0 in file
>
[jira] [Work logged] (HIVE-26939) Hive LLAP Application Master fails to come up with Hadoop 3.3.4
[
https://issues.apache.org/jira/browse/HIVE-26939?focusedWorklogId=840044=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-840044
]
ASF GitHub Bot logged work on HIVE-26939:
-
Author: ASF GitHub Bot
Created on: 18/Jan/23 18:43
Start Date: 18/Jan/23 18:43
Worklog Time Spent: 10m
Work Description: amanraj2520 commented on PR #3941:
URL: https://github.com/apache/hive/pull/3941#issuecomment-1387560380
@abstractdog Can you please review this
Issue Time Tracking
---
Worklog Id: (was: 840044)
Time Spent: 1.5h (was: 1h 20m)
> Hive LLAP Application Master fails to come up with Hadoop 3.3.4
> ---
>
> Key: HIVE-26939
> URL: https://issues.apache.org/jira/browse/HIVE-26939
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Aman Raj
>Assignee: Aman Raj
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> When current oss master hive tries to bring up the LLAP Application Master,
> it fails with this issue :
> {code:java}
> Executing the launch command\nINFO client.ServiceClient: Loading service
> definition from local FS:
> /var/lib/ambari-agent/tmp/llap-yarn-service_2023-01-10_07-56-46/Yarnfile\nERROR
> utils.JsonSerDeser: Exception while parsing json input
> stream\ncom.fasterxml.jackson.databind.exc.InvalidFormatException: Cannot
> deserialize value of type
> `org.apache.hadoop.yarn.service.api.records.PlacementScope` from String
> \"NODE\": not one of the values accepted for Enum class: [node, rack]\n at
> [Source: (org.apache.hadoop.fs.ChecksumFileSystem$FSDataBoundedInputStream);
> line: 31, column: 22] (through reference chain:
> org.apache.hadoop.yarn.service.api.records.Service[\"components\"]->java.util.ArrayList[0]->org.apache.hadoop.yarn.service.api.records.Component[\"placement_policy\"]->org.apache.hadoop.yarn.service.api.records.PlacementPolicy[\"constraints\"]->java.util.ArrayList[0]->org.apache.hadoop.yarn.service.api.records.PlacementConstraint[\"scope\"])\n\tat
>
> com.fasterxml.jackson.databind.exc.InvalidFormatException.from(InvalidFormatException.java:67)
> ~[jackson-databind-2.12.7.jar:2.12.7]\n\tat
> com.fasterxml.jackson.databind.DeserializationContext.weirdStringException(DeserializationContext.java:1851)
> ~[jackson-databind-2.12.7.jar:2.12.7]\n\tat
> com.fasterxml.jackson.databind.DeserializationContext.handleWeirdStringValue(DeserializationContext.java:1079)
> ~[jackson-databind-2.12.7.jar:2.12.7]\n\tat
> com.fasterxml.jackson.databind.deser.std.EnumDeserializer._deserializeAltString(EnumDeserializer.java:339)
> ~[jackson-databind-2.12.7.jar:2.12.7]\n\tat
> com.fasterxml.jackson.databind.deser.std.EnumDeserializer._fromString(EnumDeserializer.java:214)
> ~[jackson-databind-2.12.7.jar:2.12.7]\n\tat
> com.fasterxml.jackson.databind.deser.std.EnumDeserializer.deserialize(EnumDeserializer.java:188)
> ~[jackson-databind-2.12.7.jar:2.12.7]\n\tat
> com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:129)
> ~[jackson-databind-2.12.7.jar:2.12.7]\n\tat
> com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:324)
> ~[jackson-databind-2.12.7.jar:2.12.7]\n\tat
> com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:187)
> ~[jackson-databind-2.12.7.jar:2.12.7]\n\tat
> com.fasterxml.jackson.databind.deser.std.CollectionDeserializer._deserializeFromArray(CollectionDeserializer.java:355)
> ~[jackson-databind-2.12.7.jar:2.12.7]\n\tat
> com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:244)
> ~[jackson-databind-2.12.7.jar:2.12.7]\n\tat
> com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:28)
> ~[jackson-databind-2.12.7.jar:2.12.7]\n\tat
> com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:129)
> ~[jackson-databind-2.12.7.jar:2.12.7]\n\tat
> com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:324)
> ~[jackson-databind-2.12.7.jar:2.12.7]\n\tat
> com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:187)
> ~[jackson-databind-2.12.7.jar:2.12.7]\n\tat
> com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:129)
> ~[jackson-databind-2.12.7.jar:2.12.7]\n\tat
> com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:324)
> ~[jackson-databind-2.12.7.jar:2.12.7]\n\tat
>
[jira] [Work logged] (HIVE-26928) LlapIoImpl::getParquetFooterBuffersFromCache throws exception when metadata cache is disabled
[
https://issues.apache.org/jira/browse/HIVE-26928?focusedWorklogId=840022=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-840022
]
ASF GitHub Bot logged work on HIVE-26928:
-
Author: ASF GitHub Bot
Created on: 18/Jan/23 16:26
Start Date: 18/Jan/23 16:26
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3962:
URL: https://github.com/apache/hive/pull/3962#issuecomment-1387348148
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3962)
[](https://sonarcloud.io/project/issues?id=apache_hive=3962=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3962=false=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3962=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3962=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3962=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3962=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3962=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3962=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3962=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3962=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3962=false=CODE_SMELL)
[0 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3962=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3962=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3962=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 840022)
Time Spent: 40m (was: 0.5h)
> LlapIoImpl::getParquetFooterBuffersFromCache throws exception when metadata
> cache is disabled
> -
>
> Key: HIVE-26928
> URL: https://issues.apache.org/jira/browse/HIVE-26928
> Project: Hive
> Issue Type: Improvement
> Components: Iceberg integration
>Reporter: Rajesh Balamohan
>Assignee: Simhadri Govindappa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> When metadata / LLAP cache is disabled, "iceberg + parquet" throws the
> following error. "{color:#5a656d}hive.llap.io.memory.mode=none"{color}
> It should check for "metadatacache" correctly or fix it in LlapIoImpl.
>
> {noformat}
> Caused by: java.lang.NullPointerException: Metadata cache must not be null
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:897)
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapIoImpl.getParquetFooterBuffersFromCache(LlapIoImpl.java:467)
> at
> org.apache.iceberg.mr.hive.vector.HiveVectorizedReader.parquetRecordReader(HiveVectorizedReader.java:227)
> at
> org.apache.iceberg.mr.hive.vector.HiveVectorizedReader.reader(HiveVectorizedReader.java:162)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
>
[jira] [Resolved] (HIVE-26681) Upgrade dom4j: flexible XML framework for Java to safe version due to critical CVEs
[ https://issues.apache.org/jira/browse/HIVE-26681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam resolved HIVE-26681. -- Fix Version/s: 4.0.0 Resolution: Fixed Fix has been merged to master. Thank you for the patch [~devaspatikrishnatri] and review [~cnauroth] > Upgrade dom4j: flexible XML framework for Java to safe version due to > critical CVEs > --- > > Key: HIVE-26681 > URL: https://issues.apache.org/jira/browse/HIVE-26681 > Project: Hive > Issue Type: Task >Reporter: Devaspati Krishnatri >Assignee: Devaspati Krishnatri >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26681) Upgrade dom4j: flexible XML framework for Java to safe version due to critical CVEs
[ https://issues.apache.org/jira/browse/HIVE-26681?focusedWorklogId=840020=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-840020 ] ASF GitHub Bot logged work on HIVE-26681: - Author: ASF GitHub Bot Created on: 18/Jan/23 16:20 Start Date: 18/Jan/23 16:20 Worklog Time Spent: 10m Work Description: nrg4878 merged PR #3716: URL: https://github.com/apache/hive/pull/3716 Issue Time Tracking --- Worklog Id: (was: 840020) Time Spent: 50m (was: 40m) > Upgrade dom4j: flexible XML framework for Java to safe version due to > critical CVEs > --- > > Key: HIVE-26681 > URL: https://issues.apache.org/jira/browse/HIVE-26681 > Project: Hive > Issue Type: Task >Reporter: Devaspati Krishnatri >Assignee: Devaspati Krishnatri >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26925) MV with iceberg storage format fails when contains 'PARTITIONED ON' clause due to column number/types difference.
[
https://issues.apache.org/jira/browse/HIVE-26925?focusedWorklogId=840009=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-840009
]
ASF GitHub Bot logged work on HIVE-26925:
-
Author: ASF GitHub Bot
Created on: 18/Jan/23 15:57
Start Date: 18/Jan/23 15:57
Worklog Time Spent: 10m
Work Description: zabetak commented on code in PR #3939:
URL: https://github.com/apache/hive/pull/3939#discussion_r1073656566
##
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/create/CreateTableDesc.java:
##
@@ -958,6 +940,31 @@ public Table toTable(HiveConf conf) throws HiveException {
return tbl;
}
+ public static void setColumnsAndStorePartitionTransformSpec(
Review Comment:
Consider moving the method in `DDLUtils` or `AlterTableUtils`.
##
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/create/CreateTableDesc.java:
##
@@ -958,6 +940,31 @@ public Table toTable(HiveConf conf) throws HiveException {
return tbl;
}
+ public static void setColumnsAndStorePartitionTransformSpec(
+ List columns, List partitionColumns,
+ HiveConf conf, Table tbl, HiveStorageHandler storageHandler)
Review Comment:
`storageHandler` can be obtained from `tbl` so we could omit this argument.
##
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:
##
@@ -13720,15 +13720,7 @@ ASTNode analyzeCreateTable(
}
break;
case HiveParser.TOK_TABLEPARTCOLSBYSPEC:
-List partitionTransformSpec =
-PartitionTransform.getPartitionTransformSpec(child);
-
-if (!SessionStateUtil.addResource(conf,
hive_metastoreConstants.PARTITION_TRANSFORM_SPEC,
-partitionTransformSpec)) {
- throw new SemanticException("Query state attached to Session state
must be not null. " +
- "Partition transform metadata cannot be saved.");
-}
-
Review Comment:
Since the `if` block appears in at least three places consider refactoring
it independently. It could be done as a more generic method in
`SessionStateUtil`.
```java
public static void addResourceOrThrow(Configuration conf, String key, Object
resource) {
Optional queryState = getQueryState(conf);
if (queryState.isPresent()) {
queryState.get().addResource(key, resource);
} else {
throw new IllegalStateException("Query state is missing; failed to add
resource for " + key);
}
}
```
Then its up to you if you want to introduce
`parseAndStorePartitionTransformSpec` or not.
##
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:
##
@@ -14190,11 +14200,17 @@ protected ASTNode analyzeCreateView(ASTNode ast, QB
qb, PlannerContext plannerCt
storageFormat.getSerdeProps());
}
break;
+ case HiveParser.TOK_TABLEPARTCOLSBYSPEC:
+parseAndStorePartitionTransformSpec(child);
+partitionTransformSpecExists = true;
+break;
default:
assert false;
}
}
+validateStorageFormat(storageFormat, tblProps,
partitionTransformSpecExists);
Review Comment:
In order to test completely this code path it may be useful to add some
negative tests with views and unsupported partition by clauses.
##
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/create/CreateTableDesc.java:
##
@@ -958,6 +940,31 @@ public Table toTable(HiveConf conf) throws HiveException {
return tbl;
}
+ public static void setColumnsAndStorePartitionTransformSpec(
Review Comment:
Consider including `table` somewhere in the method name to better reflect
its purpose.
Issue Time Tracking
---
Worklog Id: (was: 840009)
Time Spent: 2h (was: 1h 50m)
> MV with iceberg storage format fails when contains 'PARTITIONED ON' clause
> due to column number/types difference.
> -
>
> Key: HIVE-26925
> URL: https://issues.apache.org/jira/browse/HIVE-26925
> Project: Hive
> Issue Type: Bug
> Components: Iceberg integration
>Reporter: Dharmik Thakkar
>Assignee: Krisztian Kasa
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> MV with iceberg storage format fails when contains 'PARTITIONED ON' clause
> due to column number/types difference.
> {code:java}
> !!! annotations iceberg
> >>> use iceberg_test_db_hive;
> No rows affected
> >>> set hive.exec.max.dynamic.partitions=2000;
> >>> set hive.exec.max.dynamic.partitions.pernode=2000;
> >>> drop materialized view if exists mv_agg_gby_col_partitioned;
> >>> create materialized view
[jira] [Work logged] (HIVE-26924) Alter materialized view enable rewrite throws SemanticException for source iceberg table
[
https://issues.apache.org/jira/browse/HIVE-26924?focusedWorklogId=839998=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-839998
]
ASF GitHub Bot logged work on HIVE-26924:
-
Author: ASF GitHub Bot
Created on: 18/Jan/23 14:58
Start Date: 18/Jan/23 14:58
Worklog Time Spent: 10m
Work Description: kasakrisz merged PR #3936:
URL: https://github.com/apache/hive/pull/3936
Issue Time Tracking
---
Worklog Id: (was: 839998)
Time Spent: 1.5h (was: 1h 20m)
> Alter materialized view enable rewrite throws SemanticException for source
> iceberg table
>
>
> Key: HIVE-26924
> URL: https://issues.apache.org/jira/browse/HIVE-26924
> Project: Hive
> Issue Type: Bug
> Components: Iceberg integration
>Reporter: Dharmik Thakkar
>Assignee: Krisztian Kasa
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> alter materialized view enable rewrite throws SemanticException for source
> iceberg table
> SQL test
> {code:java}
> >>> create materialized view mv_rewrite as select t, si from all100k where
> >>> t>115;
> >>> analyze table mv_rewrite compute statistics for columns;
> >>> set hive.explain.user=false;
> >>> explain select si,t from all100k where t>116 and t<120;
> !!! match row_contains
> alias: iceberg_test_db_hive.mv_rewrite
> >>> alter materialized view mv_rewrite disable rewrite;
> >>> explain select si,t from all100k where t>116 and t<120;
> !!! match row_contains
> alias: all100k
> >>> alter materialized view mv_rewrite enable rewrite;
> >>> explain select si,t from all100k where t>116 and t<120;
> !!! match row_contains
> alias: iceberg_test_db_hive.mv_rewrite
> >>> drop materialized view mv_rewrite; {code}
>
> Error
> {code:java}
> 2023-01-10T18:40:34,303 INFO [pool-3-thread-1] jdbc.TestDriver: Query: alter
> materialized view mv_rewrite enable rewrite
> 2023-01-10T18:40:34,365 INFO [Thread-10] jdbc.TestDriver: INFO : Compiling
> command(queryId=hive_20230110184034_f557b4a6-40a0-42ba-8e67-2f273f50af36):
> alter materialized view mv_rewrite enable rewrite
> 2023-01-10T18:40:34,426 INFO [Thread-10] jdbc.TestDriver: ERROR : FAILED:
> SemanticException Automatic rewriting for materialized view cannot be enabled
> if the materialized view uses non-transactional tables
> 2023-01-10T18:40:34,426 INFO [Thread-10] jdbc.TestDriver:
> org.apache.hadoop.hive.ql.parse.SemanticException: Automatic rewriting for
> materialized view cannot be enabled if the materialized view uses
> non-transactional tables
> 2023-01-10T18:40:34,426 INFO [Thread-10] jdbc.TestDriver: at
> org.apache.hadoop.hive.ql.ddl.view.materialized.alter.rewrite.AlterMaterializedViewRewriteAnalyzer.analyzeInternal(AlterMaterializedViewRewriteAnalyzer.java:75)
> 2023-01-10T18:40:34,426 INFO [Thread-10] jdbc.TestDriver: at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:313)
> 2023-01-10T18:40:34,427 INFO [Thread-10] jdbc.TestDriver: at
> org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:222)
> 2023-01-10T18:40:34,427 INFO [Thread-10] jdbc.TestDriver: at
> org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:105)
> 2023-01-10T18:40:34,427 INFO [Thread-10] jdbc.TestDriver: at
> org.apache.hadoop.hive.ql.Driver.compile(Driver.java:201)
> 2023-01-10T18:40:34,427 INFO [Thread-10] jdbc.TestDriver: at
> org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:657)
> 2023-01-10T18:40:34,427 INFO [Thread-10] jdbc.TestDriver: at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:603)
> 2023-01-10T18:40:34,427 INFO [Thread-10] jdbc.TestDriver: at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:597)
> 2023-01-10T18:40:34,427 INFO [Thread-10] jdbc.TestDriver: at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:127)
> 2023-01-10T18:40:34,427 INFO [Thread-10] jdbc.TestDriver: at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:206)
> 2023-01-10T18:40:34,428 INFO [Thread-10] jdbc.TestDriver: at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:336)
> 2023-01-10T18:40:34,428 INFO [Thread-10] jdbc.TestDriver: at
> java.base/java.security.AccessController.doPrivileged(Native Method)
> 2023-01-10T18:40:34,428 INFO [Thread-10] jdbc.TestDriver: at
> java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> 2023-01-10T18:40:34,428 INFO [Thread-10] jdbc.TestDriver: at
>
[jira] [Work logged] (HIVE-26924) Alter materialized view enable rewrite throws SemanticException for source iceberg table
[
https://issues.apache.org/jira/browse/HIVE-26924?focusedWorklogId=839997=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-839997
]
ASF GitHub Bot logged work on HIVE-26924:
-
Author: ASF GitHub Bot
Created on: 18/Jan/23 14:56
Start Date: 18/Jan/23 14:56
Worklog Time Spent: 10m
Work Description: kasakrisz commented on code in PR #3936:
URL: https://github.com/apache/hive/pull/3936#discussion_r1073645247
##
ql/src/java/org/apache/hadoop/hive/ql/ddl/view/materialized/alter/rewrite/AlterMaterializedViewRewriteAnalyzer.java:
##
@@ -68,10 +68,12 @@ public void analyzeInternal(ASTNode root) throws
SemanticException {
Table materializedViewTable = getTable(tableName, true);
// One last test: if we are enabling the rewrite, we need to check that
query
-// only uses transactional (MM and ACID) tables
+// only uses transactional (MM and ACID and Iceberg) tables
if (rewriteEnable) {
for (SourceTable sourceTable :
materializedViewTable.getMVMetadata().getSourceTables()) {
-if (!AcidUtils.isTransactionalTable(sourceTable.getTable())) {
+Table table = new Table(sourceTable.getTable());
+if (!AcidUtils.isTransactionalTable(sourceTable.getTable()) &&
+!(table.isNonNative() &&
table.getStorageHandler().areSnapshotsSupported())) {
throw new SemanticException("Automatic rewriting for materialized
view cannot be enabled if the " +
"materialized view uses non-transactional tables");
}
Review Comment:
Yes, Iceberg tables are transactional but transactions are handled in a
different than native transactional tables.
Issue Time Tracking
---
Worklog Id: (was: 839997)
Time Spent: 1h 20m (was: 1h 10m)
> Alter materialized view enable rewrite throws SemanticException for source
> iceberg table
>
>
> Key: HIVE-26924
> URL: https://issues.apache.org/jira/browse/HIVE-26924
> Project: Hive
> Issue Type: Bug
> Components: Iceberg integration
>Reporter: Dharmik Thakkar
>Assignee: Krisztian Kasa
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> alter materialized view enable rewrite throws SemanticException for source
> iceberg table
> SQL test
> {code:java}
> >>> create materialized view mv_rewrite as select t, si from all100k where
> >>> t>115;
> >>> analyze table mv_rewrite compute statistics for columns;
> >>> set hive.explain.user=false;
> >>> explain select si,t from all100k where t>116 and t<120;
> !!! match row_contains
> alias: iceberg_test_db_hive.mv_rewrite
> >>> alter materialized view mv_rewrite disable rewrite;
> >>> explain select si,t from all100k where t>116 and t<120;
> !!! match row_contains
> alias: all100k
> >>> alter materialized view mv_rewrite enable rewrite;
> >>> explain select si,t from all100k where t>116 and t<120;
> !!! match row_contains
> alias: iceberg_test_db_hive.mv_rewrite
> >>> drop materialized view mv_rewrite; {code}
>
> Error
> {code:java}
> 2023-01-10T18:40:34,303 INFO [pool-3-thread-1] jdbc.TestDriver: Query: alter
> materialized view mv_rewrite enable rewrite
> 2023-01-10T18:40:34,365 INFO [Thread-10] jdbc.TestDriver: INFO : Compiling
> command(queryId=hive_20230110184034_f557b4a6-40a0-42ba-8e67-2f273f50af36):
> alter materialized view mv_rewrite enable rewrite
> 2023-01-10T18:40:34,426 INFO [Thread-10] jdbc.TestDriver: ERROR : FAILED:
> SemanticException Automatic rewriting for materialized view cannot be enabled
> if the materialized view uses non-transactional tables
> 2023-01-10T18:40:34,426 INFO [Thread-10] jdbc.TestDriver:
> org.apache.hadoop.hive.ql.parse.SemanticException: Automatic rewriting for
> materialized view cannot be enabled if the materialized view uses
> non-transactional tables
> 2023-01-10T18:40:34,426 INFO [Thread-10] jdbc.TestDriver: at
> org.apache.hadoop.hive.ql.ddl.view.materialized.alter.rewrite.AlterMaterializedViewRewriteAnalyzer.analyzeInternal(AlterMaterializedViewRewriteAnalyzer.java:75)
> 2023-01-10T18:40:34,426 INFO [Thread-10] jdbc.TestDriver: at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:313)
> 2023-01-10T18:40:34,427 INFO [Thread-10] jdbc.TestDriver: at
> org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:222)
> 2023-01-10T18:40:34,427 INFO [Thread-10] jdbc.TestDriver: at
> org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:105)
> 2023-01-10T18:40:34,427 INFO [Thread-10] jdbc.TestDriver: at
> org.apache.hadoop.hive.ql.Driver.compile(Driver.java:201)
>
[jira] [Work logged] (HIVE-26887) Make sure dirPath has the correct permissions
[
https://issues.apache.org/jira/browse/HIVE-26887?focusedWorklogId=839996=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-839996
]
ASF GitHub Bot logged work on HIVE-26887:
-
Author: ASF GitHub Bot
Created on: 18/Jan/23 14:54
Start Date: 18/Jan/23 14:54
Worklog Time Spent: 10m
Work Description: skysiders commented on PR #3894:
URL: https://github.com/apache/hive/pull/3894#issuecomment-1387202288
Hi @zabetak , thanks for your review.
You mentioned that this is "kind of a breaking change", but I don't agree
with this. In setting file permissions, most of the time we can rely on the
umask of the underlying file system, such as the most commonly used fs.create
function, but in In my fix, all file permissions are explicitly set. I think
this explicit permission setting is due to the developer thinking that the file
should be set to this explicit permission. If the development of this pair of
files does not require explicit permissions, the underlying umask can indeed be
used to constrain it, but once the permissions are clear, the underlying umask
may cause the file permissions to be too strict and make the files unusable. I
would like to give an inappropriate example here. For example, the umask of the
underlying file system is 777, and the file permissions are 000, so the
upper-level files will not have any permissions. Therefore, for such files with
clearly set permissions, I think it should be Make sure they are properly
assigned permissions.
Regarding the second point "programming pattern" you mentioned, in fact, it
is also possible to use Hadoop's underlying FileSystem.create(fs, path, perm)
here. In fact, I now think that such a "programming pattern" should be adopted,
because this It is safe and more reliable than fs.create(path,perm). This kind
of repair is mainly aimed at API misuse. I have mentioned this problem in[
HBASE-26994](https://github.com/apache/hbase/pull/4391), which means that the
developer originally intended to Set special permissions here, but mistakenly
think that fs.create(path, perm) can set special permissions perm for the path.
In fact, this is wrong. In the chat with the hbase developer, I pointed out his
mistake and Got his approval.
Finally, I want to say that this fix is necessary in my opinion. I have
searched in hive and found these four API misuse problems, so I point out this
problem here.
Issue Time Tracking
---
Worklog Id: (was: 839996)
Time Spent: 1h 50m (was: 1h 40m)
> Make sure dirPath has the correct permissions
> -
>
> Key: HIVE-26887
> URL: https://issues.apache.org/jira/browse/HIVE-26887
> Project: Hive
> Issue Type: Improvement
>Reporter: Zhang Dongsheng
>Priority: Major
> Labels: backward-incompatible, pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> In the QueryResultsCache function of class QueryResultsCache, there is the
> following code segment
> {code:java}
> private QueryResultsCache(HiveConf configuration) throws IOException {
> ..
> FileSystem fs = cacheDirPath.getFileSystem(conf);
> FsPermission fsPermission = new FsPermission("700");
> fs.mkdirs(cacheDirPath, fsPermission);
> ..
> }
> {code}
> It can be seen that the function will use the mkdirs to create cacheDirPath,
> and the parameters passed in include the path variable cacheDirPath and a
> permission 700. But we haven't confirmed whether the permission is correctly
> assigned to the file.
> The above question is raised because there are two mkdir functions of hadoop,
> {code:java}
> mkdirs(Path f, FsPermission permission)
> {code}
> and
> {code:java}
> mkdirs(FileSystem fs, Path dir, FsPermission permission)
> {code}
> and the first one is used here. The permissions of this function will be
> affected by the underlying umask. Although 700 here will hardly be affected
> by umask, but I think from a rigorous point of view, we should have one more
> permission check and permission grant here.
> And I find same issue in other three methods here.
> In class Context
> {code:java}
> private Path getScratchDir(String scheme, String authority,
> boolean mkdir, String scratchDir) {
> ..
> FileSystem fs = dirPath.getFileSystem(conf);
> dirPath = new Path(fs.makeQualified(dirPath).toString());
> FsPermission fsPermission = new FsPermission(scratchDirPermission);
> if (!fs.mkdirs(dirPath, fsPermission)) {
> throw new RuntimeException("Cannot make directory: "
> + dirPath.toString());
> ..
> }
> {code}
> In class SessionState
> {code:java}
> static void createPath(HiveConf conf, Path path, String
[jira] [Work logged] (HIVE-26924) Alter materialized view enable rewrite throws SemanticException for source iceberg table
[
https://issues.apache.org/jira/browse/HIVE-26924?focusedWorklogId=839992=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-839992
]
ASF GitHub Bot logged work on HIVE-26924:
-
Author: ASF GitHub Bot
Created on: 18/Jan/23 14:49
Start Date: 18/Jan/23 14:49
Worklog Time Spent: 10m
Work Description: kasakrisz commented on PR #3936:
URL: https://github.com/apache/hive/pull/3936#issuecomment-1387194012
Yes, the doc is not up-to-date. Iceberg is also missing.
In case of create materialized view checks are performed here:
https://github.com/apache/hive/blob/55471330426c2e0a52101c2e535a66f751be76ee/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L14269
Issue Time Tracking
---
Worklog Id: (was: 839992)
Time Spent: 1h 10m (was: 1h)
> Alter materialized view enable rewrite throws SemanticException for source
> iceberg table
>
>
> Key: HIVE-26924
> URL: https://issues.apache.org/jira/browse/HIVE-26924
> Project: Hive
> Issue Type: Bug
> Components: Iceberg integration
>Reporter: Dharmik Thakkar
>Assignee: Krisztian Kasa
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> alter materialized view enable rewrite throws SemanticException for source
> iceberg table
> SQL test
> {code:java}
> >>> create materialized view mv_rewrite as select t, si from all100k where
> >>> t>115;
> >>> analyze table mv_rewrite compute statistics for columns;
> >>> set hive.explain.user=false;
> >>> explain select si,t from all100k where t>116 and t<120;
> !!! match row_contains
> alias: iceberg_test_db_hive.mv_rewrite
> >>> alter materialized view mv_rewrite disable rewrite;
> >>> explain select si,t from all100k where t>116 and t<120;
> !!! match row_contains
> alias: all100k
> >>> alter materialized view mv_rewrite enable rewrite;
> >>> explain select si,t from all100k where t>116 and t<120;
> !!! match row_contains
> alias: iceberg_test_db_hive.mv_rewrite
> >>> drop materialized view mv_rewrite; {code}
>
> Error
> {code:java}
> 2023-01-10T18:40:34,303 INFO [pool-3-thread-1] jdbc.TestDriver: Query: alter
> materialized view mv_rewrite enable rewrite
> 2023-01-10T18:40:34,365 INFO [Thread-10] jdbc.TestDriver: INFO : Compiling
> command(queryId=hive_20230110184034_f557b4a6-40a0-42ba-8e67-2f273f50af36):
> alter materialized view mv_rewrite enable rewrite
> 2023-01-10T18:40:34,426 INFO [Thread-10] jdbc.TestDriver: ERROR : FAILED:
> SemanticException Automatic rewriting for materialized view cannot be enabled
> if the materialized view uses non-transactional tables
> 2023-01-10T18:40:34,426 INFO [Thread-10] jdbc.TestDriver:
> org.apache.hadoop.hive.ql.parse.SemanticException: Automatic rewriting for
> materialized view cannot be enabled if the materialized view uses
> non-transactional tables
> 2023-01-10T18:40:34,426 INFO [Thread-10] jdbc.TestDriver: at
> org.apache.hadoop.hive.ql.ddl.view.materialized.alter.rewrite.AlterMaterializedViewRewriteAnalyzer.analyzeInternal(AlterMaterializedViewRewriteAnalyzer.java:75)
> 2023-01-10T18:40:34,426 INFO [Thread-10] jdbc.TestDriver: at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:313)
> 2023-01-10T18:40:34,427 INFO [Thread-10] jdbc.TestDriver: at
> org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:222)
> 2023-01-10T18:40:34,427 INFO [Thread-10] jdbc.TestDriver: at
> org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:105)
> 2023-01-10T18:40:34,427 INFO [Thread-10] jdbc.TestDriver: at
> org.apache.hadoop.hive.ql.Driver.compile(Driver.java:201)
> 2023-01-10T18:40:34,427 INFO [Thread-10] jdbc.TestDriver: at
> org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:657)
> 2023-01-10T18:40:34,427 INFO [Thread-10] jdbc.TestDriver: at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:603)
> 2023-01-10T18:40:34,427 INFO [Thread-10] jdbc.TestDriver: at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:597)
> 2023-01-10T18:40:34,427 INFO [Thread-10] jdbc.TestDriver: at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:127)
> 2023-01-10T18:40:34,427 INFO [Thread-10] jdbc.TestDriver: at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:206)
> 2023-01-10T18:40:34,428 INFO [Thread-10] jdbc.TestDriver: at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:336)
> 2023-01-10T18:40:34,428 INFO [Thread-10] jdbc.TestDriver: at
>
[jira] [Work logged] (HIVE-26711) The very first REPL Load should make the Target Database read-only
[ https://issues.apache.org/jira/browse/HIVE-26711?focusedWorklogId=839989=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-839989 ] ASF GitHub Bot logged work on HIVE-26711: - Author: ASF GitHub Bot Created on: 18/Jan/23 14:35 Start Date: 18/Jan/23 14:35 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3736: URL: https://github.com/apache/hive/pull/3736#issuecomment-1387173495 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3736) [](https://sonarcloud.io/project/issues?id=apache_hive=3736=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3736=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3736=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3736=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3736=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3736=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3736=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3736=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3736=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3736=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3736=false=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3736=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3736=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3736=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 839989) Time Spent: 1h 50m (was: 1h 40m) > The very first REPL Load should make the Target Database read-only > -- > > Key: HIVE-26711 > URL: https://issues.apache.org/jira/browse/HIVE-26711 > Project: Hive > Issue Type: Task >Reporter: Shreenidhi >Assignee: Shreenidhi >Priority: Major > Labels: pull-request-available > Time Spent: 1h 50m > Remaining Estimate: 0h > > Use EnforceReadOnly hook to set TARGET database read only during BootStrap > load. > Also ensure backward compatibility. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26915) Backport of HIVE-23692 TestCodahaleMetrics.testFileReporting is flaky
[ https://issues.apache.org/jira/browse/HIVE-26915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-26915: Fix Version/s: 3.2.0 > Backport of HIVE-23692 TestCodahaleMetrics.testFileReporting is flaky > - > > Key: HIVE-26915 > URL: https://issues.apache.org/jira/browse/HIVE-26915 > Project: Hive > Issue Type: Sub-task >Reporter: Aman Raj >Assignee: Aman Raj >Priority: Major > Labels: pull-request-available > Fix For: 3.2.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > This was committed in master without a HIVE Jira task. This is the commit id > : 130f80445d589cdd82904cea1073c84d1368d079 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-26915) Backport of HIVE-23692 TestCodahaleMetrics.testFileReporting is flaky
[ https://issues.apache.org/jira/browse/HIVE-26915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor resolved HIVE-26915. - Resolution: Fixed > Backport of HIVE-23692 TestCodahaleMetrics.testFileReporting is flaky > - > > Key: HIVE-26915 > URL: https://issues.apache.org/jira/browse/HIVE-26915 > Project: Hive > Issue Type: Sub-task >Reporter: Aman Raj >Assignee: Aman Raj >Priority: Major > Labels: pull-request-available > Fix For: 3.2.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > This was committed in master without a HIVE Jira task. This is the commit id > : 130f80445d589cdd82904cea1073c84d1368d079 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-26915) Backport of HIVE-23692 TestCodahaleMetrics.testFileReporting is flaky
[ https://issues.apache.org/jira/browse/HIVE-26915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17678249#comment-17678249 ] László Bodor commented on HIVE-26915: - merged to branch-3, thanks [~amanraj2520] for taking care of this! > Backport of HIVE-23692 TestCodahaleMetrics.testFileReporting is flaky > - > > Key: HIVE-26915 > URL: https://issues.apache.org/jira/browse/HIVE-26915 > Project: Hive > Issue Type: Sub-task >Reporter: Aman Raj >Assignee: Aman Raj >Priority: Major > Labels: pull-request-available > Fix For: 3.2.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > This was committed in master without a HIVE Jira task. This is the commit id > : 130f80445d589cdd82904cea1073c84d1368d079 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26915) Backport of HIVE-23692 TestCodahaleMetrics.testFileReporting is flaky
[ https://issues.apache.org/jira/browse/HIVE-26915?focusedWorklogId=839981=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-839981 ] ASF GitHub Bot logged work on HIVE-26915: - Author: ASF GitHub Bot Created on: 18/Jan/23 14:17 Start Date: 18/Jan/23 14:17 Worklog Time Spent: 10m Work Description: abstractdog merged PR #3928: URL: https://github.com/apache/hive/pull/3928 Issue Time Tracking --- Worklog Id: (was: 839981) Time Spent: 1.5h (was: 1h 20m) > Backport of HIVE-23692 TestCodahaleMetrics.testFileReporting is flaky > - > > Key: HIVE-26915 > URL: https://issues.apache.org/jira/browse/HIVE-26915 > Project: Hive > Issue Type: Sub-task >Reporter: Aman Raj >Assignee: Aman Raj >Priority: Major > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > This was committed in master without a HIVE Jira task. This is the commit id > : 130f80445d589cdd82904cea1073c84d1368d079 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26945) Test fixes for query*.q files
[ https://issues.apache.org/jira/browse/HIVE-26945?focusedWorklogId=839979=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-839979 ] ASF GitHub Bot logged work on HIVE-26945: - Author: ASF GitHub Bot Created on: 18/Jan/23 14:11 Start Date: 18/Jan/23 14:11 Worklog Time Spent: 10m Work Description: abstractdog commented on PR #3954: URL: https://github.com/apache/hive/pull/3954#issuecomment-1387136989 basically, these changes make sense to me, just one thing @amanraj2520: can you check if these bits are present on master in this form and clarify what patches caused these changes? Issue Time Tracking --- Worklog Id: (was: 839979) Time Spent: 0.5h (was: 20m) > Test fixes for query*.q files > - > > Key: HIVE-26945 > URL: https://issues.apache.org/jira/browse/HIVE-26945 > Project: Hive > Issue Type: Sub-task >Reporter: Aman Raj >Assignee: Aman Raj >Priority: Critical > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > The tests has outdated q.out files which need to be updated. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26922) Deadlock when rebuilding Materialized view stored by Iceberg
[
https://issues.apache.org/jira/browse/HIVE-26922?focusedWorklogId=839978=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-839978
]
ASF GitHub Bot logged work on HIVE-26922:
-
Author: ASF GitHub Bot
Created on: 18/Jan/23 14:11
Start Date: 18/Jan/23 14:11
Worklog Time Spent: 10m
Work Description: kasakrisz commented on code in PR #3934:
URL: https://github.com/apache/hive/pull/3934#discussion_r1073584822
##
ql/src/test/org/apache/hadoop/hive/ql/metadata/StorageHandlerMock.java:
##
@@ -70,7 +70,7 @@ public class StorageHandlerMock extends DefaultStorageHandler
{
if (writeEntity.getWriteType().equals(WriteEntity.WriteType.INSERT)) {
return LockType.SHARED_READ;
}
-return LockType.SHARED_WRITE;
+return LockType.EXCLUSIVE;
Review Comment:
This mock is used in 2 test cases:
* testLockingOnInsertIntoNonNativeTables
* testLockingOnInsertOverwriteNonNativeTables
Prior this patch this was ignored in case of insert overwrite and
`EXCLUSIVE` was set in case of any type of non transactional table.
After I altered the logic to use the lock type coming from the
`storageHandler` the mock returned `SHARED_WRITE` and test
`testLockingOnInsertOverwriteNonNativeTables` failed since it still expected
`EXCLUSIVE`.
I could alter the assertion in the test but since the default lock type
specified by the storage handler is `EXCLUSIVE` I chose altering the mock.
https://github.com/apache/hive/blob/55471330426c2e0a52101c2e535a66f751be76ee/ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveStorageHandler.java#L199-L201
Maybe `super.getLockType` would be better here.
Issue Time Tracking
---
Worklog Id: (was: 839978)
Time Spent: 2h 10m (was: 2h)
> Deadlock when rebuilding Materialized view stored by Iceberg
>
>
> Key: HIVE-26922
> URL: https://issues.apache.org/jira/browse/HIVE-26922
> Project: Hive
> Issue Type: Bug
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> {code}
> create table tbl_ice(a int, b string, c int) stored by iceberg stored as orc
> tblproperties ('format-version'='1');
> insert into tbl_ice values (1, 'one', 50), (2, 'two', 51), (3, 'three', 52),
> (4, 'four', 53), (5, 'five', 54);
> create materialized view mat1 stored by iceberg stored as orc tblproperties
> ('format-version'='1') as
> select tbl_ice.b, tbl_ice.c from tbl_ice where tbl_ice.c > 52;
> insert into tbl_ice values (10, 'ten', 60);
> alter materialized view mat1 rebuild;
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26924) Alter materialized view enable rewrite throws SemanticException for source iceberg table
[
https://issues.apache.org/jira/browse/HIVE-26924?focusedWorklogId=839975=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-839975
]
ASF GitHub Bot logged work on HIVE-26924:
-
Author: ASF GitHub Bot
Created on: 18/Jan/23 14:09
Start Date: 18/Jan/23 14:09
Worklog Time Spent: 10m
Work Description: zabetak commented on code in PR #3936:
URL: https://github.com/apache/hive/pull/3936#discussion_r1073576224
##
ql/src/java/org/apache/hadoop/hive/ql/ddl/view/materialized/alter/rewrite/AlterMaterializedViewRewriteAnalyzer.java:
##
@@ -68,10 +68,12 @@ public void analyzeInternal(ASTNode root) throws
SemanticException {
Table materializedViewTable = getTable(tableName, true);
// One last test: if we are enabling the rewrite, we need to check that
query
-// only uses transactional (MM and ACID) tables
+// only uses transactional (MM and ACID and Iceberg) tables
if (rewriteEnable) {
for (SourceTable sourceTable :
materializedViewTable.getMVMetadata().getSourceTables()) {
-if (!AcidUtils.isTransactionalTable(sourceTable.getTable())) {
+Table table = new Table(sourceTable.getTable());
+if (!AcidUtils.isTransactionalTable(sourceTable.getTable()) &&
+!(table.isNonNative() &&
table.getStorageHandler().areSnapshotsSupported())) {
throw new SemanticException("Automatic rewriting for materialized
view cannot be enabled if the " +
"materialized view uses non-transactional tables");
}
Review Comment:
Do we consider an Iceberg table as transactional? Should we change the
message to something more general. For example:
`Cannot enable materialized view rewriting cause view uses unsupported
tables`
Issue Time Tracking
---
Worklog Id: (was: 839975)
Time Spent: 1h (was: 50m)
> Alter materialized view enable rewrite throws SemanticException for source
> iceberg table
>
>
> Key: HIVE-26924
> URL: https://issues.apache.org/jira/browse/HIVE-26924
> Project: Hive
> Issue Type: Bug
> Components: Iceberg integration
>Reporter: Dharmik Thakkar
>Assignee: Krisztian Kasa
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> alter materialized view enable rewrite throws SemanticException for source
> iceberg table
> SQL test
> {code:java}
> >>> create materialized view mv_rewrite as select t, si from all100k where
> >>> t>115;
> >>> analyze table mv_rewrite compute statistics for columns;
> >>> set hive.explain.user=false;
> >>> explain select si,t from all100k where t>116 and t<120;
> !!! match row_contains
> alias: iceberg_test_db_hive.mv_rewrite
> >>> alter materialized view mv_rewrite disable rewrite;
> >>> explain select si,t from all100k where t>116 and t<120;
> !!! match row_contains
> alias: all100k
> >>> alter materialized view mv_rewrite enable rewrite;
> >>> explain select si,t from all100k where t>116 and t<120;
> !!! match row_contains
> alias: iceberg_test_db_hive.mv_rewrite
> >>> drop materialized view mv_rewrite; {code}
>
> Error
> {code:java}
> 2023-01-10T18:40:34,303 INFO [pool-3-thread-1] jdbc.TestDriver: Query: alter
> materialized view mv_rewrite enable rewrite
> 2023-01-10T18:40:34,365 INFO [Thread-10] jdbc.TestDriver: INFO : Compiling
> command(queryId=hive_20230110184034_f557b4a6-40a0-42ba-8e67-2f273f50af36):
> alter materialized view mv_rewrite enable rewrite
> 2023-01-10T18:40:34,426 INFO [Thread-10] jdbc.TestDriver: ERROR : FAILED:
> SemanticException Automatic rewriting for materialized view cannot be enabled
> if the materialized view uses non-transactional tables
> 2023-01-10T18:40:34,426 INFO [Thread-10] jdbc.TestDriver:
> org.apache.hadoop.hive.ql.parse.SemanticException: Automatic rewriting for
> materialized view cannot be enabled if the materialized view uses
> non-transactional tables
> 2023-01-10T18:40:34,426 INFO [Thread-10] jdbc.TestDriver: at
> org.apache.hadoop.hive.ql.ddl.view.materialized.alter.rewrite.AlterMaterializedViewRewriteAnalyzer.analyzeInternal(AlterMaterializedViewRewriteAnalyzer.java:75)
> 2023-01-10T18:40:34,426 INFO [Thread-10] jdbc.TestDriver: at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:313)
> 2023-01-10T18:40:34,427 INFO [Thread-10] jdbc.TestDriver: at
> org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:222)
> 2023-01-10T18:40:34,427 INFO [Thread-10] jdbc.TestDriver: at
> org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:105)
> 2023-01-10T18:40:34,427 INFO [Thread-10] jdbc.TestDriver: at
>
[jira] [Work logged] (HIVE-26400) Provide docker images for Hive
[ https://issues.apache.org/jira/browse/HIVE-26400?focusedWorklogId=839974=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-839974 ] ASF GitHub Bot logged work on HIVE-26400: - Author: ASF GitHub Bot Created on: 18/Jan/23 14:08 Start Date: 18/Jan/23 14:08 Worklog Time Spent: 10m Work Description: abstractdog commented on PR #3448: URL: https://github.com/apache/hive/pull/3448#issuecomment-1387132348 I had some comments earlier, if you can confirm you addressed those @dengzhhu653 , I'll take a second look and approve, it's time to merge this I believe please create an umbrella ticket for hive docker improvements and add this one as the first sub-jira, let's track further improvements there Issue Time Tracking --- Worklog Id: (was: 839974) Time Spent: 6h 40m (was: 6.5h) > Provide docker images for Hive > -- > > Key: HIVE-26400 > URL: https://issues.apache.org/jira/browse/HIVE-26400 > Project: Hive > Issue Type: Improvement > Components: Build Infrastructure >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Blocker > Labels: hive-4.0.0-must, pull-request-available > Time Spent: 6h 40m > Remaining Estimate: 0h > > Make Apache Hive be able to run inside docker container in pseudo-distributed > mode, with MySQL/Derby as its back database, provide the following: > * Quick-start/Debugging/Prepare a test env for Hive; > * Tools to build target image with specified version of Hive and its > dependencies; > * Images can be used as the basis for the Kubernetes operator. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26802) Create qtest running QB compaction queries
[ https://issues.apache.org/jira/browse/HIVE-26802?focusedWorklogId=839973=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-839973 ] ASF GitHub Bot logged work on HIVE-26802: - Author: ASF GitHub Bot Created on: 18/Jan/23 14:08 Start Date: 18/Jan/23 14:08 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3882: URL: https://github.com/apache/hive/pull/3882#issuecomment-1387131743 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3882) [](https://sonarcloud.io/project/issues?id=apache_hive=3882=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3882=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3882=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3882=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3882=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3882=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3882=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3882=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3882=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3882=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3882=false=CODE_SMELL) [4 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3882=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3882=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3882=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 839973) Time Spent: 5.5h (was: 5h 20m) > Create qtest running QB compaction queries > -- > > Key: HIVE-26802 > URL: https://issues.apache.org/jira/browse/HIVE-26802 > Project: Hive > Issue Type: Improvement >Reporter: Zoltán Rátkai >Assignee: Zoltán Rátkai >Priority: Minor > Labels: pull-request-available > Time Spent: 5.5h > Remaining Estimate: 0h > > Create a qtest that runs the queries that query-based compaction runs. > Not so much to check for correct data but more to check the query plans, to > simplify tracing changes in compilation that might affect QB compaction. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26922) Deadlock when rebuilding Materialized view stored by Iceberg
[
https://issues.apache.org/jira/browse/HIVE-26922?focusedWorklogId=839968=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-839968
]
ASF GitHub Bot logged work on HIVE-26922:
-
Author: ASF GitHub Bot
Created on: 18/Jan/23 13:56
Start Date: 18/Jan/23 13:56
Worklog Time Spent: 10m
Work Description: kasakrisz commented on code in PR #3934:
URL: https://github.com/apache/hive/pull/3934#discussion_r1073566847
##
ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java:
##
@@ -3122,7 +3117,19 @@ Seems much cleaner if each stmt is identified as a
particular HiveOperation (whi
}
return lockComponents;
}
-
+
+ private static LockType getLockTypeFromStorageHandler(WriteEntity output,
Table t) {
+final HiveStorageHandler storageHandler =
Preconditions.checkNotNull(t.getStorageHandler(),
+"Non-native tables must have an instance of storage handler.");
+LockType lockType = storageHandler.getLockType(output);
+if (null == LockType.findByValue(lockType.getValue())) {
Review Comment:
I haven't found any benefit :)
I just extracted these lines to a method to reuse this logic in case of
insert overwrite.
Removed the findByValue part.
Issue Time Tracking
---
Worklog Id: (was: 839968)
Time Spent: 2h (was: 1h 50m)
> Deadlock when rebuilding Materialized view stored by Iceberg
>
>
> Key: HIVE-26922
> URL: https://issues.apache.org/jira/browse/HIVE-26922
> Project: Hive
> Issue Type: Bug
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> {code}
> create table tbl_ice(a int, b string, c int) stored by iceberg stored as orc
> tblproperties ('format-version'='1');
> insert into tbl_ice values (1, 'one', 50), (2, 'two', 51), (3, 'three', 52),
> (4, 'four', 53), (5, 'five', 54);
> create materialized view mat1 stored by iceberg stored as orc tblproperties
> ('format-version'='1') as
> select tbl_ice.b, tbl_ice.c from tbl_ice where tbl_ice.c > 52;
> insert into tbl_ice values (10, 'ten', 60);
> alter materialized view mat1 rebuild;
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26922) Deadlock when rebuilding Materialized view stored by Iceberg
[
https://issues.apache.org/jira/browse/HIVE-26922?focusedWorklogId=839962=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-839962
]
ASF GitHub Bot logged work on HIVE-26922:
-
Author: ASF GitHub Bot
Created on: 18/Jan/23 13:31
Start Date: 18/Jan/23 13:31
Worklog Time Spent: 10m
Work Description: kasakrisz commented on PR #3934:
URL: https://github.com/apache/hive/pull/3934#issuecomment-1387079817
Lock requests in the code:
1.
https://github.com/apache/hive/blob/55471330426c2e0a52101c2e535a66f751be76ee/ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java#L3029
2.
https://github.com/apache/hive/blob/55471330426c2e0a52101c2e535a66f751be76ee/iceberg/iceberg-catalog/src/main/java/org/apache/iceberg/hive/HiveCommitLock.java#L118
The first lock is acquired only if these setting are present
```
set hive.support.concurrency=true;
set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
```
These are required for native acid operations.
The deadlock happens in a few lines later in the 2. place: after the lock
request is issued we check periodically the state of the lock whether it is
acquired or not. Unfortunately in this use case it is always in a waiting state
since another exclusive write lock already acquired on the same table in the 1.
place and that one is released only when the current txn ends.
In this patch my goal was to copy the behavior from the insert case: let the
table's storage handler determine the type of lock in the 1. place.
https://github.com/apache/hive/blob/55471330426c2e0a52101c2e535a66f751be76ee/ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java#L3068
In case of Iceberg it is `SHARED_READ`. Iceberg uses optimistic concurrency
control so multiple transactions can write the same table parallel only the
Iceberg commit step is serialized using exclusive write locks.
Currently we support only full rebuild of Iceberg Materialized views which
is technically an insert overwrite. So a plain insert overwrite statement is
also affected.
Issue Time Tracking
---
Worklog Id: (was: 839962)
Time Spent: 1h 50m (was: 1h 40m)
> Deadlock when rebuilding Materialized view stored by Iceberg
>
>
> Key: HIVE-26922
> URL: https://issues.apache.org/jira/browse/HIVE-26922
> Project: Hive
> Issue Type: Bug
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> {code}
> create table tbl_ice(a int, b string, c int) stored by iceberg stored as orc
> tblproperties ('format-version'='1');
> insert into tbl_ice values (1, 'one', 50), (2, 'two', 51), (3, 'three', 52),
> (4, 'four', 53), (5, 'five', 54);
> create materialized view mat1 stored by iceberg stored as orc tblproperties
> ('format-version'='1') as
> select tbl_ice.b, tbl_ice.c from tbl_ice where tbl_ice.c > 52;
> insert into tbl_ice values (10, 'ten', 60);
> alter materialized view mat1 rebuild;
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-26924) Alter materialized view enable rewrite throws SemanticException for source iceberg table
[
https://issues.apache.org/jira/browse/HIVE-26924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17678226#comment-17678226
]
Stamatis Zampetakis commented on HIVE-26924:
>From the description of the ticket it is not clear what happens after the
>SemanticException is raised. Will the queries following the exception be able
>to exploit the view or not? Naturally I would think no but I am not sure how
>to interpret the following:
{noformat}
>>> alter materialized view mv_rewrite enable rewrite;
>>> explain select si,t from all100k where t>116 and t<120;
!!! match row_contains
alias: iceberg_test_db_hive.mv_rewrite
{noformat}
Can you please clarify [~dharmikt] [~kkasa]?
> Alter materialized view enable rewrite throws SemanticException for source
> iceberg table
>
>
> Key: HIVE-26924
> URL: https://issues.apache.org/jira/browse/HIVE-26924
> Project: Hive
> Issue Type: Bug
> Components: Iceberg integration
>Reporter: Dharmik Thakkar
>Assignee: Krisztian Kasa
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> alter materialized view enable rewrite throws SemanticException for source
> iceberg table
> SQL test
> {code:java}
> >>> create materialized view mv_rewrite as select t, si from all100k where
> >>> t>115;
> >>> analyze table mv_rewrite compute statistics for columns;
> >>> set hive.explain.user=false;
> >>> explain select si,t from all100k where t>116 and t<120;
> !!! match row_contains
> alias: iceberg_test_db_hive.mv_rewrite
> >>> alter materialized view mv_rewrite disable rewrite;
> >>> explain select si,t from all100k where t>116 and t<120;
> !!! match row_contains
> alias: all100k
> >>> alter materialized view mv_rewrite enable rewrite;
> >>> explain select si,t from all100k where t>116 and t<120;
> !!! match row_contains
> alias: iceberg_test_db_hive.mv_rewrite
> >>> drop materialized view mv_rewrite; {code}
>
> Error
> {code:java}
> 2023-01-10T18:40:34,303 INFO [pool-3-thread-1] jdbc.TestDriver: Query: alter
> materialized view mv_rewrite enable rewrite
> 2023-01-10T18:40:34,365 INFO [Thread-10] jdbc.TestDriver: INFO : Compiling
> command(queryId=hive_20230110184034_f557b4a6-40a0-42ba-8e67-2f273f50af36):
> alter materialized view mv_rewrite enable rewrite
> 2023-01-10T18:40:34,426 INFO [Thread-10] jdbc.TestDriver: ERROR : FAILED:
> SemanticException Automatic rewriting for materialized view cannot be enabled
> if the materialized view uses non-transactional tables
> 2023-01-10T18:40:34,426 INFO [Thread-10] jdbc.TestDriver:
> org.apache.hadoop.hive.ql.parse.SemanticException: Automatic rewriting for
> materialized view cannot be enabled if the materialized view uses
> non-transactional tables
> 2023-01-10T18:40:34,426 INFO [Thread-10] jdbc.TestDriver: at
> org.apache.hadoop.hive.ql.ddl.view.materialized.alter.rewrite.AlterMaterializedViewRewriteAnalyzer.analyzeInternal(AlterMaterializedViewRewriteAnalyzer.java:75)
> 2023-01-10T18:40:34,426 INFO [Thread-10] jdbc.TestDriver: at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:313)
> 2023-01-10T18:40:34,427 INFO [Thread-10] jdbc.TestDriver: at
> org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:222)
> 2023-01-10T18:40:34,427 INFO [Thread-10] jdbc.TestDriver: at
> org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:105)
> 2023-01-10T18:40:34,427 INFO [Thread-10] jdbc.TestDriver: at
> org.apache.hadoop.hive.ql.Driver.compile(Driver.java:201)
> 2023-01-10T18:40:34,427 INFO [Thread-10] jdbc.TestDriver: at
> org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:657)
> 2023-01-10T18:40:34,427 INFO [Thread-10] jdbc.TestDriver: at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:603)
> 2023-01-10T18:40:34,427 INFO [Thread-10] jdbc.TestDriver: at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:597)
> 2023-01-10T18:40:34,427 INFO [Thread-10] jdbc.TestDriver: at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:127)
> 2023-01-10T18:40:34,427 INFO [Thread-10] jdbc.TestDriver: at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:206)
> 2023-01-10T18:40:34,428 INFO [Thread-10] jdbc.TestDriver: at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:336)
> 2023-01-10T18:40:34,428 INFO [Thread-10] jdbc.TestDriver: at
> java.base/java.security.AccessController.doPrivileged(Native Method)
> 2023-01-10T18:40:34,428 INFO [Thread-10] jdbc.TestDriver: at
>
[jira] [Work logged] (HIVE-26955) Select query fails when decimal column data type is changed to string/char/varchar in Parquet
[
https://issues.apache.org/jira/browse/HIVE-26955?focusedWorklogId=839946=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-839946
]
ASF GitHub Bot logged work on HIVE-26955:
-
Author: ASF GitHub Bot
Created on: 18/Jan/23 12:59
Start Date: 18/Jan/23 12:59
Worklog Time Spent: 10m
Work Description: SourabhBadhya commented on code in PR #3964:
URL: https://github.com/apache/hive/pull/3964#discussion_r1073502143
##
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ETypeConverter.java:
##
@@ -676,6 +676,39 @@ private int getScale(PrimitiveType type) {
return logicalType.getScale();
}
};
+case serdeConstants.VARCHAR_TYPE_NAME:
+ return new BinaryConverter(type, parent, index,
hiveTypeInfo) {
+@Override
+protected HiveVarcharWritable convert(Binary binary) {
+ DecimalLogicalTypeAnnotation logicalType =
(DecimalLogicalTypeAnnotation) type.getLogicalTypeAnnotation();
+ HiveDecimalWritable decimalWritable =
HiveDecimalUtils.enforcePrecisionScale(
+ new HiveDecimalWritable(binary.getBytes(),
logicalType.getScale()),
+ new DecimalTypeInfo(logicalType.getPrecision(),
logicalType.getScale()));
Review Comment:
Nice way to refactor @kasakrisz . Implemented it in a similar way. Done.
Issue Time Tracking
---
Worklog Id: (was: 839946)
Time Spent: 40m (was: 0.5h)
> Select query fails when decimal column data type is changed to
> string/char/varchar in Parquet
> -
>
> Key: HIVE-26955
> URL: https://issues.apache.org/jira/browse/HIVE-26955
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Taraka Rama Rao Lethavadla
>Assignee: Sourabh Badhya
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Steps to reproduce
> {noformat}
> create table test_parquet (id decimal) stored as parquet;
> insert into test_parquet values(238);
> alter table test_parquet change id id string;
> select * from test_parquet;
> Error: java.io.IOException: org.apache.parquet.io.ParquetDecodingException:
> Can not read value at 1 in block 0 in file
> hdfs:/namenode:8020/warehouse/tablespace/managed/hive/test_parquet/delta_001_001_/00_0
> (state=,code=0)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:624)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:531)
> at
> org.apache.hadoop.hive.ql.exec.FetchTask.executeInner(FetchTask.java:194)
> ... 55 more
> Caused by: org.apache.parquet.io.ParquetDecodingException: Can not read value
> at 1 in block 0 in file
> file:/home/centos/Apache-Hive-Tarak/itests/qtest/target/localfs/warehouse/test_parquet/00_0
> at
> org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:255)
> at
> org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:207)
> at
> org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.(ParquetRecordReaderWrapper.java:87)
> at
> org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat.getRecordReader(MapredParquetInputFormat.java:89)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator$FetchInputFormatSplit.getRecordReader(FetchOperator.java:771)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:335)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:562)
> ... 57 more
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo cannot be cast to
> org.apache.hadoop.hive.serde2.typeinfo.DecimalTypeInfo
> at
> org.apache.hadoop.hive.ql.io.parquet.convert.ETypeConverter$8$5.convert(ETypeConverter.java:669)
> at
> org.apache.hadoop.hive.ql.io.parquet.convert.ETypeConverter$8$5.convert(ETypeConverter.java:664)
> at
> org.apache.hadoop.hive.ql.io.parquet.convert.ETypeConverter$BinaryConverter.addBinary(ETypeConverter.java:977)
> at
> org.apache.parquet.column.impl.ColumnReaderBase$2$6.writeValue(ColumnReaderBase.java:360)
> at
> org.apache.parquet.column.impl.ColumnReaderBase.writeCurrentValueToConverter(ColumnReaderBase.java:410)
> at
> org.apache.parquet.column.impl.ColumnReaderImpl.writeCurrentValueToConverter(ColumnReaderImpl.java:30)
> at
> org.apache.parquet.io.RecordReaderImplementation.read(RecordReaderImplementation.java:406)
> at
>
[jira] [Work logged] (HIVE-26952) set the value of metastore.storage.schema.reader.impl to org.apache.hadoop.hive.metastore.SerDeStorageSchemaReader as default
[
https://issues.apache.org/jira/browse/HIVE-26952?focusedWorklogId=839945=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-839945
]
ASF GitHub Bot logged work on HIVE-26952:
-
Author: ASF GitHub Bot
Created on: 18/Jan/23 12:57
Start Date: 18/Jan/23 12:57
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3959:
URL: https://github.com/apache/hive/pull/3959#issuecomment-1387028071
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3959)
[](https://sonarcloud.io/project/issues?id=apache_hive=3959=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3959=false=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3959=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3959=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3959=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3959=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3959=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3959=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3959=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3959=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3959=false=CODE_SMELL)
[0 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3959=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3959=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3959=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 839945)
Time Spent: 40m (was: 0.5h)
> set the value of metastore.storage.schema.reader.impl
to
> org.apache.hadoop.hive.metastore.SerDeStorageSchemaReader as default
> --
>
> Key: HIVE-26952
> URL: https://issues.apache.org/jira/browse/HIVE-26952
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Reporter: Taraka Rama Rao Lethavadla
>Assignee: Taraka Rama Rao Lethavadla
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> With the default value of
>
> {code:java}
> DefaultStorageSchemaReader.class.getName(){code}
>
> in the Metastore Config, *metastore.storage.schema.reader.impl*
> below exception is thrown when trying to read Avro schema
> {noformat}
> Caused by: org.apache.hive.service.cli.HiveSQLException: MetaException
> (message:java.lang.UnsupportedOperationException: Storage schema reading not
> supported)
> at
> org.apache.hive.service.cli.operation.GetColumnsOperation.runInternal(GetColumnsOperation.java:213)
> at org.apache.hive.service.cli.operation.Operation.run(Operation.java:247)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.getColumns(HiveSessionImpl.java:729)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
[jira] [Work logged] (HIVE-26942) Prevent updation of "repl.target.last.id" when RESUME action initiated
[ https://issues.apache.org/jira/browse/HIVE-26942?focusedWorklogId=839943=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-839943 ] ASF GitHub Bot logged work on HIVE-26942: - Author: ASF GitHub Bot Created on: 18/Jan/23 12:52 Start Date: 18/Jan/23 12:52 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3951: URL: https://github.com/apache/hive/pull/3951#issuecomment-1387019466 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3951) [](https://sonarcloud.io/project/issues?id=apache_hive=3951=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3951=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3951=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3951=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3951=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3951=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3951=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3951=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3951=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3951=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3951=false=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3951=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3951=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3951=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 839943) Time Spent: 40m (was: 0.5h) > Prevent updation of "repl.target.last.id" when RESUME action initiated > -- > > Key: HIVE-26942 > URL: https://issues.apache.org/jira/browse/HIVE-26942 > Project: Hive > Issue Type: Task >Reporter: Shreenidhi >Assignee: Shreenidhi >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > *Problem :* > Currently when "repl.last.id" get's changed/updated this updates the > corresponding "repl.target.last.id" too. This is problematic incase of resume > because we are updating both the properties at source. > > *Solution :* > To prevent the updation of "repl.target.last.id" whenever "repl.last.id" > changed, we can introduce new db param called "repl.resume.started". This new > param will be set to "true" by orchestrator when RESUME is initiated. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26922) Deadlock when rebuilding Materialized view stored by Iceberg
[
https://issues.apache.org/jira/browse/HIVE-26922?focusedWorklogId=839933=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-839933
]
ASF GitHub Bot logged work on HIVE-26922:
-
Author: ASF GitHub Bot
Created on: 18/Jan/23 12:22
Start Date: 18/Jan/23 12:22
Worklog Time Spent: 10m
Work Description: zabetak commented on code in PR #3934:
URL: https://github.com/apache/hive/pull/3934#discussion_r1073456835
##
ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java:
##
@@ -3122,7 +3117,19 @@ Seems much cleaner if each stmt is identified as a
particular HiveOperation (whi
}
return lockComponents;
}
-
+
+ private static LockType getLockTypeFromStorageHandler(WriteEntity output,
Table t) {
+final HiveStorageHandler storageHandler =
Preconditions.checkNotNull(t.getStorageHandler(),
+"Non-native tables must have an instance of storage handler.");
+LockType lockType = storageHandler.getLockType(output);
+if (null == LockType.findByValue(lockType.getValue())) {
Review Comment:
What's the benefit of doing `findByValue`? Isn't `if(null == lockType)`
already sufficient?
##
ql/src/test/org/apache/hadoop/hive/ql/metadata/StorageHandlerMock.java:
##
@@ -70,7 +70,7 @@ public class StorageHandlerMock extends DefaultStorageHandler
{
if (writeEntity.getWriteType().equals(WriteEntity.WriteType.INSERT)) {
return LockType.SHARED_READ;
}
-return LockType.SHARED_WRITE;
+return LockType.EXCLUSIVE;
Review Comment:
Changing the lock means that we are changing the tests. Why is it necessary?
Issue Time Tracking
---
Worklog Id: (was: 839933)
Time Spent: 1h 40m (was: 1.5h)
> Deadlock when rebuilding Materialized view stored by Iceberg
>
>
> Key: HIVE-26922
> URL: https://issues.apache.org/jira/browse/HIVE-26922
> Project: Hive
> Issue Type: Bug
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> {code}
> create table tbl_ice(a int, b string, c int) stored by iceberg stored as orc
> tblproperties ('format-version'='1');
> insert into tbl_ice values (1, 'one', 50), (2, 'two', 51), (3, 'three', 52),
> (4, 'four', 53), (5, 'five', 54);
> create materialized view mat1 stored by iceberg stored as orc tblproperties
> ('format-version'='1') as
> select tbl_ice.b, tbl_ice.c from tbl_ice where tbl_ice.c > 52;
> insert into tbl_ice values (10, 'ten', 60);
> alter materialized view mat1 rebuild;
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26802) Create qtest running QB compaction queries
[
https://issues.apache.org/jira/browse/HIVE-26802?focusedWorklogId=839924=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-839924
]
ASF GitHub Bot logged work on HIVE-26802:
-
Author: ASF GitHub Bot
Created on: 18/Jan/23 12:09
Start Date: 18/Jan/23 12:09
Worklog Time Spent: 10m
Work Description: zratkai commented on code in PR #3882:
URL: https://github.com/apache/hive/pull/3882#discussion_r1073453543
##
itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java:
##
@@ -236,7 +240,39 @@ public MiniLlapLocalCliConfig() {
}
}
}
+
+ public static class MiniLlapLocalCompactorCliConfig extends
AbstractCliConfig {
+
+public MiniLlapLocalCompactorCliConfig() {
+ super(CoreCliDriver.class);
+ try {
+setQueryDir("ql/src/test/queries/clientpositive");
+
+includesFrom(testConfigProps, "compaction.query.files");
+setResultsDir("ql/src/test/results/clientpositive/llap");
+setLogDir("itests/qtest/target/qfile-results/clientpositive");
+
+setInitScript("q_test_init.sql");
+setCleanupScript("q_test_cleanup.sql");
+
+setHiveConfDir("data/conf/llap");
+setClusterType(MiniClusterType.LLAP_LOCAL);
+setCustomConfigValueMap(createConfVarsStringMap());
+ } catch (Exception e) {
+throw new RuntimeException("can't construct cliconfig", e);
+ }
+}
+private static Map createConfVarsStringMap() {
+ Map customConfigValueMap = new HashMap<>();
+ customConfigValueMap.put(HiveConf.ConfVars.HIVE_EXECUTION_ENGINE, "tez");
Review Comment:
1. It's true it should be TEZ already from config.
2. Ok, deleting this.
Issue Time Tracking
---
Worklog Id: (was: 839924)
Time Spent: 5h 20m (was: 5h 10m)
> Create qtest running QB compaction queries
> --
>
> Key: HIVE-26802
> URL: https://issues.apache.org/jira/browse/HIVE-26802
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltán Rátkai
>Assignee: Zoltán Rátkai
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 5h 20m
> Remaining Estimate: 0h
>
> Create a qtest that runs the queries that query-based compaction runs.
> Not so much to check for correct data but more to check the query plans, to
> simplify tracing changes in compilation that might affect QB compaction.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-26887) Make sure dirPath has the correct permissions
[
https://issues.apache.org/jira/browse/HIVE-26887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stamatis Zampetakis updated HIVE-26887:
---
Labels: backward-incompatible pull-request-available (was:
pull-request-available)
> Make sure dirPath has the correct permissions
> -
>
> Key: HIVE-26887
> URL: https://issues.apache.org/jira/browse/HIVE-26887
> Project: Hive
> Issue Type: Improvement
>Reporter: Zhang Dongsheng
>Priority: Major
> Labels: backward-incompatible, pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> In the QueryResultsCache function of class QueryResultsCache, there is the
> following code segment
> {code:java}
> private QueryResultsCache(HiveConf configuration) throws IOException {
> ..
> FileSystem fs = cacheDirPath.getFileSystem(conf);
> FsPermission fsPermission = new FsPermission("700");
> fs.mkdirs(cacheDirPath, fsPermission);
> ..
> }
> {code}
> It can be seen that the function will use the mkdirs to create cacheDirPath,
> and the parameters passed in include the path variable cacheDirPath and a
> permission 700. But we haven't confirmed whether the permission is correctly
> assigned to the file.
> The above question is raised because there are two mkdir functions of hadoop,
> {code:java}
> mkdirs(Path f, FsPermission permission)
> {code}
> and
> {code:java}
> mkdirs(FileSystem fs, Path dir, FsPermission permission)
> {code}
> and the first one is used here. The permissions of this function will be
> affected by the underlying umask. Although 700 here will hardly be affected
> by umask, but I think from a rigorous point of view, we should have one more
> permission check and permission grant here.
> And I find same issue in other three methods here.
> In class Context
> {code:java}
> private Path getScratchDir(String scheme, String authority,
> boolean mkdir, String scratchDir) {
> ..
> FileSystem fs = dirPath.getFileSystem(conf);
> dirPath = new Path(fs.makeQualified(dirPath).toString());
> FsPermission fsPermission = new FsPermission(scratchDirPermission);
> if (!fs.mkdirs(dirPath, fsPermission)) {
> throw new RuntimeException("Cannot make directory: "
> + dirPath.toString());
> ..
> }
> {code}
> In class SessionState
> {code:java}
> static void createPath(HiveConf conf, Path path, String permission, boolean
> isLocal,
> boolean isCleanUp) throws IOException {
> FsPermission fsPermission = new FsPermission(permission);
> FileSystem fs;
> ..
> if (!fs.mkdirs(path, fsPermission)) {
> throw new IOException("Failed to create directory " + path + " on fs "
> + fs.getUri());
> }
> ..
> }
> {code}
> and in class TezSessionState
> {code:java}
> private Path createTezDir(String sessionId, String suffix) throws IOException
> {
> ..
> Path tezDir = new Path(hdfsScratchDir, TEZ_DIR);
> FileSystem fs = tezDir.getFileSystem(conf);
> FsPermission fsPermission = new FsPermission(HiveConf.getVar(conf,
> HiveConf.ConfVars.SCRATCHDIRPERMISSION));
> fs.mkdirs(tezDir, fsPermission);
> ..
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26887) Make sure dirPath has the correct permissions
[
https://issues.apache.org/jira/browse/HIVE-26887?focusedWorklogId=839917=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-839917
]
ASF GitHub Bot logged work on HIVE-26887:
-
Author: ASF GitHub Bot
Created on: 18/Jan/23 11:17
Start Date: 18/Jan/23 11:17
Worklog Time Spent: 10m
Work Description: zabetak commented on PR #3894:
URL: https://github.com/apache/hive/pull/3894#issuecomment-1386890501
Thanks for the elaborate analysis and discussion @skysiders @cnauroth !
Looking into the changes it seems that this is kind of a breaking change
since depending on the configuration permissions will be set differently.
Moreover the proposed changes make the code more verbose and less
straightforward.
Furthermore, I am not sure we want to enforce a programming pattern where we
do `fs.mkdirs` and then `fs.setPermission` since like that we essentially
by-pass the umask that is the expected way of creating directories with the
appropriate permissions
(https://issues.apache.org/jira/browse/HDFS-1322?focusedCommentId=13072984=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-13072984).
For the reasons above, I would prefer if we didn't merge these changes.
Issue Time Tracking
---
Worklog Id: (was: 839917)
Time Spent: 1h 40m (was: 1.5h)
> Make sure dirPath has the correct permissions
> -
>
> Key: HIVE-26887
> URL: https://issues.apache.org/jira/browse/HIVE-26887
> Project: Hive
> Issue Type: Improvement
>Reporter: Zhang Dongsheng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> In the QueryResultsCache function of class QueryResultsCache, there is the
> following code segment
> {code:java}
> private QueryResultsCache(HiveConf configuration) throws IOException {
> ..
> FileSystem fs = cacheDirPath.getFileSystem(conf);
> FsPermission fsPermission = new FsPermission("700");
> fs.mkdirs(cacheDirPath, fsPermission);
> ..
> }
> {code}
> It can be seen that the function will use the mkdirs to create cacheDirPath,
> and the parameters passed in include the path variable cacheDirPath and a
> permission 700. But we haven't confirmed whether the permission is correctly
> assigned to the file.
> The above question is raised because there are two mkdir functions of hadoop,
> {code:java}
> mkdirs(Path f, FsPermission permission)
> {code}
> and
> {code:java}
> mkdirs(FileSystem fs, Path dir, FsPermission permission)
> {code}
> and the first one is used here. The permissions of this function will be
> affected by the underlying umask. Although 700 here will hardly be affected
> by umask, but I think from a rigorous point of view, we should have one more
> permission check and permission grant here.
> And I find same issue in other three methods here.
> In class Context
> {code:java}
> private Path getScratchDir(String scheme, String authority,
> boolean mkdir, String scratchDir) {
> ..
> FileSystem fs = dirPath.getFileSystem(conf);
> dirPath = new Path(fs.makeQualified(dirPath).toString());
> FsPermission fsPermission = new FsPermission(scratchDirPermission);
> if (!fs.mkdirs(dirPath, fsPermission)) {
> throw new RuntimeException("Cannot make directory: "
> + dirPath.toString());
> ..
> }
> {code}
> In class SessionState
> {code:java}
> static void createPath(HiveConf conf, Path path, String permission, boolean
> isLocal,
> boolean isCleanUp) throws IOException {
> FsPermission fsPermission = new FsPermission(permission);
> FileSystem fs;
> ..
> if (!fs.mkdirs(path, fsPermission)) {
> throw new IOException("Failed to create directory " + path + " on fs "
> + fs.getUri());
> }
> ..
> }
> {code}
> and in class TezSessionState
> {code:java}
> private Path createTezDir(String sessionId, String suffix) throws IOException
> {
> ..
> Path tezDir = new Path(hdfsScratchDir, TEZ_DIR);
> FileSystem fs = tezDir.getFileSystem(conf);
> FsPermission fsPermission = new FsPermission(HiveConf.getVar(conf,
> HiveConf.ConfVars.SCRATCHDIRPERMISSION));
> fs.mkdirs(tezDir, fsPermission);
> ..
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26955) Select query fails when decimal column data type is changed to string/char/varchar in Parquet
[
https://issues.apache.org/jira/browse/HIVE-26955?focusedWorklogId=839914=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-839914
]
ASF GitHub Bot logged work on HIVE-26955:
-
Author: ASF GitHub Bot
Created on: 18/Jan/23 11:03
Start Date: 18/Jan/23 11:03
Worklog Time Spent: 10m
Work Description: kasakrisz commented on code in PR #3964:
URL: https://github.com/apache/hive/pull/3964#discussion_r1073368036
##
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ETypeConverter.java:
##
@@ -676,6 +676,39 @@ private int getScale(PrimitiveType type) {
return logicalType.getScale();
}
};
+case serdeConstants.VARCHAR_TYPE_NAME:
+ return new BinaryConverter(type, parent, index,
hiveTypeInfo) {
+@Override
+protected HiveVarcharWritable convert(Binary binary) {
+ DecimalLogicalTypeAnnotation logicalType =
(DecimalLogicalTypeAnnotation) type.getLogicalTypeAnnotation();
+ HiveDecimalWritable decimalWritable =
HiveDecimalUtils.enforcePrecisionScale(
+ new HiveDecimalWritable(binary.getBytes(),
logicalType.getScale()),
+ new DecimalTypeInfo(logicalType.getPrecision(),
logicalType.getScale()));
Review Comment:
These 4 lines are the same in all the new cases. Could you please extract it?
Exmaple
```
abstract class BinaryConverterToCharacterType extends
BinaryConverter {
protected byte[] convertToBytes(Binary binary) {
DecimalLogicalTypeAnnotation logicalType =
(DecimalLogicalTypeAnnotation) type.getLogicalTypeAnnotation();
return HiveDecimalUtils.enforcePrecisionScale(
new HiveDecimalWritable(binary.getBytes(),
logicalType.getScale()),
new DecimalTypeInfo(logicalType.getPrecision(),
logicalType.getScale())).toString().getBytes();
}
protected abstract T convert(Binary binary);
}
```
and extend it and call `convertToBytes` in `convert`
Issue Time Tracking
---
Worklog Id: (was: 839914)
Time Spent: 0.5h (was: 20m)
> Select query fails when decimal column data type is changed to
> string/char/varchar in Parquet
> -
>
> Key: HIVE-26955
> URL: https://issues.apache.org/jira/browse/HIVE-26955
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Taraka Rama Rao Lethavadla
>Assignee: Sourabh Badhya
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Steps to reproduce
> {noformat}
> create table test_parquet (id decimal) stored as parquet;
> insert into test_parquet values(238);
> alter table test_parquet change id id string;
> select * from test_parquet;
> Error: java.io.IOException: org.apache.parquet.io.ParquetDecodingException:
> Can not read value at 1 in block 0 in file
> hdfs:/namenode:8020/warehouse/tablespace/managed/hive/test_parquet/delta_001_001_/00_0
> (state=,code=0)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:624)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:531)
> at
> org.apache.hadoop.hive.ql.exec.FetchTask.executeInner(FetchTask.java:194)
> ... 55 more
> Caused by: org.apache.parquet.io.ParquetDecodingException: Can not read value
> at 1 in block 0 in file
> file:/home/centos/Apache-Hive-Tarak/itests/qtest/target/localfs/warehouse/test_parquet/00_0
> at
> org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:255)
> at
> org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:207)
> at
> org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.(ParquetRecordReaderWrapper.java:87)
> at
> org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat.getRecordReader(MapredParquetInputFormat.java:89)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator$FetchInputFormatSplit.getRecordReader(FetchOperator.java:771)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:335)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:562)
> ... 57 more
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo cannot be cast to
> org.apache.hadoop.hive.serde2.typeinfo.DecimalTypeInfo
> at
> org.apache.hadoop.hive.ql.io.parquet.convert.ETypeConverter$8$5.convert(ETypeConverter.java:669)
> at
>
[jira] [Work logged] (HIVE-26955) Select query fails when decimal column data type is changed to string/char/varchar in Parquet
[
https://issues.apache.org/jira/browse/HIVE-26955?focusedWorklogId=839911=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-839911
]
ASF GitHub Bot logged work on HIVE-26955:
-
Author: ASF GitHub Bot
Created on: 18/Jan/23 10:56
Start Date: 18/Jan/23 10:56
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3964:
URL: https://github.com/apache/hive/pull/3964#issuecomment-1386861732
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3964)
[](https://sonarcloud.io/project/issues?id=apache_hive=3964=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3964=false=BUG)
[1
Bug](https://sonarcloud.io/project/issues?id=apache_hive=3964=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3964=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3964=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3964=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3964=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3964=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3964=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3964=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3964=false=CODE_SMELL)
[5 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3964=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3964=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3964=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 839911)
Time Spent: 20m (was: 10m)
> Select query fails when decimal column data type is changed to
> string/char/varchar in Parquet
> -
>
> Key: HIVE-26955
> URL: https://issues.apache.org/jira/browse/HIVE-26955
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Taraka Rama Rao Lethavadla
>Assignee: Sourabh Badhya
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Steps to reproduce
> {noformat}
> create table test_parquet (id decimal) stored as parquet;
> insert into test_parquet values(238);
> alter table test_parquet change id id string;
> select * from test_parquet;
> Error: java.io.IOException: org.apache.parquet.io.ParquetDecodingException:
> Can not read value at 1 in block 0 in file
> hdfs:/namenode:8020/warehouse/tablespace/managed/hive/test_parquet/delta_001_001_/00_0
> (state=,code=0)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:624)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:531)
> at
> org.apache.hadoop.hive.ql.exec.FetchTask.executeInner(FetchTask.java:194)
> ... 55 more
> Caused by: org.apache.parquet.io.ParquetDecodingException: Can not read value
> at 1 in block 0 in file
>
[jira] [Work logged] (HIVE-26915) Backport of HIVE-23692 TestCodahaleMetrics.testFileReporting is flaky
[ https://issues.apache.org/jira/browse/HIVE-26915?focusedWorklogId=839907=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-839907 ] ASF GitHub Bot logged work on HIVE-26915: - Author: ASF GitHub Bot Created on: 18/Jan/23 10:48 Start Date: 18/Jan/23 10:48 Worklog Time Spent: 10m Work Description: amanraj2520 commented on PR #3928: URL: https://github.com/apache/hive/pull/3928#issuecomment-1386851403 @zabetak @abstractdog Please review this. Issue Time Tracking --- Worklog Id: (was: 839907) Time Spent: 1h 20m (was: 1h 10m) > Backport of HIVE-23692 TestCodahaleMetrics.testFileReporting is flaky > - > > Key: HIVE-26915 > URL: https://issues.apache.org/jira/browse/HIVE-26915 > Project: Hive > Issue Type: Sub-task >Reporter: Aman Raj >Assignee: Aman Raj >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > This was committed in master without a HIVE Jira task. This is the commit id > : 130f80445d589cdd82904cea1073c84d1368d079 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26895) Backport of HIVE-22899: Make sure qtests clean up copied files from test directories
[
https://issues.apache.org/jira/browse/HIVE-26895?focusedWorklogId=839908=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-839908
]
ASF GitHub Bot logged work on HIVE-26895:
-
Author: ASF GitHub Bot
Created on: 18/Jan/23 10:48
Start Date: 18/Jan/23 10:48
Worklog Time Spent: 10m
Work Description: amanraj2520 commented on PR #3900:
URL: https://github.com/apache/hive/pull/3900#issuecomment-1386852115
@abstractdog @zabetak Please review and merge this.
Issue Time Tracking
---
Worklog Id: (was: 839908)
Time Spent: 1h 50m (was: 1h 40m)
> Backport of HIVE-22899: Make sure qtests clean up copied files from test
> directories
>
>
> Key: HIVE-26895
> URL: https://issues.apache.org/jira/browse/HIVE-26895
> Project: Hive
> Issue Type: Sub-task
>Reporter: Aman Raj
>Assignee: Aman Raj
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Tests (like avrotblsjoin.q) are failing due to the following errors:
> {code:java}
> Begin query: avrotblsjoin.qTRACE StatusLogger Log4jLoggerFactory.getContext()
> found anchor class org.apache.hadoop.hive.cli.CliDriverTRACE StatusLogger
> Log4jLoggerFactory.getContext() found anchor class
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzerTRACE StatusLogger
> Log4jLoggerFactory.getContext() found anchor class
> org.apache.curator.RetryLoopTRACE StatusLogger
> Log4jLoggerFactory.getContext() found anchor class
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzercp:
> `/home/jenkins/agent/workspace/hive-precommit_PR-3859/itests/qtest/target/tmp/table1.avsc':
> File existsDone query avrotblsjoin.q. succeeded=false, skipped=false.
> ElapsedTime(ms)=41TRACE StatusLogger Log4jLoggerFactory.getContext() found
> anchor class org.apache.curator.RetryLoop {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26802) Create qtest running QB compaction queries
[
https://issues.apache.org/jira/browse/HIVE-26802?focusedWorklogId=839897=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-839897
]
ASF GitHub Bot logged work on HIVE-26802:
-
Author: ASF GitHub Bot
Created on: 18/Jan/23 10:07
Start Date: 18/Jan/23 10:07
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3882:
URL: https://github.com/apache/hive/pull/3882#discussion_r107590
##
itests/qtest/src/test/java/org/apache/hadoop/hive/cli/TestMiniLlapLocalCompactorCliDriver.java:
##
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.cli;
+
+import org.apache.hadoop.hive.cli.control.CliAdapter;
+import org.apache.hadoop.hive.cli.control.CliConfigs;
+import org.apache.hadoop.hive.cli.control.SplitSupport;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.hive.ql.txn.compactor.Worker;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TestRule;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+import org.junit.runners.Parameterized.Parameters;
+
+import java.io.File;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+@RunWith(Parameterized.class)
+public class TestMiniLlapLocalCompactorCliDriver {
Review Comment:
was it reverted? could we try to reuse as much as possible and avoid code
duplication
Issue Time Tracking
---
Worklog Id: (was: 839897)
Time Spent: 5h 10m (was: 5h)
> Create qtest running QB compaction queries
> --
>
> Key: HIVE-26802
> URL: https://issues.apache.org/jira/browse/HIVE-26802
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltán Rátkai
>Assignee: Zoltán Rátkai
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
> Create a qtest that runs the queries that query-based compaction runs.
> Not so much to check for correct data but more to check the query plans, to
> simplify tracing changes in compilation that might affect QB compaction.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-26955) Select query fails when decimal column data type is changed to string/char/varchar in Parquet
[
https://issues.apache.org/jira/browse/HIVE-26955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sourabh Badhya updated HIVE-26955:
--
Summary: Select query fails when decimal column data type is changed to
string/char/varchar in Parquet (was: Alter table fails when decimal column
data type is changed to string/char/varchar in Parquet)
> Select query fails when decimal column data type is changed to
> string/char/varchar in Parquet
> -
>
> Key: HIVE-26955
> URL: https://issues.apache.org/jira/browse/HIVE-26955
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Taraka Rama Rao Lethavadla
>Assignee: Sourabh Badhya
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Steps to reproduce
> {noformat}
> create table test_parquet (id decimal) stored as parquet;
> insert into test_parquet values(238);
> alter table test_parquet change id id string;
> select * from test_parquet;
> Error: java.io.IOException: org.apache.parquet.io.ParquetDecodingException:
> Can not read value at 1 in block 0 in file
> hdfs:/namenode:8020/warehouse/tablespace/managed/hive/test_parquet/delta_001_001_/00_0
> (state=,code=0)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:624)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:531)
> at
> org.apache.hadoop.hive.ql.exec.FetchTask.executeInner(FetchTask.java:194)
> ... 55 more
> Caused by: org.apache.parquet.io.ParquetDecodingException: Can not read value
> at 1 in block 0 in file
> file:/home/centos/Apache-Hive-Tarak/itests/qtest/target/localfs/warehouse/test_parquet/00_0
> at
> org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:255)
> at
> org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:207)
> at
> org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.(ParquetRecordReaderWrapper.java:87)
> at
> org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat.getRecordReader(MapredParquetInputFormat.java:89)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator$FetchInputFormatSplit.getRecordReader(FetchOperator.java:771)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:335)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:562)
> ... 57 more
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo cannot be cast to
> org.apache.hadoop.hive.serde2.typeinfo.DecimalTypeInfo
> at
> org.apache.hadoop.hive.ql.io.parquet.convert.ETypeConverter$8$5.convert(ETypeConverter.java:669)
> at
> org.apache.hadoop.hive.ql.io.parquet.convert.ETypeConverter$8$5.convert(ETypeConverter.java:664)
> at
> org.apache.hadoop.hive.ql.io.parquet.convert.ETypeConverter$BinaryConverter.addBinary(ETypeConverter.java:977)
> at
> org.apache.parquet.column.impl.ColumnReaderBase$2$6.writeValue(ColumnReaderBase.java:360)
> at
> org.apache.parquet.column.impl.ColumnReaderBase.writeCurrentValueToConverter(ColumnReaderBase.java:410)
> at
> org.apache.parquet.column.impl.ColumnReaderImpl.writeCurrentValueToConverter(ColumnReaderImpl.java:30)
> at
> org.apache.parquet.io.RecordReaderImplementation.read(RecordReaderImplementation.java:406)
> at
> org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:230)
> ... 63 more{noformat}
> However the same is working as expected in ORC table
> {noformat}
> create table test_orc (id decimal) stored as orc;
> insert into test_orc values(238);
> alter table test_orc change id id string;
> select * from test_orc;
> +--+
> | test_orc.id |
> +--+
> | 238 |
> +--+{noformat}
> As well as text table
> {noformat}
> create table test_text (id decimal) stored as textfile;
> insert into test_text values(238);
> alter table test_text change id id string;
> select * from test_text;
> +---+
> | test_text.id |
> +---+
> | 238 |
> +---+{noformat}
> Similar exception is thrown when the altered datatype is varchar and char
> datatype.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26802) Create qtest running QB compaction queries
[
https://issues.apache.org/jira/browse/HIVE-26802?focusedWorklogId=839896=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-839896
]
ASF GitHub Bot logged work on HIVE-26802:
-
Author: ASF GitHub Bot
Created on: 18/Jan/23 10:04
Start Date: 18/Jan/23 10:04
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3882:
URL: https://github.com/apache/hive/pull/3882#discussion_r1073331086
##
itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java:
##
@@ -236,7 +240,39 @@ public MiniLlapLocalCliConfig() {
}
}
}
+
+ public static class MiniLlapLocalCompactorCliConfig extends
AbstractCliConfig {
+
+public MiniLlapLocalCompactorCliConfig() {
+ super(CoreCliDriver.class);
+ try {
+setQueryDir("ql/src/test/queries/clientpositive");
+
+includesFrom(testConfigProps, "compaction.query.files");
+setResultsDir("ql/src/test/results/clientpositive/llap");
+setLogDir("itests/qtest/target/qfile-results/clientpositive");
+
+setInitScript("q_test_init.sql");
+setCleanupScript("q_test_cleanup.sql");
+
+setHiveConfDir("data/conf/llap");
+setClusterType(MiniClusterType.LLAP_LOCAL);
+setCustomConfigValueMap(createConfVarsStringMap());
+ } catch (Exception e) {
+throw new RuntimeException("can't construct cliconfig", e);
+ }
+}
+private static Map createConfVarsStringMap() {
+ Map customConfigValueMap = new HashMap<>();
+ customConfigValueMap.put(HiveConf.ConfVars.HIVE_EXECUTION_ENGINE, "tez");
Review Comment:
1. Why do we need to explicitly set the exec engine? isn't it already TEZ
2. What if we want to create tests for the MR compaction, I think that one
should stay default. In q-tests, we can change to query-based. WDYT?
Issue Time Tracking
---
Worklog Id: (was: 839896)
Time Spent: 5h (was: 4h 50m)
> Create qtest running QB compaction queries
> --
>
> Key: HIVE-26802
> URL: https://issues.apache.org/jira/browse/HIVE-26802
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltán Rátkai
>Assignee: Zoltán Rátkai
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 5h
> Remaining Estimate: 0h
>
> Create a qtest that runs the queries that query-based compaction runs.
> Not so much to check for correct data but more to check the query plans, to
> simplify tracing changes in compilation that might affect QB compaction.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26928) LlapIoImpl::getParquetFooterBuffersFromCache throws exception when metadata cache is disabled
[
https://issues.apache.org/jira/browse/HIVE-26928?focusedWorklogId=839889=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-839889
]
ASF GitHub Bot logged work on HIVE-26928:
-
Author: ASF GitHub Bot
Created on: 18/Jan/23 09:55
Start Date: 18/Jan/23 09:55
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3962:
URL: https://github.com/apache/hive/pull/3962#issuecomment-1386778230
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3962)
[](https://sonarcloud.io/project/issues?id=apache_hive=3962=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3962=false=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3962=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3962=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3962=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3962=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3962=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3962=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3962=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3962=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3962=false=CODE_SMELL)
[0 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3962=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3962=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3962=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 839889)
Time Spent: 0.5h (was: 20m)
> LlapIoImpl::getParquetFooterBuffersFromCache throws exception when metadata
> cache is disabled
> -
>
> Key: HIVE-26928
> URL: https://issues.apache.org/jira/browse/HIVE-26928
> Project: Hive
> Issue Type: Improvement
> Components: Iceberg integration
>Reporter: Rajesh Balamohan
>Assignee: Simhadri Govindappa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> When metadata / LLAP cache is disabled, "iceberg + parquet" throws the
> following error. "{color:#5a656d}hive.llap.io.memory.mode=none"{color}
> It should check for "metadatacache" correctly or fix it in LlapIoImpl.
>
> {noformat}
> Caused by: java.lang.NullPointerException: Metadata cache must not be null
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:897)
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapIoImpl.getParquetFooterBuffersFromCache(LlapIoImpl.java:467)
> at
> org.apache.iceberg.mr.hive.vector.HiveVectorizedReader.parquetRecordReader(HiveVectorizedReader.java:227)
> at
> org.apache.iceberg.mr.hive.vector.HiveVectorizedReader.reader(HiveVectorizedReader.java:162)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
>
[jira] [Work logged] (HIVE-26599) Fix NPE encountered in second dump cycle of optimised bootstrap
[
https://issues.apache.org/jira/browse/HIVE-26599?focusedWorklogId=839888=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-839888
]
ASF GitHub Bot logged work on HIVE-26599:
-
Author: ASF GitHub Bot
Created on: 18/Jan/23 09:54
Start Date: 18/Jan/23 09:54
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3963:
URL: https://github.com/apache/hive/pull/3963#issuecomment-1386777389
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3963)
[](https://sonarcloud.io/project/issues?id=apache_hive=3963=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3963=false=BUG)
[2
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3963=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3963=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3963=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3963=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3963=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3963=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3963=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3963=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3963=false=CODE_SMELL)
[0 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3963=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3963=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3963=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 839888)
Time Spent: 20m (was: 10m)
> Fix NPE encountered in second dump cycle of optimised bootstrap
> ---
>
> Key: HIVE-26599
> URL: https://issues.apache.org/jira/browse/HIVE-26599
> Project: Hive
> Issue Type: Bug
>Reporter: Teddy Choi
>Assignee: Vinit Patni
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> After creating reverse replication policy after failover is completed from
> Primary to DR cluster and DR takes over. First dump and load cycle of
> optimised bootstrap is completing successfully, But We are encountering Null
> pointer exception in the second dump cycle which is halting this reverse
> replication and major blocker to test complete cycle of replication.
> {code:java}
> Scheduled Query Executor(schedule:repl_reverse, execution_id:14)]: FAILED:
> Execution Error, return code -101 from
> org.apache.hadoop.hive.ql.exec.repl.ReplDumpTask.
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.parse.repl.metric.ReplicationMetricCollector.reportStageProgress(ReplicationMetricCollector.java:192)
> at
> org.apache.hadoop.hive.ql.exec.repl.ReplDumpTask.dumpTable(ReplDumpTask.java:1458)
> at
> org.apache.hadoop.hive.ql.exec.repl.ReplDumpTask.incrementalDump(ReplDumpTask.java:961)
> at
>
[jira] [Resolved] (HIVE-26943) Fix NPE during Optimised Bootstrap when db is dropped
[ https://issues.apache.org/jira/browse/HIVE-26943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shreenidhi resolved HIVE-26943. --- Resolution: Fixed > Fix NPE during Optimised Bootstrap when db is dropped > - > > Key: HIVE-26943 > URL: https://issues.apache.org/jira/browse/HIVE-26943 > Project: Hive > Issue Type: Sub-task >Reporter: Shreenidhi >Assignee: Shreenidhi >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > Consider the steps: > 1. Current replication is from A (source) -> B(target) > 2. Failover is complete > so now A (target) <- B(source) > 3. Suppose db at A is dropped before reverse replication. > 4. Now when reverse replication triggers optimised bootstrap it will throw NPE > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26955) Alter table fails when decimal column data type is changed to string/char/varchar in Parquet
[
https://issues.apache.org/jira/browse/HIVE-26955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sourabh Badhya updated HIVE-26955:
--
Status: Patch Available (was: Open)
> Alter table fails when decimal column data type is changed to
> string/char/varchar in Parquet
>
>
> Key: HIVE-26955
> URL: https://issues.apache.org/jira/browse/HIVE-26955
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Taraka Rama Rao Lethavadla
>Assignee: Sourabh Badhya
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Steps to reproduce
> {noformat}
> create table test_parquet (id decimal) stored as parquet;
> insert into test_parquet values(238);
> alter table test_parquet change id id string;
> select * from test_parquet;
> Error: java.io.IOException: org.apache.parquet.io.ParquetDecodingException:
> Can not read value at 1 in block 0 in file
> hdfs:/namenode:8020/warehouse/tablespace/managed/hive/test_parquet/delta_001_001_/00_0
> (state=,code=0)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:624)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:531)
> at
> org.apache.hadoop.hive.ql.exec.FetchTask.executeInner(FetchTask.java:194)
> ... 55 more
> Caused by: org.apache.parquet.io.ParquetDecodingException: Can not read value
> at 1 in block 0 in file
> file:/home/centos/Apache-Hive-Tarak/itests/qtest/target/localfs/warehouse/test_parquet/00_0
> at
> org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:255)
> at
> org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:207)
> at
> org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.(ParquetRecordReaderWrapper.java:87)
> at
> org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat.getRecordReader(MapredParquetInputFormat.java:89)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator$FetchInputFormatSplit.getRecordReader(FetchOperator.java:771)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:335)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:562)
> ... 57 more
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo cannot be cast to
> org.apache.hadoop.hive.serde2.typeinfo.DecimalTypeInfo
> at
> org.apache.hadoop.hive.ql.io.parquet.convert.ETypeConverter$8$5.convert(ETypeConverter.java:669)
> at
> org.apache.hadoop.hive.ql.io.parquet.convert.ETypeConverter$8$5.convert(ETypeConverter.java:664)
> at
> org.apache.hadoop.hive.ql.io.parquet.convert.ETypeConverter$BinaryConverter.addBinary(ETypeConverter.java:977)
> at
> org.apache.parquet.column.impl.ColumnReaderBase$2$6.writeValue(ColumnReaderBase.java:360)
> at
> org.apache.parquet.column.impl.ColumnReaderBase.writeCurrentValueToConverter(ColumnReaderBase.java:410)
> at
> org.apache.parquet.column.impl.ColumnReaderImpl.writeCurrentValueToConverter(ColumnReaderImpl.java:30)
> at
> org.apache.parquet.io.RecordReaderImplementation.read(RecordReaderImplementation.java:406)
> at
> org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:230)
> ... 63 more{noformat}
> However the same is working as expected in ORC table
> {noformat}
> create table test_orc (id decimal) stored as orc;
> insert into test_orc values(238);
> alter table test_orc change id id string;
> select * from test_orc;
> +--+
> | test_orc.id |
> +--+
> | 238 |
> +--+{noformat}
> As well as text table
> {noformat}
> create table test_text (id decimal) stored as textfile;
> insert into test_text values(238);
> alter table test_text change id id string;
> select * from test_text;
> +---+
> | test_text.id |
> +---+
> | 238 |
> +---+{noformat}
> Similar exception is thrown when the altered datatype is varchar and char
> datatype.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26955) Alter table fails when decimal column data type is changed to string/char/varchar in Parquet
[
https://issues.apache.org/jira/browse/HIVE-26955?focusedWorklogId=839873=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-839873
]
ASF GitHub Bot logged work on HIVE-26955:
-
Author: ASF GitHub Bot
Created on: 18/Jan/23 09:21
Start Date: 18/Jan/23 09:21
Worklog Time Spent: 10m
Work Description: SourabhBadhya opened a new pull request, #3964:
URL: https://github.com/apache/hive/pull/3964
…d to string/char/varchar in Parquet
### What changes were proposed in this pull request?
Alter table fails when decimal column data type is changed to
string/char/varchar in Parquet
### Why are the changes needed?
To fix the failure induced by alter table when decimal column data type is
changed to string/char/varchar in Parquet.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
QTest
Issue Time Tracking
---
Worklog Id: (was: 839873)
Remaining Estimate: 0h
Time Spent: 10m
> Alter table fails when decimal column data type is changed to
> string/char/varchar in Parquet
>
>
> Key: HIVE-26955
> URL: https://issues.apache.org/jira/browse/HIVE-26955
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Taraka Rama Rao Lethavadla
>Assignee: Sourabh Badhya
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Steps to reproduce
> {noformat}
> create table test_parquet (id decimal) stored as parquet;
> insert into test_parquet values(238);
> alter table test_parquet change id id string;
> select * from test_parquet;
> Error: java.io.IOException: org.apache.parquet.io.ParquetDecodingException:
> Can not read value at 1 in block 0 in file
> hdfs:/namenode:8020/warehouse/tablespace/managed/hive/test_parquet/delta_001_001_/00_0
> (state=,code=0)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:624)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:531)
> at
> org.apache.hadoop.hive.ql.exec.FetchTask.executeInner(FetchTask.java:194)
> ... 55 more
> Caused by: org.apache.parquet.io.ParquetDecodingException: Can not read value
> at 1 in block 0 in file
> file:/home/centos/Apache-Hive-Tarak/itests/qtest/target/localfs/warehouse/test_parquet/00_0
> at
> org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:255)
> at
> org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:207)
> at
> org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.(ParquetRecordReaderWrapper.java:87)
> at
> org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat.getRecordReader(MapredParquetInputFormat.java:89)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator$FetchInputFormatSplit.getRecordReader(FetchOperator.java:771)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:335)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:562)
> ... 57 more
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo cannot be cast to
> org.apache.hadoop.hive.serde2.typeinfo.DecimalTypeInfo
> at
> org.apache.hadoop.hive.ql.io.parquet.convert.ETypeConverter$8$5.convert(ETypeConverter.java:669)
> at
> org.apache.hadoop.hive.ql.io.parquet.convert.ETypeConverter$8$5.convert(ETypeConverter.java:664)
> at
> org.apache.hadoop.hive.ql.io.parquet.convert.ETypeConverter$BinaryConverter.addBinary(ETypeConverter.java:977)
> at
> org.apache.parquet.column.impl.ColumnReaderBase$2$6.writeValue(ColumnReaderBase.java:360)
> at
> org.apache.parquet.column.impl.ColumnReaderBase.writeCurrentValueToConverter(ColumnReaderBase.java:410)
> at
> org.apache.parquet.column.impl.ColumnReaderImpl.writeCurrentValueToConverter(ColumnReaderImpl.java:30)
> at
> org.apache.parquet.io.RecordReaderImplementation.read(RecordReaderImplementation.java:406)
> at
> org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:230)
> ... 63 more{noformat}
> However the same is working as expected in ORC table
> {noformat}
> create table test_orc (id decimal) stored as orc;
> insert into test_orc values(238);
> alter table test_orc change id id string;
> select * from test_orc;
> +--+
> | test_orc.id |
> +--+
> | 238 |
> +--+{noformat}
> As well as text table
> {noformat}
> create table test_text (id decimal) stored as textfile;
> insert into
[jira] [Updated] (HIVE-26955) Alter table fails when decimal column data type is changed to string/char/varchar in Parquet
[
https://issues.apache.org/jira/browse/HIVE-26955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-26955:
--
Labels: pull-request-available (was: )
> Alter table fails when decimal column data type is changed to
> string/char/varchar in Parquet
>
>
> Key: HIVE-26955
> URL: https://issues.apache.org/jira/browse/HIVE-26955
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Taraka Rama Rao Lethavadla
>Assignee: Sourabh Badhya
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Steps to reproduce
> {noformat}
> create table test_parquet (id decimal) stored as parquet;
> insert into test_parquet values(238);
> alter table test_parquet change id id string;
> select * from test_parquet;
> Error: java.io.IOException: org.apache.parquet.io.ParquetDecodingException:
> Can not read value at 1 in block 0 in file
> hdfs:/namenode:8020/warehouse/tablespace/managed/hive/test_parquet/delta_001_001_/00_0
> (state=,code=0)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:624)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:531)
> at
> org.apache.hadoop.hive.ql.exec.FetchTask.executeInner(FetchTask.java:194)
> ... 55 more
> Caused by: org.apache.parquet.io.ParquetDecodingException: Can not read value
> at 1 in block 0 in file
> file:/home/centos/Apache-Hive-Tarak/itests/qtest/target/localfs/warehouse/test_parquet/00_0
> at
> org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:255)
> at
> org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:207)
> at
> org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.(ParquetRecordReaderWrapper.java:87)
> at
> org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat.getRecordReader(MapredParquetInputFormat.java:89)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator$FetchInputFormatSplit.getRecordReader(FetchOperator.java:771)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:335)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:562)
> ... 57 more
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo cannot be cast to
> org.apache.hadoop.hive.serde2.typeinfo.DecimalTypeInfo
> at
> org.apache.hadoop.hive.ql.io.parquet.convert.ETypeConverter$8$5.convert(ETypeConverter.java:669)
> at
> org.apache.hadoop.hive.ql.io.parquet.convert.ETypeConverter$8$5.convert(ETypeConverter.java:664)
> at
> org.apache.hadoop.hive.ql.io.parquet.convert.ETypeConverter$BinaryConverter.addBinary(ETypeConverter.java:977)
> at
> org.apache.parquet.column.impl.ColumnReaderBase$2$6.writeValue(ColumnReaderBase.java:360)
> at
> org.apache.parquet.column.impl.ColumnReaderBase.writeCurrentValueToConverter(ColumnReaderBase.java:410)
> at
> org.apache.parquet.column.impl.ColumnReaderImpl.writeCurrentValueToConverter(ColumnReaderImpl.java:30)
> at
> org.apache.parquet.io.RecordReaderImplementation.read(RecordReaderImplementation.java:406)
> at
> org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:230)
> ... 63 more{noformat}
> However the same is working as expected in ORC table
> {noformat}
> create table test_orc (id decimal) stored as orc;
> insert into test_orc values(238);
> alter table test_orc change id id string;
> select * from test_orc;
> +--+
> | test_orc.id |
> +--+
> | 238 |
> +--+{noformat}
> As well as text table
> {noformat}
> create table test_text (id decimal) stored as textfile;
> insert into test_text values(238);
> alter table test_text change id id string;
> select * from test_text;
> +---+
> | test_text.id |
> +---+
> | 238 |
> +---+{noformat}
> Similar exception is thrown when the altered datatype is varchar and char
> datatype.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26943) Fix NPE during Optimised Bootstrap when db is dropped
[ https://issues.apache.org/jira/browse/HIVE-26943?focusedWorklogId=839867=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-839867 ] ASF GitHub Bot logged work on HIVE-26943: - Author: ASF GitHub Bot Created on: 18/Jan/23 09:09 Start Date: 18/Jan/23 09:09 Worklog Time Spent: 10m Work Description: pudidic merged PR #3953: URL: https://github.com/apache/hive/pull/3953 Issue Time Tracking --- Worklog Id: (was: 839867) Time Spent: 1h 20m (was: 1h 10m) > Fix NPE during Optimised Bootstrap when db is dropped > - > > Key: HIVE-26943 > URL: https://issues.apache.org/jira/browse/HIVE-26943 > Project: Hive > Issue Type: Sub-task >Reporter: Shreenidhi >Assignee: Shreenidhi >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > Consider the steps: > 1. Current replication is from A (source) -> B(target) > 2. Failover is complete > so now A (target) <- B(source) > 3. Suppose db at A is dropped before reverse replication. > 4. Now when reverse replication triggers optimised bootstrap it will throw NPE > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26943) Fix NPE during Optimised Bootstrap when db is dropped
[ https://issues.apache.org/jira/browse/HIVE-26943?focusedWorklogId=839865=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-839865 ] ASF GitHub Bot logged work on HIVE-26943: - Author: ASF GitHub Bot Created on: 18/Jan/23 09:08 Start Date: 18/Jan/23 09:08 Worklog Time Spent: 10m Work Description: pudidic commented on PR #3953: URL: https://github.com/apache/hive/pull/3953#issuecomment-1386716704 LGTM. I will merge it. Issue Time Tracking --- Worklog Id: (was: 839865) Time Spent: 1h 10m (was: 1h) > Fix NPE during Optimised Bootstrap when db is dropped > - > > Key: HIVE-26943 > URL: https://issues.apache.org/jira/browse/HIVE-26943 > Project: Hive > Issue Type: Sub-task >Reporter: Shreenidhi >Assignee: Shreenidhi >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > Consider the steps: > 1. Current replication is from A (source) -> B(target) > 2. Failover is complete > so now A (target) <- B(source) > 3. Suppose db at A is dropped before reverse replication. > 4. Now when reverse replication triggers optimised bootstrap it will throw NPE > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26922) Deadlock when rebuilding Materialized view stored by Iceberg
[
https://issues.apache.org/jira/browse/HIVE-26922?focusedWorklogId=839862=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-839862
]
ASF GitHub Bot logged work on HIVE-26922:
-
Author: ASF GitHub Bot
Created on: 18/Jan/23 08:59
Start Date: 18/Jan/23 08:59
Worklog Time Spent: 10m
Work Description: kasakrisz commented on code in PR #3934:
URL: https://github.com/apache/hive/pull/3934#discussion_r1073259594
##
ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java:
##
@@ -3122,7 +3117,19 @@ Seems much cleaner if each stmt is identified as a
particular HiveOperation (whi
}
return lockComponents;
}
-
+
+ private static LockType getLockTypeFromStorageHandler(WriteEntity output,
Table t) {
+final HiveStorageHandler storageHandler =
Preconditions.checkNotNull(t.getStorageHandler(),
+"Non-native tables must have an instance of storage handler.");
+LockType lockType = storageHandler.getLockType(output);
+if (null == LockType.findByValue(lockType.getValue())) {
+ throw new IllegalArgumentException(String
+ .format("Lock type [%s] for Database.Table [%s.%s] is unknown",
lockType, t.getDbName(),
Review Comment:
`getCompleteName()` uses `@` as separator char. I think without quoting db
and table identifiers both separator chars (`.` and `@`) adds the same value
since both of them can be part of the identifier.
In this case this is not an issue because we just add the name to an
exception message and the message also contains the format.
Issue Time Tracking
---
Worklog Id: (was: 839862)
Time Spent: 1.5h (was: 1h 20m)
> Deadlock when rebuilding Materialized view stored by Iceberg
>
>
> Key: HIVE-26922
> URL: https://issues.apache.org/jira/browse/HIVE-26922
> Project: Hive
> Issue Type: Bug
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> {code}
> create table tbl_ice(a int, b string, c int) stored by iceberg stored as orc
> tblproperties ('format-version'='1');
> insert into tbl_ice values (1, 'one', 50), (2, 'two', 51), (3, 'three', 52),
> (4, 'four', 53), (5, 'five', 54);
> create materialized view mat1 stored by iceberg stored as orc tblproperties
> ('format-version'='1') as
> select tbl_ice.b, tbl_ice.c from tbl_ice where tbl_ice.c > 52;
> insert into tbl_ice values (10, 'ten', 60);
> alter materialized view mat1 rebuild;
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26924) Alter materialized view enable rewrite throws SemanticException for source iceberg table
[
https://issues.apache.org/jira/browse/HIVE-26924?focusedWorklogId=839859=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-839859
]
ASF GitHub Bot logged work on HIVE-26924:
-
Author: ASF GitHub Bot
Created on: 18/Jan/23 08:53
Start Date: 18/Jan/23 08:53
Worklog Time Spent: 10m
Work Description: kasakrisz commented on code in PR #3936:
URL: https://github.com/apache/hive/pull/3936#discussion_r1073252916
##
ql/src/java/org/apache/hadoop/hive/ql/ddl/view/materialized/alter/rewrite/AlterMaterializedViewRewriteAnalyzer.java:
##
@@ -68,10 +68,12 @@ public void analyzeInternal(ASTNode root) throws
SemanticException {
Table materializedViewTable = getTable(tableName, true);
// One last test: if we are enabling the rewrite, we need to check that
query
-// only uses transactional (MM and ACID) tables
+// only uses transactional (MM and ACID and Iceberg) tables
if (rewriteEnable) {
for (SourceTable sourceTable :
materializedViewTable.getMVMetadata().getSourceTables()) {
-if (!AcidUtils.isTransactionalTable(sourceTable.getTable())) {
+Table table = new Table(sourceTable.getTable());
+if (!AcidUtils.isTransactionalTable(sourceTable.getTable()) &&
+!(table.isNonNative() &&
table.getStorageHandler().areSnapshotsSupported())) {
Review Comment:
Non-native tables must have a storage handler.
Currently none of the native table formats supports snapshots.
Issue Time Tracking
---
Worklog Id: (was: 839859)
Time Spent: 50m (was: 40m)
> Alter materialized view enable rewrite throws SemanticException for source
> iceberg table
>
>
> Key: HIVE-26924
> URL: https://issues.apache.org/jira/browse/HIVE-26924
> Project: Hive
> Issue Type: Bug
> Components: Iceberg integration
>Reporter: Dharmik Thakkar
>Assignee: Krisztian Kasa
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> alter materialized view enable rewrite throws SemanticException for source
> iceberg table
> SQL test
> {code:java}
> >>> create materialized view mv_rewrite as select t, si from all100k where
> >>> t>115;
> >>> analyze table mv_rewrite compute statistics for columns;
> >>> set hive.explain.user=false;
> >>> explain select si,t from all100k where t>116 and t<120;
> !!! match row_contains
> alias: iceberg_test_db_hive.mv_rewrite
> >>> alter materialized view mv_rewrite disable rewrite;
> >>> explain select si,t from all100k where t>116 and t<120;
> !!! match row_contains
> alias: all100k
> >>> alter materialized view mv_rewrite enable rewrite;
> >>> explain select si,t from all100k where t>116 and t<120;
> !!! match row_contains
> alias: iceberg_test_db_hive.mv_rewrite
> >>> drop materialized view mv_rewrite; {code}
>
> Error
> {code:java}
> 2023-01-10T18:40:34,303 INFO [pool-3-thread-1] jdbc.TestDriver: Query: alter
> materialized view mv_rewrite enable rewrite
> 2023-01-10T18:40:34,365 INFO [Thread-10] jdbc.TestDriver: INFO : Compiling
> command(queryId=hive_20230110184034_f557b4a6-40a0-42ba-8e67-2f273f50af36):
> alter materialized view mv_rewrite enable rewrite
> 2023-01-10T18:40:34,426 INFO [Thread-10] jdbc.TestDriver: ERROR : FAILED:
> SemanticException Automatic rewriting for materialized view cannot be enabled
> if the materialized view uses non-transactional tables
> 2023-01-10T18:40:34,426 INFO [Thread-10] jdbc.TestDriver:
> org.apache.hadoop.hive.ql.parse.SemanticException: Automatic rewriting for
> materialized view cannot be enabled if the materialized view uses
> non-transactional tables
> 2023-01-10T18:40:34,426 INFO [Thread-10] jdbc.TestDriver: at
> org.apache.hadoop.hive.ql.ddl.view.materialized.alter.rewrite.AlterMaterializedViewRewriteAnalyzer.analyzeInternal(AlterMaterializedViewRewriteAnalyzer.java:75)
> 2023-01-10T18:40:34,426 INFO [Thread-10] jdbc.TestDriver: at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:313)
> 2023-01-10T18:40:34,427 INFO [Thread-10] jdbc.TestDriver: at
> org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:222)
> 2023-01-10T18:40:34,427 INFO [Thread-10] jdbc.TestDriver: at
> org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:105)
> 2023-01-10T18:40:34,427 INFO [Thread-10] jdbc.TestDriver: at
> org.apache.hadoop.hive.ql.Driver.compile(Driver.java:201)
> 2023-01-10T18:40:34,427 INFO [Thread-10] jdbc.TestDriver: at
> org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:657)
> 2023-01-10T18:40:34,427 INFO [Thread-10] jdbc.TestDriver: at
>
[jira] [Resolved] (HIVE-26717) Query based Rebalance compaction on insert-only tables
[ https://issues.apache.org/jira/browse/HIVE-26717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa resolved HIVE-26717. --- Resolution: Fixed Merged to master. Thanks [~veghlaci05] for the patch and [~dkuzmenko] for review. > Query based Rebalance compaction on insert-only tables > -- > > Key: HIVE-26717 > URL: https://issues.apache.org/jira/browse/HIVE-26717 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: László Végh >Assignee: László Végh >Priority: Major > Labels: ACID, compaction, pull-request-available > Time Spent: 2.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26717) Query based Rebalance compaction on insert-only tables
[ https://issues.apache.org/jira/browse/HIVE-26717?focusedWorklogId=839855=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-839855 ] ASF GitHub Bot logged work on HIVE-26717: - Author: ASF GitHub Bot Created on: 18/Jan/23 08:45 Start Date: 18/Jan/23 08:45 Worklog Time Spent: 10m Work Description: kasakrisz merged PR #3935: URL: https://github.com/apache/hive/pull/3935 Issue Time Tracking --- Worklog Id: (was: 839855) Time Spent: 2.5h (was: 2h 20m) > Query based Rebalance compaction on insert-only tables > -- > > Key: HIVE-26717 > URL: https://issues.apache.org/jira/browse/HIVE-26717 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: László Végh >Assignee: László Végh >Priority: Major > Labels: ACID, compaction, pull-request-available > Time Spent: 2.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-26963) Unset repl.faliover.endpoint during second cycle of optimized bootstrap.
[ https://issues.apache.org/jira/browse/HIVE-26963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rakshith C reassigned HIVE-26963: - > Unset repl.faliover.endpoint during second cycle of optimized bootstrap. > > > Key: HIVE-26963 > URL: https://issues.apache.org/jira/browse/HIVE-26963 > Project: Hive > Issue Type: Bug >Reporter: Rakshith C >Assignee: Rakshith C >Priority: Major > > Scenario: > When second cycle of optimized bootstrap is initiated from DR to PROD. > repl.failover.endpoint is not unset on DR. > Due to this background threads like statsUpdater, partitionManagement etc. > ignore the replicated db on DR. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-22173) Query with multiple lateral views hangs during compilation
[
https://issues.apache.org/jira/browse/HIVE-22173?focusedWorklogId=839849=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-839849
]
ASF GitHub Bot logged work on HIVE-22173:
-
Author: ASF GitHub Bot
Created on: 18/Jan/23 08:12
Start Date: 18/Jan/23 08:12
Worklog Time Spent: 10m
Work Description: amansinha100 commented on PR #3852:
URL: https://github.com/apache/hive/pull/3852#issuecomment-1386647977
> > Also, the commit message mentions partition pruning but I didn't see
changes related to that (I might have missed it).
>
> @amansinha100 The partition pruning optimization also relies on the
present of the synthetic `IN (...)` predicates generated by
`SyntheticJoinPredicate` transformation thus it is also affected by the changes
here. For more details:
>
>
https://github.com/apache/hive/blob/ad0ab58d9945b9a4727ab606f566e1d346bbd20b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/DynamicPartitionPruningOptimization.java#L91
Ah ok. I was looking for a direct dependency.
Issue Time Tracking
---
Worklog Id: (was: 839849)
Time Spent: 2h 10m (was: 2h)
> Query with multiple lateral views hangs during compilation
> --
>
> Key: HIVE-22173
> URL: https://issues.apache.org/jira/browse/HIVE-22173
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.1.1, 4.0.0-alpha-1
> Environment: Hive-3.1.1, Java-8
>Reporter: Rajkumar Singh
>Assignee: Stamatis Zampetakis
>Priority: Critical
> Labels: pull-request-available
> Attachments: op_plan_4_lateral_views.pdf, thread-progress.log
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> Steps To Repro:
> {code:java}
> -- create table
> CREATE EXTERNAL TABLE `jsontable`(
> `json_string` string)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.mapred.TextInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' ;
> -- Run explain of the query
> explain SELECT
> *
> FROM jsontable
> lateral view
> explode(split(regexp_replace(get_json_object(jsontable.json_string,
> '$.jsonfield.addr.city'), "\\[|\\]|\"", ""),',')) t1 as c1
> lateral view
> explode(split(regexp_replace(get_json_object(jsontable.json_string,
> '$.jsonfield.addr.country'), "\\[|\\]|\"", ""),',')) t2 as c2
> lateral view
> explode(split(regexp_replace(get_json_object(jsontable.json_string,
> '$.jsonfield.addr'), "\\[|\\]|\"", ""),',')) t3 as c3
> lateral view
> explode(split(regexp_replace(get_json_object(jsontable.json_string,
> '$.jsonfield.addr.postalCode'), "\\[|\\]|\"", ""),',')) t4 as c4
> lateral view
> explode(split(regexp_replace(get_json_object(jsontable.json_string,
> '$.jsonfield.addr.state'), "\\[|\\]|\"", ""),',')) t5 as c5
> lateral view
> explode(split(regexp_replace(get_json_object(jsontable.json_string,
> '$.jsonfield.addr.streetAddressLine'), "\\[|\\]|\"", ""),',')) t6 as c6
> lateral view
> explode(split(regexp_replace(get_json_object(jsontable.json_string,
> '$.jsonfield.dummyfield'), "\\[|\\]|\"", ""),',')) t7 as c7
> lateral view
> explode(split(regexp_replace(get_json_object(jsontable.json_string,
> '$.jsonfield.dummyfield'), "\\[|\\]|\"", ""),',')) t8 as c8
> lateral view
> explode(split(regexp_replace(get_json_object(jsontable.json_string,
> '$.jsonfield.dummyfield.name.suffix'), "\\[|\\]|\"", ""),',')) t9 as c9
> lateral view
> explode(split(regexp_replace(get_json_object(jsontable.json_string,
> '$.jsonfield.id.extension'), "\\[|\\]|\"", ""),',')) t10 as c10
> lateral view
> explode(split(regexp_replace(get_json_object(jsontable.json_string,
> '$.jsonfield.id'), "\\[|\\]|\"", ""),',')) t11 as c11
> lateral view
> explode(split(regexp_replace(get_json_object(jsontable.json_string,
> '$.jsonfield.id.root'), "\\[|\\]|\"", ""),',')) t12 as c12
> lateral view
> explode(split(regexp_replace(get_json_object(jsontable.json_string,
> '$.jsonfield.telecom.'), "\\[|\\]|\"", ""),',')) t13 as c13
> lateral view
> explode(split(regexp_replace(get_json_object(jsontable.json_string,
> '$.jsonfield.dummyfield1.use'), "\\[|\\]|\"", ""),',')) t14 as c14
> lateral view
> explode(split(regexp_replace(get_json_object(jsontable.json_string,
> '$.jsonfield.dummyfield1.value'), "\\[|\\]|\"", ""),',')) t15 as c15
> lateral view
> explode(split(regexp_replace(get_json_object(jsontable.json_string,
> '$.jsonfield1.dummyfield1.code'), "\\[|\\]|\"", ""),',')) t16 as c16
> lateral view
> explode(split(regexp_replace(get_json_object(jsontable.json_string,
> '$.jsonfield1.dummyfield1.value'), "\\[|\\]|\"", ""),',')) t17 as c17
> lateral view
>
[jira] [Work logged] (HIVE-22173) Query with multiple lateral views hangs during compilation
[
https://issues.apache.org/jira/browse/HIVE-22173?focusedWorklogId=839848=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-839848
]
ASF GitHub Bot logged work on HIVE-22173:
-
Author: ASF GitHub Bot
Created on: 18/Jan/23 08:06
Start Date: 18/Jan/23 08:06
Worklog Time Spent: 10m
Work Description: amansinha100 commented on code in PR #3852:
URL: https://github.com/apache/hive/pull/3852#discussion_r1073206272
##
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:
##
@@ -3710,7 +3710,12 @@ public static enum ConfVars {
HIVE_EXPLAIN_USER("hive.explain.user", true,
"Whether to show explain result at user level.\n" +
"When enabled, will log EXPLAIN output for the query at user level.
Tez only."),
-
+HIVE_EXPLAIN_VISIT_LIMIT("hive.explain.visit.limit", 256, new
RangeValidator(1, Integer.MAX_VALUE),
Review Comment:
Thanks for the explanation. For now, I am good to keep this as is for
EXPLAIN. A global limit can be considered in the future but it's ok to wait
for a real use case where that is needed.
Issue Time Tracking
---
Worklog Id: (was: 839848)
Time Spent: 2h (was: 1h 50m)
> Query with multiple lateral views hangs during compilation
> --
>
> Key: HIVE-22173
> URL: https://issues.apache.org/jira/browse/HIVE-22173
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.1.1, 4.0.0-alpha-1
> Environment: Hive-3.1.1, Java-8
>Reporter: Rajkumar Singh
>Assignee: Stamatis Zampetakis
>Priority: Critical
> Labels: pull-request-available
> Attachments: op_plan_4_lateral_views.pdf, thread-progress.log
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Steps To Repro:
> {code:java}
> -- create table
> CREATE EXTERNAL TABLE `jsontable`(
> `json_string` string)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.mapred.TextInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' ;
> -- Run explain of the query
> explain SELECT
> *
> FROM jsontable
> lateral view
> explode(split(regexp_replace(get_json_object(jsontable.json_string,
> '$.jsonfield.addr.city'), "\\[|\\]|\"", ""),',')) t1 as c1
> lateral view
> explode(split(regexp_replace(get_json_object(jsontable.json_string,
> '$.jsonfield.addr.country'), "\\[|\\]|\"", ""),',')) t2 as c2
> lateral view
> explode(split(regexp_replace(get_json_object(jsontable.json_string,
> '$.jsonfield.addr'), "\\[|\\]|\"", ""),',')) t3 as c3
> lateral view
> explode(split(regexp_replace(get_json_object(jsontable.json_string,
> '$.jsonfield.addr.postalCode'), "\\[|\\]|\"", ""),',')) t4 as c4
> lateral view
> explode(split(regexp_replace(get_json_object(jsontable.json_string,
> '$.jsonfield.addr.state'), "\\[|\\]|\"", ""),',')) t5 as c5
> lateral view
> explode(split(regexp_replace(get_json_object(jsontable.json_string,
> '$.jsonfield.addr.streetAddressLine'), "\\[|\\]|\"", ""),',')) t6 as c6
> lateral view
> explode(split(regexp_replace(get_json_object(jsontable.json_string,
> '$.jsonfield.dummyfield'), "\\[|\\]|\"", ""),',')) t7 as c7
> lateral view
> explode(split(regexp_replace(get_json_object(jsontable.json_string,
> '$.jsonfield.dummyfield'), "\\[|\\]|\"", ""),',')) t8 as c8
> lateral view
> explode(split(regexp_replace(get_json_object(jsontable.json_string,
> '$.jsonfield.dummyfield.name.suffix'), "\\[|\\]|\"", ""),',')) t9 as c9
> lateral view
> explode(split(regexp_replace(get_json_object(jsontable.json_string,
> '$.jsonfield.id.extension'), "\\[|\\]|\"", ""),',')) t10 as c10
> lateral view
> explode(split(regexp_replace(get_json_object(jsontable.json_string,
> '$.jsonfield.id'), "\\[|\\]|\"", ""),',')) t11 as c11
> lateral view
> explode(split(regexp_replace(get_json_object(jsontable.json_string,
> '$.jsonfield.id.root'), "\\[|\\]|\"", ""),',')) t12 as c12
> lateral view
> explode(split(regexp_replace(get_json_object(jsontable.json_string,
> '$.jsonfield.telecom.'), "\\[|\\]|\"", ""),',')) t13 as c13
> lateral view
> explode(split(regexp_replace(get_json_object(jsontable.json_string,
> '$.jsonfield.dummyfield1.use'), "\\[|\\]|\"", ""),',')) t14 as c14
> lateral view
> explode(split(regexp_replace(get_json_object(jsontable.json_string,
> '$.jsonfield.dummyfield1.value'), "\\[|\\]|\"", ""),',')) t15 as c15
> lateral view
> explode(split(regexp_replace(get_json_object(jsontable.json_string,
> '$.jsonfield1.dummyfield1.code'), "\\[|\\]|\"", ""),',')) t16 as c16
> lateral view
> explode(split(regexp_replace(get_json_object(jsontable.json_string,
> '$.jsonfield1.dummyfield1.value'), "\\[|\\]|\"", ""),',')) t17 as
[jira] [Updated] (HIVE-26955) Alter table fails when decimal column data type is changed to string/char/varchar in Parquet
[
https://issues.apache.org/jira/browse/HIVE-26955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sourabh Badhya updated HIVE-26955:
--
Description:
Steps to reproduce
{noformat}
create table test_parquet (id decimal) stored as parquet;
insert into test_parquet values(238);
alter table test_parquet change id id string;
select * from test_parquet;
Error: java.io.IOException: org.apache.parquet.io.ParquetDecodingException: Can
not read value at 1 in block 0 in file
hdfs:/namenode:8020/warehouse/tablespace/managed/hive/test_parquet/delta_001_001_/00_0
(state=,code=0)
at
org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:624)
at
org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:531)
at org.apache.hadoop.hive.ql.exec.FetchTask.executeInner(FetchTask.java:194)
... 55 more
Caused by: org.apache.parquet.io.ParquetDecodingException: Can not read value
at 1 in block 0 in file
file:/home/centos/Apache-Hive-Tarak/itests/qtest/target/localfs/warehouse/test_parquet/00_0
at
org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:255)
at
org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:207)
at
org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.(ParquetRecordReaderWrapper.java:87)
at
org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat.getRecordReader(MapredParquetInputFormat.java:89)
at
org.apache.hadoop.hive.ql.exec.FetchOperator$FetchInputFormatSplit.getRecordReader(FetchOperator.java:771)
at
org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:335)
at
org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:562)
... 57 more
Caused by: java.lang.ClassCastException:
org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo cannot be cast to
org.apache.hadoop.hive.serde2.typeinfo.DecimalTypeInfo
at
org.apache.hadoop.hive.ql.io.parquet.convert.ETypeConverter$8$5.convert(ETypeConverter.java:669)
at
org.apache.hadoop.hive.ql.io.parquet.convert.ETypeConverter$8$5.convert(ETypeConverter.java:664)
at
org.apache.hadoop.hive.ql.io.parquet.convert.ETypeConverter$BinaryConverter.addBinary(ETypeConverter.java:977)
at
org.apache.parquet.column.impl.ColumnReaderBase$2$6.writeValue(ColumnReaderBase.java:360)
at
org.apache.parquet.column.impl.ColumnReaderBase.writeCurrentValueToConverter(ColumnReaderBase.java:410)
at
org.apache.parquet.column.impl.ColumnReaderImpl.writeCurrentValueToConverter(ColumnReaderImpl.java:30)
at
org.apache.parquet.io.RecordReaderImplementation.read(RecordReaderImplementation.java:406)
at
org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:230)
... 63 more{noformat}
However the same is working as expected in ORC table
{noformat}
create table test_orc (id decimal) stored as orc;
insert into test_orc values(238);
alter table test_orc change id id string;
select * from test_orc;
+--+
| test_orc.id |
+--+
| 238 |
+--+{noformat}
As well as text table
{noformat}
create table test_text (id decimal) stored as textfile;
insert into test_text values(238);
alter table test_text change id id string;
select * from test_text;
+---+
| test_text.id |
+---+
| 238 |
+---+{noformat}
Similar exception is thrown when the altered datatype is varchar and char
datatype.
was:
Steps to reproduce
{noformat}
create table test_parquet (id decimal) stored as parquet;
insert into test_parquet values(238);
alter table test_parquet change id id string;
select * from test_parquet;
Error: java.io.IOException: org.apache.parquet.io.ParquetDecodingException: Can
not read value at 1 in block 0 in file
hdfs:/namenode:8020/warehouse/tablespace/managed/hive/test_parquet/delta_001_001_/00_0
(state=,code=0)
at
org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:624)
at
org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:531)
at org.apache.hadoop.hive.ql.exec.FetchTask.executeInner(FetchTask.java:194)
... 55 more
Caused by: org.apache.parquet.io.ParquetDecodingException: Can not read value
at 1 in block 0 in file
file:/home/centos/Apache-Hive-Tarak/itests/qtest/target/localfs/warehouse/test_parquet/00_0
at
org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:255)
at
org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:207)
at
org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.(ParquetRecordReaderWrapper.java:87)
at
org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat.getRecordReader(MapredParquetInputFormat.java:89)
[jira] [Updated] (HIVE-26955) Alter table fails when decimal column data type is changed to string/char/varchar in Parquet
[
https://issues.apache.org/jira/browse/HIVE-26955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sourabh Badhya updated HIVE-26955:
--
Summary: Alter table fails when decimal column data type is changed to
string/char/varchar in Parquet (was: Alter table change column data type of a
Parquet table throws exception)
> Alter table fails when decimal column data type is changed to
> string/char/varchar in Parquet
>
>
> Key: HIVE-26955
> URL: https://issues.apache.org/jira/browse/HIVE-26955
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Taraka Rama Rao Lethavadla
>Assignee: Sourabh Badhya
>Priority: Major
>
> Steps to reproduce
> {noformat}
> create table test_parquet (id decimal) stored as parquet;
> insert into test_parquet values(238);
> alter table test_parquet change id id string;
> select * from test_parquet;
> Error: java.io.IOException: org.apache.parquet.io.ParquetDecodingException:
> Can not read value at 1 in block 0 in file
> hdfs:/namenode:8020/warehouse/tablespace/managed/hive/test_parquet/delta_001_001_/00_0
> (state=,code=0)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:624)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:531)
> at
> org.apache.hadoop.hive.ql.exec.FetchTask.executeInner(FetchTask.java:194)
> ... 55 more
> Caused by: org.apache.parquet.io.ParquetDecodingException: Can not read value
> at 1 in block 0 in file
> file:/home/centos/Apache-Hive-Tarak/itests/qtest/target/localfs/warehouse/test_parquet/00_0
> at
> org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:255)
> at
> org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:207)
> at
> org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.(ParquetRecordReaderWrapper.java:87)
> at
> org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat.getRecordReader(MapredParquetInputFormat.java:89)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator$FetchInputFormatSplit.getRecordReader(FetchOperator.java:771)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:335)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:562)
> ... 57 more
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo cannot be cast to
> org.apache.hadoop.hive.serde2.typeinfo.DecimalTypeInfo
> at
> org.apache.hadoop.hive.ql.io.parquet.convert.ETypeConverter$8$5.convert(ETypeConverter.java:669)
> at
> org.apache.hadoop.hive.ql.io.parquet.convert.ETypeConverter$8$5.convert(ETypeConverter.java:664)
> at
> org.apache.hadoop.hive.ql.io.parquet.convert.ETypeConverter$BinaryConverter.addBinary(ETypeConverter.java:977)
> at
> org.apache.parquet.column.impl.ColumnReaderBase$2$6.writeValue(ColumnReaderBase.java:360)
> at
> org.apache.parquet.column.impl.ColumnReaderBase.writeCurrentValueToConverter(ColumnReaderBase.java:410)
> at
> org.apache.parquet.column.impl.ColumnReaderImpl.writeCurrentValueToConverter(ColumnReaderImpl.java:30)
> at
> org.apache.parquet.io.RecordReaderImplementation.read(RecordReaderImplementation.java:406)
> at
> org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:230)
> ... 63 more{noformat}
> However the same is working as expected in ORC table
> {noformat}
> create table test_orc (id decimal) stored as orc;
> insert into test_orc values(238);
> alter table test_orc change id id string;
> select * from test_orc;
> +--+
> | test_orc.id |
> +--+
> | 238 |
> +--+{noformat}
> As well as text table
> {noformat}
> create table test_text (id decimal) stored as textfile;
> insert into test_text values(238);
> alter table test_text change id id string;
> select * from test_text;
> +---+
> | test_text.id |
> +---+
> | 238 |
> +---+{noformat}
> Same exception is thrown when the altered datatype is varchar and char
> datatype.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-26955) Alter table change column data type of a Parquet table throws exception
[
https://issues.apache.org/jira/browse/HIVE-26955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sourabh Badhya updated HIVE-26955:
--
Description:
Steps to reproduce
{noformat}
create table test_parquet (id decimal) stored as parquet;
insert into test_parquet values(238);
alter table test_parquet change id id string;
select * from test_parquet;
Error: java.io.IOException: org.apache.parquet.io.ParquetDecodingException: Can
not read value at 1 in block 0 in file
hdfs:/namenode:8020/warehouse/tablespace/managed/hive/test_parquet/delta_001_001_/00_0
(state=,code=0)
at
org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:624)
at
org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:531)
at org.apache.hadoop.hive.ql.exec.FetchTask.executeInner(FetchTask.java:194)
... 55 more
Caused by: org.apache.parquet.io.ParquetDecodingException: Can not read value
at 1 in block 0 in file
file:/home/centos/Apache-Hive-Tarak/itests/qtest/target/localfs/warehouse/test_parquet/00_0
at
org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:255)
at
org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:207)
at
org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.(ParquetRecordReaderWrapper.java:87)
at
org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat.getRecordReader(MapredParquetInputFormat.java:89)
at
org.apache.hadoop.hive.ql.exec.FetchOperator$FetchInputFormatSplit.getRecordReader(FetchOperator.java:771)
at
org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:335)
at
org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:562)
... 57 more
Caused by: java.lang.ClassCastException:
org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo cannot be cast to
org.apache.hadoop.hive.serde2.typeinfo.DecimalTypeInfo
at
org.apache.hadoop.hive.ql.io.parquet.convert.ETypeConverter$8$5.convert(ETypeConverter.java:669)
at
org.apache.hadoop.hive.ql.io.parquet.convert.ETypeConverter$8$5.convert(ETypeConverter.java:664)
at
org.apache.hadoop.hive.ql.io.parquet.convert.ETypeConverter$BinaryConverter.addBinary(ETypeConverter.java:977)
at
org.apache.parquet.column.impl.ColumnReaderBase$2$6.writeValue(ColumnReaderBase.java:360)
at
org.apache.parquet.column.impl.ColumnReaderBase.writeCurrentValueToConverter(ColumnReaderBase.java:410)
at
org.apache.parquet.column.impl.ColumnReaderImpl.writeCurrentValueToConverter(ColumnReaderImpl.java:30)
at
org.apache.parquet.io.RecordReaderImplementation.read(RecordReaderImplementation.java:406)
at
org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:230)
... 63 more{noformat}
However the same is working as expected in ORC table
{noformat}
create table test_orc (id decimal) stored as orc;
insert into test_orc values(238);
alter table test_orc change id id string;
select * from test_orc;
+--+
| test_orc.id |
+--+
| 238 |
+--+{noformat}
As well as text table
{noformat}
create table test_text (id decimal) stored as textfile;
insert into test_text values(238);
alter table test_text change id id string;
select * from test_text;
+---+
| test_text.id |
+---+
| 238 |
+---+{noformat}
Same exception is thrown when the altered datatype is varchar and char datatype.
was:
Steps to reproduce
{noformat}
create table test_parquet (id decimal) stored as parquet;
insert into test_parquet values(238);
alter table test_parquet change id id string;
select * from test_parquet;
Error: java.io.IOException: org.apache.parquet.io.ParquetDecodingException: Can
not read value at 1 in block 0 in file
hdfs:/namenode:8020/warehouse/tablespace/managed/hive/test_parquet/delta_001_001_/00_0
(state=,code=0)
at
org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:624)
at
org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:531)
at org.apache.hadoop.hive.ql.exec.FetchTask.executeInner(FetchTask.java:194)
... 55 more
Caused by: org.apache.parquet.io.ParquetDecodingException: Can not read value
at 1 in block 0 in file
file:/home/centos/Apache-Hive-Tarak/itests/qtest/target/localfs/warehouse/test_parquet/00_0
at
org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:255)
at
org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:207)
at
org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.(ParquetRecordReaderWrapper.java:87)
at
org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat.getRecordReader(MapredParquetInputFormat.java:89)
at
