[jira] [Assigned] (HIVE-26177) Create a new connection pool for compaction (DataNucleus)
[ https://issues.apache.org/jira/browse/HIVE-26177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits reassigned HIVE-26177: -- Assignee: Antal Sinkovits > Create a new connection pool for compaction (DataNucleus) > - > > Key: HIVE-26177 > URL: https://issues.apache.org/jira/browse/HIVE-26177 > Project: Hive > Issue Type: Sub-task >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HIVE-26177) Create a new connection pool for compaction (DataNucleus)
[ https://issues.apache.org/jira/browse/HIVE-26177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits resolved HIVE-26177. Resolution: Fixed Pushed to master. Thanks for the review [~dkuzmenko] > Create a new connection pool for compaction (DataNucleus) > - > > Key: HIVE-26177 > URL: https://issues.apache.org/jira/browse/HIVE-26177 > Project: Hive > Issue Type: Sub-task >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HIVE-26176) Create a new connection pool for compaction (CompactionTxnHandler)
[ https://issues.apache.org/jira/browse/HIVE-26176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits resolved HIVE-26176. Resolution: Fixed Pushed to master. Thanks for the review [~dkuzmenko] and [~pvary] > Create a new connection pool for compaction (CompactionTxnHandler) > -- > > Key: HIVE-26176 > URL: https://issues.apache.org/jira/browse/HIVE-26176 > Project: Hive > Issue Type: Sub-task >Reporter: Antal Sinkovits >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Assigned] (HIVE-26155) Create a new connection pool for compaction
[ https://issues.apache.org/jira/browse/HIVE-26155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits reassigned HIVE-26155: -- > Create a new connection pool for compaction > --- > > Key: HIVE-26155 > URL: https://issues.apache.org/jira/browse/HIVE-26155 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Labels: compaction > > Currently the TxnHandler uses 2 connection pools to communicate with the HMS: > the default one and one for mutexing. If compaction is configured incorrectly > (e.g. too many Initiators are running on the same db) then compaction can use > up all the connections in the default connection pool and all user queries > can get stuck. > We should have a separate connection pool (configurable size) just for > compaction-related activities. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Work started] (HIVE-26155) Create a new connection pool for compaction
[ https://issues.apache.org/jira/browse/HIVE-26155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-26155 started by Antal Sinkovits. -- > Create a new connection pool for compaction > --- > > Key: HIVE-26155 > URL: https://issues.apache.org/jira/browse/HIVE-26155 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Labels: compaction > > Currently the TxnHandler uses 2 connection pools to communicate with the HMS: > the default one and one for mutexing. If compaction is configured incorrectly > (e.g. too many Initiators are running on the same db) then compaction can use > up all the connections in the default connection pool and all user queries > can get stuck. > We should have a separate connection pool (configurable size) just for > compaction-related activities. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HIVE-25492) Major query-based compaction is skipped if partition is empty
[ https://issues.apache.org/jira/browse/HIVE-25492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits resolved HIVE-25492. Fix Version/s: 4.0.0 Resolution: Fixed Pushed to master. Thanks for the review [~dkuzmenko] > Major query-based compaction is skipped if partition is empty > - > > Key: HIVE-25492 > URL: https://issues.apache.org/jira/browse/HIVE-25492 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0-alpha-1, 4.0.0-alpha-2 >Reporter: Karen Coppage >Assignee: Antal Sinkovits >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > Currently if the result of query-based compaction is an empty base, delta, or > delete delta, the empty directory is deleted. > This is because of minor compaction – if there are only deltas to compact, > then no compacted delete delta should be created (only a compacted delta). In > the same way, if there are only delete deltas to compact, then no compacted > delta should be created (only a compacted delete delta). > There is an issue with major compaction. If all the data in the partition has > been deleted, then we should get an empty base directory after compaction. > Instead, the empty base directory is deleted because it's empty and > compaction claims to succeed but we end up with the same deltas/delete deltas > we started with – basically compaction does not run. > Where to start? MajorQueryCompactor#commitCompaction -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-25492) Major query-based compaction is skipped if partition is empty
[ https://issues.apache.org/jira/browse/HIVE-25492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-25492: --- Affects Version/s: 4.0.0-alpha-1 4.0.0-alpha-2 > Major query-based compaction is skipped if partition is empty > - > > Key: HIVE-25492 > URL: https://issues.apache.org/jira/browse/HIVE-25492 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0-alpha-1, 4.0.0-alpha-2 >Reporter: Karen Coppage >Assignee: Antal Sinkovits >Priority: Major > Labels: pull-request-available > Time Spent: 2h 20m > Remaining Estimate: 0h > > Currently if the result of query-based compaction is an empty base, delta, or > delete delta, the empty directory is deleted. > This is because of minor compaction – if there are only deltas to compact, > then no compacted delete delta should be created (only a compacted delta). In > the same way, if there are only delete deltas to compact, then no compacted > delta should be created (only a compacted delete delta). > There is an issue with major compaction. If all the data in the partition has > been deleted, then we should get an empty base directory after compaction. > Instead, the empty base directory is deleted because it's empty and > compaction claims to succeed but we end up with the same deltas/delete deltas > we started with – basically compaction does not run. > Where to start? MajorQueryCompactor#commitCompaction -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HIVE-25492) Major query-based compaction is skipped if partition is empty
[ https://issues.apache.org/jira/browse/HIVE-25492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits reassigned HIVE-25492: -- Assignee: Antal Sinkovits > Major query-based compaction is skipped if partition is empty > - > > Key: HIVE-25492 > URL: https://issues.apache.org/jira/browse/HIVE-25492 > Project: Hive > Issue Type: Bug >Reporter: Karen Coppage >Assignee: Antal Sinkovits >Priority: Major > > Currently if the result of query-based compaction is an empty base, delta, or > delete delta, the empty directory is deleted. > This is because of minor compaction – if there are only deltas to compact, > then no compacted delete delta should be created (only a compacted delta). In > the same way, if there are only delete deltas to compact, then no compacted > delta should be created (only a compacted delete delta). > There is an issue with major compaction. If all the data in the partition has > been deleted, then we should get an empty base directory after compaction. > Instead, the empty base directory is deleted because it's empty and > compaction claims to succeed but we end up with the same deltas/delete deltas > we started with – basically compaction does not run. > Where to start? MajorQueryCompactor#commitCompaction -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work started] (HIVE-25492) Major query-based compaction is skipped if partition is empty
[ https://issues.apache.org/jira/browse/HIVE-25492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-25492 started by Antal Sinkovits. -- > Major query-based compaction is skipped if partition is empty > - > > Key: HIVE-25492 > URL: https://issues.apache.org/jira/browse/HIVE-25492 > Project: Hive > Issue Type: Bug >Reporter: Karen Coppage >Assignee: Antal Sinkovits >Priority: Major > > Currently if the result of query-based compaction is an empty base, delta, or > delete delta, the empty directory is deleted. > This is because of minor compaction – if there are only deltas to compact, > then no compacted delete delta should be created (only a compacted delta). In > the same way, if there are only delete deltas to compact, then no compacted > delta should be created (only a compacted delete delta). > There is an issue with major compaction. If all the data in the partition has > been deleted, then we should get an empty base directory after compaction. > Instead, the empty base directory is deleted because it's empty and > compaction claims to succeed but we end up with the same deltas/delete deltas > we started with – basically compaction does not run. > Where to start? MajorQueryCompactor#commitCompaction -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HIVE-26060) Invalidate acid table directory cache on drop table
[ https://issues.apache.org/jira/browse/HIVE-26060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits resolved HIVE-26060. Fix Version/s: 4.0.0 Resolution: Fixed Pushed to master. Thanks for the review [~dkuzmenko] > Invalidate acid table directory cache on drop table > --- > > Key: HIVE-26060 > URL: https://issues.apache.org/jira/browse/HIVE-26060 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0-alpha-1 >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Minor > Labels: ACID, pull-request-available > Fix For: 4.0.0 > > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work started] (HIVE-26060) Invalidate acid table directory cache on drop table
[ https://issues.apache.org/jira/browse/HIVE-26060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-26060 started by Antal Sinkovits. -- > Invalidate acid table directory cache on drop table > --- > > Key: HIVE-26060 > URL: https://issues.apache.org/jira/browse/HIVE-26060 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0-alpha-1 >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Minor > Labels: ACID > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HIVE-26060) Invalidate acid table directory cache on drop table
[ https://issues.apache.org/jira/browse/HIVE-26060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits reassigned HIVE-26060: -- > Invalidate acid table directory cache on drop table > --- > > Key: HIVE-26060 > URL: https://issues.apache.org/jira/browse/HIVE-26060 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0-alpha-1 >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Minor > Labels: ACID > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-26048) Missing quotation mark in findReadyToClean query
[
https://issues.apache.org/jira/browse/HIVE-26048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Antal Sinkovits updated HIVE-26048:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Missing quotation mark in findReadyToClean query
>
>
> Key: HIVE-26048
> URL: https://issues.apache.org/jira/browse/HIVE-26048
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0-alpha-1
>Reporter: Antal Sinkovits
>Assignee: Antal Sinkovits
>Priority: Major
> Labels: ACID, pull-request-available
> Fix For: 4.0.0-alpha-1
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Missing quotation mark causes postgres column error
> {code}
> 2022-03-18T00:53:43,314 ERROR [Thread-651] compactor.Cleaner: Caught an
> exception in the main loop of compactor cleaner, MetaException(message:Unable
> to connect to transaction database org.postgresql.util.PSQLException: ERROR:
> column "cq_retry_retention" does not exist
> Position: 485
> at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2433)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2178)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:306)
> at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441)
> at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365)
> at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:307)
> at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:293)
> at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:270)
> at org.postgresql.jdbc.PgStatement.executeQuery(PgStatement.java:224)
> at
> org.apache.hive.com.zaxxer.hikari.pool.ProxyStatement.executeQuery(ProxyStatement.java:108)
> at
> org.apache.hive.com.zaxxer.hikari.pool.HikariProxyStatement.executeQuery(HikariProxyStatement.java)
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findReadyToClean(CompactionTxnHandler.java:374)
> at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.run(Cleaner.java:146)
> )
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findReadyToClean(CompactionTxnHandler.java:397)
> at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.run(Cleaner.java:146)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HIVE-26048) Missing quotation mark in findReadyToClean query
[
https://issues.apache.org/jira/browse/HIVE-26048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17509814#comment-17509814
]
Antal Sinkovits commented on HIVE-26048:
Pushed to master. Thanks for the. review [~klcopp] and [~dkuzmenko].
> Missing quotation mark in findReadyToClean query
>
>
> Key: HIVE-26048
> URL: https://issues.apache.org/jira/browse/HIVE-26048
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0-alpha-1
>Reporter: Antal Sinkovits
>Assignee: Antal Sinkovits
>Priority: Major
> Labels: ACID, pull-request-available
> Fix For: 4.0.0-alpha-1
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Missing quotation mark causes postgres column error
> {code}
> 2022-03-18T00:53:43,314 ERROR [Thread-651] compactor.Cleaner: Caught an
> exception in the main loop of compactor cleaner, MetaException(message:Unable
> to connect to transaction database org.postgresql.util.PSQLException: ERROR:
> column "cq_retry_retention" does not exist
> Position: 485
> at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2433)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2178)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:306)
> at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441)
> at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365)
> at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:307)
> at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:293)
> at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:270)
> at org.postgresql.jdbc.PgStatement.executeQuery(PgStatement.java:224)
> at
> org.apache.hive.com.zaxxer.hikari.pool.ProxyStatement.executeQuery(ProxyStatement.java:108)
> at
> org.apache.hive.com.zaxxer.hikari.pool.HikariProxyStatement.executeQuery(HikariProxyStatement.java)
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findReadyToClean(CompactionTxnHandler.java:374)
> at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.run(Cleaner.java:146)
> )
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findReadyToClean(CompactionTxnHandler.java:397)
> at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.run(Cleaner.java:146)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HIVE-26048) Missing quotation mark in findReadyToClean query
[
https://issues.apache.org/jira/browse/HIVE-26048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Antal Sinkovits updated HIVE-26048:
---
Affects Version/s: 4.0.0-alpha-1
(was: 4.0.0)
> Missing quotation mark in findReadyToClean query
>
>
> Key: HIVE-26048
> URL: https://issues.apache.org/jira/browse/HIVE-26048
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0-alpha-1
>Reporter: Antal Sinkovits
>Assignee: Antal Sinkovits
>Priority: Major
> Labels: ACID, pull-request-available
> Fix For: 4.0.0-alpha-1
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Missing quotation mark causes postgres column error
> {code}
> 2022-03-18T00:53:43,314 ERROR [Thread-651] compactor.Cleaner: Caught an
> exception in the main loop of compactor cleaner, MetaException(message:Unable
> to connect to transaction database org.postgresql.util.PSQLException: ERROR:
> column "cq_retry_retention" does not exist
> Position: 485
> at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2433)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2178)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:306)
> at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441)
> at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365)
> at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:307)
> at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:293)
> at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:270)
> at org.postgresql.jdbc.PgStatement.executeQuery(PgStatement.java:224)
> at
> org.apache.hive.com.zaxxer.hikari.pool.ProxyStatement.executeQuery(ProxyStatement.java:108)
> at
> org.apache.hive.com.zaxxer.hikari.pool.HikariProxyStatement.executeQuery(HikariProxyStatement.java)
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findReadyToClean(CompactionTxnHandler.java:374)
> at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.run(Cleaner.java:146)
> )
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findReadyToClean(CompactionTxnHandler.java:397)
> at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.run(Cleaner.java:146)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HIVE-26048) Missing quotation mark in findReadyToClean query
[
https://issues.apache.org/jira/browse/HIVE-26048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Antal Sinkovits updated HIVE-26048:
---
Fix Version/s: 4.0.0-alpha-1
> Missing quotation mark in findReadyToClean query
>
>
> Key: HIVE-26048
> URL: https://issues.apache.org/jira/browse/HIVE-26048
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Antal Sinkovits
>Assignee: Antal Sinkovits
>Priority: Major
> Labels: ACID, pull-request-available
> Fix For: 4.0.0-alpha-1
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Missing quotation mark causes postgres column error
> {code}
> 2022-03-18T00:53:43,314 ERROR [Thread-651] compactor.Cleaner: Caught an
> exception in the main loop of compactor cleaner, MetaException(message:Unable
> to connect to transaction database org.postgresql.util.PSQLException: ERROR:
> column "cq_retry_retention" does not exist
> Position: 485
> at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2433)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2178)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:306)
> at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441)
> at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365)
> at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:307)
> at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:293)
> at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:270)
> at org.postgresql.jdbc.PgStatement.executeQuery(PgStatement.java:224)
> at
> org.apache.hive.com.zaxxer.hikari.pool.ProxyStatement.executeQuery(ProxyStatement.java:108)
> at
> org.apache.hive.com.zaxxer.hikari.pool.HikariProxyStatement.executeQuery(HikariProxyStatement.java)
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findReadyToClean(CompactionTxnHandler.java:374)
> at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.run(Cleaner.java:146)
> )
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findReadyToClean(CompactionTxnHandler.java:397)
> at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.run(Cleaner.java:146)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HIVE-26048) Missing quotation mark in findReadyToClean query
[
https://issues.apache.org/jira/browse/HIVE-26048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Antal Sinkovits updated HIVE-26048:
---
Labels: ACID (was: )
> Missing quotation mark in findReadyToClean query
>
>
> Key: HIVE-26048
> URL: https://issues.apache.org/jira/browse/HIVE-26048
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Antal Sinkovits
>Assignee: Antal Sinkovits
>Priority: Major
> Labels: ACID
>
> Missing quotation mark causes postgres column error
> {code}
> 2022-03-18T00:53:43,314 ERROR [Thread-651] compactor.Cleaner: Caught an
> exception in the main loop of compactor cleaner, MetaException(message:Unable
> to connect to transaction database org.postgresql.util.PSQLException: ERROR:
> column "cq_retry_retention" does not exist
> Position: 485
> at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2433)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2178)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:306)
> at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441)
> at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365)
> at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:307)
> at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:293)
> at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:270)
> at org.postgresql.jdbc.PgStatement.executeQuery(PgStatement.java:224)
> at
> org.apache.hive.com.zaxxer.hikari.pool.ProxyStatement.executeQuery(ProxyStatement.java:108)
> at
> org.apache.hive.com.zaxxer.hikari.pool.HikariProxyStatement.executeQuery(HikariProxyStatement.java)
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findReadyToClean(CompactionTxnHandler.java:374)
> at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.run(Cleaner.java:146)
> )
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findReadyToClean(CompactionTxnHandler.java:397)
> at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.run(Cleaner.java:146)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HIVE-26048) Missing quotation mark in findReadyToClean query
[
https://issues.apache.org/jira/browse/HIVE-26048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Antal Sinkovits updated HIVE-26048:
---
Status: Patch Available (was: In Progress)
> Missing quotation mark in findReadyToClean query
>
>
> Key: HIVE-26048
> URL: https://issues.apache.org/jira/browse/HIVE-26048
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Antal Sinkovits
>Assignee: Antal Sinkovits
>Priority: Major
> Labels: ACID, pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Missing quotation mark causes postgres column error
> {code}
> 2022-03-18T00:53:43,314 ERROR [Thread-651] compactor.Cleaner: Caught an
> exception in the main loop of compactor cleaner, MetaException(message:Unable
> to connect to transaction database org.postgresql.util.PSQLException: ERROR:
> column "cq_retry_retention" does not exist
> Position: 485
> at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2433)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2178)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:306)
> at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441)
> at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365)
> at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:307)
> at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:293)
> at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:270)
> at org.postgresql.jdbc.PgStatement.executeQuery(PgStatement.java:224)
> at
> org.apache.hive.com.zaxxer.hikari.pool.ProxyStatement.executeQuery(ProxyStatement.java:108)
> at
> org.apache.hive.com.zaxxer.hikari.pool.HikariProxyStatement.executeQuery(HikariProxyStatement.java)
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findReadyToClean(CompactionTxnHandler.java:374)
> at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.run(Cleaner.java:146)
> )
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findReadyToClean(CompactionTxnHandler.java:397)
> at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.run(Cleaner.java:146)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work started] (HIVE-26048) Missing quotation mark in findReadyToClean query
[
https://issues.apache.org/jira/browse/HIVE-26048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-26048 started by Antal Sinkovits.
--
> Missing quotation mark in findReadyToClean query
>
>
> Key: HIVE-26048
> URL: https://issues.apache.org/jira/browse/HIVE-26048
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Antal Sinkovits
>Assignee: Antal Sinkovits
>Priority: Major
> Labels: ACID
>
> Missing quotation mark causes postgres column error
> {code}
> 2022-03-18T00:53:43,314 ERROR [Thread-651] compactor.Cleaner: Caught an
> exception in the main loop of compactor cleaner, MetaException(message:Unable
> to connect to transaction database org.postgresql.util.PSQLException: ERROR:
> column "cq_retry_retention" does not exist
> Position: 485
> at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2433)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2178)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:306)
> at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441)
> at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365)
> at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:307)
> at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:293)
> at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:270)
> at org.postgresql.jdbc.PgStatement.executeQuery(PgStatement.java:224)
> at
> org.apache.hive.com.zaxxer.hikari.pool.ProxyStatement.executeQuery(ProxyStatement.java:108)
> at
> org.apache.hive.com.zaxxer.hikari.pool.HikariProxyStatement.executeQuery(HikariProxyStatement.java)
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findReadyToClean(CompactionTxnHandler.java:374)
> at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.run(Cleaner.java:146)
> )
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findReadyToClean(CompactionTxnHandler.java:397)
> at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.run(Cleaner.java:146)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HIVE-26048) Missing quotation mark in findReadyToClean query
[
https://issues.apache.org/jira/browse/HIVE-26048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Antal Sinkovits updated HIVE-26048:
---
Affects Version/s: 4.0.0
> Missing quotation mark in findReadyToClean query
>
>
> Key: HIVE-26048
> URL: https://issues.apache.org/jira/browse/HIVE-26048
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Antal Sinkovits
>Assignee: Antal Sinkovits
>Priority: Major
>
> Missing quotation mark causes postgres column error
> {code}
> 2022-03-18T00:53:43,314 ERROR [Thread-651] compactor.Cleaner: Caught an
> exception in the main loop of compactor cleaner, MetaException(message:Unable
> to connect to transaction database org.postgresql.util.PSQLException: ERROR:
> column "cq_retry_retention" does not exist
> Position: 485
> at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2433)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2178)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:306)
> at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441)
> at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365)
> at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:307)
> at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:293)
> at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:270)
> at org.postgresql.jdbc.PgStatement.executeQuery(PgStatement.java:224)
> at
> org.apache.hive.com.zaxxer.hikari.pool.ProxyStatement.executeQuery(ProxyStatement.java:108)
> at
> org.apache.hive.com.zaxxer.hikari.pool.HikariProxyStatement.executeQuery(HikariProxyStatement.java)
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findReadyToClean(CompactionTxnHandler.java:374)
> at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.run(Cleaner.java:146)
> )
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findReadyToClean(CompactionTxnHandler.java:397)
> at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.run(Cleaner.java:146)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Assigned] (HIVE-26048) Missing quotation mark in findReadyToClean query
[
https://issues.apache.org/jira/browse/HIVE-26048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Antal Sinkovits reassigned HIVE-26048:
--
> Missing quotation mark in findReadyToClean query
>
>
> Key: HIVE-26048
> URL: https://issues.apache.org/jira/browse/HIVE-26048
> Project: Hive
> Issue Type: Bug
>Reporter: Antal Sinkovits
>Assignee: Antal Sinkovits
>Priority: Major
>
> Missing quotation mark causes postgres column error
> {code}
> 2022-03-18T00:53:43,314 ERROR [Thread-651] compactor.Cleaner: Caught an
> exception in the main loop of compactor cleaner, MetaException(message:Unable
> to connect to transaction database org.postgresql.util.PSQLException: ERROR:
> column "cq_retry_retention" does not exist
> Position: 485
> at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2433)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2178)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:306)
> at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441)
> at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365)
> at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:307)
> at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:293)
> at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:270)
> at org.postgresql.jdbc.PgStatement.executeQuery(PgStatement.java:224)
> at
> org.apache.hive.com.zaxxer.hikari.pool.ProxyStatement.executeQuery(ProxyStatement.java:108)
> at
> org.apache.hive.com.zaxxer.hikari.pool.HikariProxyStatement.executeQuery(HikariProxyStatement.java)
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findReadyToClean(CompactionTxnHandler.java:374)
> at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.run(Cleaner.java:146)
> )
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findReadyToClean(CompactionTxnHandler.java:397)
> at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.run(Cleaner.java:146)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HIVE-25986) Statement id is incorrect in case of load in path to MM table
[ https://issues.apache.org/jira/browse/HIVE-25986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-25986: --- Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. Thanks for the review [~pvary] > Statement id is incorrect in case of load in path to MM table > - > > Key: HIVE-25986 > URL: https://issues.apache.org/jira/browse/HIVE-25986 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Labels: ACID, pull-request-available > Fix For: 4.0.0 > > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-25986) Statement id is incorrect in case of load in path to MM table
[ https://issues.apache.org/jira/browse/HIVE-25986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-25986: --- Status: Patch Available (was: Open) > Statement id is incorrect in case of load in path to MM table > - > > Key: HIVE-25986 > URL: https://issues.apache.org/jira/browse/HIVE-25986 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Labels: ACID, pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-25986) Statement id is incorrect in case of load in path to MM table
[ https://issues.apache.org/jira/browse/HIVE-25986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-25986: --- Summary: Statement id is incorrect in case of load in path to MM table (was: statement id is incorrect in case of load in path to MM table) > Statement id is incorrect in case of load in path to MM table > - > > Key: HIVE-25986 > URL: https://issues.apache.org/jira/browse/HIVE-25986 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Labels: ACID > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-25986) statement id is incorrect in case of load in path to MM table
[ https://issues.apache.org/jira/browse/HIVE-25986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-25986: --- Summary: statement id is incorrect in case of load in path to MM table (was: statement id in incorrect in case of load in path to MM table) > statement id is incorrect in case of load in path to MM table > - > > Key: HIVE-25986 > URL: https://issues.apache.org/jira/browse/HIVE-25986 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Labels: ACID > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-25986) statement id in incorrect in case of load in path to MM table
[ https://issues.apache.org/jira/browse/HIVE-25986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-25986: --- Labels: ACID (was: ) > statement id in incorrect in case of load in path to MM table > - > > Key: HIVE-25986 > URL: https://issues.apache.org/jira/browse/HIVE-25986 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Labels: ACID > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-25986) statement id in incorrect in case of load in path to MM table
[ https://issues.apache.org/jira/browse/HIVE-25986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-25986: --- Affects Version/s: 4.0.0 > statement id in incorrect in case of load in path to MM table > - > > Key: HIVE-25986 > URL: https://issues.apache.org/jira/browse/HIVE-25986 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HIVE-25986) statement id in incorrect in case of load in path to MM table
[ https://issues.apache.org/jira/browse/HIVE-25986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits reassigned HIVE-25986: -- > statement id in incorrect in case of load in path to MM table > - > > Key: HIVE-25986 > URL: https://issues.apache.org/jira/browse/HIVE-25986 > Project: Hive > Issue Type: Bug >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HIVE-24805) Compactor: Initiator shouldn't fetch table details again and again for partitioned tables
[
https://issues.apache.org/jira/browse/HIVE-24805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Antal Sinkovits resolved HIVE-24805.
Fix Version/s: 4.0.0
Resolution: Fixed
Pushed to master. Thanks for the review [~dkuzmenko].
> Compactor: Initiator shouldn't fetch table details again and again for
> partitioned tables
> -
>
> Key: HIVE-24805
> URL: https://issues.apache.org/jira/browse/HIVE-24805
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Rajesh Balamohan
>Assignee: Antal Sinkovits
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> Initiator shouldn't be fetch table details for all its partitions. When there
> are large number of databases/tables, it takes lot of time for Initiator to
> complete its initial iteration and load on DB also goes higher.
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java#L129
> https://github.com/apache/hive/blob/64bb52316f19426ebea0087ee15e282cbde1d852/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java#L456
> For all the following partitions, table details would be the same. However,
> it ends up fetching table details from HMS again and again.
> {noformat}
> 2021-02-22 08:13:16,106 INFO
> org.apache.hadoop.hive.ql.txn.compactor.Initiator: [Thread-11]: Checking to
> see if we should compact
> tpcds_bin_partitioned_orc_1000.store_returns_tmp2.sr_returned_date_sk=2451899
> 2021-02-22 08:13:16,124 INFO
> org.apache.hadoop.hive.ql.txn.compactor.Initiator: [Thread-11]: Checking to
> see if we should compact
> tpcds_bin_partitioned_orc_1000.store_returns_tmp2.sr_returned_date_sk=2451830
> 2021-02-22 08:13:16,140 INFO
> org.apache.hadoop.hive.ql.txn.compactor.Initiator: [Thread-11]: Checking to
> see if we should compact
> tpcds_bin_partitioned_orc_1000.store_returns_tmp2.sr_returned_date_sk=2452586
> 2021-02-22 08:13:16,149 INFO
> org.apache.hadoop.hive.ql.txn.compactor.Initiator: [Thread-11]: Checking to
> see if we should compact
> tpcds_bin_partitioned_orc_1000.store_returns_tmp2.sr_returned_date_sk=2452698
> 2021-02-22 08:13:16,158 INFO
> org.apache.hadoop.hive.ql.txn.compactor.Initiator: [Thread-11]: Checking to
> see if we should compact
> tpcds_bin_partitioned_orc_1000.store_returns_tmp2.sr_returned_date_sk=2452063
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (HIVE-25862) Persist the time of last run of the initiator
[ https://issues.apache.org/jira/browse/HIVE-25862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits resolved HIVE-25862. Fix Version/s: 4.0.0 Resolution: Fixed Pushed to master. Thanks for the. review [~dkuzmenko]. > Persist the time of last run of the initiator > - > > Key: HIVE-25862 > URL: https://issues.apache.org/jira/browse/HIVE-25862 > Project: Hive > Issue Type: Improvement >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1h > Remaining Estimate: 0h > > The time of last run is used as a filter when finding compaction candidates. > Because its only stored in memory, we lose this filtering capability if the > service restarts, so it would make sense to persist it. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-25862) Persist the time of last run of the initiator
[ https://issues.apache.org/jira/browse/HIVE-25862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-25862: --- Summary: Persist the time of last run of the initiator (was: Persist the time of last run in the initiator) > Persist the time of last run of the initiator > - > > Key: HIVE-25862 > URL: https://issues.apache.org/jira/browse/HIVE-25862 > Project: Hive > Issue Type: Improvement >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > > The time of last run is used as a filter when finding compaction candidates. > Because its only stored in memory, we lose this filtering capability if the > service restarts, so it would make sense to persist it. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HIVE-25862) Persist the time of last run in the initiator
[ https://issues.apache.org/jira/browse/HIVE-25862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits reassigned HIVE-25862: -- > Persist the time of last run in the initiator > - > > Key: HIVE-25862 > URL: https://issues.apache.org/jira/browse/HIVE-25862 > Project: Hive > Issue Type: Improvement >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > > The time of last run is used as a filter when finding compaction candidates. > Because its only stored in memory, we lose this filtering capability if the > service restarts, so it would make sense to persist it. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HIVE-24805) Compactor: Initiator shouldn't fetch table details again and again for partitioned tables
[
https://issues.apache.org/jira/browse/HIVE-24805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Antal Sinkovits reassigned HIVE-24805:
--
Assignee: Antal Sinkovits
> Compactor: Initiator shouldn't fetch table details again and again for
> partitioned tables
> -
>
> Key: HIVE-24805
> URL: https://issues.apache.org/jira/browse/HIVE-24805
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Rajesh Balamohan
>Assignee: Antal Sinkovits
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Initiator shouldn't be fetch table details for all its partitions. When there
> are large number of databases/tables, it takes lot of time for Initiator to
> complete its initial iteration and load on DB also goes higher.
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java#L129
> https://github.com/apache/hive/blob/64bb52316f19426ebea0087ee15e282cbde1d852/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java#L456
> For all the following partitions, table details would be the same. However,
> it ends up fetching table details from HMS again and again.
> {noformat}
> 2021-02-22 08:13:16,106 INFO
> org.apache.hadoop.hive.ql.txn.compactor.Initiator: [Thread-11]: Checking to
> see if we should compact
> tpcds_bin_partitioned_orc_1000.store_returns_tmp2.sr_returned_date_sk=2451899
> 2021-02-22 08:13:16,124 INFO
> org.apache.hadoop.hive.ql.txn.compactor.Initiator: [Thread-11]: Checking to
> see if we should compact
> tpcds_bin_partitioned_orc_1000.store_returns_tmp2.sr_returned_date_sk=2451830
> 2021-02-22 08:13:16,140 INFO
> org.apache.hadoop.hive.ql.txn.compactor.Initiator: [Thread-11]: Checking to
> see if we should compact
> tpcds_bin_partitioned_orc_1000.store_returns_tmp2.sr_returned_date_sk=2452586
> 2021-02-22 08:13:16,149 INFO
> org.apache.hadoop.hive.ql.txn.compactor.Initiator: [Thread-11]: Checking to
> see if we should compact
> tpcds_bin_partitioned_orc_1000.store_returns_tmp2.sr_returned_date_sk=2452698
> 2021-02-22 08:13:16,158 INFO
> org.apache.hadoop.hive.ql.txn.compactor.Initiator: [Thread-11]: Checking to
> see if we should compact
> tpcds_bin_partitioned_orc_1000.store_returns_tmp2.sr_returned_date_sk=2452063
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work started] (HIVE-24805) Compactor: Initiator shouldn't fetch table details again and again for partitioned tables
[
https://issues.apache.org/jira/browse/HIVE-24805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-24805 started by Antal Sinkovits.
--
> Compactor: Initiator shouldn't fetch table details again and again for
> partitioned tables
> -
>
> Key: HIVE-24805
> URL: https://issues.apache.org/jira/browse/HIVE-24805
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Rajesh Balamohan
>Assignee: Antal Sinkovits
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Initiator shouldn't be fetch table details for all its partitions. When there
> are large number of databases/tables, it takes lot of time for Initiator to
> complete its initial iteration and load on DB also goes higher.
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java#L129
> https://github.com/apache/hive/blob/64bb52316f19426ebea0087ee15e282cbde1d852/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java#L456
> For all the following partitions, table details would be the same. However,
> it ends up fetching table details from HMS again and again.
> {noformat}
> 2021-02-22 08:13:16,106 INFO
> org.apache.hadoop.hive.ql.txn.compactor.Initiator: [Thread-11]: Checking to
> see if we should compact
> tpcds_bin_partitioned_orc_1000.store_returns_tmp2.sr_returned_date_sk=2451899
> 2021-02-22 08:13:16,124 INFO
> org.apache.hadoop.hive.ql.txn.compactor.Initiator: [Thread-11]: Checking to
> see if we should compact
> tpcds_bin_partitioned_orc_1000.store_returns_tmp2.sr_returned_date_sk=2451830
> 2021-02-22 08:13:16,140 INFO
> org.apache.hadoop.hive.ql.txn.compactor.Initiator: [Thread-11]: Checking to
> see if we should compact
> tpcds_bin_partitioned_orc_1000.store_returns_tmp2.sr_returned_date_sk=2452586
> 2021-02-22 08:13:16,149 INFO
> org.apache.hadoop.hive.ql.txn.compactor.Initiator: [Thread-11]: Checking to
> see if we should compact
> tpcds_bin_partitioned_orc_1000.store_returns_tmp2.sr_returned_date_sk=2452698
> 2021-02-22 08:13:16,158 INFO
> org.apache.hadoop.hive.ql.txn.compactor.Initiator: [Thread-11]: Checking to
> see if we should compact
> tpcds_bin_partitioned_orc_1000.store_returns_tmp2.sr_returned_date_sk=2452063
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (HIVE-25716) Fix flaky test TestCompactionMetrics#testOldestReadyForCleaningAge
[
https://issues.apache.org/jira/browse/HIVE-25716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Antal Sinkovits resolved HIVE-25716.
Fix Version/s: 4.0.0
Resolution: Fixed
Committed to master branch. Thanks for your contribution [~vcsomor]
> Fix flaky test TestCompactionMetrics#testOldestReadyForCleaningAge
> --
>
> Key: HIVE-25716
> URL: https://issues.apache.org/jira/browse/HIVE-25716
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Karen Coppage
>Assignee: Viktor Csomor
>Priority: Major
> Labels: flaky-test, pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Flaky check failed on run #59:
> [http://ci.hive.apache.org/job/hive-flaky-check/467/|http://ci.hive.apache.org/job/hive-flaky-check/467/]
> {code:java}
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:87)
> at org.junit.Assert.assertTrue(Assert.java:42)
> at org.junit.Assert.assertTrue(Assert.java:53)
> at
> org.apache.hadoop.hive.ql.txn.compactor.TestCompactionMetrics.testOldestReadyForCleaningAge(TestCompactionMetrics.java:214)
> {code}
> (!) After turning off the test the problematic line is actually 215 in the
> codebase
> {code}
> Assert.assertTrue(Metrics.getOrCreateGauge(MetricsConstants.OLDEST_READY_FOR_CLEANING_AGE).intValue()
> >= youngDiff);
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HIVE-25252) All new compaction metrics should be lower case
[ https://issues.apache.org/jira/browse/HIVE-25252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-25252: --- Summary: All new compaction metrics should be lower case (was: All new compaction metrics should be lower cased) > All new compaction metrics should be lower case > --- > > Key: HIVE-25252 > URL: https://issues.apache.org/jira/browse/HIVE-25252 > Project: Hive > Issue Type: Sub-task >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > > E.g: > compaction_worker_cycle_MINOR -> compaction_worker_cycle_minor > compaction_worker_cycle_MAJOR -> compaction_worker_cycle_major > compaction_cleaner_cycle_MINOR -> compaction_cleaner_cycle_minor > compaction_cleaner_cycle_MAJOR -> compaction_cleaner_cycle_major -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-25252) All new compaction metrics should be lower cased
[ https://issues.apache.org/jira/browse/HIVE-25252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-25252: --- Summary: All new compaction metrics should be lower cased (was: All new metrics should be lower cased) > All new compaction metrics should be lower cased > > > Key: HIVE-25252 > URL: https://issues.apache.org/jira/browse/HIVE-25252 > Project: Hive > Issue Type: Sub-task >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > > E.g: > compaction_worker_cycle_MINOR -> compaction_worker_cycle_minor > compaction_worker_cycle_MAJOR -> compaction_worker_cycle_major > compaction_cleaner_cycle_MINOR -> compaction_cleaner_cycle_minor > compaction_cleaner_cycle_MAJOR -> compaction_cleaner_cycle_major -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-25252) All new metrics should be lower cased
[ https://issues.apache.org/jira/browse/HIVE-25252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-25252: --- Description: E.g: compaction_worker_cycle_MINOR -> compaction_worker_cycle_minor compaction_worker_cycle_MAJOR -> compaction_worker_cycle_major compaction_cleaner_cycle_MINOR -> compaction_cleaner_cycle_minor compaction_cleaner_cycle_MAJOR -> compaction_cleaner_cycle_major was: E.g: compaction_worker_cycle_MINOR -> compaction_worker_cycle_minor compaction_worker_cycle_MAJOR -> compaction_worker_cycle_major > All new metrics should be lower cased > - > > Key: HIVE-25252 > URL: https://issues.apache.org/jira/browse/HIVE-25252 > Project: Hive > Issue Type: Sub-task >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > > E.g: > compaction_worker_cycle_MINOR -> compaction_worker_cycle_minor > compaction_worker_cycle_MAJOR -> compaction_worker_cycle_major > compaction_cleaner_cycle_MINOR -> compaction_cleaner_cycle_minor > compaction_cleaner_cycle_MAJOR -> compaction_cleaner_cycle_major -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-25252) All new metrics should be lower cased
[ https://issues.apache.org/jira/browse/HIVE-25252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-25252 started by Antal Sinkovits. -- > All new metrics should be lower cased > - > > Key: HIVE-25252 > URL: https://issues.apache.org/jira/browse/HIVE-25252 > Project: Hive > Issue Type: Sub-task >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > > compaction_worker_cycle_MINOR -> compaction_worker_cycle_minor > compaction_worker_cycle_MAJOR -> compaction_worker_cycle_major -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-25252) All new metrics should be lower cased
[ https://issues.apache.org/jira/browse/HIVE-25252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-25252: --- Description: E.g: compaction_worker_cycle_MINOR -> compaction_worker_cycle_minor compaction_worker_cycle_MAJOR -> compaction_worker_cycle_major was: compaction_worker_cycle_MINOR -> compaction_worker_cycle_minor compaction_worker_cycle_MAJOR -> compaction_worker_cycle_major > All new metrics should be lower cased > - > > Key: HIVE-25252 > URL: https://issues.apache.org/jira/browse/HIVE-25252 > Project: Hive > Issue Type: Sub-task >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > > E.g: > compaction_worker_cycle_MINOR -> compaction_worker_cycle_minor > compaction_worker_cycle_MAJOR -> compaction_worker_cycle_major -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-25252) All new metrics should be lower cased
[ https://issues.apache.org/jira/browse/HIVE-25252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-25252: --- Parent: HIVE-24824 Issue Type: Sub-task (was: Bug) > All new metrics should be lower cased > - > > Key: HIVE-25252 > URL: https://issues.apache.org/jira/browse/HIVE-25252 > Project: Hive > Issue Type: Sub-task >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > > compaction_worker_cycle_MINOR -> compaction_worker_cycle_minor > compaction_worker_cycle_MAJOR -> compaction_worker_cycle_major -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-25252) All new mewLower case
[ https://issues.apache.org/jira/browse/HIVE-25252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits reassigned HIVE-25252: -- > All new mewLower case > - > > Key: HIVE-25252 > URL: https://issues.apache.org/jira/browse/HIVE-25252 > Project: Hive > Issue Type: Bug >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > > compaction_worker_cycle_MINOR -> compaction_worker_cycle_minor > compaction_worker_cycle_MAJOR -> compaction_worker_cycle_major -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-25252) All new metrics should be lower cased
[ https://issues.apache.org/jira/browse/HIVE-25252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-25252: --- Summary: All new metrics should be lower cased (was: All new mewLower case) > All new metrics should be lower cased > - > > Key: HIVE-25252 > URL: https://issues.apache.org/jira/browse/HIVE-25252 > Project: Hive > Issue Type: Bug >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > > compaction_worker_cycle_MINOR -> compaction_worker_cycle_minor > compaction_worker_cycle_MAJOR -> compaction_worker_cycle_major -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-25215) tables_with_x_aborted_transactions should count partition/unpartitioned tables
[ https://issues.apache.org/jira/browse/HIVE-25215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-25215 started by Antal Sinkovits. -- > tables_with_x_aborted_transactions should count partition/unpartitioned tables > -- > > Key: HIVE-25215 > URL: https://issues.apache.org/jira/browse/HIVE-25215 > Project: Hive > Issue Type: Sub-task >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > > Initiator compare's each partition's number of aborts to > hive.compactor.abortedtxn.threshold, so tables_with_x_aborted_transactions > should reflect the number of partitions/unpartitioned tables with >x aborts, > instead of the number of tables with >x aborts. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-25215) tables_with_x_aborted_transactions should count partition/unpartitioned tables
[ https://issues.apache.org/jira/browse/HIVE-25215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits reassigned HIVE-25215: -- > tables_with_x_aborted_transactions should count partition/unpartitioned tables > -- > > Key: HIVE-25215 > URL: https://issues.apache.org/jira/browse/HIVE-25215 > Project: Hive > Issue Type: Sub-task >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > > Initiator compare's each partition's number of aborts to > hive.compactor.abortedtxn.threshold, so tables_with_x_aborted_transactions > should reflect the number of partitions/unpartitioned tables with >x aborts, > instead of the number of tables with >x aborts. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-25081) Put metrics collection behind a feature flag
[ https://issues.apache.org/jira/browse/HIVE-25081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-25081: --- Parent: HIVE-24824 Issue Type: Sub-task (was: Bug) > Put metrics collection behind a feature flag > > > Key: HIVE-25081 > URL: https://issues.apache.org/jira/browse/HIVE-25081 > Project: Hive > Issue Type: Sub-task >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Most metrics we're creating are collected in AcidMetricsService, which is > behind a feature flag. However there are some metrics that are collected > outside of the service. These should be behind a feature flag in addition to > hive.metastore.metrics.enabled. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-25081) Put metrics collection behind a feature flag
[ https://issues.apache.org/jira/browse/HIVE-25081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-25081 started by Antal Sinkovits. -- > Put metrics collection behind a feature flag > > > Key: HIVE-25081 > URL: https://issues.apache.org/jira/browse/HIVE-25081 > Project: Hive > Issue Type: Bug >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > > Most metrics we're creating are collected in AcidMetricsService, which is > behind a feature flag. However there are some metrics that are collected > outside of the service. These should be behind a feature flag in addition to > hive.metastore.metrics.enabled. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-25080) Create metric about oldest entry in "ready for cleaning" state
[ https://issues.apache.org/jira/browse/HIVE-25080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-25080 started by Antal Sinkovits. -- > Create metric about oldest entry in "ready for cleaning" state > -- > > Key: HIVE-25080 > URL: https://issues.apache.org/jira/browse/HIVE-25080 > Project: Hive > Issue Type: Bug >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > > When a compaction txn commits, COMPACTION_QUEUE.CQ_COMMIT_TIME is updated > with the current time. Then the compaction state is set to "ready for > cleaning". (... and then the Cleaner runs and the state is set to "succeeded" > hopefully) > Based on this we know (roughly) how long a compaction has been in state > "ready for cleaning". > We should create a metric similar to compaction_oldest_enqueue_age_in_sec > that would show that the cleaner is blocked by something i.e. find the > compaction in "ready for cleaning" that has the oldest commit time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-25079) Create new metric about number of writes to tables with manually disabled compaction
[ https://issues.apache.org/jira/browse/HIVE-25079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-25079 started by Antal Sinkovits. -- > Create new metric about number of writes to tables with manually disabled > compaction > > > Key: HIVE-25079 > URL: https://issues.apache.org/jira/browse/HIVE-25079 > Project: Hive > Issue Type: Bug >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > > Create a new metric that measures the number of writes tables that has > compaction turned off manually. It does not matter if the write is committed > or aborted (both are bad...) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-25081) Put metrics collection behind a feature flag
[ https://issues.apache.org/jira/browse/HIVE-25081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits reassigned HIVE-25081: -- > Put metrics collection behind a feature flag > > > Key: HIVE-25081 > URL: https://issues.apache.org/jira/browse/HIVE-25081 > Project: Hive > Issue Type: Bug >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > > Most metrics we're creating are collected in AcidMetricsService, which is > behind a feature flag. However there are some metrics that are collected > outside of the service. These should be behind a feature flag in addition to > hive.metastore.metrics.enabled. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-25080) Create metric about oldest entry in "ready for cleaning" state
[ https://issues.apache.org/jira/browse/HIVE-25080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits reassigned HIVE-25080: -- > Create metric about oldest entry in "ready for cleaning" state > -- > > Key: HIVE-25080 > URL: https://issues.apache.org/jira/browse/HIVE-25080 > Project: Hive > Issue Type: Bug >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > > When a compaction txn commits, COMPACTION_QUEUE.CQ_COMMIT_TIME is updated > with the current time. Then the compaction state is set to "ready for > cleaning". (... and then the Cleaner runs and the state is set to "succeeded" > hopefully) > Based on this we know (roughly) how long a compaction has been in state > "ready for cleaning". > We should create a metric similar to compaction_oldest_enqueue_age_in_sec > that would show that the cleaner is blocked by something i.e. find the > compaction in "ready for cleaning" that has the oldest commit time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-25079) Create new metric about number of writes to tables with manually disabled compaction
[ https://issues.apache.org/jira/browse/HIVE-25079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits reassigned HIVE-25079: -- > Create new metric about number of writes to tables with manually disabled > compaction > > > Key: HIVE-25079 > URL: https://issues.apache.org/jira/browse/HIVE-25079 > Project: Hive > Issue Type: Bug >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > > Create a new metric that measures the number of writes tables that has > compaction turned off manually. It does not matter if the write is committed > or aborted (both are bad...) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-24722) LLAP cache hydration
[ https://issues.apache.org/jira/browse/HIVE-24722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits resolved HIVE-24722. Fix Version/s: 4.0.0 Resolution: Fixed All subtasks are committed, closing this. > LLAP cache hydration > > > Key: HIVE-24722 > URL: https://issues.apache.org/jira/browse/HIVE-24722 > Project: Hive > Issue Type: Improvement >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Labels: llap > Fix For: 4.0.0 > > > Provide a way to save and reload the contents of the cache in the llap > daemons. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-25009) Compaction worker and initiator version check can cause NPE if the COMPACTION_QUEUE is empty
[ https://issues.apache.org/jira/browse/HIVE-25009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits resolved HIVE-25009. Fix Version/s: 4.0.0 Resolution: Fixed > Compaction worker and initiator version check can cause NPE if the > COMPACTION_QUEUE is empty > > > Key: HIVE-25009 > URL: https://issues.apache.org/jira/browse/HIVE-25009 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-25018) Create new metrics about Initiator / Cleaner failures
[ https://issues.apache.org/jira/browse/HIVE-25018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-25018: --- Fix Version/s: 4.0.0 > Create new metrics about Initiator / Cleaner failures > - > > Key: HIVE-25018 > URL: https://issues.apache.org/jira/browse/HIVE-25018 > Project: Hive > Issue Type: Sub-task >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 40m > Remaining Estimate: 0h > > 2 new metrics should be defined: > Failed Initiator cycles > Failed Cleaner cycles > They should be measured as part of the error handling in the services, the > lock timeout on AUX lock, should be ignored. > These should be RatioGauges (fail / success) > A RatioGauge implementation is available in the metrics package in common, a > similar one should be created in the metastore. The common is build on top of > MetricsVariable interface, where someone provides the metric from outside, in > the metastore it should be done like the Gauge implementation, where the > metrics class handles the AtomicIntegers -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-25018) Create new metrics about Initiator / Cleaner failures
[ https://issues.apache.org/jira/browse/HIVE-25018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits resolved HIVE-25018. Target Version/s: 4.0.0 Resolution: Fixed > Create new metrics about Initiator / Cleaner failures > - > > Key: HIVE-25018 > URL: https://issues.apache.org/jira/browse/HIVE-25018 > Project: Hive > Issue Type: Sub-task >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > 2 new metrics should be defined: > Failed Initiator cycles > Failed Cleaner cycles > They should be measured as part of the error handling in the services, the > lock timeout on AUX lock, should be ignored. > These should be RatioGauges (fail / success) > A RatioGauge implementation is available in the metrics package in common, a > similar one should be created in the metastore. The common is build on top of > MetricsVariable interface, where someone provides the metric from outside, in > the metastore it should be done like the Gauge implementation, where the > metrics class handles the AtomicIntegers -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-25018) Create new metrics about Initiator / Cleaner failures
[ https://issues.apache.org/jira/browse/HIVE-25018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-25018: --- Target Version/s: (was: 4.0.0) > Create new metrics about Initiator / Cleaner failures > - > > Key: HIVE-25018 > URL: https://issues.apache.org/jira/browse/HIVE-25018 > Project: Hive > Issue Type: Sub-task >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > 2 new metrics should be defined: > Failed Initiator cycles > Failed Cleaner cycles > They should be measured as part of the error handling in the services, the > lock timeout on AUX lock, should be ignored. > These should be RatioGauges (fail / success) > A RatioGauge implementation is available in the metrics package in common, a > similar one should be created in the metastore. The common is build on top of > MetricsVariable interface, where someone provides the metric from outside, in > the metastore it should be done like the Gauge implementation, where the > metrics class handles the AtomicIntegers -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-25037) Create metric: Number of tables with > x aborts
[ https://issues.apache.org/jira/browse/HIVE-25037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-25037 started by Antal Sinkovits. -- > Create metric: Number of tables with > x aborts > --- > > Key: HIVE-25037 > URL: https://issues.apache.org/jira/browse/HIVE-25037 > Project: Hive > Issue Type: Sub-task >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > > Create metric about number of tables with > x aborts. > x should be settable and default to 1500. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-25037) Create metric: Number of tables with > x aborts
[ https://issues.apache.org/jira/browse/HIVE-25037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits reassigned HIVE-25037: -- > Create metric: Number of tables with > x aborts > --- > > Key: HIVE-25037 > URL: https://issues.apache.org/jira/browse/HIVE-25037 > Project: Hive > Issue Type: Sub-task >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > > Create metric about number of tables with > x aborts. > x should be settable and default to 1500. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-25021) Divide oldest_open_txn into oldest replication and non-replication transactions
[ https://issues.apache.org/jira/browse/HIVE-25021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits reassigned HIVE-25021: -- > Divide oldest_open_txn into oldest replication and non-replication > transactions > --- > > Key: HIVE-25021 > URL: https://issues.apache.org/jira/browse/HIVE-25021 > Project: Hive > Issue Type: Sub-task >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > > We should have different metrics (age and txn id) for > oldest replication txn (TXN_TYPE==1) > oldest non-replication txn (TXN_TYPE!=1) > so recommendations can be tailored to the different cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-25021) Divide oldest_open_txn into oldest replication and non-replication transactions
[ https://issues.apache.org/jira/browse/HIVE-25021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-25021 started by Antal Sinkovits. -- > Divide oldest_open_txn into oldest replication and non-replication > transactions > --- > > Key: HIVE-25021 > URL: https://issues.apache.org/jira/browse/HIVE-25021 > Project: Hive > Issue Type: Sub-task >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > > We should have different metrics (age and txn id) for > oldest replication txn (TXN_TYPE==1) > oldest non-replication txn (TXN_TYPE!=1) > so recommendations can be tailored to the different cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-25019) Rename metrics that have spaces in the name
[ https://issues.apache.org/jira/browse/HIVE-25019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-25019 started by Antal Sinkovits. -- > Rename metrics that have spaces in the name > --- > > Key: HIVE-25019 > URL: https://issues.apache.org/jira/browse/HIVE-25019 > Project: Hive > Issue Type: Sub-task >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > > Metrics "num_compactions_ready for cleaning" and "num_compactions_not > initiated" contain spaces. > They should be renamed to "num_compactions_ready_for_cleaning" and > "num_compactions_not_initiated" respectively. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-25019) Rename metrics that have spaces in the name
[ https://issues.apache.org/jira/browse/HIVE-25019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits reassigned HIVE-25019: -- > Rename metrics that have spaces in the name > --- > > Key: HIVE-25019 > URL: https://issues.apache.org/jira/browse/HIVE-25019 > Project: Hive > Issue Type: Sub-task >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > > Metrics "num_compactions_ready for cleaning" and "num_compactions_not > initiated" contain spaces. > They should be renamed to "num_compactions_ready_for_cleaning" and > "num_compactions_not_initiated" respectively. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-25018) Create new metrics about Initiator / Cleaner failures
[ https://issues.apache.org/jira/browse/HIVE-25018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-25018 started by Antal Sinkovits. -- > Create new metrics about Initiator / Cleaner failures > - > > Key: HIVE-25018 > URL: https://issues.apache.org/jira/browse/HIVE-25018 > Project: Hive > Issue Type: Sub-task >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > > 2 new metrics should be defined: > Failed Initiator cycles > Failed Cleaner cycles > They should be measured as part of the error handling in the services, the > lock timeout on AUX lock, should be ignored. > These should be RatioGauges (fail / success) > A RatioGauge implementation is available in the metrics package in common, a > similar one should be created in the metastore. The common is build on top of > MetricsVariable interface, where someone provides the metric from outside, in > the metastore it should be done like the Gauge implementation, where the > metrics class handles the AtomicIntegers -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-25018) Create new metrics about Initiator / Cleaner failures
[ https://issues.apache.org/jira/browse/HIVE-25018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits reassigned HIVE-25018: -- > Create new metrics about Initiator / Cleaner failures > - > > Key: HIVE-25018 > URL: https://issues.apache.org/jira/browse/HIVE-25018 > Project: Hive > Issue Type: Sub-task >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > > 2 new metrics should be defined: > Failed Initiator cycles > Failed Cleaner cycles > They should be measured as part of the error handling in the services, the > lock timeout on AUX lock, should be ignored. > These should be RatioGauges (fail / success) > A RatioGauge implementation is available in the metrics package in common, a > similar one should be created in the metastore. The common is build on top of > MetricsVariable interface, where someone provides the metric from outside, in > the metastore it should be done like the Gauge implementation, where the > metrics class handles the AtomicIntegers -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-25009) Compaction worker and initiator version check can cause NPE if the COMPACTION_QUEUE is empty
[ https://issues.apache.org/jira/browse/HIVE-25009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-25009: --- Affects Version/s: 4.0.0 > Compaction worker and initiator version check can cause NPE if the > COMPACTION_QUEUE is empty > > > Key: HIVE-25009 > URL: https://issues.apache.org/jira/browse/HIVE-25009 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-25009) Compaction worker and initiator version check can cause NPE if the COMPACTION_QUEUE is empty
[ https://issues.apache.org/jira/browse/HIVE-25009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-25009 started by Antal Sinkovits. -- > Compaction worker and initiator version check can cause NPE if the > COMPACTION_QUEUE is empty > > > Key: HIVE-25009 > URL: https://issues.apache.org/jira/browse/HIVE-25009 > Project: Hive > Issue Type: Bug >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-25009) Compaction worker and initiator version check can cause NPE if the COMPACTION_QUEUE is empty
[ https://issues.apache.org/jira/browse/HIVE-25009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits reassigned HIVE-25009: -- > Compaction worker and initiator version check can cause NPE if the > COMPACTION_QUEUE is empty > > > Key: HIVE-25009 > URL: https://issues.apache.org/jira/browse/HIVE-25009 > Project: Hive > Issue Type: Bug >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-25009) Compaction worker and initiator version check can cause NPE if the COMPACTION_QUEUE is empty
[ https://issues.apache.org/jira/browse/HIVE-25009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-25009: --- Component/s: Transactions > Compaction worker and initiator version check can cause NPE if the > COMPACTION_QUEUE is empty > > > Key: HIVE-25009 > URL: https://issues.apache.org/jira/browse/HIVE-25009 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24727) Cache hydration api in llap proto
[ https://issues.apache.org/jira/browse/HIVE-24727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits reassigned HIVE-24727: -- Assignee: Antal Sinkovits > Cache hydration api in llap proto > - > > Key: HIVE-24727 > URL: https://issues.apache.org/jira/browse/HIVE-24727 > Project: Hive > Issue Type: Sub-task >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24725) Collect top priority items from llap cache policy
[ https://issues.apache.org/jira/browse/HIVE-24725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits reassigned HIVE-24725: -- Assignee: Antal Sinkovits > Collect top priority items from llap cache policy > - > > Key: HIVE-24725 > URL: https://issues.apache.org/jira/browse/HIVE-24725 > Project: Hive > Issue Type: Sub-task >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24726) Track required data for cache hydration
[ https://issues.apache.org/jira/browse/HIVE-24726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits reassigned HIVE-24726: -- Assignee: Antal Sinkovits > Track required data for cache hydration > --- > > Key: HIVE-24726 > URL: https://issues.apache.org/jira/browse/HIVE-24726 > Project: Hive > Issue Type: Sub-task >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24729) Implement strategy for llap cache hydration
[ https://issues.apache.org/jira/browse/HIVE-24729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits reassigned HIVE-24729: -- Assignee: Antal Sinkovits > Implement strategy for llap cache hydration > --- > > Key: HIVE-24729 > URL: https://issues.apache.org/jira/browse/HIVE-24729 > Project: Hive > Issue Type: Sub-task >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24728) Low level reader for llap cache hydration
[ https://issues.apache.org/jira/browse/HIVE-24728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits reassigned HIVE-24728: -- Assignee: Antal Sinkovits > Low level reader for llap cache hydration > - > > Key: HIVE-24728 > URL: https://issues.apache.org/jira/browse/HIVE-24728 > Project: Hive > Issue Type: Sub-task >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24722) LLAP cache hydration
[ https://issues.apache.org/jira/browse/HIVE-24722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits reassigned HIVE-24722: -- > LLAP cache hydration > > > Key: HIVE-24722 > URL: https://issues.apache.org/jira/browse/HIVE-24722 > Project: Hive > Issue Type: Improvement >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Labels: llap > > Provide a way to save and reload the contents of the cache in the llap > daemons. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-24653) Race condition between compactor marker generation and get splits
[ https://issues.apache.org/jira/browse/HIVE-24653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits resolved HIVE-24653. Fix Version/s: 3.1.3 Resolution: Fixed > Race condition between compactor marker generation and get splits > - > > Key: HIVE-24653 > URL: https://issues.apache.org/jira/browse/HIVE-24653 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.2 >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Minor > Labels: pull-request-available > Fix For: 3.1.3 > > Time Spent: 1h > Remaining Estimate: 0h > > In a rear scenario it's possible that the compactor moved the files in the > final location before creating the compactor marker, so it can be fetched by > get splits before the marker is created. > 2020-09-14 04:55:25,978 [ERROR] ORC_GET_SPLITS #4 |io.AcidUtils|: Failed to > read > hdfs://host/warehouse/tablespace/managed/hive/database.db/table/partition=x/base_0011535/_metadata_acid: > No content to map to Object due to end of input > java.io.EOFException: No content to map to Object due to end of input -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24653) Race condition between compactor marker generation and get splits
[ https://issues.apache.org/jira/browse/HIVE-24653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits reassigned HIVE-24653: -- > Race condition between compactor marker generation and get splits > - > > Key: HIVE-24653 > URL: https://issues.apache.org/jira/browse/HIVE-24653 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.2 >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Minor > > In a rear scenario it's possible that the compactor moved the files in the > final location before creating the compactor marker, so it can be fetched by > get splits before the marker is created. > 2020-09-14 04:55:25,978 [ERROR] ORC_GET_SPLITS #4 |io.AcidUtils|: Failed to > read > hdfs://host/warehouse/tablespace/managed/hive/database.db/table/partition=x/base_0011535/_metadata_acid: > No content to map to Object due to end of input > java.io.EOFException: No content to map to Object due to end of input -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-24475) Generalize fixacidkeyindex utility
[ https://issues.apache.org/jira/browse/HIVE-24475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits resolved HIVE-24475. Resolution: Fixed > Generalize fixacidkeyindex utility > -- > > Key: HIVE-24475 > URL: https://issues.apache.org/jira/browse/HIVE-24475 > Project: Hive > Issue Type: Improvement > Components: ORC, Transactions >Affects Versions: 3.0.0 >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > There is a utility in hive which can validate/fix corrupted > hive.acid.key.index. > hive --service fixacidkeyindex > Unfortunately it is only tailored for a specific problem > (https://issues.apache.org/jira/browse/HIVE-18907), instead of generally > validating and recovering the hive.acid.key.index from the stripe data itself. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24475) Generalize fixacidkeyindex utility
[ https://issues.apache.org/jira/browse/HIVE-24475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-24475: --- Fix Version/s: 4.0.0 > Generalize fixacidkeyindex utility > -- > > Key: HIVE-24475 > URL: https://issues.apache.org/jira/browse/HIVE-24475 > Project: Hive > Issue Type: Improvement > Components: ORC, Transactions >Affects Versions: 3.0.0 >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > There is a utility in hive which can validate/fix corrupted > hive.acid.key.index. > hive --service fixacidkeyindex > Unfortunately it is only tailored for a specific problem > (https://issues.apache.org/jira/browse/HIVE-18907), instead of generally > validating and recovering the hive.acid.key.index from the stripe data itself. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-24475) Generalize fixacidkeyindex utility
[ https://issues.apache.org/jira/browse/HIVE-24475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-24475 started by Antal Sinkovits. -- > Generalize fixacidkeyindex utility > -- > > Key: HIVE-24475 > URL: https://issues.apache.org/jira/browse/HIVE-24475 > Project: Hive > Issue Type: Improvement > Components: ORC, Transactions >Affects Versions: 3.0.0 >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > > There is a utility in hive which can validate/fix corrupted > hive.acid.key.index. > hive --service fixacidkeyindex > Unfortunately it is only tailored for a specific problem > (https://issues.apache.org/jira/browse/HIVE-18907), instead of generally > validating and recovering the hive.acid.key.index from the stripe data itself. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24475) Generalize fixacidkeyindex utility
[ https://issues.apache.org/jira/browse/HIVE-24475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-24475: --- Affects Version/s: (was: 4.0.0) 3.0.0 > Generalize fixacidkeyindex utility > -- > > Key: HIVE-24475 > URL: https://issues.apache.org/jira/browse/HIVE-24475 > Project: Hive > Issue Type: Improvement > Components: ORC, Transactions >Affects Versions: 3.0.0 >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > > There is a utility in hive which can validate/fix corrupted > hive.acid.key.index. > hive --service fixacidkeyindex > Unfortunately it is only tailored for a specific problem > (https://issues.apache.org/jira/browse/HIVE-18907), instead of generally > validating and recovering the hive.acid.key.index from the stripe data itself. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24475) Generalize fixacidkeyindex utility
[ https://issues.apache.org/jira/browse/HIVE-24475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits reassigned HIVE-24475: -- > Generalize fixacidkeyindex utility > -- > > Key: HIVE-24475 > URL: https://issues.apache.org/jira/browse/HIVE-24475 > Project: Hive > Issue Type: Improvement > Components: ORC, Transactions >Affects Versions: 4.0.0 >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > > There is a utility in hive which can validate/fix corrupted > hive.acid.key.index. > hive --service fixacidkeyindex > Unfortunately it is only tailored for a specific problem > (https://issues.apache.org/jira/browse/HIVE-18907), instead of generally > validating and recovering the hive.acid.key.index from the stripe data itself. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24293) Integer overflow in llap collision mask
[ https://issues.apache.org/jira/browse/HIVE-24293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-24293: --- Description: If multiple threads put the same buffer to the cache, only one succeeds. The other one detects this, and replaces its own buffer. This is marked by a bit mask encoded in a long, where the collided buffers are marked with a 1. > Integer overflow in llap collision mask > --- > > Key: HIVE-24293 > URL: https://issues.apache.org/jira/browse/HIVE-24293 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > > If multiple threads put the same buffer to the cache, only one succeeds. The > other one detects this, and replaces its own buffer. This is marked by a bit > mask encoded in a long, where the collided buffers are marked with a 1. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-24293) Integer overflow in llap collision mask
[ https://issues.apache.org/jira/browse/HIVE-24293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-24293 started by Antal Sinkovits. -- > Integer overflow in llap collision mask > --- > > Key: HIVE-24293 > URL: https://issues.apache.org/jira/browse/HIVE-24293 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > > If multiple threads put the same buffer to the cache, only one succeeds. The > other one detects this, and replaces its own buffer. This is marked by a bit > mask encoded in a long, where the collided buffers are marked with a 1. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24293) Integer overflow in llap collision mask
[ https://issues.apache.org/jira/browse/HIVE-24293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits reassigned HIVE-24293: -- > Integer overflow in llap collision mask > --- > > Key: HIVE-24293 > URL: https://issues.apache.org/jira/browse/HIVE-24293 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23851) MSCK REPAIR Command With Partition Filtering Fails While Dropping Partitions
[
https://issues.apache.org/jira/browse/HIVE-23851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17158126#comment-17158126
]
Antal Sinkovits commented on HIVE-23851:
[~srahman] I'm a bit confused now. Is this the same issue as
https://issues.apache.org/jira/browse/HIVE-23808 ?
> MSCK REPAIR Command With Partition Filtering Fails While Dropping Partitions
>
>
> Key: HIVE-23851
> URL: https://issues.apache.org/jira/browse/HIVE-23851
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Syed Shameerur Rahman
>Assignee: Syed Shameerur Rahman
>Priority: Major
> Fix For: 4.0.0
>
>
> *Steps to reproduce:*
> # Create external table
> # Run msck command to sync all the partitions with metastore
> # Remove one of the partition path
> # Run msck repair with partition filtering
> *Stack Trace:*
> {code:java}
> 2020-07-15T02:10:29,045 ERROR [4dad298b-28b1-4e6b-94b6-aa785b60c576 main]
> ppr.PartitionExpressionForMetastore: Failed to deserialize the expression
> java.lang.IndexOutOfBoundsException: Index: 110, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:657) ~[?:1.8.0_192]
> at java.util.ArrayList.get(ArrayList.java:433) ~[?:1.8.0_192]
> at
> org.apache.hive.com.esotericsoftware.kryo.util.MapReferenceResolver.getReadObject(MapReferenceResolver.java:60)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readReferenceOrNull(Kryo.java:857)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:707)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:211)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities.deserializeObjectFromKryo(SerializationUtilities.java:806)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities.deserializeExpressionFromKryo(SerializationUtilities.java:775)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.optimizer.ppr.PartitionExpressionForMetastore.deserializeExpr(PartitionExpressionForMetastore.java:96)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.optimizer.ppr.PartitionExpressionForMetastore.convertExprToFilter(PartitionExpressionForMetastore.java:52)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.PartFilterExprUtil.makeExpressionTree(PartFilterExprUtil.java:48)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByExprInternal(ObjectStore.java:3593)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.VerifyingObjectStore.getPartitionsByExpr(VerifyingObjectStore.java:80)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT-tests.jar:4.0.0-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_192]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_192]
> {code}
> *Cause:*
> In case of msck repair with partition filtering we expect expression proxy
> class to be set as PartitionExpressionForMetastore (
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/ddl/misc/msck/MsckAnalyzer.java#L78
> ), While dropping partition we serialize the drop partition filter
> expression as (
> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java#L589
> ) which is incompatible during deserializtion happening in
> PartitionExpressionForMetastore (
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionExpressionForMetastore.java#L52
> ) hence the query fails with Failed to deserialize the expression.
> *Solutions*:
> I could think of two approaches to this problem
> # Since PartitionExpressionForMetastore is required only during parition
> pruning step, We can switch back the expression proxy class to
> MsckPartitionExpressionProxy once the partition pruning step is done.
> # The other solution is to make serialization process in msck drop partition
> filter expression compatible with the one with
> PartitionExpressionForMetastore, We can do this via Reflection since the drop
> partition serialization happens in Msck class (standadlone-metatsore) by this
> way we can completely remove the need for class MsckPartitionExpressionProxy
> and this also helps to
[jira] [Work started] (HIVE-23847) Extracting hive-parser module broke exec jar upload in tez
[
https://issues.apache.org/jira/browse/HIVE-23847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-23847 started by Antal Sinkovits.
--
> Extracting hive-parser module broke exec jar upload in tez
> --
>
> Key: HIVE-23847
> URL: https://issues.apache.org/jira/browse/HIVE-23847
> Project: Hive
> Issue Type: Bug
>Reporter: Antal Sinkovits
>Assignee: Antal Sinkovits
>Priority: Major
>
> 2020-07-13 16:53:50,551 [INFO] [Dispatcher thread {Central}]
> |HistoryEventHandler.criticalEvents|:
> [HISTORY][DAG:dag_1594632473849_0001_1][Event:TASK_ATTEMPT_FINISHED]:
> vertexName=Map 1, taskAttemptId=attempt_1594632473849_0001_1_00_00_0,
> creationTime=1594652027059, allocationTime=1594652028460,
> startTime=1594652029356, finishTime=1594652030546, timeTaken=1190,
> status=FAILED, taskFailureType=NON_FATAL, errorEnum=FRAMEWORK_ERROR,
> diagnostics=Error: Error while running task ( failure ) :
> attempt_1594632473849_0001_1_00_00_0:java.lang.RuntimeException:
> java.lang.RuntimeException: Map operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:381)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:75)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:62)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:62)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:38)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
> at
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.RuntimeException: Map operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:340)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266)
> ... 16 more
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/hadoop/hive/ql/parse/ParseException
> at java.lang.Class.getDeclaredConstructors0(Native Method)
> at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
> at java.lang.Class.getConstructor0(Class.java:3075)
> at java.lang.Class.getDeclaredConstructor(Class.java:2178)
> at
> org.apache.hive.common.util.ReflectionUtil.newInstance(ReflectionUtil.java:79)
> at
> org.apache.hadoop.hive.ql.exec.Registry.registerGenericUDTF(Registry.java:225)
> at
> org.apache.hadoop.hive.ql.exec.Registry.registerGenericUDTF(Registry.java:217)
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.(FunctionRegistry.java:544)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.isDeterministic(ExprNodeGenericFuncEvaluator.java:154)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.isConsistentWithinQuery(ExprNodeEvaluator.java:117)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.iterate(ExprNodeEvaluatorFactory.java:102)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.toCachedEvals(ExprNodeEvaluatorFactory.java:76)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:69)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:359)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:548)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:502)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:368)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.initializeMapOperator(MapOperator.java:506)
> at
>
[jira] [Assigned] (HIVE-23847) Extracting hive-parser module broke exec jar upload in tez
[
https://issues.apache.org/jira/browse/HIVE-23847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Antal Sinkovits reassigned HIVE-23847:
--
Assignee: Antal Sinkovits
> Extracting hive-parser module broke exec jar upload in tez
> --
>
> Key: HIVE-23847
> URL: https://issues.apache.org/jira/browse/HIVE-23847
> Project: Hive
> Issue Type: Bug
>Reporter: Antal Sinkovits
>Assignee: Antal Sinkovits
>Priority: Major
>
> 2020-07-13 16:53:50,551 [INFO] [Dispatcher thread {Central}]
> |HistoryEventHandler.criticalEvents|:
> [HISTORY][DAG:dag_1594632473849_0001_1][Event:TASK_ATTEMPT_FINISHED]:
> vertexName=Map 1, taskAttemptId=attempt_1594632473849_0001_1_00_00_0,
> creationTime=1594652027059, allocationTime=1594652028460,
> startTime=1594652029356, finishTime=1594652030546, timeTaken=1190,
> status=FAILED, taskFailureType=NON_FATAL, errorEnum=FRAMEWORK_ERROR,
> diagnostics=Error: Error while running task ( failure ) :
> attempt_1594632473849_0001_1_00_00_0:java.lang.RuntimeException:
> java.lang.RuntimeException: Map operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:381)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:75)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:62)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:62)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:38)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
> at
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.RuntimeException: Map operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:340)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266)
> ... 16 more
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/hadoop/hive/ql/parse/ParseException
> at java.lang.Class.getDeclaredConstructors0(Native Method)
> at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
> at java.lang.Class.getConstructor0(Class.java:3075)
> at java.lang.Class.getDeclaredConstructor(Class.java:2178)
> at
> org.apache.hive.common.util.ReflectionUtil.newInstance(ReflectionUtil.java:79)
> at
> org.apache.hadoop.hive.ql.exec.Registry.registerGenericUDTF(Registry.java:225)
> at
> org.apache.hadoop.hive.ql.exec.Registry.registerGenericUDTF(Registry.java:217)
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.(FunctionRegistry.java:544)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.isDeterministic(ExprNodeGenericFuncEvaluator.java:154)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.isConsistentWithinQuery(ExprNodeEvaluator.java:117)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.iterate(ExprNodeEvaluatorFactory.java:102)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.toCachedEvals(ExprNodeEvaluatorFactory.java:76)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:69)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:359)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:548)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:502)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:368)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.initializeMapOperator(MapOperator.java:506)
> at
>
[jira] [Commented] (HIVE-23808) "MSCK REPAIR.. DROP Partitions fail" with kryo Exception
[
https://issues.apache.org/jira/browse/HIVE-23808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17156919#comment-17156919
]
Antal Sinkovits commented on HIVE-23808:
[~srahman] this was reproducible on master.
> "MSCK REPAIR.. DROP Partitions fail" with kryo Exception
> -
>
> Key: HIVE-23808
> URL: https://issues.apache.org/jira/browse/HIVE-23808
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.2.0
>Reporter: Rajkumar Singh
>Assignee: Antal Sinkovits
>Priority: Major
>
> Steps to the repo:
> 1. Create External partition table
> 2. Remove some partition manually be using hdfs dfs -rm command
> 3. run "MSCK REPAIR.. DROP Partitions" and it will fail with following
> exception
> {code:java}
> 2020-07-06 10:42:11,434 WARN
> org.apache.hadoop.hive.metastore.utils.RetryUtilities$ExponentiallyDecayingBatchWork:
> [HiveServer2-Background-Pool: Thread-210]: Exception thrown while processing
> using a batch size 2
> org.apache.hadoop.hive.metastore.utils.MetastoreException:
> MetaException(message:Index: 117, Size: 0)
> at org.apache.hadoop.hive.metastore.Msck$2.execute(Msck.java:479)
> ~[hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.metastore.Msck$2.execute(Msck.java:432)
> ~[hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at
> org.apache.hadoop.hive.metastore.utils.RetryUtilities$ExponentiallyDecayingBatchWork.run(RetryUtilities.java:91)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at
> org.apache.hadoop.hive.metastore.Msck.dropPartitionsInBatches(Msck.java:496)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.metastore.Msck.repair(Msck.java:223)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at
> org.apache.hadoop.hive.ql.ddl.misc.msck.MsckOperation.execute(MsckOperation.java:74)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.ql.ddl.DDLTask.execute(DDLTask.java:80)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:359)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:330)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:721)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:488)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:482)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:166)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:225)
> [hive-service-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:87)
> [hive-service-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:322)
> [hive-service-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at java.security.AccessController.doPrivileged(Native Method)
> [?:1.8.0_242]
> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_242]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> [hadoop-common-3.1.1.7.1.1.0-565.jar:?]
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:340)
> [hive-service-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [?:1.8.0_242]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [?:1.8.0_242]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>
[jira] [Resolved] (HIVE-23808) "MSCK REPAIR.. DROP Partitions fail" with kryo Exception
[
https://issues.apache.org/jira/browse/HIVE-23808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Antal Sinkovits resolved HIVE-23808.
Resolution: Duplicate
> "MSCK REPAIR.. DROP Partitions fail" with kryo Exception
> -
>
> Key: HIVE-23808
> URL: https://issues.apache.org/jira/browse/HIVE-23808
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.2.0
>Reporter: Rajkumar Singh
>Assignee: Antal Sinkovits
>Priority: Major
>
> Steps to the repo:
> 1. Create External partition table
> 2. Remove some partition manually be using hdfs dfs -rm command
> 3. run "MSCK REPAIR.. DROP Partitions" and it will fail with following
> exception
> {code:java}
> 2020-07-06 10:42:11,434 WARN
> org.apache.hadoop.hive.metastore.utils.RetryUtilities$ExponentiallyDecayingBatchWork:
> [HiveServer2-Background-Pool: Thread-210]: Exception thrown while processing
> using a batch size 2
> org.apache.hadoop.hive.metastore.utils.MetastoreException:
> MetaException(message:Index: 117, Size: 0)
> at org.apache.hadoop.hive.metastore.Msck$2.execute(Msck.java:479)
> ~[hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.metastore.Msck$2.execute(Msck.java:432)
> ~[hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at
> org.apache.hadoop.hive.metastore.utils.RetryUtilities$ExponentiallyDecayingBatchWork.run(RetryUtilities.java:91)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at
> org.apache.hadoop.hive.metastore.Msck.dropPartitionsInBatches(Msck.java:496)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.metastore.Msck.repair(Msck.java:223)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at
> org.apache.hadoop.hive.ql.ddl.misc.msck.MsckOperation.execute(MsckOperation.java:74)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.ql.ddl.DDLTask.execute(DDLTask.java:80)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:359)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:330)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:721)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:488)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:482)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:166)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:225)
> [hive-service-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:87)
> [hive-service-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:322)
> [hive-service-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at java.security.AccessController.doPrivileged(Native Method)
> [?:1.8.0_242]
> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_242]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> [hadoop-common-3.1.1.7.1.1.0-565.jar:?]
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:340)
> [hive-service-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [?:1.8.0_242]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [?:1.8.0_242]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [?:1.8.0_242]
> at
[jira] [Commented] (HIVE-23808) "MSCK REPAIR.. DROP Partitions fail" with kryo Exception
[
https://issues.apache.org/jira/browse/HIVE-23808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17156772#comment-17156772
]
Antal Sinkovits commented on HIVE-23808:
It seems that https://issues.apache.org/jira/browse/HIVE-22957 fixed this issue
> "MSCK REPAIR.. DROP Partitions fail" with kryo Exception
> -
>
> Key: HIVE-23808
> URL: https://issues.apache.org/jira/browse/HIVE-23808
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.2.0
>Reporter: Rajkumar Singh
>Assignee: Antal Sinkovits
>Priority: Major
>
> Steps to the repo:
> 1. Create External partition table
> 2. Remove some partition manually be using hdfs dfs -rm command
> 3. run "MSCK REPAIR.. DROP Partitions" and it will fail with following
> exception
> {code:java}
> 2020-07-06 10:42:11,434 WARN
> org.apache.hadoop.hive.metastore.utils.RetryUtilities$ExponentiallyDecayingBatchWork:
> [HiveServer2-Background-Pool: Thread-210]: Exception thrown while processing
> using a batch size 2
> org.apache.hadoop.hive.metastore.utils.MetastoreException:
> MetaException(message:Index: 117, Size: 0)
> at org.apache.hadoop.hive.metastore.Msck$2.execute(Msck.java:479)
> ~[hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.metastore.Msck$2.execute(Msck.java:432)
> ~[hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at
> org.apache.hadoop.hive.metastore.utils.RetryUtilities$ExponentiallyDecayingBatchWork.run(RetryUtilities.java:91)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at
> org.apache.hadoop.hive.metastore.Msck.dropPartitionsInBatches(Msck.java:496)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.metastore.Msck.repair(Msck.java:223)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at
> org.apache.hadoop.hive.ql.ddl.misc.msck.MsckOperation.execute(MsckOperation.java:74)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.ql.ddl.DDLTask.execute(DDLTask.java:80)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:359)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:330)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:721)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:488)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:482)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:166)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:225)
> [hive-service-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:87)
> [hive-service-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:322)
> [hive-service-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at java.security.AccessController.doPrivileged(Native Method)
> [?:1.8.0_242]
> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_242]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> [hadoop-common-3.1.1.7.1.1.0-565.jar:?]
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:340)
> [hive-service-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [?:1.8.0_242]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [?:1.8.0_242]
> at
>
[jira] [Work started] (HIVE-23808) "MSCK REPAIR.. DROP Partitions fail" with kryo Exception
[
https://issues.apache.org/jira/browse/HIVE-23808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-23808 started by Antal Sinkovits.
--
> "MSCK REPAIR.. DROP Partitions fail" with kryo Exception
> -
>
> Key: HIVE-23808
> URL: https://issues.apache.org/jira/browse/HIVE-23808
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.2.0
>Reporter: Rajkumar Singh
>Assignee: Antal Sinkovits
>Priority: Major
>
> Steps to the repo:
> 1. Create External partition table
> 2. Remove some partition manually be using hdfs dfs -rm command
> 3. run "MSCK REPAIR.. DROP Partitions" and it will fail with following
> exception
> {code:java}
> 2020-07-06 10:42:11,434 WARN
> org.apache.hadoop.hive.metastore.utils.RetryUtilities$ExponentiallyDecayingBatchWork:
> [HiveServer2-Background-Pool: Thread-210]: Exception thrown while processing
> using a batch size 2
> org.apache.hadoop.hive.metastore.utils.MetastoreException:
> MetaException(message:Index: 117, Size: 0)
> at org.apache.hadoop.hive.metastore.Msck$2.execute(Msck.java:479)
> ~[hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.metastore.Msck$2.execute(Msck.java:432)
> ~[hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at
> org.apache.hadoop.hive.metastore.utils.RetryUtilities$ExponentiallyDecayingBatchWork.run(RetryUtilities.java:91)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at
> org.apache.hadoop.hive.metastore.Msck.dropPartitionsInBatches(Msck.java:496)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.metastore.Msck.repair(Msck.java:223)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at
> org.apache.hadoop.hive.ql.ddl.misc.msck.MsckOperation.execute(MsckOperation.java:74)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.ql.ddl.DDLTask.execute(DDLTask.java:80)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:359)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:330)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:721)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:488)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:482)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:166)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:225)
> [hive-service-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:87)
> [hive-service-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:322)
> [hive-service-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at java.security.AccessController.doPrivileged(Native Method)
> [?:1.8.0_242]
> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_242]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> [hadoop-common-3.1.1.7.1.1.0-565.jar:?]
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:340)
> [hive-service-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [?:1.8.0_242]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [?:1.8.0_242]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [?:1.8.0_242]
> at
[jira] [Assigned] (HIVE-23808) "MSCK REPAIR.. DROP Partitions fail" with kryo Exception
[
https://issues.apache.org/jira/browse/HIVE-23808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Antal Sinkovits reassigned HIVE-23808:
--
Assignee: Antal Sinkovits
> "MSCK REPAIR.. DROP Partitions fail" with kryo Exception
> -
>
> Key: HIVE-23808
> URL: https://issues.apache.org/jira/browse/HIVE-23808
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.2.0
>Reporter: Rajkumar Singh
>Assignee: Antal Sinkovits
>Priority: Major
>
> Steps to the repo:
> 1. Create External partition table
> 2. Remove some partition manually be using hdfs dfs -rm command
> 3. run "MSCK REPAIR.. DROP Partitions" and it will fail with following
> exception
> {code:java}
> 2020-07-06 10:42:11,434 WARN
> org.apache.hadoop.hive.metastore.utils.RetryUtilities$ExponentiallyDecayingBatchWork:
> [HiveServer2-Background-Pool: Thread-210]: Exception thrown while processing
> using a batch size 2
> org.apache.hadoop.hive.metastore.utils.MetastoreException:
> MetaException(message:Index: 117, Size: 0)
> at org.apache.hadoop.hive.metastore.Msck$2.execute(Msck.java:479)
> ~[hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.metastore.Msck$2.execute(Msck.java:432)
> ~[hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at
> org.apache.hadoop.hive.metastore.utils.RetryUtilities$ExponentiallyDecayingBatchWork.run(RetryUtilities.java:91)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at
> org.apache.hadoop.hive.metastore.Msck.dropPartitionsInBatches(Msck.java:496)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.metastore.Msck.repair(Msck.java:223)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at
> org.apache.hadoop.hive.ql.ddl.misc.msck.MsckOperation.execute(MsckOperation.java:74)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.ql.ddl.DDLTask.execute(DDLTask.java:80)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:359)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:330)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:721)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:488)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:482)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:166)
> [hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:225)
> [hive-service-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:87)
> [hive-service-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:322)
> [hive-service-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at java.security.AccessController.doPrivileged(Native Method)
> [?:1.8.0_242]
> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_242]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> [hadoop-common-3.1.1.7.1.1.0-565.jar:?]
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:340)
> [hive-service-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [?:1.8.0_242]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [?:1.8.0_242]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [?:1.8.0_242]
> at
[jira] [Commented] (HIVE-23741) Store CacheTags in the file cache level
[ https://issues.apache.org/jira/browse/HIVE-23741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17142877#comment-17142877 ] Antal Sinkovits commented on HIVE-23741: [~odraese] [~szita] could you please review? > Store CacheTags in the file cache level > --- > > Key: HIVE-23741 > URL: https://issues.apache.org/jira/browse/HIVE-23741 > Project: Hive > Issue Type: Improvement >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > CacheTags are currently stored for every data buffer. The strings are > internalized, but the number of cache tag objects can be reduced by moving > them to the file cache level, and back referencing them. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23741) Store CacheTags in the file cache level
[ https://issues.apache.org/jira/browse/HIVE-23741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-23741: --- Status: Patch Available (was: Open) > Store CacheTags in the file cache level > --- > > Key: HIVE-23741 > URL: https://issues.apache.org/jira/browse/HIVE-23741 > Project: Hive > Issue Type: Improvement >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > CacheTags are currently stored for every data buffer. The strings are > internalized, but the number of cache tag objects can be reduced by moving > them to the file cache level, and back referencing them. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23741) Store CacheTags in the file cache level
[ https://issues.apache.org/jira/browse/HIVE-23741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-23741: --- Attachment: (was: HIVE-23741.01.patch) > Store CacheTags in the file cache level > --- > > Key: HIVE-23741 > URL: https://issues.apache.org/jira/browse/HIVE-23741 > Project: Hive > Issue Type: Improvement >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > > CacheTags are currently stored for every data buffer. The strings are > internalized, but the number of cache tag objects can be reduced by moving > them to the file cache level, and back referencing them. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23741) Store CacheTags in the file cache level
[ https://issues.apache.org/jira/browse/HIVE-23741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-23741: --- Status: Open (was: Patch Available) > Store CacheTags in the file cache level > --- > > Key: HIVE-23741 > URL: https://issues.apache.org/jira/browse/HIVE-23741 > Project: Hive > Issue Type: Improvement >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > > CacheTags are currently stored for every data buffer. The strings are > internalized, but the number of cache tag objects can be reduced by moving > them to the file cache level, and back referencing them. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23741) Store CacheTags in the file cache level
[ https://issues.apache.org/jira/browse/HIVE-23741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-23741: --- Status: Patch Available (was: Open) > Store CacheTags in the file cache level > --- > > Key: HIVE-23741 > URL: https://issues.apache.org/jira/browse/HIVE-23741 > Project: Hive > Issue Type: Improvement >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-23741.01.patch > > > CacheTags are currently stored for every data buffer. The strings are > internalized, but the number of cache tag objects can be reduced by moving > them to the file cache level, and back referencing them. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23741) Store CacheTags in the file cache level
[ https://issues.apache.org/jira/browse/HIVE-23741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-23741: --- Attachment: HIVE-23741.01.patch > Store CacheTags in the file cache level > --- > > Key: HIVE-23741 > URL: https://issues.apache.org/jira/browse/HIVE-23741 > Project: Hive > Issue Type: Improvement >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-23741.01.patch > > > CacheTags are currently stored for every data buffer. The strings are > internalized, but the number of cache tag objects can be reduced by moving > them to the file cache level, and back referencing them. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23741) Store CacheTags in the file cache level
[ https://issues.apache.org/jira/browse/HIVE-23741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits reassigned HIVE-23741: -- > Store CacheTags in the file cache level > --- > > Key: HIVE-23741 > URL: https://issues.apache.org/jira/browse/HIVE-23741 > Project: Hive > Issue Type: Improvement >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > > CacheTags are currently stored for every data buffer. The strings are > internalized, but the number of cache tag objects can be reduced by moving > them to the file cache level, and back referencing them. -- This message was sent by Atlassian Jira (v8.3.4#803005)
