[jira] [Commented] (HIVE-22818) Preparation for jetty 9.4.26 upgrade
[ https://issues.apache.org/jira/browse/HIVE-22818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029699#comment-17029699 ] Laszlo Pinter commented on HIVE-22818: -- @Thanks [~pvary] for review & commit! > Preparation for jetty 9.4.26 upgrade > > > Key: HIVE-22818 > URL: https://issues.apache.org/jira/browse/HIVE-22818 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Minor > Fix For: 4.0.0 > > Attachments: HIVE-22818.01.patch, HIVE-22818.02.patch, > HIVE-22818.03.patch > > > Make some code adjustment, before upgrading jetty to 9.4.26. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22818) Preparation for jetty 9.4.26 upgrade
[ https://issues.apache.org/jira/browse/HIVE-22818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22818: - Attachment: HIVE-22818.03.patch > Preparation for jetty 9.4.26 upgrade > > > Key: HIVE-22818 > URL: https://issues.apache.org/jira/browse/HIVE-22818 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Minor > Attachments: HIVE-22818.01.patch, HIVE-22818.02.patch, > HIVE-22818.03.patch > > > Make some code adjustment, before upgrading jetty to 9.4.26. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22818) Preparation for jetty 9.4.26 upgrade
[ https://issues.apache.org/jira/browse/HIVE-22818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22818: - Attachment: HIVE-22818.02.patch > Preparation for jetty 9.4.26 upgrade > > > Key: HIVE-22818 > URL: https://issues.apache.org/jira/browse/HIVE-22818 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Minor > Attachments: HIVE-22818.01.patch, HIVE-22818.02.patch > > > Make some code adjustment, before upgrading jetty to 9.4.26. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22729) Provide a failure reason for failed compactions
[
https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17028807#comment-17028807
]
Laszlo Pinter commented on HIVE-22729:
--
Thank you [~pvary], [~klcopp] [~dkuzmenko]!
> Provide a failure reason for failed compactions
> ---
>
> Key: HIVE-22729
> URL: https://issues.apache.org/jira/browse/HIVE-22729
> Project: Hive
> Issue Type: Improvement
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22729.01.patch, HIVE-22729.02.patch,
> HIVE-22729.03.patch, HIVE-22729.04.patch, HIVE-22729.05.patch,
> HIVE-22729.06.patch, HIVE-22729.07.patch, HIVE-22729.08.patch,
> HIVE-22729.09.patch, HIVE-22729.10.patch, HIVE-22729.11.patch,
> HIVE-22729.12.patch, HIVE-22729.13.patch
>
>
> We should provide a compaction failure reason as easily accessible as
> possible. Like in the result of the {{SHOW COMPACTIONS}} command.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22818) Preparation for jetty 9.4.26 upgrade
[ https://issues.apache.org/jira/browse/HIVE-22818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22818: - Attachment: HIVE-22818.01.patch > Preparation for jetty 9.4.26 upgrade > > > Key: HIVE-22818 > URL: https://issues.apache.org/jira/browse/HIVE-22818 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Minor > Attachments: HIVE-22818.01.patch > > > Make some code adjustment, before upgrading jetty to 9.4.26. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22818) Preparation for jetty 9.4.26 upgrade
[ https://issues.apache.org/jira/browse/HIVE-22818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22818: - Status: Patch Available (was: Open) > Preparation for jetty 9.4.26 upgrade > > > Key: HIVE-22818 > URL: https://issues.apache.org/jira/browse/HIVE-22818 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Minor > Attachments: HIVE-22818.01.patch > > > Make some code adjustment, before upgrading jetty to 9.4.26. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22818) Preparation for jetty 9.4.26 upgrade
[ https://issues.apache.org/jira/browse/HIVE-22818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter reassigned HIVE-22818: > Preparation for jetty 9.4.26 upgrade > > > Key: HIVE-22818 > URL: https://issues.apache.org/jira/browse/HIVE-22818 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Minor > > Make some code adjustment, before upgrading jetty to 9.4.26. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22729) Provide a failure reason for failed compactions
[
https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter updated HIVE-22729:
-
Attachment: HIVE-22729.13.patch
> Provide a failure reason for failed compactions
> ---
>
> Key: HIVE-22729
> URL: https://issues.apache.org/jira/browse/HIVE-22729
> Project: Hive
> Issue Type: Improvement
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-22729.01.patch, HIVE-22729.02.patch,
> HIVE-22729.03.patch, HIVE-22729.04.patch, HIVE-22729.05.patch,
> HIVE-22729.06.patch, HIVE-22729.07.patch, HIVE-22729.08.patch,
> HIVE-22729.09.patch, HIVE-22729.10.patch, HIVE-22729.11.patch,
> HIVE-22729.12.patch, HIVE-22729.13.patch
>
>
> We should provide a compaction failure reason as easily accessible as
> possible. Like in the result of the {{SHOW COMPACTIONS}} command.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22729) Provide a failure reason for failed compactions
[
https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter updated HIVE-22729:
-
Attachment: HIVE-22729.12.patch
> Provide a failure reason for failed compactions
> ---
>
> Key: HIVE-22729
> URL: https://issues.apache.org/jira/browse/HIVE-22729
> Project: Hive
> Issue Type: Improvement
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-22729.01.patch, HIVE-22729.02.patch,
> HIVE-22729.03.patch, HIVE-22729.04.patch, HIVE-22729.05.patch,
> HIVE-22729.06.patch, HIVE-22729.07.patch, HIVE-22729.08.patch,
> HIVE-22729.09.patch, HIVE-22729.10.patch, HIVE-22729.11.patch,
> HIVE-22729.12.patch
>
>
> We should provide a compaction failure reason as easily accessible as
> possible. Like in the result of the {{SHOW COMPACTIONS}} command.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22729) Provide a failure reason for failed compactions
[
https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter updated HIVE-22729:
-
Attachment: HIVE-22729.11.patch
> Provide a failure reason for failed compactions
> ---
>
> Key: HIVE-22729
> URL: https://issues.apache.org/jira/browse/HIVE-22729
> Project: Hive
> Issue Type: Improvement
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-22729.01.patch, HIVE-22729.02.patch,
> HIVE-22729.03.patch, HIVE-22729.04.patch, HIVE-22729.05.patch,
> HIVE-22729.06.patch, HIVE-22729.07.patch, HIVE-22729.08.patch,
> HIVE-22729.09.patch, HIVE-22729.10.patch, HIVE-22729.11.patch
>
>
> We should provide a compaction failure reason as easily accessible as
> possible. Like in the result of the {{SHOW COMPACTIONS}} command.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21487) COMPLETED_COMPACTIONS and COMPACTION_QUEUE table missing appropriate indexes
[
https://issues.apache.org/jira/browse/HIVE-21487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026641#comment-17026641
]
Laszlo Pinter commented on HIVE-21487:
--
I created a couple of performance tests using the HMS benchmarking tool, and I
noticed a slight performance improvement on Initiator side with new indexes on
COMPLETED_COMPACTIONS: (cc_database, cc_table, cc_partition).
[~pvary] [~dkuzmenko] [~klcopp] Could you please review? Thanks
> COMPLETED_COMPACTIONS and COMPACTION_QUEUE table missing appropriate indexes
>
>
> Key: HIVE-21487
> URL: https://issues.apache.org/jira/browse/HIVE-21487
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.1
>Reporter: Todd Lipcon
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-21847.01.patch, HIVE-21847.02.patch
>
>
> Looking at a MySQL install where HMS is pointed on Hive 3.1, I see a constant
> stream of queries of the form:
> {code}
> select CC_STATE from COMPLETED_COMPACTIONS where CC_DATABASE =
> 'tpcds_orc_exact_1000' and CC_TABLE = 'catalog_returns' and CC_PARTITION =
> 'cr_returned_date_sk=2452851' and CC_STATE != 'a' order by CC_ID desc;
> {code}
> but the COMPLETED_COMPACTIONS table has no index. In this case it's resulting
> in a full table scan over 115k rows, which takes around 100ms.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22729) Provide a failure reason for failed compactions
[
https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter updated HIVE-22729:
-
Attachment: HIVE-22729.10.patch
> Provide a failure reason for failed compactions
> ---
>
> Key: HIVE-22729
> URL: https://issues.apache.org/jira/browse/HIVE-22729
> Project: Hive
> Issue Type: Improvement
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-22729.01.patch, HIVE-22729.02.patch,
> HIVE-22729.03.patch, HIVE-22729.04.patch, HIVE-22729.05.patch,
> HIVE-22729.06.patch, HIVE-22729.07.patch, HIVE-22729.08.patch,
> HIVE-22729.09.patch, HIVE-22729.10.patch
>
>
> We should provide a compaction failure reason as easily accessible as
> possible. Like in the result of the {{SHOW COMPACTIONS}} command.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22793) Update default settings in HMS Benchmarking tool
[ https://issues.apache.org/jira/browse/HIVE-22793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22793: - Status: Patch Available (was: Open) > Update default settings in HMS Benchmarking tool > > > Key: HIVE-22793 > URL: https://issues.apache.org/jira/browse/HIVE-22793 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Trivial > Attachments: HIVE-22793.01.patch > > > HMS Benchmarking tool has invalid default setting values. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22793) Update default settings in HMS Benchmarking tool
[ https://issues.apache.org/jira/browse/HIVE-22793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22793: - Attachment: HIVE-22793.01.patch > Update default settings in HMS Benchmarking tool > > > Key: HIVE-22793 > URL: https://issues.apache.org/jira/browse/HIVE-22793 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Trivial > Attachments: HIVE-22793.01.patch > > > HMS Benchmarking tool has invalid default setting values. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22793) Update default settings in HMS Benchmarking tool
[ https://issues.apache.org/jira/browse/HIVE-22793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter reassigned HIVE-22793: > Update default settings in HMS Benchmarking tool > > > Key: HIVE-22793 > URL: https://issues.apache.org/jira/browse/HIVE-22793 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Trivial > > HMS Benchmarking tool has invalid default setting values. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22729) Provide a failure reason for failed compactions
[
https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17025736#comment-17025736
]
Laszlo Pinter commented on HIVE-22729:
--
[~pvary] [~dkuzmenko] [~klcopp] Can I get a +1 from you? Thank you
> Provide a failure reason for failed compactions
> ---
>
> Key: HIVE-22729
> URL: https://issues.apache.org/jira/browse/HIVE-22729
> Project: Hive
> Issue Type: Improvement
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-22729.01.patch, HIVE-22729.02.patch,
> HIVE-22729.03.patch, HIVE-22729.04.patch, HIVE-22729.05.patch,
> HIVE-22729.06.patch, HIVE-22729.07.patch, HIVE-22729.08.patch,
> HIVE-22729.09.patch
>
>
> We should provide a compaction failure reason as easily accessible as
> possible. Like in the result of the {{SHOW COMPACTIONS}} command.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22729) Provide a failure reason for failed compactions
[
https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter updated HIVE-22729:
-
Attachment: HIVE-22729.09.patch
> Provide a failure reason for failed compactions
> ---
>
> Key: HIVE-22729
> URL: https://issues.apache.org/jira/browse/HIVE-22729
> Project: Hive
> Issue Type: Improvement
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-22729.01.patch, HIVE-22729.02.patch,
> HIVE-22729.03.patch, HIVE-22729.04.patch, HIVE-22729.05.patch,
> HIVE-22729.06.patch, HIVE-22729.07.patch, HIVE-22729.08.patch,
> HIVE-22729.09.patch
>
>
> We should provide a compaction failure reason as easily accessible as
> possible. Like in the result of the {{SHOW COMPACTIONS}} command.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22729) Provide a failure reason for failed compactions
[
https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter updated HIVE-22729:
-
Attachment: HIVE-22729.08.patch
> Provide a failure reason for failed compactions
> ---
>
> Key: HIVE-22729
> URL: https://issues.apache.org/jira/browse/HIVE-22729
> Project: Hive
> Issue Type: Improvement
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-22729.01.patch, HIVE-22729.02.patch,
> HIVE-22729.03.patch, HIVE-22729.04.patch, HIVE-22729.05.patch,
> HIVE-22729.06.patch, HIVE-22729.07.patch, HIVE-22729.08.patch
>
>
> We should provide a compaction failure reason as easily accessible as
> possible. Like in the result of the {{SHOW COMPACTIONS}} command.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-22765) Quote table names in CompactionTxnHandler
[
https://issues.apache.org/jira/browse/HIVE-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter reassigned HIVE-22765:
> Quote table names in CompactionTxnHandler
> -
>
> Key: HIVE-22765
> URL: https://issues.apache.org/jira/browse/HIVE-22765
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0
>Reporter: Laszlo Pinter
>Assignee: Zoltan Chovan
>Priority: Major
>
> When I run a metastore backed up by a postgres db, I get the following
> exception:
> {code:java}
> MetaException(message:Unable to connect to transaction database
> org.postgresql.util.PSQLException: ERROR: relation "compaction_queue" does
> not existMetaException(message:Unable to connect to transaction database
> org.postgresql.util.PSQLException: ERROR: relation "compaction_queue" does
> not exist Position: 83 at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2468)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2211)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:309)
> at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:446) at
> org.postgresql.jdbc.PgStatement.execute(PgStatement.java:370) at
> org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:311) at
> org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:297) at
> org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:274) at
> org.postgresql.jdbc.PgStatement.executeQuery(PgStatement.java:225) at
> org.apache.hive.com.zaxxer.hikari.pool.ProxyStatement.executeQuery(ProxyStatement.java:108)
> at
> org.apache.hive.com.zaxxer.hikari.pool.HikariProxyStatement.executeQuery(HikariProxyStatement.java)
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findNextToCompact(CompactionTxnHandler.java:157)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.find_next_compact(HiveMetaStore.java:8061)
> at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy28.find_next_compact(Unknown Source) at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.findNextCompact(HiveMetaStoreClient.java:4046)
> at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:208)
> at com.sun.proxy.$Proxy29.findNextCompact(Unknown Source) at
> org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:102){code}
> This is due to missing quotes around table names.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21487) COMPLETED_COMPACTIONS and COMPACTION_QUEUE table missing appropriate indexes
[
https://issues.apache.org/jira/browse/HIVE-21487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter updated HIVE-21487:
-
Attachment: HIVE-21847.02.patch
> COMPLETED_COMPACTIONS and COMPACTION_QUEUE table missing appropriate indexes
>
>
> Key: HIVE-21487
> URL: https://issues.apache.org/jira/browse/HIVE-21487
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.1
>Reporter: Todd Lipcon
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-21847.01.patch, HIVE-21847.02.patch
>
>
> Looking at a MySQL install where HMS is pointed on Hive 3.1, I see a constant
> stream of queries of the form:
> {code}
> select CC_STATE from COMPLETED_COMPACTIONS where CC_DATABASE =
> 'tpcds_orc_exact_1000' and CC_TABLE = 'catalog_returns' and CC_PARTITION =
> 'cr_returned_date_sk=2452851' and CC_STATE != 'a' order by CC_ID desc;
> {code}
> but the COMPLETED_COMPACTIONS table has no index. In this case it's resulting
> in a full table scan over 115k rows, which takes around 100ms.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22764) Create new command for "optimize" compaction and have basic implementation.
[ https://issues.apache.org/jira/browse/HIVE-22764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021905#comment-17021905 ] Laszlo Pinter commented on HIVE-22764: -- [~aditya-shah] Please provide a review board/pull request link to make the review easier. > Create new command for "optimize" compaction and have basic implementation. > --- > > Key: HIVE-22764 > URL: https://issues.apache.org/jira/browse/HIVE-22764 > Project: Hive > Issue Type: Sub-task >Reporter: Aditya Shah >Assignee: Aditya Shah >Priority: Major > Attachments: HIVE-22764.patch > > > Created new blocking compaction (added compaction type "optimize") by adding > a lock request on the compaction's transaction. It works mostly like > mmMajorCompaction and writes files w/o row_IDs. I have added an additional > table property to provide optimize columns that is used by the compactor to > cluster the data by. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22729) Provide a failure reason for failed compactions
[
https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter updated HIVE-22729:
-
Attachment: HIVE-22729.07.patch
> Provide a failure reason for failed compactions
> ---
>
> Key: HIVE-22729
> URL: https://issues.apache.org/jira/browse/HIVE-22729
> Project: Hive
> Issue Type: Improvement
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-22729.01.patch, HIVE-22729.02.patch,
> HIVE-22729.03.patch, HIVE-22729.04.patch, HIVE-22729.05.patch,
> HIVE-22729.06.patch, HIVE-22729.07.patch
>
>
> We should provide a compaction failure reason as easily accessible as
> possible. Like in the result of the {{SHOW COMPACTIONS}} command.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Reopened] (HIVE-22126) hive-exec packaging should shade guava
[ https://issues.apache.org/jira/browse/HIVE-22126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter reopened HIVE-22126: -- Assignee: Eugene Chung (was: Laszlo Pinter) > hive-exec packaging should shade guava > -- > > Key: HIVE-22126 > URL: https://issues.apache.org/jira/browse/HIVE-22126 > Project: Hive > Issue Type: Bug >Reporter: Vihang Karajgaonkar >Assignee: Eugene Chung >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22126.01.patch > > > The ql/pom.xml includes complete guava library into hive-exec.jar > https://github.com/apache/hive/blob/master/ql/pom.xml#L990 This causes a > problems for downstream clients of hive which have hive-exec.jar in their > classpath since they are pinned to the same guava version as that of hive. > We should shade guava classes so that other components which depend on > hive-exec can independently use a different version of guava as needed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22126) hive-exec packaging should shade guava
[ https://issues.apache.org/jira/browse/HIVE-22126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021223#comment-17021223 ] Laszlo Pinter commented on HIVE-22126: -- [~euigeun_chung] feel free to reopen and take over the jira. > hive-exec packaging should shade guava > -- > > Key: HIVE-22126 > URL: https://issues.apache.org/jira/browse/HIVE-22126 > Project: Hive > Issue Type: Bug >Reporter: Vihang Karajgaonkar >Assignee: Laszlo Pinter >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22126.01.patch > > > The ql/pom.xml includes complete guava library into hive-exec.jar > https://github.com/apache/hive/blob/master/ql/pom.xml#L990 This causes a > problems for downstream clients of hive which have hive-exec.jar in their > classpath since they are pinned to the same guava version as that of hive. > We should shade guava classes so that other components which depend on > hive-exec can independently use a different version of guava as needed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22729) Provide a failure reason for failed compactions
[
https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter updated HIVE-22729:
-
Attachment: HIVE-22729.06.patch
> Provide a failure reason for failed compactions
> ---
>
> Key: HIVE-22729
> URL: https://issues.apache.org/jira/browse/HIVE-22729
> Project: Hive
> Issue Type: Improvement
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-22729.01.patch, HIVE-22729.02.patch,
> HIVE-22729.03.patch, HIVE-22729.04.patch, HIVE-22729.05.patch,
> HIVE-22729.06.patch
>
>
> We should provide a compaction failure reason as easily accessible as
> possible. Like in the result of the {{SHOW COMPACTIONS}} command.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22255) Hive don't trigger Major Compaction automatically if table contains only base files
[
https://issues.apache.org/jira/browse/HIVE-22255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020240#comment-17020240
]
Laszlo Pinter commented on HIVE-22255:
--
[~Rajkumar Singh] Could you please provide a review board/pull request where I
can post my remarks? Thanks.
> Hive don't trigger Major Compaction automatically if table contains only base
> files
>
>
> Key: HIVE-22255
> URL: https://issues.apache.org/jira/browse/HIVE-22255
> Project: Hive
> Issue Type: Bug
> Components: Hive, Transactions
>Affects Versions: 3.1.2
> Environment: Hive-3.1.1
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
> Attachments: HIVE-22255.01.patch, HIVE-22255.patch
>
>
> user may run into the issue if the table consists of all base files but no
> delta, then the following condition will yield false and automatic major
> compaction will be skipped.
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java#L313]
>
> Steps to Reproduce:
> # create Acid table
> {code:java}
> // create table myacid(id int);
> {code}
> # Run multiple insert table
> {code:java}
> // insert overwrite table myacid values(1);insert overwrite table myacid
> values(2),(3),(4){code}
> # DFS ls output
> {code:java}
> // dfs -ls -R /warehouse/tablespace/managed/hive/myacid;
> ++
> | DFS Output |
> ++
> | drwxrwx---+ - hive hadoop 0 2019-09-27 16:42
> /warehouse/tablespace/managed/hive/myacid/base_001 |
> | -rw-rw+ 3 hive hadoop 1 2019-09-27 16:42
> /warehouse/tablespace/managed/hive/myacid/base_001/_orc_acid_version |
> | -rw-rw+ 3 hive hadoop 610 2019-09-27 16:42
> /warehouse/tablespace/managed/hive/myacid/base_001/bucket_0 |
> | drwxrwx---+ - hive hadoop 0 2019-09-27 16:43
> /warehouse/tablespace/managed/hive/myacid/base_002 |
> | -rw-rw+ 3 hive hadoop 1 2019-09-27 16:43
> /warehouse/tablespace/managed/hive/myacid/base_002/_orc_acid_version |
> | -rw-rw+ 3 hive hadoop 633 2019-09-27 16:43
> /warehouse/tablespace/managed/hive/myacid/base_002/bucket_0 |
> ++{code}
>
> you will see that Major compaction will not be trigger until you run alter
> table compact MAJOR.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22729) Provide a failure reason for failed compactions
[
https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter updated HIVE-22729:
-
Attachment: HIVE-22729.05.patch
> Provide a failure reason for failed compactions
> ---
>
> Key: HIVE-22729
> URL: https://issues.apache.org/jira/browse/HIVE-22729
> Project: Hive
> Issue Type: Improvement
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-22729.01.patch, HIVE-22729.02.patch,
> HIVE-22729.03.patch, HIVE-22729.04.patch, HIVE-22729.05.patch
>
>
> We should provide a compaction failure reason as easily accessible as
> possible. Like in the result of the {{SHOW COMPACTIONS}} command.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21487) COMPLETED_COMPACTIONS and COMPACTION_QUEUE table missing appropriate indexes
[
https://issues.apache.org/jira/browse/HIVE-21487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter updated HIVE-21487:
-
Status: Patch Available (was: Open)
> COMPLETED_COMPACTIONS and COMPACTION_QUEUE table missing appropriate indexes
>
>
> Key: HIVE-21487
> URL: https://issues.apache.org/jira/browse/HIVE-21487
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.1
>Reporter: Todd Lipcon
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-21847.01.patch
>
>
> Looking at a MySQL install where HMS is pointed on Hive 3.1, I see a constant
> stream of queries of the form:
> {code}
> select CC_STATE from COMPLETED_COMPACTIONS where CC_DATABASE =
> 'tpcds_orc_exact_1000' and CC_TABLE = 'catalog_returns' and CC_PARTITION =
> 'cr_returned_date_sk=2452851' and CC_STATE != 'a' order by CC_ID desc;
> {code}
> but the COMPLETED_COMPACTIONS table has no index. In this case it's resulting
> in a full table scan over 115k rows, which takes around 100ms.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21487) COMPLETED_COMPACTIONS and COMPACTION_QUEUE table missing appropriate indexes
[
https://issues.apache.org/jira/browse/HIVE-21487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter updated HIVE-21487:
-
Attachment: HIVE-21847.01.patch
> COMPLETED_COMPACTIONS and COMPACTION_QUEUE table missing appropriate indexes
>
>
> Key: HIVE-21487
> URL: https://issues.apache.org/jira/browse/HIVE-21487
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.1
>Reporter: Todd Lipcon
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-21847.01.patch
>
>
> Looking at a MySQL install where HMS is pointed on Hive 3.1, I see a constant
> stream of queries of the form:
> {code}
> select CC_STATE from COMPLETED_COMPACTIONS where CC_DATABASE =
> 'tpcds_orc_exact_1000' and CC_TABLE = 'catalog_returns' and CC_PARTITION =
> 'cr_returned_date_sk=2452851' and CC_STATE != 'a' order by CC_ID desc;
> {code}
> but the COMPLETED_COMPACTIONS table has no index. In this case it's resulting
> in a full table scan over 115k rows, which takes around 100ms.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22729) Provide a failure reason for failed compactions
[
https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter updated HIVE-22729:
-
Attachment: HIVE-22729.04.patch
> Provide a failure reason for failed compactions
> ---
>
> Key: HIVE-22729
> URL: https://issues.apache.org/jira/browse/HIVE-22729
> Project: Hive
> Issue Type: Improvement
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-22729.01.patch, HIVE-22729.02.patch,
> HIVE-22729.03.patch, HIVE-22729.04.patch
>
>
> We should provide a compaction failure reason as easily accessible as
> possible. Like in the result of the {{SHOW COMPACTIONS}} command.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22729) Provide a failure reason for failed compactions
[
https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter updated HIVE-22729:
-
Attachment: HIVE-22729.03.patch
> Provide a failure reason for failed compactions
> ---
>
> Key: HIVE-22729
> URL: https://issues.apache.org/jira/browse/HIVE-22729
> Project: Hive
> Issue Type: Improvement
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-22729.01.patch, HIVE-22729.02.patch,
> HIVE-22729.03.patch
>
>
> We should provide a compaction failure reason as easily accessible as
> possible. Like in the result of the {{SHOW COMPACTIONS}} command.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22729) Provide a failure reason for failed compactions
[
https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter updated HIVE-22729:
-
Attachment: (was: HIVE-22729.03.patch)
> Provide a failure reason for failed compactions
> ---
>
> Key: HIVE-22729
> URL: https://issues.apache.org/jira/browse/HIVE-22729
> Project: Hive
> Issue Type: Improvement
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-22729.01.patch, HIVE-22729.02.patch
>
>
> We should provide a compaction failure reason as easily accessible as
> possible. Like in the result of the {{SHOW COMPACTIONS}} command.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22729) Provide a failure reason for failed compactions
[
https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter updated HIVE-22729:
-
Attachment: HIVE-22729.03.patch
> Provide a failure reason for failed compactions
> ---
>
> Key: HIVE-22729
> URL: https://issues.apache.org/jira/browse/HIVE-22729
> Project: Hive
> Issue Type: Improvement
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-22729.01.patch, HIVE-22729.02.patch,
> HIVE-22729.03.patch
>
>
> We should provide a compaction failure reason as easily accessible as
> possible. Like in the result of the {{SHOW COMPACTIONS}} command.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22729) Provide a failure reason for failed compactions
[
https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter updated HIVE-22729:
-
Attachment: HIVE-22729.02.patch
> Provide a failure reason for failed compactions
> ---
>
> Key: HIVE-22729
> URL: https://issues.apache.org/jira/browse/HIVE-22729
> Project: Hive
> Issue Type: Improvement
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-22729.01.patch, HIVE-22729.02.patch
>
>
> We should provide a compaction failure reason as easily accessible as
> possible. Like in the result of the {{SHOW COMPACTIONS}} command.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21487) COMPLETED_COMPACTIONS and COMPACTION_QUEUE table missing appropriate indexes
[
https://issues.apache.org/jira/browse/HIVE-21487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019560#comment-17019560
]
Laszlo Pinter commented on HIVE-21487:
--
We should add the following indexes to compaction related tables:
COMPACTION_QUEUE: cq_id and (cq_state, cq_id)
COMPLETED_COMPACTIONS: (cc_database, cc_table, cc_partition)
> COMPLETED_COMPACTIONS and COMPACTION_QUEUE table missing appropriate indexes
>
>
> Key: HIVE-21487
> URL: https://issues.apache.org/jira/browse/HIVE-21487
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.1
>Reporter: Todd Lipcon
>Assignee: Laszlo Pinter
>Priority: Major
>
> Looking at a MySQL install where HMS is pointed on Hive 3.1, I see a constant
> stream of queries of the form:
> {code}
> select CC_STATE from COMPLETED_COMPACTIONS where CC_DATABASE =
> 'tpcds_orc_exact_1000' and CC_TABLE = 'catalog_returns' and CC_PARTITION =
> 'cr_returned_date_sk=2452851' and CC_STATE != 'a' order by CC_ID desc;
> {code}
> but the COMPLETED_COMPACTIONS table has no index. In this case it's resulting
> in a full table scan over 115k rows, which takes around 100ms.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-21487) COMPLETED_COMPACTIONS table missing appropriate indexes
[
https://issues.apache.org/jira/browse/HIVE-21487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter reassigned HIVE-21487:
Assignee: Laszlo Pinter
> COMPLETED_COMPACTIONS table missing appropriate indexes
> ---
>
> Key: HIVE-21487
> URL: https://issues.apache.org/jira/browse/HIVE-21487
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.1
>Reporter: Todd Lipcon
>Assignee: Laszlo Pinter
>Priority: Major
>
> Looking at a MySQL install where HMS is pointed on Hive 3.1, I see a constant
> stream of queries of the form:
> {code}
> select CC_STATE from COMPLETED_COMPACTIONS where CC_DATABASE =
> 'tpcds_orc_exact_1000' and CC_TABLE = 'catalog_returns' and CC_PARTITION =
> 'cr_returned_date_sk=2452851' and CC_STATE != 'a' order by CC_ID desc;
> {code}
> but the COMPLETED_COMPACTIONS table has no index. In this case it's resulting
> in a full table scan over 115k rows, which takes around 100ms.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21487) COMPLETED_COMPACTIONS and COMPACTION_QUEUE table missing appropriate indexes
[
https://issues.apache.org/jira/browse/HIVE-21487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter updated HIVE-21487:
-
Summary: COMPLETED_COMPACTIONS and COMPACTION_QUEUE table missing
appropriate indexes (was: COMPLETED_COMPACTIONS table missing appropriate
indexes)
> COMPLETED_COMPACTIONS and COMPACTION_QUEUE table missing appropriate indexes
>
>
> Key: HIVE-21487
> URL: https://issues.apache.org/jira/browse/HIVE-21487
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.1
>Reporter: Todd Lipcon
>Assignee: Laszlo Pinter
>Priority: Major
>
> Looking at a MySQL install where HMS is pointed on Hive 3.1, I see a constant
> stream of queries of the form:
> {code}
> select CC_STATE from COMPLETED_COMPACTIONS where CC_DATABASE =
> 'tpcds_orc_exact_1000' and CC_TABLE = 'catalog_returns' and CC_PARTITION =
> 'cr_returned_date_sk=2452851' and CC_STATE != 'a' order by CC_ID desc;
> {code}
> but the COMPLETED_COMPACTIONS table has no index. In this case it's resulting
> in a full table scan over 115k rows, which takes around 100ms.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21931) Slow compaction for tiny tables
[
https://issues.apache.org/jira/browse/HIVE-21931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019534#comment-17019534
]
Laszlo Pinter commented on HIVE-21931:
--
[~csringhofer] It's a session property.
> Slow compaction for tiny tables
> ---
>
> Key: HIVE-21931
> URL: https://issues.apache.org/jira/browse/HIVE-21931
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Csaba Ringhofer
>Priority: Major
> Labels: compaction
>
> I observed the issue in Impala development environment when (major)
> compacting insert_only transactional tables in Hive. The compaction could
> take ~10 minutes even when it only had to merge 2 rows from 2 inserts. The
> actual work was done much earlier, the new base file was correctly written to
> HDFS, and Hive seemed to wait without doing any work.
> The compactions are started manually, hive.compactor.initiator.on=false to
> avoid "surprise compaction" during tests.
> {code}
> hive.compactor.abortedtxn.threshold=1000
> hive.compactor.check.interval=300s
> hive.compactor.cleaner.run.interval=5000ms
> hive.compactor.compact.insert.only=true
> hive.compactor.crud.query.based=false
> hive.compactor.delta.num.threshold=10
> hive.compactor.delta.pct.threshold=0.1
> hive.compactor.history.reaper.interval=2m
> hive.compactor.history.retention.attempted=2
> hive.compactor.history.retention.failed=3
> hive.compactor.history.retention.succeeded=3
> hive.compactor.initiator.failed.compacts.threshold=2
> hive.compactor.initiator.on=false
> hive.compactor.max.num.delta=500
> hive.compactor.worker.threads=4
> hive.compactor.worker.timeout=86400s
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21931) Slow compaction for tiny tables
[
https://issues.apache.org/jira/browse/HIVE-21931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019501#comment-17019501
]
Laszlo Pinter commented on HIVE-21931:
--
[~csringhofer] With HIVE-22554 you can configure the initial wait time out to a
much lower value, like 2000 millisec. Does that satisfies your needs?
> Slow compaction for tiny tables
> ---
>
> Key: HIVE-21931
> URL: https://issues.apache.org/jira/browse/HIVE-21931
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Csaba Ringhofer
>Priority: Major
> Labels: compaction
>
> I observed the issue in Impala development environment when (major)
> compacting insert_only transactional tables in Hive. The compaction could
> take ~10 minutes even when it only had to merge 2 rows from 2 inserts. The
> actual work was done much earlier, the new base file was correctly written to
> HDFS, and Hive seemed to wait without doing any work.
> The compactions are started manually, hive.compactor.initiator.on=false to
> avoid "surprise compaction" during tests.
> {code}
> hive.compactor.abortedtxn.threshold=1000
> hive.compactor.check.interval=300s
> hive.compactor.cleaner.run.interval=5000ms
> hive.compactor.compact.insert.only=true
> hive.compactor.crud.query.based=false
> hive.compactor.delta.num.threshold=10
> hive.compactor.delta.pct.threshold=0.1
> hive.compactor.history.reaper.interval=2m
> hive.compactor.history.retention.attempted=2
> hive.compactor.history.retention.failed=3
> hive.compactor.history.retention.succeeded=3
> hive.compactor.initiator.failed.compacts.threshold=2
> hive.compactor.initiator.on=false
> hive.compactor.max.num.delta=500
> hive.compactor.worker.threads=4
> hive.compactor.worker.timeout=86400s
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21931) Slow compaction for tiny tables
[
https://issues.apache.org/jira/browse/HIVE-21931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019458#comment-17019458
]
Laszlo Pinter commented on HIVE-21931:
--
[~csringhofer] Did you run compaction in blocking mode?
HIVE-22554 provides a way to configure the wait time out.
> Slow compaction for tiny tables
> ---
>
> Key: HIVE-21931
> URL: https://issues.apache.org/jira/browse/HIVE-21931
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Csaba Ringhofer
>Priority: Major
> Labels: compaction
>
> I observed the issue in Impala development environment when (major)
> compacting insert_only transactional tables in Hive. The compaction could
> take ~10 minutes even when it only had to merge 2 rows from 2 inserts. The
> actual work was done much earlier, the new base file was correctly written to
> HDFS, and Hive seemed to wait without doing any work.
> The compactions are started manually, hive.compactor.initiator.on=false to
> avoid "surprise compaction" during tests.
> {code}
> hive.compactor.abortedtxn.threshold=1000
> hive.compactor.check.interval=300s
> hive.compactor.cleaner.run.interval=5000ms
> hive.compactor.compact.insert.only=true
> hive.compactor.crud.query.based=false
> hive.compactor.delta.num.threshold=10
> hive.compactor.delta.pct.threshold=0.1
> hive.compactor.history.reaper.interval=2m
> hive.compactor.history.retention.attempted=2
> hive.compactor.history.retention.failed=3
> hive.compactor.history.retention.succeeded=3
> hive.compactor.initiator.failed.compacts.threshold=2
> hive.compactor.initiator.on=false
> hive.compactor.max.num.delta=500
> hive.compactor.worker.threads=4
> hive.compactor.worker.timeout=86400s
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22729) Provide a failure reason for failed compactions
[
https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter updated HIVE-22729:
-
Status: Patch Available (was: Open)
> Provide a failure reason for failed compactions
> ---
>
> Key: HIVE-22729
> URL: https://issues.apache.org/jira/browse/HIVE-22729
> Project: Hive
> Issue Type: Improvement
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-22729.01.patch
>
>
> We should provide a compaction failure reason as easily accessible as
> possible. Like in the result of the {{SHOW COMPACTIONS}} command.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22729) Provide a failure reason for failed compactions
[
https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter updated HIVE-22729:
-
Attachment: HIVE-22729.01.patch
> Provide a failure reason for failed compactions
> ---
>
> Key: HIVE-22729
> URL: https://issues.apache.org/jira/browse/HIVE-22729
> Project: Hive
> Issue Type: Improvement
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-22729.01.patch
>
>
> We should provide a compaction failure reason as easily accessible as
> possible. Like in the result of the {{SHOW COMPACTIONS}} command.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22703) Compaction configuration check when starting HMS/HS2
[ https://issues.apache.org/jira/browse/HIVE-22703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019340#comment-17019340 ] Laszlo Pinter commented on HIVE-22703: -- Thanks [~pvary] and [~dkuzmenko] > Compaction configuration check when starting HMS/HS2 > > > Key: HIVE-22703 > URL: https://issues.apache.org/jira/browse/HIVE-22703 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Minor > Fix For: 4.0.0 > > Attachments: HIVE-22703.01.patch, HIVE-22703.02.patch > > > Currently when starting HMS we can have bugous configuration which prevents > compatction to work. We should find a way to inform the admin about the > configuration error, or even prevent HMS to start in this case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22705) LLAP cache is polluted by query-based compactor
[ https://issues.apache.org/jira/browse/HIVE-22705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017764#comment-17017764 ] Laszlo Pinter commented on HIVE-22705: -- +1 (non-binding and pending tests) > LLAP cache is polluted by query-based compactor > --- > > Key: HIVE-22705 > URL: https://issues.apache.org/jira/browse/HIVE-22705 > Project: Hive > Issue Type: Improvement >Reporter: Ádám Szita >Assignee: Ádám Szita >Priority: Major > Attachments: HIVE-22705.0.patch > > > One of the steps that query-based compaction does is the verification of ACID > sort order by using the _validate_acid_sort_order_ UDF. This is a > prerequisite before the actual compaction can happen, and is done by a [query > that reads the whole table > content|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MajorQueryCompactor.java#L161-L167]. > This results in the whole table content being populated into the cache. The > problem is that this content is not useful and will rather pollute the cache > space, as it can never be used again: cache content binds to files (file IDs) > that obviously will be changed in this case by compaction. > I propose we disable LLAP caching in the session of query-based compaction's > queries. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22729) Provide a failure reason for failed compactions
[
https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter reassigned HIVE-22729:
> Provide a failure reason for failed compactions
> ---
>
> Key: HIVE-22729
> URL: https://issues.apache.org/jira/browse/HIVE-22729
> Project: Hive
> Issue Type: Improvement
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
>
> We should provide a compaction failure reason as easily accessible as
> possible. Like in the result of the {{SHOW COMPACTIONS}} command.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22703) Compaction configuration check when starting HMS/HS2
[ https://issues.apache.org/jira/browse/HIVE-22703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22703: - Attachment: HIVE-22703.02.patch > Compaction configuration check when starting HMS/HS2 > > > Key: HIVE-22703 > URL: https://issues.apache.org/jira/browse/HIVE-22703 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Minor > Attachments: HIVE-22703.01.patch, HIVE-22703.02.patch > > > Currently when starting HMS we can have bugous configuration which prevents > compatction to work. We should find a way to inform the admin about the > configuration error, or even prevent HMS to start in this case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22703) Compaction configuration check when starting HMS/HS2
[ https://issues.apache.org/jira/browse/HIVE-22703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22703: - Status: Patch Available (was: Open) > Compaction configuration check when starting HMS/HS2 > > > Key: HIVE-22703 > URL: https://issues.apache.org/jira/browse/HIVE-22703 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-22703.01.patch > > > Currently when starting HMS we can have bugous configuration which prevents > compatction to work. We should find a way to inform the admin about the > configuration error, or even prevent HMS to start in this case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22703) Compaction configuration check when starting HMS/HS2
[ https://issues.apache.org/jira/browse/HIVE-22703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22703: - Attachment: HIVE-22703.01.patch > Compaction configuration check when starting HMS/HS2 > > > Key: HIVE-22703 > URL: https://issues.apache.org/jira/browse/HIVE-22703 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-22703.01.patch > > > Currently when starting HMS we can have bugous configuration which prevents > compatction to work. We should find a way to inform the admin about the > configuration error, or even prevent HMS to start in this case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22703) Compaction configuration check when starting HMS/HS2
[ https://issues.apache.org/jira/browse/HIVE-22703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22703: - Priority: Minor (was: Major) > Compaction configuration check when starting HMS/HS2 > > > Key: HIVE-22703 > URL: https://issues.apache.org/jira/browse/HIVE-22703 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Minor > Attachments: HIVE-22703.01.patch > > > Currently when starting HMS we can have bugous configuration which prevents > compatction to work. We should find a way to inform the admin about the > configuration error, or even prevent HMS to start in this case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-20934) ACID: Query based compactor for minor compaction
[ https://issues.apache.org/jira/browse/HIVE-20934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-20934: - Attachment: HIVE-20934.14.patch > ACID: Query based compactor for minor compaction > > > Key: HIVE-20934 > URL: https://issues.apache.org/jira/browse/HIVE-20934 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Vaibhav Gumashta >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-20934.01.patch, HIVE-20934.02.patch, > HIVE-20934.03.patch, HIVE-20934.04.patch, HIVE-20934.05.patch, > HIVE-20934.06.patch, HIVE-20934.07.patch, HIVE-20934.08.patch, > HIVE-20934.09.patch, HIVE-20934.10.patch, HIVE-20934.11.patch, > HIVE-20934.12.patch, HIVE-20934.13.patch, HIVE-20934.14.patch > > > Follow up of HIVE-20699. This is to enable running minor compactions as a > HiveQL query -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-20934) ACID: Query based compactor for minor compaction
[ https://issues.apache.org/jira/browse/HIVE-20934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-20934: - Attachment: HIVE-20934.13.patch > ACID: Query based compactor for minor compaction > > > Key: HIVE-20934 > URL: https://issues.apache.org/jira/browse/HIVE-20934 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Vaibhav Gumashta >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-20934.01.patch, HIVE-20934.02.patch, > HIVE-20934.03.patch, HIVE-20934.04.patch, HIVE-20934.05.patch, > HIVE-20934.06.patch, HIVE-20934.07.patch, HIVE-20934.08.patch, > HIVE-20934.09.patch, HIVE-20934.10.patch, HIVE-20934.11.patch, > HIVE-20934.12.patch, HIVE-20934.13.patch > > > Follow up of HIVE-20699. This is to enable running minor compactions as a > HiveQL query -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-20934) ACID: Query based compactor for minor compaction
[ https://issues.apache.org/jira/browse/HIVE-20934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-20934: - Attachment: HIVE-20934.12.patch > ACID: Query based compactor for minor compaction > > > Key: HIVE-20934 > URL: https://issues.apache.org/jira/browse/HIVE-20934 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Vaibhav Gumashta >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-20934.01.patch, HIVE-20934.02.patch, > HIVE-20934.03.patch, HIVE-20934.04.patch, HIVE-20934.05.patch, > HIVE-20934.06.patch, HIVE-20934.07.patch, HIVE-20934.08.patch, > HIVE-20934.09.patch, HIVE-20934.10.patch, HIVE-20934.11.patch, > HIVE-20934.12.patch > > > Follow up of HIVE-20699. This is to enable running minor compactions as a > HiveQL query -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-20934) ACID: Query based compactor for minor compaction
[ https://issues.apache.org/jira/browse/HIVE-20934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-20934: - Attachment: HIVE-20934.11.patch > ACID: Query based compactor for minor compaction > > > Key: HIVE-20934 > URL: https://issues.apache.org/jira/browse/HIVE-20934 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Vaibhav Gumashta >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-20934.01.patch, HIVE-20934.02.patch, > HIVE-20934.03.patch, HIVE-20934.04.patch, HIVE-20934.05.patch, > HIVE-20934.06.patch, HIVE-20934.07.patch, HIVE-20934.08.patch, > HIVE-20934.09.patch, HIVE-20934.10.patch, HIVE-20934.11.patch > > > Follow up of HIVE-20699. This is to enable running minor compactions as a > HiveQL query -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-20934) ACID: Query based compactor for minor compaction
[ https://issues.apache.org/jira/browse/HIVE-20934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-20934: - Attachment: HIVE-20934.10.patch > ACID: Query based compactor for minor compaction > > > Key: HIVE-20934 > URL: https://issues.apache.org/jira/browse/HIVE-20934 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Vaibhav Gumashta >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-20934.01.patch, HIVE-20934.02.patch, > HIVE-20934.03.patch, HIVE-20934.04.patch, HIVE-20934.05.patch, > HIVE-20934.06.patch, HIVE-20934.07.patch, HIVE-20934.08.patch, > HIVE-20934.09.patch, HIVE-20934.10.patch > > > Follow up of HIVE-20699. This is to enable running minor compactions as a > HiveQL query -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-20934) ACID: Query based compactor for minor compaction
[ https://issues.apache.org/jira/browse/HIVE-20934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-20934: - Attachment: HIVE-20934.09.patch > ACID: Query based compactor for minor compaction > > > Key: HIVE-20934 > URL: https://issues.apache.org/jira/browse/HIVE-20934 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Vaibhav Gumashta >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-20934.01.patch, HIVE-20934.02.patch, > HIVE-20934.03.patch, HIVE-20934.04.patch, HIVE-20934.05.patch, > HIVE-20934.06.patch, HIVE-20934.07.patch, HIVE-20934.08.patch, > HIVE-20934.09.patch > > > Follow up of HIVE-20699. This is to enable running minor compactions as a > HiveQL query -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22703) Compaction configuration check when starting HMS/HS2
[ https://issues.apache.org/jira/browse/HIVE-22703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter reassigned HIVE-22703: > Compaction configuration check when starting HMS/HS2 > > > Key: HIVE-22703 > URL: https://issues.apache.org/jira/browse/HIVE-22703 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > > Currently when starting HMS we can have bugous configuration which prevents > compatction to work. We should find a way to inform the admin about the > configuration error, or even prevent HMS to start in this case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-20934) ACID: Query based compactor for minor compaction
[ https://issues.apache.org/jira/browse/HIVE-20934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-20934: - Attachment: HIVE-20934.08.patch > ACID: Query based compactor for minor compaction > > > Key: HIVE-20934 > URL: https://issues.apache.org/jira/browse/HIVE-20934 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Vaibhav Gumashta >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-20934.01.patch, HIVE-20934.02.patch, > HIVE-20934.03.patch, HIVE-20934.04.patch, HIVE-20934.05.patch, > HIVE-20934.06.patch, HIVE-20934.07.patch, HIVE-20934.08.patch > > > Follow up of HIVE-20699. This is to enable running minor compactions as a > HiveQL query -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-20934) ACID: Query based compactor for minor compaction
[ https://issues.apache.org/jira/browse/HIVE-20934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-20934: - Attachment: HIVE-20934.07.patch > ACID: Query based compactor for minor compaction > > > Key: HIVE-20934 > URL: https://issues.apache.org/jira/browse/HIVE-20934 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Vaibhav Gumashta >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-20934.01.patch, HIVE-20934.02.patch, > HIVE-20934.03.patch, HIVE-20934.04.patch, HIVE-20934.05.patch, > HIVE-20934.06.patch, HIVE-20934.07.patch > > > Follow up of HIVE-20699. This is to enable running minor compactions as a > HiveQL query -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-20934) ACID: Query based compactor for minor compaction
[ https://issues.apache.org/jira/browse/HIVE-20934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-20934: - Attachment: HIVE-20934.06.patch > ACID: Query based compactor for minor compaction > > > Key: HIVE-20934 > URL: https://issues.apache.org/jira/browse/HIVE-20934 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Vaibhav Gumashta >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-20934.01.patch, HIVE-20934.02.patch, > HIVE-20934.03.patch, HIVE-20934.04.patch, HIVE-20934.05.patch, > HIVE-20934.06.patch > > > Follow up of HIVE-20699. This is to enable running minor compactions as a > HiveQL query -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-20934) ACID: Query based compactor for minor compaction
[ https://issues.apache.org/jira/browse/HIVE-20934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-20934: - Attachment: HIVE-20934.05.patch > ACID: Query based compactor for minor compaction > > > Key: HIVE-20934 > URL: https://issues.apache.org/jira/browse/HIVE-20934 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Vaibhav Gumashta >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-20934.01.patch, HIVE-20934.02.patch, > HIVE-20934.03.patch, HIVE-20934.04.patch, HIVE-20934.05.patch > > > Follow up of HIVE-20699. This is to enable running minor compactions as a > HiveQL query -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22700) Compactions may leak memory when unauthorized
[ https://issues.apache.org/jira/browse/HIVE-22700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22700: - Attachment: HIVE-22700.01.patch > Compactions may leak memory when unauthorized > - > > Key: HIVE-22700 > URL: https://issues.apache.org/jira/browse/HIVE-22700 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-22700.01.patch > > > Initiator class determines compaction type periodically. Initiator either > runs as hive user or impersonates the owner of the table. When impersonation > is used, Initiator#checkForCompaction may leak memory. If impersonation > (ugi.doAs()) call fails, FileSystem.closeAllForUGI does not run, therefore > does not clean the file system cache. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22700) Compactions may leak memory when unauthorized
[ https://issues.apache.org/jira/browse/HIVE-22700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22700: - Status: Patch Available (was: Open) > Compactions may leak memory when unauthorized > - > > Key: HIVE-22700 > URL: https://issues.apache.org/jira/browse/HIVE-22700 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-22700.01.patch > > > Initiator class determines compaction type periodically. Initiator either > runs as hive user or impersonates the owner of the table. When impersonation > is used, Initiator#checkForCompaction may leak memory. If impersonation > (ugi.doAs()) call fails, FileSystem.closeAllForUGI does not run, therefore > does not clean the file system cache. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22700) Compactions may leak memory when unauthorized
[ https://issues.apache.org/jira/browse/HIVE-22700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter reassigned HIVE-22700: > Compactions may leak memory when unauthorized > - > > Key: HIVE-22700 > URL: https://issues.apache.org/jira/browse/HIVE-22700 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > > Initiator class determines compaction type periodically. Initiator either > runs as hive user or impersonates the owner of the table. When impersonation > is used, Initiator#checkForCompaction may leak memory. If impersonation > (ugi.doAs()) call fails, FileSystem.closeAllForUGI does not run, therefore > does not clean the file system cache. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22661) Compaction fails on non bucketed table with data loaded inpath
[
https://issues.apache.org/jira/browse/HIVE-22661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009540#comment-17009540
]
Laszlo Pinter commented on HIVE-22661:
--
[~szita] Thanks for the patch. +1 (non-binding)
> Compaction fails on non bucketed table with data loaded inpath
> --
>
> Key: HIVE-22661
> URL: https://issues.apache.org/jira/browse/HIVE-22661
> Project: Hive
> Issue Type: Bug
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
> Attachments: HIVE-22661.0.patch, HIVE-22661.1.patch,
> HIVE-22661.2.patch
>
>
> Compaction cannot handle situations where:
> * data was ingested with {{LOAD DATA INPATH}}
> * this ingest method is run multiple times, and
> ** with different number of files getting created in the delta directories
> Therefore, for file/dir structures such as:
> {code:java}
> /warehouse/tablespace/managed/hive/comp3/delta_001_001_
> /warehouse/tablespace/managed/hive/comp3/delta_001_001_/00_0
> /warehouse/tablespace/managed/hive/comp3/delta_001_001_/01_0
> /warehouse/tablespace/managed/hive/comp3/delta_002_002_
> /warehouse/tablespace/managed/hive/comp3/delta_002_002_/00_0
> /warehouse/tablespace/managed/hive/comp3/delta_002_002_/01_0
> /warehouse/tablespace/managed/hive/comp3/delta_002_002_/02_0
> {code}
> Although the table is not bucketed, bucket is calculated from the (raw)
> files' names. Compaction in the above case will fail on delta1-1 not having
> data for 'bucket' 2.
> Steps to repro using small dataset:
> {code:java}
> set tez.grouping.min-size=8;
> set tez.grouping.max-size=8;
> set mapreduce.input.fileinputformat.split.minsize=8;
> set mapreduce.input.fileinputformat.split.minsize=8;
> create external table comp0 (a string);
> insert into comp0 values ("qwertyuiopasdfghjklzxcvbnm");
> insert into comp0 values ("qwertyuiopasdfghjklzxcvbnm");
> create external table comp1 stored as orc as select * from comp0;
> insert into comp0 values ("qwertyuiopasdfghjklzxcvbnm");
> create external table comp2 stored as orc as select * from comp0;
> create table comp3 (a string);
> load data inpath '/warehouse/tablespace/external/hive/comp1' into table comp3;
> load data inpath '/warehouse/tablespace/external/hive/comp2' into table
> comp3;{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22535) ACID: Check for already running compactions in Worker
[ https://issues.apache.org/jira/browse/HIVE-22535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17008729#comment-17008729 ] Laszlo Pinter commented on HIVE-22535: -- Meanwhile it turned out, that CompactionTxnHandler#findNextToCompact() already does this check, therefore I'm closing this jira. > ACID: Check for already running compactions in Worker > -- > > Key: HIVE-22535 > URL: https://issues.apache.org/jira/browse/HIVE-22535 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-22535.01.patch, HIVE-22535.02.patch > > > Compactor worker doesn't check if a compaction is already running for a given > resource. Initiator does this, but in case of manual compaction, this check > is missing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22535) ACID: Check for already running compactions in Worker
[ https://issues.apache.org/jira/browse/HIVE-22535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22535: - Resolution: Won't Fix Status: Resolved (was: Patch Available) > ACID: Check for already running compactions in Worker > -- > > Key: HIVE-22535 > URL: https://issues.apache.org/jira/browse/HIVE-22535 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-22535.01.patch, HIVE-22535.02.patch > > > Compactor worker doesn't check if a compaction is already running for a given > resource. Initiator does this, but in case of manual compaction, this check > is missing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-20934) ACID: Query based compactor for minor compaction
[ https://issues.apache.org/jira/browse/HIVE-20934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-20934: - Attachment: HIVE-20934.04.patch > ACID: Query based compactor for minor compaction > > > Key: HIVE-20934 > URL: https://issues.apache.org/jira/browse/HIVE-20934 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Vaibhav Gumashta >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-20934.01.patch, HIVE-20934.02.patch, > HIVE-20934.03.patch, HIVE-20934.04.patch > > > Follow up of HIVE-20699. This is to enable running minor compactions as a > HiveQL query -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22280) Q tests for partitioned temporary tables
[ https://issues.apache.org/jira/browse/HIVE-22280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22280: - Attachment: HIVE-22280.12.patch > Q tests for partitioned temporary tables > > > Key: HIVE-22280 > URL: https://issues.apache.org/jira/browse/HIVE-22280 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22280.01.patch, HIVE-22280.02.patch, > HIVE-22280.03.patch, HIVE-22280.04.patch, HIVE-22280.05.patch, > HIVE-22280.06.patch, HIVE-22280.07.patch, HIVE-22280.08.patch, > HIVE-22280.09.patch, HIVE-22280.10.patch, HIVE-22280.11.patch, > HIVE-22280.12.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22535) ACID: Check for already running compactions in Worker
[ https://issues.apache.org/jira/browse/HIVE-22535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22535: - Attachment: HIVE-22535.02.patch > ACID: Check for already running compactions in Worker > -- > > Key: HIVE-22535 > URL: https://issues.apache.org/jira/browse/HIVE-22535 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-22535.01.patch, HIVE-22535.02.patch > > > Compactor worker doesn't check if a compaction is already running for a given > resource. Initiator does this, but in case of manual compaction, this check > is missing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-20934) ACID: Query based compactor for minor compaction
[ https://issues.apache.org/jira/browse/HIVE-20934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-20934: - Attachment: HIVE-20934.03.patch > ACID: Query based compactor for minor compaction > > > Key: HIVE-20934 > URL: https://issues.apache.org/jira/browse/HIVE-20934 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Vaibhav Gumashta >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-20934.01.patch, HIVE-20934.02.patch, > HIVE-20934.03.patch > > > Follow up of HIVE-20699. This is to enable running minor compactions as a > HiveQL query -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22280) Q tests for partitioned temporary tables
[ https://issues.apache.org/jira/browse/HIVE-22280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22280: - Attachment: HIVE-22280.11.patch > Q tests for partitioned temporary tables > > > Key: HIVE-22280 > URL: https://issues.apache.org/jira/browse/HIVE-22280 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22280.01.patch, HIVE-22280.02.patch, > HIVE-22280.03.patch, HIVE-22280.04.patch, HIVE-22280.05.patch, > HIVE-22280.06.patch, HIVE-22280.07.patch, HIVE-22280.08.patch, > HIVE-22280.09.patch, HIVE-22280.10.patch, HIVE-22280.11.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22535) ACID: Check for already running compactions in Worker
[ https://issues.apache.org/jira/browse/HIVE-22535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22535: - Attachment: HIVE-22535.01.patch > ACID: Check for already running compactions in Worker > -- > > Key: HIVE-22535 > URL: https://issues.apache.org/jira/browse/HIVE-22535 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-22535.01.patch > > > Compactor worker doesn't check if a compaction is already running for a given > resource. Initiator does this, but in case of manual compaction, this check > is missing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22535) ACID: Check for already running compactions in Worker
[ https://issues.apache.org/jira/browse/HIVE-22535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22535: - Status: Patch Available (was: Open) > ACID: Check for already running compactions in Worker > -- > > Key: HIVE-22535 > URL: https://issues.apache.org/jira/browse/HIVE-22535 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-22535.01.patch > > > Compactor worker doesn't check if a compaction is already running for a given > resource. Initiator does this, but in case of manual compaction, this check > is missing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-21603) Java 11 preparation: update powermock version
[ https://issues.apache.org/jira/browse/HIVE-21603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter reassigned HIVE-21603: Assignee: (was: Laszlo Pinter) > Java 11 preparation: update powermock version > - > > Key: HIVE-21603 > URL: https://issues.apache.org/jira/browse/HIVE-21603 > Project: Hive > Issue Type: Task > Components: Hive >Affects Versions: 4.0.0 >Reporter: Laszlo Pinter >Priority: Major > > PowerMock1 has no support for Java11, therefore we need to bump its version > to 2.0.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-20934) ACID: Query based compactor for minor compaction
[ https://issues.apache.org/jira/browse/HIVE-20934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-20934: - Attachment: HIVE-20934.02.patch > ACID: Query based compactor for minor compaction > > > Key: HIVE-20934 > URL: https://issues.apache.org/jira/browse/HIVE-20934 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Vaibhav Gumashta >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-20934.01.patch, HIVE-20934.02.patch > > > Follow up of HIVE-20699. This is to enable running minor compactions as a > HiveQL query -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22280) Q tests for partitioned temporary tables
[ https://issues.apache.org/jira/browse/HIVE-22280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22280: - Attachment: HIVE-22280.10.patch > Q tests for partitioned temporary tables > > > Key: HIVE-22280 > URL: https://issues.apache.org/jira/browse/HIVE-22280 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22280.01.patch, HIVE-22280.02.patch, > HIVE-22280.03.patch, HIVE-22280.04.patch, HIVE-22280.05.patch, > HIVE-22280.06.patch, HIVE-22280.07.patch, HIVE-22280.08.patch, > HIVE-22280.09.patch, HIVE-22280.10.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22582) Avoid reading table as ACID when table name is starting with "delta" , but table is not transactional and BI Split Strategy is used
[
https://issues.apache.org/jira/browse/HIVE-22582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988684#comment-16988684
]
Laszlo Pinter commented on HIVE-22582:
--
+1 (non-binding)
> Avoid reading table as ACID when table name is starting with "delta" , but
> table is not transactional and BI Split Strategy is used
> ---
>
> Key: HIVE-22582
> URL: https://issues.apache.org/jira/browse/HIVE-22582
> Project: Hive
> Issue Type: Bug
>Reporter: Aditya Shah
>Assignee: Aditya Shah
>Priority: Major
> Attachments: HIVE-22582.patch
>
>
> The issue is fixed in HIVE-22473 but missed a check for BI Split Strategy.
> Steps to reproduce:
> {code:java}
> set hive.exec.orc.split.strategy=BI;
> create table delta_result (a int) stored as orc
> tblproperties('transactional'='false');
> insert into delta_result select 1;
> select * from delta_result;
> {code}
> Exception Stack Trace:
> {code:java}
> Caused by: java.lang.RuntimeException: ORC split generation failed with
> exception: String index out of range: -1
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1929)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:2016)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.generateWrappedSplits(FetchOperator.java:461)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextSplits(FetchOperator.java:430)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:336)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:576)
> ... 50 more
> Caused by: java.lang.StringIndexOutOfBoundsException: String index out of
> range: -1
> at java.lang.String.substring(String.java:1967)
> at
> org.apache.hadoop.hive.ql.io.AcidUtils.parsedDelta(AcidUtils.java:1128)
> at
> org.apache.hadoop.hive.ql.io.AcidUtils$ParsedDeltaLight.parse(AcidUtils.java:921)
> at
> org.apache.hadoop.hive.ql.io.AcidUtils.getLogicalLength(AcidUtils.java:2084)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$BISplitStrategy.getSplits(OrcInputFormat.java:1115)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1905)
> ... 55 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22473) Avoid reading table as ACID when table name is starting with "delta", but table is not transactional
[
https://issues.apache.org/jira/browse/HIVE-22473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988597#comment-16988597
]
Laszlo Pinter commented on HIVE-22473:
--
[~aditya-shah] Please open a new jira, and describe the details. Thanks
> Avoid reading table as ACID when table name is starting with "delta", but
> table is not transactional
>
>
> Key: HIVE-22473
> URL: https://issues.apache.org/jira/browse/HIVE-22473
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22473.01.patch, HIVE-22473.02.patch,
> HIVE-22473.03.patch, HIVE-22473.04.patch
>
>
> {code:sql}
> create table delta_result (a int) stored as orc
> tblproperties('transactional'='false');
> insert into delta_result select 1;
> select * from delta_result;
> {code}
> The above query will result in the following exception:
> 2019-11-08T13:49:05,780 WARN [HiveServer2-Handler-Pool: Thread-7906]
> thrift.ThriftCLIService: Error fetching results:2019-11-08T13:49:05,780 WARN
> [HiveServer2-Handler-Pool: Thread-7906] thrift.ThriftCLIService: Error
> fetching results:org.apache.hive.service.cli.HiveSQLException:
> java.io.IOException: java.lang.RuntimeException: ORC split generation failed
> with exception: java.lang.StringIndexOutOfBoundsException: String index out
> of range: -1 at
> org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:481)
> ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
> org.apache.hive.service.cli.operation.OperationManager.getOperationNextRowSet(OperationManager.java:331)
> ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
> org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:946)
> ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
> org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:567)
> ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
> org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:801)
> ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1837)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1822)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
> org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
> ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_211] at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_211] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_211]Caused
> by: java.io.IOException: java.lang.RuntimeException: ORC split generation
> failed with exception: java.lang.StringIndexOutOfBoundsException: String
> index out of range: -1 at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:638)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:545)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
> org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:151)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
> org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2142)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:241)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
> org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:476)
> ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] ... 13 moreCaused by:
> java.lang.RuntimeException: ORC split generation failed with exception:
> java.lang.StringIndexOutOfBoundsException: String index out of range: -1 at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1929)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:2016)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
>
[jira] [Updated] (HIVE-22534) ACID: Improve Compactor thread logging
[ https://issues.apache.org/jira/browse/HIVE-22534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22534: - Attachment: HIVE-22534.07.patch > ACID: Improve Compactor thread logging > -- > > Key: HIVE-22534 > URL: https://issues.apache.org/jira/browse/HIVE-22534 > Project: Hive > Issue Type: Improvement >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Minor > Attachments: HIVE-22534.01.patch, HIVE-22534.02.patch, > HIVE-22534.03.patch, HIVE-22534.04.patch, HIVE-22534.05.patch, > HIVE-22534.06.patch, HIVE-22534.07.patch > > > Make sure that it is easy to find issues when one of the compactor thread > fails. > Maybe: > * MDC - with iteration / threadname - so we can easily grep the logs for a > given run > * MDC with table/partition data on which the worker is working -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22280) Q tests for partitioned temporary tables
[ https://issues.apache.org/jira/browse/HIVE-22280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22280: - Attachment: HIVE-22280.09.patch > Q tests for partitioned temporary tables > > > Key: HIVE-22280 > URL: https://issues.apache.org/jira/browse/HIVE-22280 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22280.01.patch, HIVE-22280.02.patch, > HIVE-22280.03.patch, HIVE-22280.04.patch, HIVE-22280.05.patch, > HIVE-22280.06.patch, HIVE-22280.07.patch, HIVE-22280.08.patch, > HIVE-22280.09.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22534) ACID: Improve Compactor thread logging
[ https://issues.apache.org/jira/browse/HIVE-22534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22534: - Attachment: HIVE-22534.06.patch > ACID: Improve Compactor thread logging > -- > > Key: HIVE-22534 > URL: https://issues.apache.org/jira/browse/HIVE-22534 > Project: Hive > Issue Type: Improvement >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Minor > Attachments: HIVE-22534.01.patch, HIVE-22534.02.patch, > HIVE-22534.03.patch, HIVE-22534.04.patch, HIVE-22534.05.patch, > HIVE-22534.06.patch > > > Make sure that it is easy to find issues when one of the compactor thread > fails. > Maybe: > * MDC - with iteration / threadname - so we can easily grep the logs for a > given run > * MDC with table/partition data on which the worker is working -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22554) ACID: Wait timeout for blocking compaction should be configurable
[
https://issues.apache.org/jira/browse/HIVE-22554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16986845#comment-16986845
]
Laszlo Pinter commented on HIVE-22554:
--
Thanks [~pvary], for the review & commit!
> ACID: Wait timeout for blocking compaction should be configurable
> -
>
> Key: HIVE-22554
> URL: https://issues.apache.org/jira/browse/HIVE-22554
> Project: Hive
> Issue Type: Task
> Components: Hive
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-22554.01.patch, HIVE-22554.02.patch
>
>
> The wait timeout for blocking compaction is hardcoded to 5 minutes.
> {code:java}
> public class AlterTableCompactOperation extends
> DDLOperation {
> private static final int FIVE_MINUTES_IN_MILLIES = 5*60*1000;
> ...
> }{code}
> This should be configurable via a Hive Configuration parameter.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-20934) ACID: Query based compactor for minor compaction
[ https://issues.apache.org/jira/browse/HIVE-20934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-20934: - Status: Patch Available (was: Open) > ACID: Query based compactor for minor compaction > > > Key: HIVE-20934 > URL: https://issues.apache.org/jira/browse/HIVE-20934 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Vaibhav Gumashta >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-20934.01.patch > > > Follow up of HIVE-20699. This is to enable running minor compactions as a > HiveQL query -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-20934) ACID: Query based compactor for minor compaction
[ https://issues.apache.org/jira/browse/HIVE-20934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-20934: - Attachment: HIVE-20934.01.patch > ACID: Query based compactor for minor compaction > > > Key: HIVE-20934 > URL: https://issues.apache.org/jira/browse/HIVE-20934 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Vaibhav Gumashta >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-20934.01.patch > > > Follow up of HIVE-20699. This is to enable running minor compactions as a > HiveQL query -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22280) Q tests for partitioned temporary tables
[ https://issues.apache.org/jira/browse/HIVE-22280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22280: - Status: Patch Available (was: Reopened) > Q tests for partitioned temporary tables > > > Key: HIVE-22280 > URL: https://issues.apache.org/jira/browse/HIVE-22280 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22280.01.patch, HIVE-22280.02.patch, > HIVE-22280.03.patch, HIVE-22280.04.patch, HIVE-22280.05.patch, > HIVE-22280.06.patch, HIVE-22280.07.patch, HIVE-22280.08.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22280) Q tests for partitioned temporary tables
[ https://issues.apache.org/jira/browse/HIVE-22280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22280: - Attachment: HIVE-22280.08.patch > Q tests for partitioned temporary tables > > > Key: HIVE-22280 > URL: https://issues.apache.org/jira/browse/HIVE-22280 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22280.01.patch, HIVE-22280.02.patch, > HIVE-22280.03.patch, HIVE-22280.04.patch, HIVE-22280.05.patch, > HIVE-22280.06.patch, HIVE-22280.07.patch, HIVE-22280.08.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (HIVE-22280) Q tests for partitioned temporary tables
[ https://issues.apache.org/jira/browse/HIVE-22280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter reopened HIVE-22280: -- > Q tests for partitioned temporary tables > > > Key: HIVE-22280 > URL: https://issues.apache.org/jira/browse/HIVE-22280 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22280.01.patch, HIVE-22280.02.patch, > HIVE-22280.03.patch, HIVE-22280.04.patch, HIVE-22280.05.patch, > HIVE-22280.06.patch, HIVE-22280.07.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22554) ACID: Wait timeout for blocking compaction should be configurable
[
https://issues.apache.org/jira/browse/HIVE-22554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter updated HIVE-22554:
-
Attachment: HIVE-22554.02.patch
> ACID: Wait timeout for blocking compaction should be configurable
> -
>
> Key: HIVE-22554
> URL: https://issues.apache.org/jira/browse/HIVE-22554
> Project: Hive
> Issue Type: Task
> Components: Hive
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Minor
> Attachments: HIVE-22554.01.patch, HIVE-22554.02.patch
>
>
> The wait timeout for blocking compaction is hardcoded to 5 minutes.
> {code:java}
> public class AlterTableCompactOperation extends
> DDLOperation {
> private static final int FIVE_MINUTES_IN_MILLIES = 5*60*1000;
> ...
> }{code}
> This should be configurable via a Hive Configuration parameter.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22554) ACID: Wait timeout for blocking compaction should be configurable
[
https://issues.apache.org/jira/browse/HIVE-22554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter updated HIVE-22554:
-
Attachment: HIVE-22554.01.patch
> ACID: Wait timeout for blocking compaction should be configurable
> -
>
> Key: HIVE-22554
> URL: https://issues.apache.org/jira/browse/HIVE-22554
> Project: Hive
> Issue Type: Task
> Components: Hive
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Minor
> Attachments: HIVE-22554.01.patch
>
>
> The wait timeout for blocking compaction is hardcoded to 5 minutes.
> {code:java}
> public class AlterTableCompactOperation extends
> DDLOperation {
> private static final int FIVE_MINUTES_IN_MILLIES = 5*60*1000;
> ...
> }{code}
> This should be configurable via a Hive Configuration parameter.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22554) ACID: Wait timeout for blocking compaction should be configurable
[
https://issues.apache.org/jira/browse/HIVE-22554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter updated HIVE-22554:
-
Attachment: (was: HIVE-22554.01.patch)
> ACID: Wait timeout for blocking compaction should be configurable
> -
>
> Key: HIVE-22554
> URL: https://issues.apache.org/jira/browse/HIVE-22554
> Project: Hive
> Issue Type: Task
> Components: Hive
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Minor
> Attachments: HIVE-22554.01.patch
>
>
> The wait timeout for blocking compaction is hardcoded to 5 minutes.
> {code:java}
> public class AlterTableCompactOperation extends
> DDLOperation {
> private static final int FIVE_MINUTES_IN_MILLIES = 5*60*1000;
> ...
> }{code}
> This should be configurable via a Hive Configuration parameter.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22554) ACID: Wait timeout for blocking compaction should be configurable
[
https://issues.apache.org/jira/browse/HIVE-22554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter updated HIVE-22554:
-
Attachment: HIVE-22554.01.patch
> ACID: Wait timeout for blocking compaction should be configurable
> -
>
> Key: HIVE-22554
> URL: https://issues.apache.org/jira/browse/HIVE-22554
> Project: Hive
> Issue Type: Task
> Components: Hive
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Minor
> Attachments: HIVE-22554.01.patch
>
>
> The wait timeout for blocking compaction is hardcoded to 5 minutes.
> {code:java}
> public class AlterTableCompactOperation extends
> DDLOperation {
> private static final int FIVE_MINUTES_IN_MILLIES = 5*60*1000;
> ...
> }{code}
> This should be configurable via a Hive Configuration parameter.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22554) ACID: Wait timeout for blocking compaction should be configurable
[
https://issues.apache.org/jira/browse/HIVE-22554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter updated HIVE-22554:
-
Status: Patch Available (was: Open)
> ACID: Wait timeout for blocking compaction should be configurable
> -
>
> Key: HIVE-22554
> URL: https://issues.apache.org/jira/browse/HIVE-22554
> Project: Hive
> Issue Type: Task
> Components: Hive
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Minor
> Attachments: HIVE-22554.01.patch
>
>
> The wait timeout for blocking compaction is hardcoded to 5 minutes.
> {code:java}
> public class AlterTableCompactOperation extends
> DDLOperation {
> private static final int FIVE_MINUTES_IN_MILLIES = 5*60*1000;
> ...
> }{code}
> This should be configurable via a Hive Configuration parameter.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HIVE-21765) Support partitions on temporary tables
[ https://issues.apache.org/jira/browse/HIVE-21765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter resolved HIVE-21765. -- Resolution: Fixed > Support partitions on temporary tables > -- > > Key: HIVE-21765 > URL: https://issues.apache.org/jira/browse/HIVE-21765 > Project: Hive > Issue Type: New Feature >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > > HIVE-18739 and HIVE-20661 added partial support for partition columns on > temporary tables, but it was not complete and it was available only for > internal usage. The scope of this jira, is to implement the missing parts and > expose this functionality. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22280) Q tests for partitioned temporary tables
[ https://issues.apache.org/jira/browse/HIVE-22280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16984403#comment-16984403 ] Laszlo Pinter commented on HIVE-22280: -- [~pvary] Thanks for the review and commit. > Q tests for partitioned temporary tables > > > Key: HIVE-22280 > URL: https://issues.apache.org/jira/browse/HIVE-22280 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22280.01.patch, HIVE-22280.02.patch, > HIVE-22280.03.patch, HIVE-22280.04.patch, HIVE-22280.05.patch, > HIVE-22280.06.patch, HIVE-22280.07.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-22542) Update orc version to 1.5.8
[ https://issues.apache.org/jira/browse/HIVE-22542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter resolved HIVE-22542. -- Resolution: Duplicate > Update orc version to 1.5.8 > --- > > Key: HIVE-22542 > URL: https://issues.apache.org/jira/browse/HIVE-22542 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-22542.01.patch > > > Update orc to 1.5.8 in root pom -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22542) Update orc version to 1.5.8
[ https://issues.apache.org/jira/browse/HIVE-22542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22542: - Status: Open (was: Patch Available) > Update orc version to 1.5.8 > --- > > Key: HIVE-22542 > URL: https://issues.apache.org/jira/browse/HIVE-22542 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-22542.01.patch > > > Update orc to 1.5.8 in root pom -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22554) ACID: Wait timeout for blocking compaction should be configurable
[
https://issues.apache.org/jira/browse/HIVE-22554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter reassigned HIVE-22554:
> ACID: Wait timeout for blocking compaction should be configurable
> -
>
> Key: HIVE-22554
> URL: https://issues.apache.org/jira/browse/HIVE-22554
> Project: Hive
> Issue Type: Task
> Components: Hive
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Minor
>
> The wait timeout for blocking compaction is hardcoded to 5 minutes.
> {code:java}
> public class AlterTableCompactOperation extends
> DDLOperation {
> private static final int FIVE_MINUTES_IN_MILLIES = 5*60*1000;
> ...
> }{code}
> This should be configurable via a Hive Configuration parameter.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22534) ACID: Improve Compactor thread logging
[ https://issues.apache.org/jira/browse/HIVE-22534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22534: - Attachment: HIVE-22534.05.patch > ACID: Improve Compactor thread logging > -- > > Key: HIVE-22534 > URL: https://issues.apache.org/jira/browse/HIVE-22534 > Project: Hive > Issue Type: Improvement >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Minor > Attachments: HIVE-22534.01.patch, HIVE-22534.02.patch, > HIVE-22534.03.patch, HIVE-22534.04.patch, HIVE-22534.05.patch > > > Make sure that it is easy to find issues when one of the compactor thread > fails. > Maybe: > * MDC - with iteration / threadname - so we can easily grep the logs for a > given run > * MDC with table/partition data on which the worker is working -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22534) ACID: Improve Compactor thread logging
[ https://issues.apache.org/jira/browse/HIVE-22534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22534: - Attachment: HIVE-22534.04.patch > ACID: Improve Compactor thread logging > -- > > Key: HIVE-22534 > URL: https://issues.apache.org/jira/browse/HIVE-22534 > Project: Hive > Issue Type: Improvement >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Minor > Attachments: HIVE-22534.01.patch, HIVE-22534.02.patch, > HIVE-22534.03.patch, HIVE-22534.04.patch > > > Make sure that it is easy to find issues when one of the compactor thread > fails. > Maybe: > * MDC - with iteration / threadname - so we can easily grep the logs for a > given run > * MDC with table/partition data on which the worker is working -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22542) Update orc version to 1.5.8
[ https://issues.apache.org/jira/browse/HIVE-22542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22542: - Component/s: Hive > Update orc version to 1.5.8 > --- > > Key: HIVE-22542 > URL: https://issues.apache.org/jira/browse/HIVE-22542 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-22542.01.patch > > > Update orc to 1.5.8 in root pom -- This message was sent by Atlassian Jira (v8.3.4#803005)
