[
https://issues.apache.org/jira/browse/HIVE-12389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-12389:
----------------------------------
Description:
in extreme situations, due to misconfigurations, it may be possible to have
100Ks or even 1Ms of aborted txns.
This causes delete from TXNS where txn_id in (...) to have a huge IN clause and
DB chokes.
Should use something like TxnHandler.TIMED_OUT_TXN_ABORT_BATCH_SIZE to break up
delete into multiple queries. (Incidentally the batch size should likely be
1000, not 100, maybe even configurable).
On MySQL for example, it can cause query to fail with
bq. Packet for query is too large (9288598 > 1048576). You can change this
value on the server by setting the max_allowed_packet' variable.
was:
in extreme situations, due to misconfigurations, it may be possible to have
100Ks or even 1Ms of aborted txns.
This causes delete from TXNS where txn_id in (...) to have a huge IN clause and
DB chokes.
Should use something like TxnHandler.TIMED_OUT_TXN_ABORT_BATCH_SIZE to break up
delete into multiple queries. (Incidentally the batch size should likely be
1000, not 100, maybe even configurable).
On MySQL for example, it can cause
bq. Packet for query is too large (9288598 > 1048576). You can change this
value on the server by setting the max_allowed_packet' variable.
> CompactionTxnHandler.cleanEmptyAbortedTxns() should safeguard against huge IN
> clauses
> -------------------------------------------------------------------------------------
>
> Key: HIVE-12389
> URL: https://issues.apache.org/jira/browse/HIVE-12389
> Project: Hive
> Issue Type: Bug
> Components: Metastore, Transactions
> Affects Versions: 1.0.0
> Reporter: Eugene Koifman
> Assignee: Eugene Koifman
> Attachments: HIVE-12389.patch
>
>
> in extreme situations, due to misconfigurations, it may be possible to have
> 100Ks or even 1Ms of aborted txns.
> This causes delete from TXNS where txn_id in (...) to have a huge IN clause
> and DB chokes.
> Should use something like TxnHandler.TIMED_OUT_TXN_ABORT_BATCH_SIZE to break
> up delete into multiple queries. (Incidentally the batch size should likely
> be 1000, not 100, maybe even configurable).
> On MySQL for example, it can cause query to fail with
> bq. Packet for query is too large (9288598 > 1048576). You can change this
> value on the server by setting the max_allowed_packet' variable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
