[
https://issues.apache.org/jira/browse/HIVE-16564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman reassigned HIVE-16564:
-------------------------------------
> StreamingAPI is locking too much?
> ---------------------------------
>
> Key: HIVE-16564
> URL: https://issues.apache.org/jira/browse/HIVE-16564
> Project: Hive
> Issue Type: Bug
> Components: HCatalog, Transactions
> Affects Versions: 1.0.0
> Reporter: Eugene Koifman
> Assignee: Eugene Koifman
>
> Currently _TransactionBatchImpl.beginNextTransactionImpl()_ acquires Shared
> locks for each Transaction in the batch.
> Especially under high load this creates pressure on the LockManager (i.e.
> Metastore) and degrades performance of Ingest itself.
> Because all transactions in a batch write to the same physical file and the
> fact that for Acid tables (which are required for Streaming Ingest) shared
> locks only protect against Exclusive locks (like drop table),
> acquiring/releasing locks doesn't for each txn doesn't achieve much.
> One possibility to acquire all locks (i.e. for all txns) at the time the
> batch is created (same as is done for openTxn() for all txns in the batch).
> Locks for each txn in the batch will be released automatically when commit is
> called for the respective txn.
> Alternatively, don't acquire any locks - this means someone may drop a table
> while it's written to but using locks here doesn't buy much. Say a Drop
> request is issued when a write is in progress. It will block until the write
> releases it's lock and execute immediately after that. Thus none of the data
> of that write is visible for any meaningful length of time anyway.
> Allow a "meta lock" - a lock not associated with any specific txn, that is
> held for the duration of the TransactionBatch. This sort of breaks the model
> (especially since HIVE-12636). Perhaps each batch can open one "extra" txn
> for internal purposes, just to acquire this "meta lock". No data will ever
> be tagged with this "extra" txn.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
