[
https://issues.apache.org/jira/browse/HIVE-23107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17071246#comment-17071246
]
Peter Vary commented on HIVE-23107:
-----------------------------------
Thanks for the patch [~lpinter]!
Some suggestions for squashing jdbc queries:
* Maybe using this query for setting the next transaction id in markCompacted:
{code}
UPDATE "COMPACTION_QUEUE" SET "CQ_STATE" = 'r', "CQ_WORKER_ID" = NULL,
"CQ_NEXT_TXN_ID" = (SELECT "NTXN_NEXT" FROM "NEXT_TXN_ID")
WHERE "CQ_ID" = 11;
{code}
* Maybe pushing all of the queries to the db in one go in
cleanTxnToWriteIdTable, like:
{code}
SELECT MIN("RES"."ID") AS "ID" FROM (
SELECT MIN("NTXN_NEXT") AS "ID" FROM "NEXT_TXN_ID"
UNION
SELECT MIN("WS_COMMIT_ID") AS "ID" FROM "WRITE_SET"
UNION
SELECT MIN("TXN_ID") AS "ID" FROM "TXNS" WHERE "TXN_STATE" = 'a'
) "RES"
{code}
* Maybe pushing all of the queries to the db in one go in
findMinOpenTxnIdForCleaner, like:
{code}
SELECT MIN("RES"."ID") AS "ID" FROM (
SELECT MIN("TXN_ID") AS "ID" FROM "TXNS" WHERE "TXN_STATE" = 'o'
UNION
SELECT MIN("CQ_NEXT_TXN_ID") AS "ID" FROM "COMPACTION_QUEUE" WHERE "CQ_STATE"
= 'r'
UNION
SELECT NTXN_NEXT AS "ID" FROM "NEXT_TXN_ID"
) "RES"
{code}
One more suggestion: Maybe we should drop the MIN_HISTORY_LEVEL table in
upgrade queries?
Do we have test coverage for the changed methods? I have seen that you made
plenty of tests during previous patches, just asking to make sure :D
Thanks for the patch! This will help trendemously!
Peter
> Remove MIN_HISTORY_LEVEL table
> ------------------------------
>
> Key: HIVE-23107
> URL: https://issues.apache.org/jira/browse/HIVE-23107
> Project: Hive
> Issue Type: Improvement
> Components: Hive
> Reporter: László Pintér
> Assignee: László Pintér
> Priority: Major
> Attachments: HIVE-23107.01.patch
>
>
> MIN_HISTORY_LEVEL table is used in two places:
> * Cleaner uses it to decide if the files can be removed - this could be
> replaced by adding a new column to compaction_queue storing the next_txn_id
> when the change was committed, and before cleaning checking the minimum open
> transaction id in the TXNS table
> * Initiator uses it to decide if some items from TXN_TO_WRITE_ID table can
> be removed. This could be replaced by using the WRITE_SET.WS_COMMIT_ID.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
