[
https://issues.apache.org/jira/browse/HIVE-18948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16408416#comment-16408416
]
Jesus Camacho Rodriguez commented on HIVE-18948:
------------------------------------------------
A bit more detail on the work needed to move lock acquisition before the
optimization phase.
Call to acquire locks from Driver.java lands in {{acquireLocks}} method in
DbTxnManager.
The method takes as parameters {{QueryPlan plan, Context ctx, String username,
boolean isBlocking}}. Currently, {{queryPlan}} is not available until query is
fully compiled / executed. The following information in the queryPlan is
accessed during lock acquisition: query id, operation, read entities, and write
entities.
It seems when they query is parsed (after parsing and before optimization), we
will already have query id, operation, and read entities.
However, new read entities might be added during compilation/optimization,
e.g., when views are used.
In turn, write entities are generated while we are compiling/optimizing the
plan though (e.g. when we generate the file sink operator in semantic
analyzer), hence we would need to move that logic so it is done while parsing
the query (at least we should generate the information that we need). The
information that we use from read/write entities is the name of the entity, the
type (DATABASE, TABLE, PARTITION, etc.), and sometimes that table object itself
(e.g., to check if table is transactional).
Thus, most of the work consists on moving the read entities and write entities
generation (or the properties that are needed from them) when we are parsing
the query.
> Acquire locks before generating the valid transaction list
> ----------------------------------------------------------
>
> Key: HIVE-18948
> URL: https://issues.apache.org/jira/browse/HIVE-18948
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
> Affects Versions: 3.0.0
> Reporter: Jesus Camacho Rodriguez
> Priority: Critical
>
> HIVE-18825 moves the valid transaction list generation logic before query
> optimization. In order to support lock-based concurrency control correctly,
> the logic to acquire the locks has to be moved before query optimization too,
> and before the valid transaction list is generated.
> This requires a bit of work/refactoring, since lock acquisition logic relies
> on read/write entities, and it is heavily dependent on the QueryPlan object
> too, e.g., it sets some properties in the file sink descriptors for the plan.
> Currently, all these data structures (except for read entities) are only
> available after query has been optimized, hence we will need to 1) generate
> some of this data structures before query optimization, e.g., write entities,
> and 2) create and propagate some of the properties so they are set during
> query optimization, e.g., those properties contained in the file sink
> descriptors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
