[ 
https://issues.apache.org/jira/browse/HIVE-18940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16408289#comment-16408289
 ] 

Vihang Karajgaonkar commented on HIVE-18940:
--------------------------------------------

Another improvement based on [~akolb]'s suggestion above is to break the lock 
for each unique {{dbname.tblname}} pair and generate the commit ids specific to 
that individual entity. So instead of having one row blocking the whole world 
in NOTIFICATION_SEQUENCE table, it would now have multiple rows one for each 
table. This would obviously mean that clients will have to smarter and will 
need to store the last commit id on an individual table basis. Since this will 
be a change of behavior, this will have to be done using a configurable option 
so that existing clients can still use the current way and move to the later 
when they are ready by switching the config. The biggest advantage of this 
approach is that practically no transaction will ever block since conflicting 
transactions are not allowed in the first place using ZK locks at the query 
compilation time. The disadvantage here is added complexity on both server and 
client side. Server needs to handle corner cases well (create/drop database 
events). Clients will need added logic to store the commit id on an individual 
table level.

> Hive notifications serialize all write DDL operations
> -----------------------------------------------------
>
>                 Key: HIVE-18940
>                 URL: https://issues.apache.org/jira/browse/HIVE-18940
>             Project: Hive
>          Issue Type: Bug
>          Components: Metastore
>    Affects Versions: 3.0.0
>            Reporter: Alexander Kolbasov
>            Priority: Major
>
> The implementation of DbNotificationListener uses a single row to store 
> current notification ID and uses {{SELECT FOR UPDATE}} to lock the row. This 
> serializes all write DDL operations which isn't good.
> We should consider using database auto-increment for notification ID instead. 
> Especially on mMySQL/innoDb it is supported natively with relatively 
> light-weight locking. 
> This creates potential issue for consumers though because such IDs may have 
> holes. There are two types of holes - transient hole for a transaction which 
> have not committed yet and will be committed shortly and permanent holes for 
> transactions that fail. Consumers need to deal with it. It may be useful to 
> add DB-generated timestamp as well to assist in recovery from holes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to