[
https://issues.apache.org/jira/browse/HIVE-18940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16408289#comment-16408289
]
Vihang Karajgaonkar commented on HIVE-18940:
--------------------------------------------
Another improvement based on [~akolb]'s suggestion above is to break the lock
for each unique {{dbname.tblname}} pair and generate the commit ids specific to
that individual entity. So instead of having one row blocking the whole world
in NOTIFICATION_SEQUENCE table, it would now have multiple rows one for each
table. This would obviously mean that clients will have to smarter and will
need to store the last commit id on an individual table basis. Since this will
be a change of behavior, this will have to be done using a configurable option
so that existing clients can still use the current way and move to the later
when they are ready by switching the config. The biggest advantage of this
approach is that practically no transaction will ever block since conflicting
transactions are not allowed in the first place using ZK locks at the query
compilation time. The disadvantage here is added complexity on both server and
client side. Server needs to handle corner cases well (create/drop database
events). Clients will need added logic to store the commit id on an individual
table level.
> Hive notifications serialize all write DDL operations
> -----------------------------------------------------
>
> Key: HIVE-18940
> URL: https://issues.apache.org/jira/browse/HIVE-18940
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 3.0.0
> Reporter: Alexander Kolbasov
> Priority: Major
>
> The implementation of DbNotificationListener uses a single row to store
> current notification ID and uses {{SELECT FOR UPDATE}} to lock the row. This
> serializes all write DDL operations which isn't good.
> We should consider using database auto-increment for notification ID instead.
> Especially on mMySQL/innoDb it is supported natively with relatively
> light-weight locking.
> This creates potential issue for consumers though because such IDs may have
> holes. There are two types of holes - transient hole for a transaction which
> have not committed yet and will be committed shortly and permanent holes for
> transactions that fail. Consumers need to deal with it. It may be useful to
> add DB-generated timestamp as well to assist in recovery from holes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
