[
https://issues.apache.org/jira/browse/HIVE-18940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16399924#comment-16399924
]
Alexander Kolbasov commented on HIVE-18940:
-------------------------------------------
[~vihangk1] The major problem with all auto-increment approaches is that
clients have to deal with holes in the stream of IDs and the hole can be
temporary (there is transaction in flight) or permanent (transaction failed). I
am not convinced that we must produce events in the commit order but it is
important to not loose events (and guarantee that within a session the order is
preserved).
Sessions usually have a nice property that they send some operations to HMS,
wait for completion and only then send another one. Interleaving ordering
between different sessions should be fine. Unfortunately, HMS APIs do not
include any session ID (otherwise we could use it to partition locks).
> Hive notifications serialize all write DDL operations
> -----------------------------------------------------
>
> Key: HIVE-18940
> URL: https://issues.apache.org/jira/browse/HIVE-18940
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 3.0.0
> Reporter: Alexander Kolbasov
> Priority: Major
>
> The implementation of DbNotificationListener uses a single row to store
> current notification ID and uses {{SELECT FOR UPDATE}} to lock the row. This
> serializes all write DDL operations which isn't good.
> We should consider using database auto-increment for notification ID instead.
> Especially on mMySQL/innoDb it is supported natively with relatively
> light-weight locking.
> This creates potential issue for consumers though because such IDs may have
> holes. There are two types of holes - transient hole for a transaction which
> have not committed yet and will be committed shortly and permanent holes for
> transactions that fail. Consumers need to deal with it. It may be useful to
> add DB-generated timestamp as well to assist in recovery from holes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
