Vihang Karajgaonkar commented on HIVE-18885:

bq. but it allows for missed notifications since creating notification and 
storing it may fail. This means that there is some chance that certain 
operations succeed but do not have corresponding notifications.

Isn't it possible even now? I see that transactional listeners are called 
within a transaction block but non-transaction listeners are called outside the 
transaction block.

bq. Another issue is that we have bulk operations which create many 
notifications. This means that the code should be restructured to save all 
these notification after transaction completes rather then inline.

Yes, this also gives us an opportunity to generate aggregate notifications 
where they make sense. If a table has thousands of partitions does it make 
sense to generate thousands of (expensive) notifications when cascade is true?

> Cascaded alter table + notifications = disaster
> -----------------------------------------------
>                 Key: HIVE-18885
>                 URL: https://issues.apache.org/jira/browse/HIVE-18885
>             Project: Hive
>          Issue Type: Bug
>          Components: Hive, Metastore
>    Affects Versions: 3.0.0
>            Reporter: Alexander Kolbasov
>            Priority: Major
> You can see the problem from looking at the code, but it actually created 
> severe problems for real life Hive user.
> When {{alter table}} has {{cascade}} option it does the following:
> {code:java}
>          msdb.openTransaction()
>           ...
>           List<Partition> parts = msdb.getPartitions(dbname, name, -1);
>           for (Partition part : parts) {
>             List<FieldSchema> oldCols = part.getSd().getCols();
>             part.getSd().setCols(newt.getSd().getCols());
>             String oldPartName = 
> Warehouse.makePartName(oldt.getPartitionKeys(), part.getValues());
>             updatePartColumnStatsForAlterColumns(msdb, part, oldPartName, 
> part.getValues(), oldCols, part);
>             msdb.alterPartition(dbname, name, part.getValues(), part);
>           }
>  {code}
> So it walks all partitions (and this may be huge list) and does some 
> non-trivial operations in one single uber-transaction.
> When DbNotificationListener is enabled, it adds an event for each partition, 
> all while
> holding a row lock on NOTIFICATION_SEQUENCE table. As a result, while this is 
> happening no other write DDL can proceed. This can sometimes cause DB lock 
> timeouts which cause HMS level operation retries which make things even worse.
> In one particular case this pretty much made HMS unusable.

This message was sent by Atlassian JIRA

Reply via email to