[jira] [Commented] (HIVE-18783) Set Notification ID for Alter Table event

2018-02-23 Thread JIRA

[ 
https://issues.apache.org/jira/browse/HIVE-18783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374612#comment-16374612
 ] 

Sergio Peña commented on HIVE-18783:


[~akolb] I think the problem is about the transactionalListenerResponses does 
not exist for ALTER_TABLE events. For other events, such as CREATE_TABLE, the 
transactionalListenerResponses is obtained from the transactional listener 
which contains the EVENT_ID of the notification generated, and this is passed 
to the post-listener so that clients listening later can do something with that 
EVENT_ID. For ALTER_TABLE events, this EVENT_ID is not passed to the 
post-listener, so clients can fail if they expect such EVENT_ID on the 
post-listener.

> Set Notification ID for Alter Table event
> -
>
> Key: HIVE-18783
> URL: https://issues.apache.org/jira/browse/HIVE-18783
> Project: Hive
>  Issue Type: Bug
>Reporter: Na Li
>Assignee: Alexander Kolbasov
>Priority: Major
>
>  in HiveMetaStore, alter_table_core does NOT call transactional listener, and 
> the notification ID corresponding to the alter table event is NOT set in the 
> event parameters.
> {code}
> + alter_table_core
>   
>   try {
> Table oldt = this.get_table_core(dbname, name);
> this.firePreEvent(new PreAlterTableEvent(oldt, newTable, this));
> this.alterHandler.alterTable(this.getMS(), this.wh, dbname, name, 
> newTable, envContext, this);
> success = true;
> if (!this.listeners.isEmpty()) {
>   MetaStoreListenerNotifier.notifyEvent(this.listeners, 
> EventType.ALTER_TABLE, new AlterTableEvent(oldt, newTable, true, this), 
> envContext);
> }
>   } catch (NoSuchObjectException var12) {
> ex = var12;
> throw new InvalidOperationException(var12.getMessage());
>   } catch (Exception var13) {
> ex = var13;
> if (var13 instanceof MetaException) {
>   throw (MetaException)var13;
> }
> if (var13 instanceof InvalidOperationException) {
>   throw (InvalidOperationException)var13;
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-18783) Set Notification ID for Alter Table event

2018-02-23 Thread JIRA

[ 
https://issues.apache.org/jira/browse/HIVE-18783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374565#comment-16374565
 ] 

Sergio Peña commented on HIVE-18783:


[~LinaAtAustin] [~akolb] The transactional listener for alter table is on the 
HiveAlterHandler class. This should be called and generate an event, otherwise, 
the client consuming notifications wouldn't see the alter table event.

> Set Notification ID for Alter Table event
> -
>
> Key: HIVE-18783
> URL: https://issues.apache.org/jira/browse/HIVE-18783
> Project: Hive
>  Issue Type: Bug
>Reporter: Na Li
>Assignee: Alexander Kolbasov
>Priority: Major
>
>  in HiveMetaStore, alter_table_core does NOT call transactional listener, and 
> the notification ID corresponding to the alter table event is NOT set in the 
> event parameters.
> {code}
> + alter_table_core
>   
>   try {
> Table oldt = this.get_table_core(dbname, name);
> this.firePreEvent(new PreAlterTableEvent(oldt, newTable, this));
> this.alterHandler.alterTable(this.getMS(), this.wh, dbname, name, 
> newTable, envContext, this);
> success = true;
> if (!this.listeners.isEmpty()) {
>   MetaStoreListenerNotifier.notifyEvent(this.listeners, 
> EventType.ALTER_TABLE, new AlterTableEvent(oldt, newTable, true, this), 
> envContext);
> }
>   } catch (NoSuchObjectException var12) {
> ex = var12;
> throw new InvalidOperationException(var12.getMessage());
>   } catch (Exception var13) {
> ex = var13;
> if (var13 instanceof MetaException) {
>   throw (MetaException)var13;
> }
> if (var13 instanceof InvalidOperationException) {
>   throw (InvalidOperationException)var13;
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-18783) Set Notification ID for Alter Table event

2018-02-22 Thread Alexander Kolbasov (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-18783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374020#comment-16374020
 ] 

Alexander Kolbasov commented on HIVE-18783:
---

Here is alter_table_core with a bit more core:

{code}
  try {
Table oldt = get_table_core(dbname, name);
firePreEvent(new PreAlterTableEvent(oldt, newTable, this));
alterHandler.alterTable(getMS(), wh, dbname, name, newTable,  // 
<--Here!
envContext, this);
{code}

The alterHandler is {{HiveALterHandler}} so you call {{alterTable()}} there 
there is this piece:

{code}
  if (transactionalListeners != null && !transactionalListeners.isEmpty()) {
MetaStoreListenerNotifier.notifyEvent(transactionalListeners,
  
EventMessage.EventType.ALTER_TABLE,
  new AlterTableEvent(oldt, newt, 
true, handler),
  environmentContext);
{code}

This is the part that is executed inside transaction. The part that you quoted 
is, indeed, post-transactional listeners invocation. Does it make sense?


> Set Notification ID for Alter Table event
> -
>
> Key: HIVE-18783
> URL: https://issues.apache.org/jira/browse/HIVE-18783
> Project: Hive
>  Issue Type: Bug
>Reporter: Na Li
>Assignee: Alexander Kolbasov
>Priority: Major
>
>  in HiveMetaStore, alter_table_core does NOT call transactional listener, and 
> the notification ID corresponding to the alter table event is NOT set in the 
> event parameters.
> {code}
> + alter_table_core
>   
>   try {
> Table oldt = this.get_table_core(dbname, name);
> this.firePreEvent(new PreAlterTableEvent(oldt, newTable, this));
> this.alterHandler.alterTable(this.getMS(), this.wh, dbname, name, 
> newTable, envContext, this);
> success = true;
> if (!this.listeners.isEmpty()) {
>   MetaStoreListenerNotifier.notifyEvent(this.listeners, 
> EventType.ALTER_TABLE, new AlterTableEvent(oldt, newTable, true, this), 
> envContext);
> }
>   } catch (NoSuchObjectException var12) {
> ex = var12;
> throw new InvalidOperationException(var12.getMessage());
>   } catch (Exception var13) {
> ex = var13;
> if (var13 instanceof MetaException) {
>   throw (MetaException)var13;
> }
> if (var13 instanceof InvalidOperationException) {
>   throw (InvalidOperationException)var13;
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-18783) Set Notification ID for Alter Table event

2018-02-22 Thread Alexander Kolbasov (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-18783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374011#comment-16374011
 ] 

Alexander Kolbasov commented on HIVE-18783:
---

[~LinaAtAustin] if notification ID is not set for an event, how it propagates 
to Sentry at all? Or you are saying that none of the ALTER TABLE events ever 
reach SENTRY?

> Set Notification ID for Alter Table event
> -
>
> Key: HIVE-18783
> URL: https://issues.apache.org/jira/browse/HIVE-18783
> Project: Hive
>  Issue Type: Bug
>Reporter: Na Li
>Assignee: Alexander Kolbasov
>Priority: Major
>
>  in HiveMetaStore, alter_table_core does NOT call transactional listener, and 
> the notification ID corresponding to the alter table event is NOT set in the 
> event parameters.
> {code}
> + alter_table_core
>   
>   try {
> Table oldt = this.get_table_core(dbname, name);
> this.firePreEvent(new PreAlterTableEvent(oldt, newTable, this));
> this.alterHandler.alterTable(this.getMS(), this.wh, dbname, name, 
> newTable, envContext, this);
> success = true;
> if (!this.listeners.isEmpty()) {
>   MetaStoreListenerNotifier.notifyEvent(this.listeners, 
> EventType.ALTER_TABLE, new AlterTableEvent(oldt, newTable, true, this), 
> envContext);
> }
>   } catch (NoSuchObjectException var12) {
> ex = var12;
> throw new InvalidOperationException(var12.getMessage());
>   } catch (Exception var13) {
> ex = var13;
> if (var13 instanceof MetaException) {
>   throw (MetaException)var13;
> }
> if (var13 instanceof InvalidOperationException) {
>   throw (InvalidOperationException)var13;
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-18783) Set Notification ID for Alter Table event

2018-02-22 Thread Na Li (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-18783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373986#comment-16373986
 ] 

Na Li commented on HIVE-18783:
--

[~akolb] The hive branch I mentioned is 2.3.2. I found this issue by stepping 
into the code. 

"this.listeners" is for post event lisener processing.  
"this.transactionalListeners" is for transactional listener.

For example, create table event contains the notification ID because it calls 
both listeners.
  
{code:java}
+  HiveMetaStore.create_table_core  
if (primaryKeys == null && foreignKeys == null) {
  ms.createTable(tbl);
} else {
  ms.createTableWithConstraints(tbl, primaryKeys, foreignKeys);
}

if (!this.transactionalListeners.isEmpty()) {
  transactionalListenerResponses = 
MetaStoreListenerNotifier.notifyEvent(this.transactionalListeners, 
EventType.CREATE_TABLE, new CreateTableEvent(tbl, true, this), envContext);
}

success = ms.commitTransaction();
  } finally {
if (!success) {
  ms.rollbackTransaction();
  if (madeDir) {
this.wh.deleteDir(tblPath, true);
  }
}

if (!this.listeners.isEmpty()) {
  MetaStoreListenerNotifier.notifyEvent(this.listeners, 
EventType.CREATE_TABLE, new CreateTableEvent(tbl, success, this), envContext, 
transactionalListenerResponses, ms);
}{code}
 

 

 

> Set Notification ID for Alter Table event
> -
>
> Key: HIVE-18783
> URL: https://issues.apache.org/jira/browse/HIVE-18783
> Project: Hive
>  Issue Type: Bug
>Reporter: Na Li
>Assignee: Alexander Kolbasov
>Priority: Major
>
>  in HiveMetaStore, alter_table_core does NOT call transactional listener, and 
> the notification ID corresponding to the alter table event is NOT set in the 
> event parameters.
> {code}
> + alter_table_core
>   
>   try {
> Table oldt = this.get_table_core(dbname, name);
> this.firePreEvent(new PreAlterTableEvent(oldt, newTable, this));
> this.alterHandler.alterTable(this.getMS(), this.wh, dbname, name, 
> newTable, envContext, this);
> success = true;
> if (!this.listeners.isEmpty()) {
>   MetaStoreListenerNotifier.notifyEvent(this.listeners, 
> EventType.ALTER_TABLE, new AlterTableEvent(oldt, newTable, true, this), 
> envContext);
> }
>   } catch (NoSuchObjectException var12) {
> ex = var12;
> throw new InvalidOperationException(var12.getMessage());
>   } catch (Exception var13) {
> ex = var13;
> if (var13 instanceof MetaException) {
>   throw (MetaException)var13;
> }
> if (var13 instanceof InvalidOperationException) {
>   throw (InvalidOperationException)var13;
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-18783) Set Notification ID for Alter Table event

2018-02-22 Thread Alexander Kolbasov (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-18783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373984#comment-16373984
 ] 

Alexander Kolbasov commented on HIVE-18783:
---

[~spena] What do you think - does this work or not?

> Set Notification ID for Alter Table event
> -
>
> Key: HIVE-18783
> URL: https://issues.apache.org/jira/browse/HIVE-18783
> Project: Hive
>  Issue Type: Bug
>Reporter: Na Li
>Assignee: Alexander Kolbasov
>Priority: Major
>
>  in HiveMetaStore, alter_table_core does NOT call transactional listener, and 
> the notification ID corresponding to the alter table event is NOT set in the 
> event parameters.
> {code}
> + alter_table_core
>   
>   try {
> Table oldt = this.get_table_core(dbname, name);
> this.firePreEvent(new PreAlterTableEvent(oldt, newTable, this));
> this.alterHandler.alterTable(this.getMS(), this.wh, dbname, name, 
> newTable, envContext, this);
> success = true;
> if (!this.listeners.isEmpty()) {
>   MetaStoreListenerNotifier.notifyEvent(this.listeners, 
> EventType.ALTER_TABLE, new AlterTableEvent(oldt, newTable, true, this), 
> envContext);
> }
>   } catch (NoSuchObjectException var12) {
> ex = var12;
> throw new InvalidOperationException(var12.getMessage());
>   } catch (Exception var13) {
> ex = var13;
> if (var13 instanceof MetaException) {
>   throw (MetaException)var13;
> }
> if (var13 instanceof InvalidOperationException) {
>   throw (InvalidOperationException)var13;
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-18783) Set Notification ID for Alter Table event

2018-02-22 Thread Alexander Kolbasov (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-18783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373976#comment-16373976
 ] 

Alexander Kolbasov commented on HIVE-18783:
---

[~spena] FYI.

[~LinaAtAustin] What branch did you use to produce snippet above? Why do you 
think that alter_table_core does not call transactional listener? Isn't it what 
{{MetaStoreListenerNotifier.notifyEvent()}} call does?

> Set Notification ID for Alter Table event
> -
>
> Key: HIVE-18783
> URL: https://issues.apache.org/jira/browse/HIVE-18783
> Project: Hive
>  Issue Type: Bug
>Reporter: Na Li
>Assignee: Alexander Kolbasov
>Priority: Major
>
>  in HiveMetaStore, alter_table_core does NOT call transactional listener, and 
> the notification ID corresponding to the alter table event is NOT set in the 
> event parameters.
> {code}
> + alter_table_core
>   
>   try {
> Table oldt = this.get_table_core(dbname, name);
> this.firePreEvent(new PreAlterTableEvent(oldt, newTable, this));
> this.alterHandler.alterTable(this.getMS(), this.wh, dbname, name, 
> newTable, envContext, this);
> success = true;
> if (!this.listeners.isEmpty()) {
>   MetaStoreListenerNotifier.notifyEvent(this.listeners, 
> EventType.ALTER_TABLE, new AlterTableEvent(oldt, newTable, true, this), 
> envContext);
> }
>   } catch (NoSuchObjectException var12) {
> ex = var12;
> throw new InvalidOperationException(var12.getMessage());
>   } catch (Exception var13) {
> ex = var13;
> if (var13 instanceof MetaException) {
>   throw (MetaException)var13;
> }
> if (var13 instanceof InvalidOperationException) {
>   throw (InvalidOperationException)var13;
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)