Re: hooks in metastore functions
Cool. I will prepare a patch and post on HIVE-2038. Ashutosh On Wed, Mar 9, 2011 at 16:59, John Sichi wrote: > Yeah, thinking about it more, they're likely to end up looking different. > The listener should cover most possible repository changes, whereas > HiveMetaHook is focused on a narrower set of object definitions. > > JVS > > On Mar 9, 2011, at 1:48 PM, Ashutosh Chauhan wrote: > >> It might be possible to extend and modify the HiveMetaHook interface. >> But, I think keeping them separate is better because MetaHook and >> MetaStoreListener are interfaces for two different functionalities. >> MetaHook is for communicating with external system if there is a need >> for it. MetaStoreListener observe changes on metastore and run some >> logic in response to those changes. What do you think? >> >> Ashutosh >> >> On Wed, Mar 9, 2011 at 13:36, John Sichi wrote: >>> Couldn't we reuse HiveMetaHook for this new purpose (with an instance >>> loaded via global config vs associated with the table handler)? >>> >>> JVS >>> >>> On Mar 8, 2011, at 2:12 PM, Ashutosh Chauhan wrote: >>> Hi all, I have a requirement that every time some change on metastore takes place, we have some logic which needs to be run. For example, if a new table is getting created in metastore I want to send a message to a message bus. Easiest way for this to work is to add the logic in createTable(). Control it by a hiveConf param and turn it off by default. Alternative way is via hooks. Have this extra logic in hook and then load and fire the hook if its available. Does anyone has an opinion which of these two is preferable. Second one requires new hook loading and execution logic. I am currently interested in four functions: createTable() dropTable() addPartition() dropPartition(). Current, HiveMetaHook which exists in createTable() doesn't perfectly fit the bill, since it is fired only when user expresses it in his create table statement (i.e., if he has specified a storage handler) Instead I want to have this logic always run. If it is unclear, let me know, I can post the code which can demonstrate my usecase. Ashutosh >>> >>> > >
Re: hooks in metastore functions
Yeah, thinking about it more, they're likely to end up looking different. The listener should cover most possible repository changes, whereas HiveMetaHook is focused on a narrower set of object definitions. JVS On Mar 9, 2011, at 1:48 PM, Ashutosh Chauhan wrote: > It might be possible to extend and modify the HiveMetaHook interface. > But, I think keeping them separate is better because MetaHook and > MetaStoreListener are interfaces for two different functionalities. > MetaHook is for communicating with external system if there is a need > for it. MetaStoreListener observe changes on metastore and run some > logic in response to those changes. What do you think? > > Ashutosh > > On Wed, Mar 9, 2011 at 13:36, John Sichi wrote: >> Couldn't we reuse HiveMetaHook for this new purpose (with an instance loaded >> via global config vs associated with the table handler)? >> >> JVS >> >> On Mar 8, 2011, at 2:12 PM, Ashutosh Chauhan wrote: >> >>> Hi all, >>> >>> I have a requirement that every time some change on metastore takes >>> place, we have some logic which needs to be run. For example, if a new >>> table is getting created in metastore I want to send a message to a >>> message bus. Easiest way for this to work is to add the logic in >>> createTable(). Control it by a hiveConf param and turn it off by >>> default. Alternative way is via hooks. Have this extra logic in hook >>> and then load and fire the hook if its available. Does anyone has an >>> opinion which of these two is preferable. Second one requires new hook >>> loading and execution logic. I am currently interested in four >>> functions: createTable() dropTable() addPartition() dropPartition(). >>> Current, HiveMetaHook which exists in createTable() doesn't perfectly >>> fit the bill, since it is fired only when user expresses it in his >>> create table statement (i.e., if he has specified a storage handler) >>> Instead I want to have this logic always run. >>> If it is unclear, let me know, I can post the code which can >>> demonstrate my usecase. >>> >>> Ashutosh >> >>
Re: hooks in metastore functions
It might be possible to extend and modify the HiveMetaHook interface. But, I think keeping them separate is better because MetaHook and MetaStoreListener are interfaces for two different functionalities. MetaHook is for communicating with external system if there is a need for it. MetaStoreListener observe changes on metastore and run some logic in response to those changes. What do you think? Ashutosh On Wed, Mar 9, 2011 at 13:36, John Sichi wrote: > Couldn't we reuse HiveMetaHook for this new purpose (with an instance loaded > via global config vs associated with the table handler)? > > JVS > > On Mar 8, 2011, at 2:12 PM, Ashutosh Chauhan wrote: > >> Hi all, >> >> I have a requirement that every time some change on metastore takes >> place, we have some logic which needs to be run. For example, if a new >> table is getting created in metastore I want to send a message to a >> message bus. Easiest way for this to work is to add the logic in >> createTable(). Control it by a hiveConf param and turn it off by >> default. Alternative way is via hooks. Have this extra logic in hook >> and then load and fire the hook if its available. Does anyone has an >> opinion which of these two is preferable. Second one requires new hook >> loading and execution logic. I am currently interested in four >> functions: createTable() dropTable() addPartition() dropPartition(). >> Current, HiveMetaHook which exists in createTable() doesn't perfectly >> fit the bill, since it is fired only when user expresses it in his >> create table statement (i.e., if he has specified a storage handler) >> Instead I want to have this logic always run. >> If it is unclear, let me know, I can post the code which can >> demonstrate my usecase. >> >> Ashutosh > >
Re: hooks in metastore functions
Couldn't we reuse HiveMetaHook for this new purpose (with an instance loaded via global config vs associated with the table handler)? JVS On Mar 8, 2011, at 2:12 PM, Ashutosh Chauhan wrote: > Hi all, > > I have a requirement that every time some change on metastore takes > place, we have some logic which needs to be run. For example, if a new > table is getting created in metastore I want to send a message to a > message bus. Easiest way for this to work is to add the logic in > createTable(). Control it by a hiveConf param and turn it off by > default. Alternative way is via hooks. Have this extra logic in hook > and then load and fire the hook if its available. Does anyone has an > opinion which of these two is preferable. Second one requires new hook > loading and execution logic. I am currently interested in four > functions: createTable() dropTable() addPartition() dropPartition(). > Current, HiveMetaHook which exists in createTable() doesn't perfectly > fit the bill, since it is fired only when user expresses it in his > create table statement (i.e., if he has specified a storage handler) > Instead I want to have this logic always run. > If it is unclear, let me know, I can post the code which can > demonstrate my usecase. > > Ashutosh
