RE: FW: Cassandra trigger to send notifications

2016-12-20 Thread Oren Yekutieli 
Got it. Thanks!

Regards,
Oren

From: DuyHai Doan [mailto:doanduy...@gmail.com]
Sent: Tuesday, December 20, 2016 10:11 AM
To: user@cassandra.apache.org
Subject: Re: FW: Cassandra trigger to send notifications

Right now CDC is very very LOW LEVEL, you'll need to

1. Use a CommitLogReader to read the changes from the CDC files
2. Perform de-duplication by yourself

I'll blog post a technical deep dive about CDC in the future once I have time 
to play with it

On Mon, Dec 19, 2016 at 9:57 PM, Oren Yekutieli 
<oren.yekuti...@ecitele.com<mailto:oren.yekuti...@ecitele.com>> wrote:
Thank you both Matija and Eric.
I have read somewhere about CDC but I didn’t know that it is already released.
I also didn’t find good documentation of it except for the design doc and Jira 
ticket, where in both cases it is hard to know what really was released.
As a newbie to Cassandra I will probably do need a more ‘user friendly’ CDC 
documentation. Is it available somewhere?

Regards,
Oren

From: Eric Stevens [mailto:migh...@gmail.com<mailto:migh...@gmail.com>]
Sent: Saturday, December 17, 2016 12:32 AM
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: Re: FW: Cassandra trigger to send notifications


You probably want to look at change data capture rather than triggers: 
http://cassandra.apache.org/doc/latest/operating/cdc.html

Be aware that one of your criteria regarding operation order is going to be 
very difficult to guarantee due to eventual consistency.

On Fri, Dec 16, 2016, 2:43 AM Matija Gobec 
<matija0...@gmail.com<mailto:matija0...@gmail.com>> wrote:
Hi Oren,

I've spent a reasonable time working out triggers and I would say that your 
best bet is doing this in the app.
Just publish a rabbitmq message from the app when you execute a statement.
If your goal is to have an audit then try batch writing data to the tables and 
delta to their audit counterparts.

Matija

On Fri, Dec 16, 2016 at 9:14 AM, Oren Yekutieli 
<oren.yekuti...@ecitele.com<mailto:oren.yekuti...@ecitele.com>> wrote:
Hello,

In a Cassandra cluster I want to push a notification to rabbitmq whenever a 
change (insert/update/delete) was made to some Cassandra tables, with the 
following requirements:
The notifications should:

1.  Be ordered in the same order the changes were stored.

2.  Be sent only if the change was (or going to be) committed successfully.
And in case of insert/update:

3.  Include all the row values.

4.  Indicate what values were changed.

I’m trying to figure out if Cassandra triggers are the correct way to go.
According to 
http://www.datastax.com/dev/blog/whats-new-in-cassandra-2-0-prototype-triggers-support,
 one of the potential use cases for triggers is:
•>> implementing alerts/notifications
I also played a bit with the example given here: 
http://stackoverflow.com/questions/35161747/example-of-creating-triggers-in-cassandra-and-does-this-support-only-for-java
 and from what I read and learned so far I assume that:
Regarding #1, I doubt if the notification order can be kept since the trigger 
is invoked on the coordinator node while there are multiple clients that can 
connect to the various cluster nodes (coordinators).
Regarding #2, I’m not sure that notification will be sent only for changes that 
are going to be committed successfully since the trigger is invoked before the 
change is being applied.
Regarding #3, I seen that this requirement is fulfilled.
Regarding #4, I didn’t find a way to figure out what values are actually 
changed, without re-reading the current (old) values from the table which of 
course impose a significant performance penalty.

Are my assumptions correct?
If yes then triggers are not the right approach. What other ways can be taken?

Regards,
Oren

Re: FW: Cassandra trigger to send notifications

2016-12-20 Thread DuyHai Doan 
Right now CDC is very very LOW LEVEL, you'll need to

1. Use a CommitLogReader to read the changes from the CDC files
2. Perform de-duplication by yourself

I'll blog post a technical deep dive about CDC in the future once I have
time to play with it

On Mon, Dec 19, 2016 at 9:57 PM, Oren Yekutieli <oren.yekuti...@ecitele.com>
wrote:

> Thank you both Matija and Eric.
>
> I have read somewhere about CDC but I didn’t know that it is already
> released.
>
> I also didn’t find good documentation of it except for the design doc and
> Jira ticket, where in both cases it is hard to know what really was
> released.
>
> As a newbie to Cassandra I will probably do need a more ‘user friendly’
> CDC documentation. Is it available somewhere?
>
>
>
> Regards,
>
> Oren
>
>
>
> *From:* Eric Stevens [mailto:migh...@gmail.com]
> *Sent:* Saturday, December 17, 2016 12:32 AM
> *To:* user@cassandra.apache.org
> *Subject:* Re: FW: Cassandra trigger to send notifications
>
>
>
> You probably want to look at change data capture rather than triggers:
> http://cassandra.apache.org/doc/latest/operating/cdc.html
>
> Be aware that one of your criteria regarding operation order is going to
> be very difficult to guarantee due to eventual consistency.
>
>
>
> On Fri, Dec 16, 2016, 2:43 AM Matija Gobec <matija0...@gmail.com> wrote:
>
> Hi Oren,
>
>
>
> I've spent a reasonable time working out triggers and I would say that
> your best bet is doing this in the app.
>
> Just publish a rabbitmq message from the app when you execute a statement.
>
> If your goal is to have an audit then try batch writing data to the tables
> and delta to their audit counterparts.
>
>
>
> Matija
>
>
>
> On Fri, Dec 16, 2016 at 9:14 AM, Oren Yekutieli <
> oren.yekuti...@ecitele.com> wrote:
>
> Hello,
>
>
>
> In a Cassandra cluster I want to push a notification to rabbitmq whenever
> a change (insert/update/delete) was made to some Cassandra tables, with the
> following requirements:
>
> The notifications should:
>
> 1.  Be ordered in the same order the changes were stored.
>
> 2.  Be sent only if the change was (or going to be) committed
> successfully.
>
> And in case of insert/update:
>
> 3.  Include all the row values.
>
> 4.  Indicate what values were changed.
>
>
>
> I’m trying to figure out if Cassandra triggers are the correct way to go.
>
> According to http://www.datastax.com/dev/blog/whats-new-in-cassandra-2-
> 0-prototype-triggers-support, one of the potential use cases for triggers
> is:
>
> ·>> implementing alerts/notifications
>
> I also played a bit with the example given here: http://stackoverflow.com/
> questions/35161747/example-of-creating-triggers-in-
> cassandra-and-does-this-support-only-for-java and from what I read and
> learned so far I assume that:
>
> Regarding #1, I doubt if the notification order can be kept since the
> trigger is invoked on the coordinator node while there are multiple clients
> that can connect to the various cluster nodes (coordinators).
>
> Regarding #2, I’m not sure that notification will be sent only for changes
> that are going to be committed successfully since the trigger is invoked
> before the change is being applied.
>
> Regarding #3, I seen that this requirement is fulfilled.
>
> Regarding #4, I didn’t find a way to figure out what values are actually
> changed, without re-reading the current (old) values from the table which
> of course impose a significant performance penalty.
>
>
>
> Are my assumptions correct?
>
> If yes then triggers are not the right approach. What other ways can be
> taken?
>
>
>
> Regards,
>
> Oren
>
>
>
>
>
>

RE: FW: Cassandra trigger to send notifications

2016-12-19 Thread Oren Yekutieli 
Thank you both Matija and Eric.
I have read somewhere about CDC but I didn’t know that it is already released.
I also didn’t find good documentation of it except for the design doc and Jira 
ticket, where in both cases it is hard to know what really was released.
As a newbie to Cassandra I will probably do need a more ‘user friendly’ CDC 
documentation. Is it available somewhere?

Regards,
Oren

From: Eric Stevens [mailto:migh...@gmail.com]
Sent: Saturday, December 17, 2016 12:32 AM
To: user@cassandra.apache.org
Subject: Re: FW: Cassandra trigger to send notifications


You probably want to look at change data capture rather than triggers: 
http://cassandra.apache.org/doc/latest/operating/cdc.html

Be aware that one of your criteria regarding operation order is going to be 
very difficult to guarantee due to eventual consistency.

On Fri, Dec 16, 2016, 2:43 AM Matija Gobec 
<matija0...@gmail.com<mailto:matija0...@gmail.com>> wrote:
Hi Oren,

I've spent a reasonable time working out triggers and I would say that your 
best bet is doing this in the app.
Just publish a rabbitmq message from the app when you execute a statement.
If your goal is to have an audit then try batch writing data to the tables and 
delta to their audit counterparts.

Matija

On Fri, Dec 16, 2016 at 9:14 AM, Oren Yekutieli 
<oren.yekuti...@ecitele.com<mailto:oren.yekuti...@ecitele.com>> wrote:
Hello,

In a Cassandra cluster I want to push a notification to rabbitmq whenever a 
change (insert/update/delete) was made to some Cassandra tables, with the 
following requirements:
The notifications should:

1.  Be ordered in the same order the changes were stored.

2.  Be sent only if the change was (or going to be) committed successfully.
And in case of insert/update:

3.  Include all the row values.

4.  Indicate what values were changed.

I’m trying to figure out if Cassandra triggers are the correct way to go.
According to 
http://www.datastax.com/dev/blog/whats-new-in-cassandra-2-0-prototype-triggers-support,
 one of the potential use cases for triggers is:
•>> implementing alerts/notifications
I also played a bit with the example given here: 
http://stackoverflow.com/questions/35161747/example-of-creating-triggers-in-cassandra-and-does-this-support-only-for-java
 and from what I read and learned so far I assume that:
Regarding #1, I doubt if the notification order can be kept since the trigger 
is invoked on the coordinator node while there are multiple clients that can 
connect to the various cluster nodes (coordinators).
Regarding #2, I’m not sure that notification will be sent only for changes that 
are going to be committed successfully since the trigger is invoked before the 
change is being applied.
Regarding #3, I seen that this requirement is fulfilled.
Regarding #4, I didn’t find a way to figure out what values are actually 
changed, without re-reading the current (old) values from the table which of 
course impose a significant performance penalty.

Are my assumptions correct?
If yes then triggers are not the right approach. What other ways can be taken?

Regards,
Oren

Re: FW: Cassandra trigger to send notifications

2016-12-16 Thread Eric Stevens 
You probably want to look at change data capture rather than triggers:
http://cassandra.apache.org/doc/latest/operating/cdc.html

Be aware that one of your criteria regarding operation order is going to be
very difficult to guarantee due to eventual consistency.

On Fri, Dec 16, 2016, 2:43 AM Matija Gobec  wrote:

> Hi Oren,
>
> I've spent a reasonable time working out triggers and I would say that
> your best bet is doing this in the app.
> Just publish a rabbitmq message from the app when you execute a statement.
> If your goal is to have an audit then try batch writing data to the tables
> and delta to their audit counterparts.
>
> Matija
>
> On Fri, Dec 16, 2016 at 9:14 AM, Oren Yekutieli <
> oren.yekuti...@ecitele.com> wrote:
>
> Hello,
>
>
>
> In a Cassandra cluster I want to push a notification to rabbitmq whenever
> a change (insert/update/delete) was made to some Cassandra tables, with the
> following requirements:
>
> The notifications should:
>
> 1.  Be ordered in the same order the changes were stored.
>
> 2.  Be sent only if the change was (or going to be) committed
> successfully.
>
> And in case of insert/update:
>
> 3.  Include all the row values.
>
> 4.  Indicate what values were changed.
>
>
>
> I’m trying to figure out if Cassandra triggers are the correct way to go.
>
> According to
> http://www.datastax.com/dev/blog/whats-new-in-cassandra-2-0-prototype-triggers-support,
> one of the potential use cases for triggers is:
>
> ·>> implementing alerts/notifications
>
> I also played a bit with the example given here:
> http://stackoverflow.com/questions/35161747/example-of-creating-triggers-in-cassandra-and-does-this-support-only-for-java
> and from what I read and learned so far I assume that:
>
> Regarding #1, I doubt if the notification order can be kept since the
> trigger is invoked on the coordinator node while there are multiple clients
> that can connect to the various cluster nodes (coordinators).
>
> Regarding #2, I’m not sure that notification will be sent only for changes
> that are going to be committed successfully since the trigger is invoked
> before the change is being applied.
>
> Regarding #3, I seen that this requirement is fulfilled.
>
> Regarding #4, I didn’t find a way to figure out what values are actually
> changed, without re-reading the current (old) values from the table which
> of course impose a significant performance penalty.
>
>
>
> Are my assumptions correct?
>
> If yes then triggers are not the right approach. What other ways can be
> taken?
>
>
>
> Regards,
>
> Oren
>
>
>
>
>

Re: FW: Cassandra trigger to send notifications

2016-12-16 Thread Matija Gobec 
Hi Oren,

I've spent a reasonable time working out triggers and I would say that your
best bet is doing this in the app.
Just publish a rabbitmq message from the app when you execute a statement.
If your goal is to have an audit then try batch writing data to the tables
and delta to their audit counterparts.

Matija

On Fri, Dec 16, 2016 at 9:14 AM, Oren Yekutieli 
wrote:

> Hello,
>
>
>
> In a Cassandra cluster I want to push a notification to rabbitmq whenever
> a change (insert/update/delete) was made to some Cassandra tables, with the
> following requirements:
>
> The notifications should:
>
> 1.  Be ordered in the same order the changes were stored.
>
> 2.  Be sent only if the change was (or going to be) committed
> successfully.
>
> And in case of insert/update:
>
> 3.  Include all the row values.
>
> 4.  Indicate what values were changed.
>
>
>
> I’m trying to figure out if Cassandra triggers are the correct way to go.
>
> According to http://www.datastax.com/dev/blog/whats-new-in-cassandra-2-
> 0-prototype-triggers-support, one of the potential use cases for triggers
> is:
>
> ·>> implementing alerts/notifications
>
> I also played a bit with the example given here: http://stackoverflow.com/
> questions/35161747/example-of-creating-triggers-in-
> cassandra-and-does-this-support-only-for-java and from what I read and
> learned so far I assume that:
>
> Regarding #1, I doubt if the notification order can be kept since the
> trigger is invoked on the coordinator node while there are multiple clients
> that can connect to the various cluster nodes (coordinators).
>
> Regarding #2, I’m not sure that notification will be sent only for changes
> that are going to be committed successfully since the trigger is invoked
> before the change is being applied.
>
> Regarding #3, I seen that this requirement is fulfilled.
>
> Regarding #4, I didn’t find a way to figure out what values are actually
> changed, without re-reading the current (old) values from the table which
> of course impose a significant performance penalty.
>
>
>
> Are my assumptions correct?
>
> If yes then triggers are not the right approach. What other ways can be
> taken?
>
>
>
> Regards,
>
> Oren
>
>
>

FW: Cassandra trigger to send notifications

2016-12-16 Thread Oren Yekutieli 
Hello,

In a Cassandra cluster I want to push a notification to rabbitmq whenever a 
change (insert/update/delete) was made to some Cassandra tables, with the 
following requirements:
The notifications should:

1.  Be ordered in the same order the changes were stored.

2.  Be sent only if the change was (or going to be) committed successfully.
And in case of insert/update:

3.  Include all the row values.

4.  Indicate what values were changed.

I'm trying to figure out if Cassandra triggers are the correct way to go.
According to 
http://www.datastax.com/dev/blog/whats-new-in-cassandra-2-0-prototype-triggers-support,
 one of the potential use cases for triggers is:
*>> implementing alerts/notifications
I also played a bit with the example given here: 
http://stackoverflow.com/questions/35161747/example-of-creating-triggers-in-cassandra-and-does-this-support-only-for-java
 and from what I read and learned so far I assume that:
Regarding #1, I doubt if the notification order can be kept since the trigger 
is invoked on the coordinator node while there are multiple clients that can 
connect to the various cluster nodes (coordinators).
Regarding #2, I'm not sure that notification will be sent only for changes that 
are going to be committed successfully since the trigger is invoked before the 
change is being applied.
Regarding #3, I seen that this requirement is fulfilled.
Regarding #4, I didn't find a way to figure out what values are actually 
changed, without re-reading the current (old) values from the table which of 
course impose a significant performance penalty.

Are my assumptions correct?
If yes then triggers are not the right approach. What other ways can be taken?

Regards,
Oren