Re: Best approach in Cassandra (+ Spark?) for Continuous Queries?

2015-01-04 Thread Hugo José Pinto
Many thanks once again. I rethought the target data structure, and things started coming together to allow for really elegant, compact ESP preprocessing and storage. Best. Enviado do meu iPhone No dia 03/01/2015, às 23:53, Peter Lin wool...@gmail.com escreveu: if you like SQL dialect, try

Re: Best approach in Cassandra (+ Spark?) for Continuous Queries?

2015-01-03 Thread Peter Lin
It looks like you're using the wrong tool and architecture. If the use case really needs continuous query like event processing, use an ESP product to do that. You can still store data in Cassandra for persistence . The design you want is to have two paths: event stream and persistence. At the

Best approach in Cassandra (+ Spark?) for Continuous Queries?

2015-01-03 Thread Hugo José Pinto
Hello. We're currently using Hazelcast (http://hazelcast.org/) as a distributed in-memory data grid. That's been working sort-of-well for us, but going solely in-memory has exhausted its path in our use case, and we're considering porting our application to a NoSQL persistent store. After the

Re: Best approach in Cassandra (+ Spark?) for Continuous Queries?

2015-01-03 Thread DuyHai Doan
Hello Hugo I was facing the same kind of requirement from some users. Long story short, below are the possible strategies with advantages and draw-backs of each 1) Put Spark in front of the back-end, every incoming modification/update/insert goes into Spark first, then Spark will forward it to

Re: Best approach in Cassandra (+ Spark?) for Continuous Queries?

2015-01-03 Thread Jabbar Azam
Hello, Or you can have a look at akka http://www.akka.io for event processing and use cassandra for persistence(Peters suggestion). On Sat Jan 03 2015 at 11:59:45 AM Peter Lin wool...@gmail.com wrote: It looks like you're using the wrong tool and architecture. If the use case really needs

Re: Best approach in Cassandra (+ Spark?) for Continuous Queries?

2015-01-03 Thread Hugo José Pinto
Thank you all for your answers. It seems I'll have to go with some event-driven processing before/during the Cassandra write path. My concern would be that I'd love to first guarantee the disk write of the Cassandra persistence and then do the event processing (which is mostly CRUD

Re: Best approach in Cassandra (+ Spark?) for Continuous Queries?

2015-01-03 Thread Colin
Use a message bus with a transactional get, get the message, send to cassandra, upon write success, submit to esp, commit get on bus. Messaging systems like rabbitmq support this semantic. Using cassandra as a queuing mechanism is an anti-pattern. -- Colin Clark +1-320-221-9531 On Jan 3,

Re: Best approach in Cassandra (+ Spark?) for Continuous Queries?

2015-01-03 Thread Peter Lin
listen to colin's advice, avoid the temptation of anti-patterns. On Sat, Jan 3, 2015 at 6:10 PM, Colin colpcl...@gmail.com wrote: Use a message bus with a transactional get, get the message, send to cassandra, upon write success, submit to esp, commit get on bus. Messaging systems like

Re: Best approach in Cassandra (+ Spark?) for Continuous Queries?

2015-01-03 Thread Hugo José Pinto
Thanks :) Duly noted - this is all uncharted territory for us, hence the value of seasoned advice. Best -- Hugo José Pinto No dia 03/01/2015, às 23:43, Peter Lin wool...@gmail.com escreveu: listen to colin's advice, avoid the temptation of anti-patterns. On Sat, Jan 3, 2015 at 6:10

Re: Best approach in Cassandra (+ Spark?) for Continuous Queries?

2015-01-03 Thread Peter Lin
if you like SQL dialect, try out products that use streamSQL to do continuous queries. Espers comes to mind. Google to see what other products support streamSQL On Sat, Jan 3, 2015 at 6:48 PM, Hugo José Pinto hugo.pi...@inovaworks.com wrote: Thanks :) Duly noted - this is all uncharted