This seems a bit overkill. We run far more than 100mps (closer to 600) in
rabbit with very good latency on a 3 node cluster. It has been very reliable as
well.
Thunder
----- Reply message -----
From: "Jagan Ranganathan" <ja...@zohocorp.com>
To: <user@cassandra.apache.org>
Subject: Queuing System
Date: Sat, Feb 22, 2014 9:06 AM
Thanks Tupshin for your assistance. As I mentioned in the other mail, Yes I am
planning to use RabbitMQ for my messaging system. But I wonder which will give
better performance if writing directly into Rabbit with Ack support Vs a
temporary Queue in Cassandra first and then dequeue and publish in Rabbit.I use
Rabbit for Messaging because of the Routing and Push model communication etc.
So I am thinking of using Cassandra as a temporary Queue which will give fast
write performance with no data loss Vs waiting for Rabbit Ack @ application
level or handling Rabbit re-connection Vs Cassandra hinted handoff writes.
So Cassandra might aggregate all my msg queue temporarily before I publish them
to Rabbit. Is this fine? If so, please share your insight on which model &
access pattern will be a better fit for this usage. Throughput requirements may
be around say 100 ops/sec.
Regards,
Jagan
---- On Sat, 22 Feb 2014 21:10:36 +0530 Tupshin Harper<tups...@tupshin.com>
wrote ----
While, historically, it has been true that queuing in Cassandra has been an
anti-pattern, it is also true that Leveled Compaction addresses the worst
aspect of frequent deletes in Cassandra, and that overall, queuing in
Cassandra is nowhere near the anti-pattern that it used to be. This is
something that I've been meaning to write about more extensively.
If your requirements are more around availability (particularly multi-dc)
and relability with moderate (not extreme) performance, it is quite
possible to build a pretty decent system on top of Cassandra. You don't
mention your throughput requirements, nor additional semantics that might
be necessary (e.g. deliver at-least-once vs deliver exactly once), but
Cassandra 2.0's lightweight transactions provide a CAS primitive that
can be used to ensure deliver-once if that is a requirement.
I'd be happy to continue discussing appropriate data-models and access
patterns if you decide to go down this path.
-Tupshin
On Sat, Feb 22, 2014 at 10:03 AM, Jagan Ranganathan
<ja...@zohocorp.com> wrote:
Hi,
I need to decouple some of the work being processed from the
user thread to provide better user experience. For that I
need a queuing system with the following needs,
High Availability No Data Loss
Better Performance.
Following are some libraries that were considered along with
the limitation I see,
Redis - Data Loss ZooKeeper - Not
advised for Queue system.
TokyoCabinet/SQLite/LevelDB - of this Level
DB seem to be performing better. With replication
requirement, I probably have to look at Apache
ActiveMQ+LevelDB.
After checking on the third option above, I kind of wonder if
Cassandra with Leveled Compaction offer a similar system. Do
you see any issues in such a usage or is there other better
solutions available.
Will be great to get insights on this.
Regards,
Jagan
