Re: Requesting Feedback for Cassandra as a backup solution.
Thanks for your valuable reply, will check. Thanks and regards, Gowtham S On Mon, 19 Feb 2024 at 15:46, Bowen Song via user wrote: > You can have a read at > https://www.datastax.com/blog/cassandra-anti-patterns-queues-and-queue-datasets > > Your table schema does not include the most important piece of information > - the partition keys (and clustering keys, if any). Keep in mind that you > can only efficiently query Cassandra by the exact partition key or the > token of a partition key, otherwise you will have to rely on MV or > secondary index, or worse, scan the entire table (all the nodes) to find > your data. > > A Cassandra schema should look like this: > CREATE TABLE xyz ( > a text, > b text, > c timeuuid, > d int, > e text, > PRIMARY KEY ((a, b), c, d) > ); > > The line "PRIMARY KEY" contains arguably the most important piece of > information of the table schema. > > > On 19/02/2024 06:52, Gowtham S wrote: > > Hi Bowen > > which is a well documented anti-pattern. >> > Can you please explain more on this, I'm not aware of it. It will be > helpful to make decisions. > Please find the below table schema > > *Table schema* > TopicName - text > Partition - int > MessageUUID - text > Actual data - text > OccurredTime - Timestamp > Status - boolean > > We are planning to read the table with the topic name and the status is > not true. And produce those to the respective topic when Kafka is live. > > Thanks and regards, > Gowtham S > > > On Sat, 17 Feb 2024 at 18:10, Bowen Song via user < > user@cassandra.apache.org> wrote: > >> Hi Gowtham, >> >> On the face of it, it sounds like you are planning to use Cassandra for a >> queue-like application, which is a well documented anti-pattern. If that's >> not the case, can you please show the table schema and some example queries? >> >> Cheers, >> Bowen >> On 17/02/2024 08:44, Gowtham S wrote: >> >> Dear Cassandra Community, >> >> I am reaching out to seek your valuable feedback and insights on a >> proposed solution we are considering for managing Kafka outages using >> Cassandra. >> >> At our organization, we heavily rely on Kafka for real-time data >> processing and messaging. However, like any technology, Kafka is >> susceptible to occasional outages which can disrupt our operations and >> impact our services. To mitigate the impact of such outages and ensure >> continuity, we are exploring the possibility of leveraging Cassandra as a >> backup solution. >> >> Our proposed approach involves storing messages in Cassandra during Kafka >> outages. Subsequently, we plan to implement a scheduler that will read from >> Cassandra and attempt to write these messages back into Kafka once it is >> operational again. >> >> We believe that by adopting this strategy, we can achieve the following >> benefits: >> >>1. >> >>Improved Fault Tolerance: By having a backup mechanism in place, we >>can reduce the risk of data loss and ensure continuity of operations >> during >>Kafka outages. >>2. >> >>Enhanced Reliability: Cassandra's distributed architecture and >>built-in replication features make it well-suited for storing data >>reliably, even in the face of failures. >>3. >> >>Scalability: Both Cassandra and Kafka are designed to scale >>horizontally, allowing us to handle increased loads seamlessly. >> >> Before proceeding further with this approach, we would greatly appreciate >> any feedback, suggestions, or concerns from the community. Specifically, we >> are interested in hearing about: >> >>- Potential challenges or drawbacks of using Cassandra as a backup >>solution for Kafka outages. >>- Best practices or recommendations for implementing such a backup >>mechanism effectively. >>- Any alternative approaches or technologies that we should consider? >> >> Your expertise and insights are invaluable to us, and we are eager to >> learn from your experiences and perspectives. Please feel free to share >> your thoughts or reach out to us with any questions or clarifications. >> >> Thank you for taking the time to consider our proposal, and we look >> forward to hearing from you soon. >> Thanks and regards, >> Gowtham S >> >>
Re: Requesting Feedback for Cassandra as a backup solution.
You can have a read at https://www.datastax.com/blog/cassandra-anti-patterns-queues-and-queue-datasets Your table schema does not include the most important piece of information - the partition keys (and clustering keys, if any). Keep in mind that you can only efficiently query Cassandra by the exact partition key or the token of a partition key, otherwise you will have to rely on MV or secondary index, or worse, scan the entire table (all the nodes) to find your data. A Cassandra schema should look like this: CREATE TABLE xyz ( a text, b text, c timeuuid, d int, e text, PRIMARY KEY ((a, b), c, d) ); The line "PRIMARY KEY" contains arguably the most important piece of information of the table schema. On 19/02/2024 06:52, Gowtham S wrote: Hi Bowen which is a well documented anti-pattern. Can you please explain more on this, I'm not aware of it. It will be helpful to make decisions. Please find the below table schema *Table schema* TopicName - text Partition - int MessageUUID - text Actual data - text OccurredTime - Timestamp Status - boolean We are planning to read the table with the topic name and the status is not true. And produce those to the respective topic when Kafka is live. Thanks and regards, Gowtham S On Sat, 17 Feb 2024 at 18:10, Bowen Song via user wrote: Hi Gowtham, On the face of it, it sounds like you are planning to use Cassandra for a queue-like application, which is a well documented anti-pattern. If that's not the case, can you please show the table schema and some example queries? Cheers, Bowen On 17/02/2024 08:44, Gowtham S wrote: Dear Cassandra Community, I am reaching out to seek your valuable feedback and insights on a proposed solution we are considering for managing Kafka outages using Cassandra. At our organization, we heavily rely on Kafka for real-time data processing and messaging. However, like any technology, Kafka is susceptible to occasional outages which can disrupt our operations and impact our services. To mitigate the impact of such outages and ensure continuity, we are exploring the possibility of leveraging Cassandra as a backup solution. Our proposed approach involves storing messages in Cassandra during Kafka outages. Subsequently, we plan to implement a scheduler that will read from Cassandra and attempt to write these messages back into Kafka once it is operational again. We believe that by adopting this strategy, we can achieve the following benefits: 1. Improved Fault Tolerance: By having a backup mechanism in place, we can reduce the risk of data loss and ensure continuity of operations during Kafka outages. 2. Enhanced Reliability: Cassandra's distributed architecture and built-in replication features make it well-suited for storing data reliably, even in the face of failures. 3. Scalability: Both Cassandra and Kafka are designed to scale horizontally, allowing us to handle increased loads seamlessly. Before proceeding further with this approach, we would greatly appreciate any feedback, suggestions, or concerns from the community. Specifically, we are interested in hearing about: * Potential challenges or drawbacks of using Cassandra as a backup solution for Kafka outages. * Best practices or recommendations for implementing such a backup mechanism effectively. * Any alternative approaches or technologies that we should consider? Your expertise and insights are invaluable to us, and we are eager to learn from your experiences and perspectives. Please feel free to share your thoughts or reach out to us with any questions or clarifications. Thank you for taking the time to consider our proposal, and we look forward to hearing from you soon. Thanks and regards, Gowtham S
Re: Requesting Feedback for Cassandra as a backup solution.
Hi Bowen which is a well documented anti-pattern. > Can you please explain more on this, I'm not aware of it. It will be helpful to make decisions. Please find the below table schema *Table schema* TopicName - text Partition - int MessageUUID - text Actual data - text OccurredTime - Timestamp Status - boolean We are planning to read the table with the topic name and the status is not true. And produce those to the respective topic when Kafka is live. Thanks and regards, Gowtham S On Sat, 17 Feb 2024 at 18:10, Bowen Song via user wrote: > Hi Gowtham, > > On the face of it, it sounds like you are planning to use Cassandra for a > queue-like application, which is a well documented anti-pattern. If that's > not the case, can you please show the table schema and some example queries? > > Cheers, > Bowen > On 17/02/2024 08:44, Gowtham S wrote: > > Dear Cassandra Community, > > I am reaching out to seek your valuable feedback and insights on a > proposed solution we are considering for managing Kafka outages using > Cassandra. > > At our organization, we heavily rely on Kafka for real-time data > processing and messaging. However, like any technology, Kafka is > susceptible to occasional outages which can disrupt our operations and > impact our services. To mitigate the impact of such outages and ensure > continuity, we are exploring the possibility of leveraging Cassandra as a > backup solution. > > Our proposed approach involves storing messages in Cassandra during Kafka > outages. Subsequently, we plan to implement a scheduler that will read from > Cassandra and attempt to write these messages back into Kafka once it is > operational again. > > We believe that by adopting this strategy, we can achieve the following > benefits: > >1. > >Improved Fault Tolerance: By having a backup mechanism in place, we >can reduce the risk of data loss and ensure continuity of operations during >Kafka outages. >2. > >Enhanced Reliability: Cassandra's distributed architecture and >built-in replication features make it well-suited for storing data >reliably, even in the face of failures. >3. > >Scalability: Both Cassandra and Kafka are designed to scale >horizontally, allowing us to handle increased loads seamlessly. > > Before proceeding further with this approach, we would greatly appreciate > any feedback, suggestions, or concerns from the community. Specifically, we > are interested in hearing about: > >- Potential challenges or drawbacks of using Cassandra as a backup >solution for Kafka outages. >- Best practices or recommendations for implementing such a backup >mechanism effectively. >- Any alternative approaches or technologies that we should consider? > > Your expertise and insights are invaluable to us, and we are eager to > learn from your experiences and perspectives. Please feel free to share > your thoughts or reach out to us with any questions or clarifications. > > Thank you for taking the time to consider our proposal, and we look > forward to hearing from you soon. > Thanks and regards, > Gowtham S > >
Re: Requesting Feedback for Cassandra as a backup solution.
TBH, this sounds to me like a very expensive (in terms of effort) way to deal with whatever Kafka unreliability you’re having. We have lots of both Kafka and Cassandra clusters under management and I have no doubt that Kafka is capable of being as reliable as Cassandra (and both are capable of achieving 99.99%+ availability) and, if anything, is easier to achieve that reliability with Kafka. Adding an additional distributed tech to manage is a whole lot of new learning if you’re not already expert at it. I think someone else suggest just running parallel Kafka cluster – I’ve certainly seen that be succesful. However, a really good recommendation probably requires a bit more understand of just what kind of issues you’re worried about with Kafka. Cheers Ben From: Bowen Song via user Date: Saturday, 17 February 2024 at 23:40 To: user@cassandra.apache.org Cc: Bowen Song Subject: Re: Requesting Feedback for Cassandra as a backup solution. EXTERNAL EMAIL - USE CAUTION when clicking links or attachments Hi Gowtham, On the face of it, it sounds like you are planning to use Cassandra for a queue-like application, which is a well documented anti-pattern. If that's not the case, can you please show the table schema and some example queries? Cheers, Bowen On 17/02/2024 08:44, Gowtham S wrote: Dear Cassandra Community, I am reaching out to seek your valuable feedback and insights on a proposed solution we are considering for managing Kafka outages using Cassandra. At our organization, we heavily rely on Kafka for real-time data processing and messaging. However, like any technology, Kafka is susceptible to occasional outages which can disrupt our operations and impact our services. To mitigate the impact of such outages and ensure continuity, we are exploring the possibility of leveraging Cassandra as a backup solution. Our proposed approach involves storing messages in Cassandra during Kafka outages. Subsequently, we plan to implement a scheduler that will read from Cassandra and attempt to write these messages back into Kafka once it is operational again. We believe that by adopting this strategy, we can achieve the following benefits: 1. Improved Fault Tolerance: By having a backup mechanism in place, we can reduce the risk of data loss and ensure continuity of operations during Kafka outages. 2. Enhanced Reliability: Cassandra's distributed architecture and built-in replication features make it well-suited for storing data reliably, even in the face of failures. 3. Scalability: Both Cassandra and Kafka are designed to scale horizontally, allowing us to handle increased loads seamlessly. Before proceeding further with this approach, we would greatly appreciate any feedback, suggestions, or concerns from the community. Specifically, we are interested in hearing about: * Potential challenges or drawbacks of using Cassandra as a backup solution for Kafka outages. * Best practices or recommendations for implementing such a backup mechanism effectively. * Any alternative approaches or technologies that we should consider? Your expertise and insights are invaluable to us, and we are eager to learn from your experiences and perspectives. Please feel free to share your thoughts or reach out to us with any questions or clarifications. Thank you for taking the time to consider our proposal, and we look forward to hearing from you soon. Thanks and regards, Gowtham S
Re: Requesting Feedback for Cassandra as a backup solution.
Hi Gowtham, On the face of it, it sounds like you are planning to use Cassandra for a queue-like application, which is a well documented anti-pattern. If that's not the case, can you please show the table schema and some example queries? Cheers, Bowen On 17/02/2024 08:44, Gowtham S wrote: Dear Cassandra Community, I am reaching out to seek your valuable feedback and insights on a proposed solution we are considering for managing Kafka outages using Cassandra. At our organization, we heavily rely on Kafka for real-time data processing and messaging. However, like any technology, Kafka is susceptible to occasional outages which can disrupt our operations and impact our services. To mitigate the impact of such outages and ensure continuity, we are exploring the possibility of leveraging Cassandra as a backup solution. Our proposed approach involves storing messages in Cassandra during Kafka outages. Subsequently, we plan to implement a scheduler that will read from Cassandra and attempt to write these messages back into Kafka once it is operational again. We believe that by adopting this strategy, we can achieve the following benefits: 1. Improved Fault Tolerance: By having a backup mechanism in place, we can reduce the risk of data loss and ensure continuity of operations during Kafka outages. 2. Enhanced Reliability: Cassandra's distributed architecture and built-in replication features make it well-suited for storing data reliably, even in the face of failures. 3. Scalability: Both Cassandra and Kafka are designed to scale horizontally, allowing us to handle increased loads seamlessly. Before proceeding further with this approach, we would greatly appreciate any feedback, suggestions, or concerns from the community. Specifically, we are interested in hearing about: * Potential challenges or drawbacks of using Cassandra as a backup solution for Kafka outages. * Best practices or recommendations for implementing such a backup mechanism effectively. * Any alternative approaches or technologies that we should consider? Your expertise and insights are invaluable to us, and we are eager to learn from your experiences and perspectives. Please feel free to share your thoughts or reach out to us with any questions or clarifications. Thank you for taking the time to consider our proposal, and we look forward to hearing from you soon. Thanks and regards, Gowtham S
Re: Requesting Feedback for Cassandra as a backup solution.
Thanks for your suggestion Thanks and regards, Gowtham S On Sat, 17 Feb 2024 at 14:58, CPC wrote: > hi, > > We implemented same strategy in one of our customers. Since 2016 we had > one downtime in one DC because of high temperature(whole physical DC > shutdown). > > With that approach I assume you will use Cassandra as a queue. You have to > be careful about modeling and should use multiple partitions may be based > on hour or fixed size partitions. > > Another thing is that Kafka has really high throughput so you should plan > how many Cassandra node you need to meet same throughput. > > Another approach would be to use another Kafka cluster or queue technology > as backup. > > > > On Sat, Feb 17, 2024, 11:45 AM Gowtham S wrote: > >> Dear Cassandra Community, >> >> I am reaching out to seek your valuable feedback and insights on a >> proposed solution we are considering for managing Kafka outages using >> Cassandra. >> >> At our organization, we heavily rely on Kafka for real-time data >> processing and messaging. However, like any technology, Kafka is >> susceptible to occasional outages which can disrupt our operations and >> impact our services. To mitigate the impact of such outages and ensure >> continuity, we are exploring the possibility of leveraging Cassandra as a >> backup solution. >> >> Our proposed approach involves storing messages in Cassandra during Kafka >> outages. Subsequently, we plan to implement a scheduler that will read from >> Cassandra and attempt to write these messages back into Kafka once it is >> operational again. >> >> We believe that by adopting this strategy, we can achieve the following >> benefits: >> >>1. >> >>Improved Fault Tolerance: By having a backup mechanism in place, we >>can reduce the risk of data loss and ensure continuity of operations >> during >>Kafka outages. >>2. >> >>Enhanced Reliability: Cassandra's distributed architecture and >>built-in replication features make it well-suited for storing data >>reliably, even in the face of failures. >>3. >> >>Scalability: Both Cassandra and Kafka are designed to scale >>horizontally, allowing us to handle increased loads seamlessly. >> >> Before proceeding further with this approach, we would greatly appreciate >> any feedback, suggestions, or concerns from the community. Specifically, we >> are interested in hearing about: >> >>- Potential challenges or drawbacks of using Cassandra as a backup >>solution for Kafka outages. >>- Best practices or recommendations for implementing such a backup >>mechanism effectively. >>- Any alternative approaches or technologies that we should consider? >> >> Your expertise and insights are invaluable to us, and we are eager to >> learn from your experiences and perspectives. Please feel free to share >> your thoughts or reach out to us with any questions or clarifications. >> >> Thank you for taking the time to consider our proposal, and we look >> forward to hearing from you soon. >> Thanks and regards, >> Gowtham S >> >
Re: Requesting Feedback for Cassandra as a backup solution.
hi, We implemented same strategy in one of our customers. Since 2016 we had one downtime in one DC because of high temperature(whole physical DC shutdown). With that approach I assume you will use Cassandra as a queue. You have to be careful about modeling and should use multiple partitions may be based on hour or fixed size partitions. Another thing is that Kafka has really high throughput so you should plan how many Cassandra node you need to meet same throughput. Another approach would be to use another Kafka cluster or queue technology as backup. On Sat, Feb 17, 2024, 11:45 AM Gowtham S wrote: > Dear Cassandra Community, > > I am reaching out to seek your valuable feedback and insights on a > proposed solution we are considering for managing Kafka outages using > Cassandra. > > At our organization, we heavily rely on Kafka for real-time data > processing and messaging. However, like any technology, Kafka is > susceptible to occasional outages which can disrupt our operations and > impact our services. To mitigate the impact of such outages and ensure > continuity, we are exploring the possibility of leveraging Cassandra as a > backup solution. > > Our proposed approach involves storing messages in Cassandra during Kafka > outages. Subsequently, we plan to implement a scheduler that will read from > Cassandra and attempt to write these messages back into Kafka once it is > operational again. > > We believe that by adopting this strategy, we can achieve the following > benefits: > >1. > >Improved Fault Tolerance: By having a backup mechanism in place, we >can reduce the risk of data loss and ensure continuity of operations during >Kafka outages. >2. > >Enhanced Reliability: Cassandra's distributed architecture and >built-in replication features make it well-suited for storing data >reliably, even in the face of failures. >3. > >Scalability: Both Cassandra and Kafka are designed to scale >horizontally, allowing us to handle increased loads seamlessly. > > Before proceeding further with this approach, we would greatly appreciate > any feedback, suggestions, or concerns from the community. Specifically, we > are interested in hearing about: > >- Potential challenges or drawbacks of using Cassandra as a backup >solution for Kafka outages. >- Best practices or recommendations for implementing such a backup >mechanism effectively. >- Any alternative approaches or technologies that we should consider? > > Your expertise and insights are invaluable to us, and we are eager to > learn from your experiences and perspectives. Please feel free to share > your thoughts or reach out to us with any questions or clarifications. > > Thank you for taking the time to consider our proposal, and we look > forward to hearing from you soon. > Thanks and regards, > Gowtham S >
Requesting Feedback for Cassandra as a backup solution.
Dear Cassandra Community, I am reaching out to seek your valuable feedback and insights on a proposed solution we are considering for managing Kafka outages using Cassandra. At our organization, we heavily rely on Kafka for real-time data processing and messaging. However, like any technology, Kafka is susceptible to occasional outages which can disrupt our operations and impact our services. To mitigate the impact of such outages and ensure continuity, we are exploring the possibility of leveraging Cassandra as a backup solution. Our proposed approach involves storing messages in Cassandra during Kafka outages. Subsequently, we plan to implement a scheduler that will read from Cassandra and attempt to write these messages back into Kafka once it is operational again. We believe that by adopting this strategy, we can achieve the following benefits: 1. Improved Fault Tolerance: By having a backup mechanism in place, we can reduce the risk of data loss and ensure continuity of operations during Kafka outages. 2. Enhanced Reliability: Cassandra's distributed architecture and built-in replication features make it well-suited for storing data reliably, even in the face of failures. 3. Scalability: Both Cassandra and Kafka are designed to scale horizontally, allowing us to handle increased loads seamlessly. Before proceeding further with this approach, we would greatly appreciate any feedback, suggestions, or concerns from the community. Specifically, we are interested in hearing about: - Potential challenges or drawbacks of using Cassandra as a backup solution for Kafka outages. - Best practices or recommendations for implementing such a backup mechanism effectively. - Any alternative approaches or technologies that we should consider? Your expertise and insights are invaluable to us, and we are eager to learn from your experiences and perspectives. Please feel free to share your thoughts or reach out to us with any questions or clarifications. Thank you for taking the time to consider our proposal, and we look forward to hearing from you soon. Thanks and regards, Gowtham S
