Re: Will this use-case can be handled with spark-sql streaming and cassandra?
1) this is not a use case, but a technical solution. Hence nobody can tell you if it make sense or not 2) do an upsert in Cassandra. However keep in mind that the application submitting to the Kafka topic and the one consuming from the Kafka topic need to ensure that they process messages in the right order. This may not be always guaranteed, eg in case of errors, and they need to avoid overwriting new data with old data. This is also not a Kafka setting that has to be dealt with at producer and consumer level > Am 29.08.2019 um 13:21 schrieb Shyam P : > > Hi, > I need to do a PoC for a business use-case. > > Use case : Need to update a record in Cassandra table if exists. > > Will spark streaming support compare each record and update existing > Cassandra record ? > > For each record received from kakfa topic , If I want to check and compare > each record whether its already there in Cassandra or not , if yes , update > the record else insert a new record. > > How can be this done using spark-structured streaming and cassandra? any > snippet or sample if you have. > > Thank you, > > Shyam
Re: Will this use-case can be handled with spark-sql streaming and cassandra?
What exactly is your requirement? Is the read before write mandatory? Are you maintaining states in Cassandra? Regards Prathmesh Ranaut https://linkedin.com/in/prathmeshranaut > On Aug 29, 2019, at 3:35 PM, Shyam P wrote: > > > thanks Aayush. For every record I need to get the data from cassandra > table and update it ? Else it may not update the existing record. > > What is this datastax-spark-connector ? is that not a "Cassandra > connector library written for spark"? > If not , how to write ourselves. > Where and how to start ? Can you please guide me. > > > > Thank you. > Shyam > > > > > On Thu, Aug 29, 2019 at 5:03 PM Aayush Ranaut > wrote: > >> Cassandra is upsert, you should be able to do what you need with a single >> statement unless you’re looking to maintain counters. >> >> I’m not sure if there is a Cassandra connector library written for spark >> streaming because we wrote one ourselves when we wanted to do the same. >> >> Regards >> Prathmesh Ranaut >> https://linkedin.com/in/prathmeshranaut >> >> >> On Aug 29, 2019, at 7:21 AM, Shyam P >> wrote: >> >> >>> Hi, >>> I need to do a PoC for a business use-case. >>> >>> Use case : Need to update a record in Cassandra table if exists. >>> >>> Will spark streaming support compare each record and update existing >>> Cassandra record ? >>> >>> For each record received from kakfa topic , If I want to check and compare >>> each record whether its already there in Cassandra or not , if yes , update >>> the record else insert a new record. >>> >>> How can be this done using spark-structured streaming and cassandra? any >>> snippet or sample if you have. >>> >>> Thank you, >>> >>> Shyam >>> >>> >>> >>> >>> >> >> >
Re: Will this use-case can be handled with spark-sql streaming and cassandra?
thanks Aayush. For every record I need to get the data from cassandra table and update it ? Else it may not update the existing record. What is this datastax-spark-connector ? is that not a "Cassandra connector library written for spark"? If not , how to write ourselves. Where and how to start ? Can you please guide me. Thank you. Shyam On Thu, Aug 29, 2019 at 5:03 PM Aayush Ranaut wrote: > Cassandra is upsert, you should be able to do what you need with a single > statement unless you’re looking to maintain counters. > > I’m not sure if there is a Cassandra connector library written for spark > streaming because we wrote one ourselves when we wanted to do the same. > > Regards > Prathmesh Ranaut > https://linkedin.com/in/prathmeshranaut > > On Aug 29, 2019, at 7:21 AM, Shyam P wrote: > > Hi, > > I need to do a PoC for a business use-case. > > *Use case :* Need to update a record in Cassandra table if exists. > > Will spark streaming support compare each record and update existing > Cassandra record ? > > For each record received from kakfa topic , If I want to check and compare > each record whether its already there in Cassandra or not , if yes , update > the record else insert a new record. > > How can be this done using spark-structured streaming and cassandra? any > snippet or sample if you have. > > Thank you, > > Shyam > >
Re: Will this use-case can be handled with spark-sql streaming and cassandra?
Cassandra is upsert, you should be able to do what you need with a single statement unless you’re looking to maintain counters. I’m not sure if there is a Cassandra connector library written for spark streaming because we wrote one ourselves when we wanted to do the same. Regards Prathmesh Ranaut https://linkedin.com/in/prathmeshranaut > On Aug 29, 2019, at 7:21 AM, Shyam P wrote: > > Hi, > I need to do a PoC for a business use-case. > > Use case : Need to update a record in Cassandra table if exists. > > Will spark streaming support compare each record and update existing > Cassandra record ? > > For each record received from kakfa topic , If I want to check and compare > each record whether its already there in Cassandra or not , if yes , update > the record else insert a new record. > > How can be this done using spark-structured streaming and cassandra? any > snippet or sample if you have. > > Thank you, > > Shyam
Will this use-case can be handled with spark-sql streaming and cassandra?
Hi, I need to do a PoC for a business use-case. *Use case :* Need to update a record in Cassandra table if exists. Will spark streaming support compare each record and update existing Cassandra record ? For each record received from kakfa topic , If I want to check and compare each record whether its already there in Cassandra or not , if yes , update the record else insert a new record. How can be this done using spark-structured streaming and cassandra? any snippet or sample if you have. Thank you, Shyam
