Hi, we are using structured streaming for stream processing and for each message to do some data enrichment i have to lookup data from cassandra and that data in cassandra gets periodically (like once in a day) updated. I want to look at the option of loading it as a dataset and then register it as a temp table and query the required data from that and avoid making a call to cassandra for each message. But, not sure how to refresh the loaded data from cassandra periodically since its gets updated once in a day? I am looking for any suggestions here, thanks.
regards Arvind -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Stream-Processing-how-to-refresh-a-loaded-dataset-periodically-tp28769.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
