Yes, you can certainly use spark streaming, but reading from the original source table may still be time consuming and resource intensive.
Having some context on the RDBMS platform, data size/volumes involved and the tolerable lag (between changes being created and it being processed by Spark) will help people give you better recommendations/best practices. All the same, one approach is to create triggers on the source table and insert data into a different table and then read from there. Another approach is to push the delta data into something like Kafka and then use Spark streaming against that. Taking that Kafka approach further, you can capture the delta upstream so that the processing that pushes it into the RDBMS can also push it to Kafka directly. On 12/27/18, 4:52 PM, "Nicolas Paris" <[email protected]> wrote: Hi I have this living RDBMS and I d'like to apply a spark job on several tables once new data get in. I could run batch spark jobs thought cron jobs every minutes. But the job takes time and resources to begin (sparkcontext, yarn....) I wonder if I could run one instance of a spark streaming job to save those resources. However I haven't seen about structured streaming from jdbc source in the documentation. Any recommendation ? -- nicolas --------------------------------------------------------------------- To unsubscribe e-mail: [email protected]
