On Wed, Jun 8, 2016 at 2:38 AM, Mohit Anchlia <[email protected]> wrote: > I am looking to write an ETL job using spark that reads data from the > source, perform transformation and insert it into the destination.
Is this going to be one-time job or you want it to run every time interval? > 1. Source becomes slow or un-responsive. How to control such a situation so > that it doesn't cause DDoS on the source? Why do you think Spark would DDoS the source? I'm reading it as if Spark tried to open a new connection after the currently-open one became slow. I don't think it's how Spark does connections. What is the source in your use case? > Also, at the same time how to make it resilient that it does pick up from > where it left? It sounds like checkpointing. It's available in Core and Streaming. So, what's your source and how often do you want to query for data? You may also benefit from the recent additions to Spark in 2.0 called Structured Streaming (aka Streaming Datasets) - see https://issues.apache.org/jira/browse/SPARK-8360. > 2. In the same context when destination becomes slow or un-responsive. What is a destination? It appears as if you were doing streaming and want to use checkpointing and back-pressure. But you haven't said much about your use case to be precise. Jacek --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
