Re: Local spark context on an executor

2017-03-22 Thread Shashank Mandil
Sqoop doesn't work on sharded database. Thanks, Shashank On Wed, Mar 22, 2017 at 5:43 AM Reynier González Tejeda wrote: > Why are you using spark instead of sqoop? > > 2017-03-21 21:29 GMT-03:00 ayan guha : > > For JDBC to work, you can start

Re: Local spark context on an executor

2017-03-22 Thread Reynier González Tejeda
Why are you using spark instead of sqoop? 2017-03-21 21:29 GMT-03:00 ayan guha : > For JDBC to work, you can start spark-submit with appropriate jdbc driver > jars (using --jars), then you will have the driver available on executors. > > For acquiring connections, create a

Re: Local spark context on an executor

2017-03-21 Thread ayan guha
For JDBC to work, you can start spark-submit with appropriate jdbc driver jars (using --jars), then you will have the driver available on executors. For acquiring connections, create a singleton connection per executor. I think dataframe's jdbc reader (sqlContext.read.jdbc) already take care of

Re: Local spark context on an executor

2017-03-21 Thread Shashank Mandil
I am using spark to dump data from mysql into hdfs. The way I am doing this is by creating a spark dataframe with the metadata of different mysql tables to dump from multiple mysql hosts and then running a map over that data frame to dump each mysql table data into hdfs inside the executor. The

Re: Local spark context on an executor

2017-03-21 Thread ayan guha
What is your use case? I am sure there must be a better way to solve it On Wed, Mar 22, 2017 at 9:34 AM, Shashank Mandil wrote: > Hi All, > > I am using spark in a yarn cluster mode. > When I run a yarn application it creates multiple executors on the hadoop >