Sqoop doesn't work on sharded database.
Thanks,
Shashank
On Wed, Mar 22, 2017 at 5:43 AM Reynier González Tejeda
wrote:
> Why are you using spark instead of sqoop?
>
> 2017-03-21 21:29 GMT-03:00 ayan guha :
>
> For JDBC to work, you can start
Why are you using spark instead of sqoop?
2017-03-21 21:29 GMT-03:00 ayan guha :
> For JDBC to work, you can start spark-submit with appropriate jdbc driver
> jars (using --jars), then you will have the driver available on executors.
>
> For acquiring connections, create a
For JDBC to work, you can start spark-submit with appropriate jdbc driver
jars (using --jars), then you will have the driver available on executors.
For acquiring connections, create a singleton connection per executor. I
think dataframe's jdbc reader (sqlContext.read.jdbc) already take care of
I am using spark to dump data from mysql into hdfs.
The way I am doing this is by creating a spark dataframe with the metadata
of different mysql tables to dump from multiple mysql hosts and then
running a map over that data frame to dump each mysql table data into hdfs
inside the executor.
The
What is your use case? I am sure there must be a better way to solve it
On Wed, Mar 22, 2017 at 9:34 AM, Shashank Mandil
wrote:
> Hi All,
>
> I am using spark in a yarn cluster mode.
> When I run a yarn application it creates multiple executors on the hadoop
>