I was wondering what people generally do about doing database operations from
executor nodes. I’m (at least for now) avoiding doing database updates from
executor nodes to avoid proliferation of database connections on the cluster.
The general pattern I adopt is to collect queries (or tuples) on the executors
and write to the database on the driver.
// Executes on the executor
rdd.foreach(s => {
val query = s"insert into .... ${s}";
accumulator += query;
});
// Executes on the driver
acclumulator.value.foreach(query => {
// get connection
// update database
});
I’m obviously trading database connections for driver heap. How do other spark
users do it?
Cheers
Praveen
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
