Re: DataFrameWriter.jdbc is very slow
It's good Thanks for your reply Michael. On Thu, Aug 20, 2015 at 11:03 PM, Michael Armbrust wrote: > We will probably fix this in Spark 1.6 > > https://issues.apache.org/jira/browse/SPARK-10040 > > On Thu, Aug 20, 2015 at 5:18 AM, Aram Mkrtchyan < > aram.mkrtchyan...@gmail.com> wrote: > >> We want to migrate our data (approximately 20M rows) from parquet to >> postgres, >> when we are using dataframe writer's jdbc method the execution time is very >> large, we have tried the same with batch insert it was much effective. >> Is it intentionally implemented in that way? >> > >
Re: DataFrameWriter.jdbc is very slow
We will probably fix this in Spark 1.6 https://issues.apache.org/jira/browse/SPARK-10040 On Thu, Aug 20, 2015 at 5:18 AM, Aram Mkrtchyan wrote: > We want to migrate our data (approximately 20M rows) from parquet to postgres, > when we are using dataframe writer's jdbc method the execution time is very > large, we have tried the same with batch insert it was much effective. > Is it intentionally implemented in that way? >
DataFrameWriter.jdbc is very slow
We want to migrate our data (approximately 20M rows) from parquet to postgres, when we are using dataframe writer's jdbc method the execution time is very large, we have tried the same with batch insert it was much effective. Is it intentionally implemented in that way?
