Re: writing into oracle database is very slow
Thanks for interesting ideas! Looks like spark directly writing to relational
database is not as straight forward as I expected.
Sent from my iPhone
> On Apr 19, 2019, at 06:58, Khare, Ankit wrote:
>
> Hi Jiang
>
> We faced similar issue so we write the file and then use sqoop to export data
> to mssql.
>
> We achieved a great time benefit with this strategy.
>
> Sent from my iPhone
>
> On 19. Apr 2019, at 10:47, spark receiver wrote:
>
>> hi Jiang,
>>
>> i was facing the very same issue ,the solution is write to file and using
>> oracle external table to do the insert.
>>
>> hope this could help.
>>
>> Dalin
>>
>>> On Thu, Apr 18, 2019 at 11:43 AM Jörn Franke wrote:
>>> What is the size of the data? How much time does it need on HDFS and how
>>> much on Oracle? How many partitions do you have on Oracle side?
>>>
>>> Am 06.04.2019 um 16:59 schrieb Lian Jiang :
>>>
Hi,
My spark job writes into oracle db using:
df.coalesce(10).write.format("jdbc").option("url", url)
.option("driver", driver).option("user", user)
.option("batchsize", 2000)
.option("password", password).option("dbtable",
tableName).mode("append").save()
It is much slow than writting into HDFS. The data to write is small.
Is this expected? Thanks for any clue.
Re: writing into oracle database is very slow
Hi Jiang
We faced similar issue so we write the file and then use sqoop to export data
to mssql.
We achieved a great time benefit with this strategy.
Sent from my iPhone
On 19. Apr 2019, at 10:47, spark receiver
mailto:spark.recei...@gmail.com>> wrote:
hi Jiang,
i was facing the very same issue ,the solution is write to file and using
oracle external table to do the insert.
hope this could help.
Dalin
On Thu, Apr 18, 2019 at 11:43 AM Jörn Franke
mailto:jornfra...@gmail.com>> wrote:
What is the size of the data? How much time does it need on HDFS and how much
on Oracle? How many partitions do you have on Oracle side?
Am 06.04.2019 um 16:59 schrieb Lian Jiang
mailto:jiangok2...@gmail.com>>:
Hi,
My spark job writes into oracle db using:
df.coalesce(10).write.format("jdbc").option("url", url)
.option("driver", driver).option("user", user)
.option("batchsize", 2000)
.option("password", password).option("dbtable",
tableName).mode("append").save()
It is much slow than writting into HDFS. The data to write is small.
Is this expected? Thanks for any clue.
Re: writing into oracle database is very slow
hi Jiang,
i was facing the very same issue ,the solution is write to file and using
oracle external table to do the insert.
hope this could help.
Dalin
On Thu, Apr 18, 2019 at 11:43 AM Jörn Franke wrote:
> What is the size of the data? How much time does it need on HDFS and how
> much on Oracle? How many partitions do you have on Oracle side?
>
> Am 06.04.2019 um 16:59 schrieb Lian Jiang :
>
> Hi,
>
> My spark job writes into oracle db using:
>
> df.coalesce(10).write.format("jdbc").option("url", url)
> .option("driver", driver).option("user", user)
> .option("batchsize", 2000)
> .option("password", password).option("dbtable",
> tableName).mode("append").save()
>
> It is much slow than writting into HDFS. The data to write is small.
>
> Is this expected? Thanks for any clue.
>
>
>
Re: writing into oracle database is very slow
What is the size of the data? How much time does it need on HDFS and how much
on Oracle? How many partitions do you have on Oracle side?
> Am 06.04.2019 um 16:59 schrieb Lian Jiang :
>
> Hi,
>
> My spark job writes into oracle db using:
> df.coalesce(10).write.format("jdbc").option("url", url)
> .option("driver", driver).option("user", user)
> .option("batchsize", 2000)
> .option("password", password).option("dbtable",
> tableName).mode("append").save()
> It is much slow than writting into HDFS. The data to write is small.
> Is this expected? Thanks for any clue.
>
Re: writing into oracle database is very slow
Are you sure you only need 10 partitions? Do you get the same performance writing to HDFS with 10 partitions? -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ - To unsubscribe e-mail: user-unsubscr...@spark.apache.org
writing into oracle database is very slow
Hi,
My spark job writes into oracle db using:
df.coalesce(10).write.format("jdbc").option("url", url)
.option("driver", driver).option("user", user)
.option("batchsize", 2000)
.option("password", password).option("dbtable",
tableName).mode("append").save()
It is much slow than writting into HDFS. The data to write is small.
Is this expected? Thanks for any clue.
