Actually I found a solution to this issue
*Challenge*
Insert data from Spark dataframe when one or more columns in theOracle
table rely on some derived_colums dependent on data in one or more
dataframe columns.
Standard JDBC from Spark to Oracle does batch insert of dataframe into
Oracle *so it
Another option is to just use plain jdbc (if in java) in a foreachPartition
call on the dataframe/dataset then you get full control of the insert
statement but need to open the connection/transaction yourself
On Sat, 19 Jun 2021 at 19:33, Mich Talebzadeh
wrote:
> Hi,
>
> I did some research on
Hi,
I did some research on this.
The only way one can write to Oracle from Spark is through JDBC (excluding
other options outside of Spark).
The challenge here is that you have a column based on function
get_function() column that Spark needs to insert. Currently there is no
way of inserting
Hi
Why this can be done by oracle insert trigger? Or even a view?
On Sat, 19 Jun 2021 at 7:17 am, Mich Talebzadeh
wrote:
> Well the challenge is that Spark is best suited to insert a dataframe into
> the Oracle table, i.e. a bulk insert
>
> that insert into table (column list) values (..) is
Well the challenge is that Spark is best suited to insert a dataframe into
the Oracle table, i.e. a bulk insert
that insert into table (column list) values (..) is a single record insert
.. Can you try creating a staging table in oracle without get_function()
column and do a bulk insert from
Hi Mich,
Thanks for your reply. Please advise the insert query that I need to
substitute should be like below:
Insert into table(a,b,c) values(?,get_function_value(?),?)
In the statement above :
? : refers to value from dataframe column values
get_function_value : refers to be the function
I gather you mean using JDBC to write to the Oracle table?
Spark provides a unified framework to write to any JDBC compliant database.
def writeTableWithJDBC(dataFrame, url, tableName, user, password, driver,
mode):
try:
dataFrame. \
write. \
format("jdbc"). \
