Generally, the problem is that I don’t find a way to automatically create a JDBC table in the JDBC database when I want to insert data into it using Spark SQL only, not DataFrames API.
> On 2 Feb 2023, at 21:22, Harut Martirosyan <harut.martiros...@gmail.com> > wrote: > > Hi, thanks for the reply. > > Let’s imagine we have a parquet based table called parquet_table, now I want > to insert it into a new JDBC table, all using pure SQL. > > If the JDBC table already exists, it’s easy, we do CREATE TABLE USING JDBC > and then we do INSERT INTO that table. > > If the table doesn’t exist, is there a way to create it using Spark SQL only? > I don’t want to use DataFrames API, I know that I can use .write() for that, > but I want to keep it in pure SQL, since that is more comprehendible for data > analysts. > >> On 2 Feb 2023, at 02:08, Mich Talebzadeh <mich.talebza...@gmail.com> wrote: >> >> Hi, >> >> It is not very clear your statement below: >> >> ".. If the table existed, I would create a table using JDBC in spark SQL and >> then insert into it, but I can't create a table if it doesn't exist in JDBC >> database..." >> >> If the table exists in your JDBC database, why do you need to create it? >> >> How do you verify if it exists? Can you share the code and the doc link? >> >> HTH >> >> >> >> view my Linkedin profile >> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >> >> https://en.everybodywiki.com/Mich_Talebzadeh >> >> >> Disclaimer: Use it at your own risk. Any and all responsibility for any >> loss, damage or destruction of data or any other property which may arise >> from relying on this email's technical content is explicitly disclaimed. The >> author will in no case be liable for any monetary damages arising from such >> loss, damage or destruction. >> >> >> >> On Wed, 1 Feb 2023 at 19:33, Harut Martirosyan <harut.martiros...@gmail.com >> <mailto:harut.martiros...@gmail.com>> wrote: >>> I have a resultset (defined in SQL), and I want to insert it into my JDBC >>> database using only SQL, not dataframes API. >>> >>> If the table existed, I would create a table using JDBC in spark SQL and >>> then insert into it, but I can't create a table if it doesn't exist in JDBC >>> database. >>> >>> How to do that using pure SQL (no python/scala/java)? >>> >>> I am trying to use INSERT OVERWRITE DIRECTORY with JDBC file format >>> (according to the documentation) but as expected this functionality is >>> available only for File-based storage systems. >>> >>> -- >>> RGRDZ Harut >
