Flink scala data sink code from CSV to postgres.
Hi Guys, We have a requirement like loading data from CSV file to postgress database In Flink scala. We need very basic script how to read data from file and write it to postgress database. Could you please some one help me on that guys -- Regards. Jagan.
Re: Flink Scala - data load from CSV to postgress database.
Hi Flavio,
Thanks this script is very useful, however we need Flink scala script to
load data from csv to postgress.
if we have following java equivalent code in scala that will be really
helpful...thanks.
myResult.output(
// build and configure OutputFormat
JDBCOutputFormat.buildJDBCOutputFormat()
.setDrivername(" ")
.setDBUrl(" ")
.setQuery("insert into persons (name, age, height)
values (?,?,?)")
.finish()
);
On 23 September 2016 at 08:35, Flavio Pompermaier <pomperma...@okkam.it>
wrote:
> Hi Jagan, I have a Java version of such a Job if you want (that requires
> table-api in order to handle null values), I hope this could help!
> Obviously you'll have to set all parameters properly..(e.g.
> jdbcOutputFormat ,
> fieldTypes, etc...).
>
>
> public static void main(String[] args) throws Exception {
> String path = "file:/tmp/myFile.csv";
> String rowDelim = CsvInputFormat.DEFAULT_LINE_DELIMITER;
> String fieldDelim = CsvInputFormat.DEFAULT_FIELD_DELIMITER;
> String[] fieldNames = "Column 1,Column 2,Column
> 3,Column4".split(fieldDelim);
>
> Character quoteCharacter = '"';
> boolean ignoreFirstLine = Boolean.TRUE;
> String ignoreComments = null;
> boolean lenient = Boolean.FALSE;
>
> TypeInformation[] fieldTypes = new TypeInformation[] {
> BasicTypeInfo.STRING_TYPE_INFO,
> BasicTypeInfo.INT_TYPE_INFO,
> BasicTypeInfo.STRING_TYPE_INFO,
> BasicTypeInfo.STRING_TYPE_INFO
> };
>
> ExecutionEnvironment env =
> ExecutionEnvironment.getExecutionEnvironment();
>
> CsvTableSource csvTableSource = new
> CsvTableSource(path,fieldNames,fieldTypes,fieldDelim,
>rowDelim, quoteCharacter, ignoreFirstLine, ignoreComments, lenient);
> DataSet csvDataSet = csvTableSource.getDataSet(env);
> JDBCOutputFormat jdbcOutputFormat =
> JDBCOutputFormat.buildJDBCOutputFormat()
> .setDrivername("org.postgresql.Driver")
> .setDBUrl("jdbc:postgresql://localhost/test?user=xxx=xxx")
> .setQuery("insert into %s (id, title, author, price, qty) values
> (?,?,?,?,?)") .finish(); csvDataSet.output(jdbcOutputFormat);
> }
>
> Best,
> Flavio
>
> On Fri, Sep 23, 2016 at 12:52 AM, Jagan <jaga...@gmail.com> wrote:
>
> > Hi Guys,
> >
> > We have requirement like – loading data from local CSV file to Postgress
> > database using Flink Scala…
> >
> > Do you have any sample Flink scala code for this?
> >
> > We have tried and searched in Google/Flinkweb website for data load, we
> > haven’t found any sample code for this requisite.
> >
> > Code: Flink Scala.
> >
> > Load: from CSV to local postgres database.
> >
> > Thanks
> >
> > Jagan
> >
> > On 22 September 2016 at 20:37, Jagan <jaga...@gmail.com> wrote:
> >
> > > Hi Team,
> > >
> > > Will you be able to guide me on this? Is this a known issue that
> we
> > > can't implement dataload in flink scala ?
> > >
> > >data load from csv to postgress or any relational database in
> > Flink
> > > Scala
> > >
> > > Thanks
> > >
> > > Jagan.
> > >
> > > On 22 September 2016 at 20:15, Jagan <jaga...@gmail.com> wrote:
> > >
> > >> Thanks Suneel,
> > >>
> > >> but client want to implement the data load in Flink Scala..
> > >>
> > >>
> > >> On 22 September 2016 at 20:07, Suneel Marthi <smar...@apache.org>
> > wrote:
> > >>
> > >>> Couldn't u use SQLLoader or something for doing that?
> > >>>
> > >>> http://stackoverflow.com/questions/2987433/how-to-import-csv
> > >>> -file-data-into-a-postgresql-table
> > >>>
> > >>>
> > >>>
> > >>> On Thu, Sep 22, 2016 at 3:01 PM, Jagan <jaga...@gmail.com> wrote:
> > >>>
> > >>> > Hi Guys,
> > >>> >
> > >>> > We have a requirement like – loading data from local CSV file to
> > >>> Postgress
> > >>> > database using Flink Scala…We have tried number of ways all failed
> > >>> >
> > >>> > Do you have any example for this? With dependency libraries to
> > >>> understand
> > >>> > how to load data from CSV to postgres
> > >>> >
> > >>> > We have tried and searched in Google/Flinkweb website
Re: Flink Scala - data load from CSV to postgress database.
Hi Guys, We have requirement like – loading data from local CSV file to Postgress database using Flink Scala… Do you have any sample Flink scala code for this? We have tried and searched in Google/Flinkweb website for data load, we haven’t found any sample code for this requisite. Code: Flink Scala. Load: from CSV to local postgres database. Thanks Jagan On 22 September 2016 at 20:37, Jagan <jaga...@gmail.com> wrote: > Hi Team, > > Will you be able to guide me on this? Is this a known issue that we > can't implement dataload in flink scala ? > >data load from csv to postgress or any relational database in Flink > Scala > > Thanks > > Jagan. > > On 22 September 2016 at 20:15, Jagan <jaga...@gmail.com> wrote: > >> Thanks Suneel, >> >> but client want to implement the data load in Flink Scala.. >> >> >> On 22 September 2016 at 20:07, Suneel Marthi <smar...@apache.org> wrote: >> >>> Couldn't u use SQLLoader or something for doing that? >>> >>> http://stackoverflow.com/questions/2987433/how-to-import-csv >>> -file-data-into-a-postgresql-table >>> >>> >>> >>> On Thu, Sep 22, 2016 at 3:01 PM, Jagan <jaga...@gmail.com> wrote: >>> >>> > Hi Guys, >>> > >>> > We have a requirement like – loading data from local CSV file to >>> Postgress >>> > database using Flink Scala…We have tried number of ways all failed >>> > >>> > Do you have any example for this? With dependency libraries to >>> understand >>> > how to load data from CSV to postgres >>> > >>> > We have tried and searched in Google/Flinkweb website for data load, we >>> > haven’t found any sample code for this requisite. >>> > >>> > Code: Flink Scala. >>> > >>> > Load: from CSV to local postgres database. >>> > >>> > Thanks >>> > >>> > Jagan >>> > >>> > 0044-7411239688 >>> > >>> > >>> > >>> > -- >>> > Regards. >>> > Jagan. >>> > >>> > >>> > - >>> > * The Good You Do Today, People will often forget tomorrow: Do >>> Good >>> > Anyway >>> > >>> >> >> >> >> -- >> Regards. >> Jagan. >> >> >> - >> * The Good You Do Today, People will often forget tomorrow: Do Good >> Anyway >> > > > > -- > Regards. > Jagan. > > > - > * The Good You Do Today, People will often forget tomorrow: Do Good > Anyway > -- Regards. Jagan. - * The Good You Do Today, People will often forget tomorrow: Do Good Anyway
Flink Scala data load from CSV to postgress database.
Hi Guys, We have requirement like – loading data from local CSV file to Postgress database using Flink Scala…We have tried number of ways all faile Do you have any example for this? With dependency libraries to understand how to load data from CSV to postgres We have tried and searched in Google/Flinkweb website for data load, we haven’t found any sample code for this requisite. Code: Flink Scala. Load: from CSV to local postgres database. Thanks Jagan 0044-7411239688
Re: Flink Scala - data load from CSV to postgress database.
Hi Team, Will you be able to guide me on this? Is this a known issue that we can't implement dataload in flink scala ? data load from csv to postgress or any relational database in Flink Scala Thanks Jagan. On 22 September 2016 at 20:15, Jagan <jaga...@gmail.com> wrote: > Thanks Suneel, > > but client want to implement the data load in Flink Scala.. > > > On 22 September 2016 at 20:07, Suneel Marthi <smar...@apache.org> wrote: > >> Couldn't u use SQLLoader or something for doing that? >> >> http://stackoverflow.com/questions/2987433/how-to-import- >> csv-file-data-into-a-postgresql-table >> >> >> >> On Thu, Sep 22, 2016 at 3:01 PM, Jagan <jaga...@gmail.com> wrote: >> >> > Hi Guys, >> > >> > We have a requirement like – loading data from local CSV file to >> Postgress >> > database using Flink Scala…We have tried number of ways all failed >> > >> > Do you have any example for this? With dependency libraries to >> understand >> > how to load data from CSV to postgres >> > >> > We have tried and searched in Google/Flinkweb website for data load, we >> > haven’t found any sample code for this requisite. >> > >> > Code: Flink Scala. >> > >> > Load: from CSV to local postgres database. >> > >> > Thanks >> > >> > Jagan >> > >> > 0044-7411239688 >> > >> > >> > >> > -- >> > Regards. >> > Jagan. >> > >> > >> > - >> > * The Good You Do Today, People will often forget tomorrow: Do Good >> > Anyway >> > >> > > > > -- > Regards. > Jagan. > > > - > * The Good You Do Today, People will often forget tomorrow: Do Good > Anyway > -- Regards. Jagan. - * The Good You Do Today, People will often forget tomorrow: Do Good Anyway
Re: Flink Scala - data load from CSV to postgress database.
Thanks Suneel, but client want to implement the data load in Flink Scala.. On 22 September 2016 at 20:07, Suneel Marthi <smar...@apache.org> wrote: > Couldn't u use SQLLoader or something for doing that? > > http://stackoverflow.com/questions/2987433/how-to- > import-csv-file-data-into-a-postgresql-table > > > > On Thu, Sep 22, 2016 at 3:01 PM, Jagan <jaga...@gmail.com> wrote: > > > Hi Guys, > > > > We have a requirement like – loading data from local CSV file to > Postgress > > database using Flink Scala…We have tried number of ways all failed > > > > Do you have any example for this? With dependency libraries to understand > > how to load data from CSV to postgres > > > > We have tried and searched in Google/Flinkweb website for data load, we > > haven’t found any sample code for this requisite. > > > > Code: Flink Scala. > > > > Load: from CSV to local postgres database. > > > > Thanks > > > > Jagan > > > > 0044-7411239688 > > > > > > > > -- > > Regards. > > Jagan. > > > > ---- > > - > > * The Good You Do Today, People will often forget tomorrow: Do Good > > Anyway > > > -- Regards. Jagan. - * The Good You Do Today, People will often forget tomorrow: Do Good Anyway
Fwd: Flink Scala - data load from CSV to postgress database.
Hi Guys, We have a requirement like – loading data from local CSV file to Postgress database using Flink Scala…We have tried number of ways all failed Do you have any example for this? With dependency libraries to understand how to load data from CSV to postgres We have tried and searched in Google/Flinkweb website for data load, we haven’t found any sample code for this requisite. Code: Flink Scala. Load: from CSV to local postgres database. Thanks Jagan 0044-7411239688 -- Regards. Jagan. - * The Good You Do Today, People will often forget tomorrow: Do Good Anyway
