Re: Flink Scala - data load from CSV to postgress database.

[Jagan]

Hi Flavio,

Thanks this script is very useful, however we need Flink scala script to
load data from csv to postgress.

if we have following java equivalent code in scala that will be really
helpful...thanks.

myResult.output(
// build and configure OutputFormat
JDBCOutputFormat.buildJDBCOutputFormat()
.setDrivername(" ")
.setDBUrl(" ")
.setQuery("insert into persons (name, age, height)
values (?,?,?)")
.finish()
);



On 23 September 2016 at 08:35, Flavio Pompermaier 
wrote:

> Hi Jagan, I have a Java version of such a Job if you want (that requires
> table-api in order to handle null values), I hope this could help!
> Obviously you'll have to set all parameters properly..(e.g.
> jdbcOutputFormat ,
> fieldTypes, etc...).
>
>
> public static void main(String[] args) throws Exception {
> String path = "file:/tmp/myFile.csv";
> String rowDelim = CsvInputFormat.DEFAULT_LINE_DELIMITER;
> String fieldDelim = CsvInputFormat.DEFAULT_FIELD_DELIMITER;
> String[] fieldNames = "Column 1,Column 2,Column
> 3,Column4".split(fieldDelim);
>
> Character quoteCharacter = '"';
> boolean ignoreFirstLine = Boolean.TRUE;
> String ignoreComments = null;
> boolean lenient = Boolean.FALSE;
>
> TypeInformation[] fieldTypes = new TypeInformation[] {
> BasicTypeInfo.STRING_TYPE_INFO,
> BasicTypeInfo.INT_TYPE_INFO,
> BasicTypeInfo.STRING_TYPE_INFO,
> BasicTypeInfo.STRING_TYPE_INFO
> };
>
> ExecutionEnvironment env =
> ExecutionEnvironment.getExecutionEnvironment();
>
> CsvTableSource csvTableSource = new
> CsvTableSource(path,fieldNames,fieldTypes,fieldDelim,
>rowDelim, quoteCharacter, ignoreFirstLine, ignoreComments, lenient);
> DataSet csvDataSet = csvTableSource.getDataSet(env);
> JDBCOutputFormat jdbcOutputFormat =
> JDBCOutputFormat.buildJDBCOutputFormat()
> .setDrivername("org.postgresql.Driver")
> .setDBUrl("jdbc:postgresql://localhost/test?user=xxx=xxx")
> .setQuery("insert into %s (id, title, author, price, qty) values
> (?,?,?,?,?)") .finish(); csvDataSet.output(jdbcOutputFormat);
> }
>
> Best,
> Flavio
>
> On Fri, Sep 23, 2016 at 12:52 AM, Jagan  wrote:
>
> > Hi Guys,
> >
> > We have requirement like – loading data from local CSV file to Postgress
> > database using Flink Scala…
> >
> > Do you have any sample Flink scala code for this?
> >
> > We have tried and searched in Google/Flinkweb website for data load, we
> > haven’t found any sample code for this requisite.
> >
> > Code: Flink Scala.
> >
> > Load: from CSV to local postgres database.
> >
> > Thanks
> >
> > Jagan
> >
> > On 22 September 2016 at 20:37, Jagan  wrote:
> >
> > > Hi Team,
> > >
> > >   Will you be able to guide me on this? Is this a known issue that
> we
> > > can't implement  dataload in flink scala ?
> > >
> > >data load from csv to postgress or any relational database in
> > Flink
> > > Scala
> > >
> > > Thanks
> > >
> > > Jagan.
> > >
> > > On 22 September 2016 at 20:15, Jagan  wrote:
> > >
> > >> Thanks Suneel,
> > >>
> > >> but client want to implement the data load in Flink Scala..
> > >>
> > >>
> > >> On 22 September 2016 at 20:07, Suneel Marthi 
> > wrote:
> > >>
> > >>> Couldn't u use SQLLoader or something for doing that?
> > >>>
> > >>> http://stackoverflow.com/questions/2987433/how-to-import-csv
> > >>> -file-data-into-a-postgresql-table
> > >>>
> > >>>
> > >>>
> > >>> On Thu, Sep 22, 2016 at 3:01 PM, Jagan  wrote:
> > >>>
> > >>> > Hi Guys,
> > >>> >
> > >>> > We have a requirement like – loading data from local CSV file to
> > >>> Postgress
> > >>> > database using Flink Scala…We have tried number of ways all failed
> > >>> >
> > >>> > Do you have any example for this? With dependency libraries to
> > >>> understand
> > >>> > how to load data from CSV to postgres
> > >>> >
> > >>> > We have tried and searched in Google/Flinkweb website for data
> load,
> > we
> > >>> > haven’t found any sample code for this requisite.
> > >>> >
> > >>> > Code: Flink Scala.
> > >>> >
> > >>> > Load: from CSV to local postgres database.
> > >>> >
> > >>> > Thanks
> > >>> >
> > >>> > Jagan
> > >>> >
> > >>> > 0044-7411239688
> > >>> >
> > >>> >
> > >>> >
> > >>> > --
> > >>> > Regards.
> > >>> > Jagan.
> > >>> >
> > >>> > 
> > >>> > -
> > >>> > *  The Good You Do Today, People will often forget tomorrow: Do
> > >>> Good
> > >>> > Anyway 
> > >>> >
> > >>>
> > >>
> > >>
> > >>
> > >> --
> > >> Regards.
> > >> Jagan.
> > >>
> > >> 
> > >> -
> > >> *  The Good You Do Today, People will often forget tomorrow: Do
> Good
> > >> Anyway 
> > >>
> > >
> > 

Re: Flink Scala - data load from CSV to postgress database.

[Flavio Pompermaier]

Hi Jagan, I have a Java version of such a Job if you want (that requires
table-api in order to handle null values), I hope this could help!
Obviously you'll have to set all parameters properly..(e.g. jdbcOutputFormat ,
fieldTypes, etc...).


public static void main(String[] args) throws Exception {
String path = "file:/tmp/myFile.csv";
String rowDelim = CsvInputFormat.DEFAULT_LINE_DELIMITER;
String fieldDelim = CsvInputFormat.DEFAULT_FIELD_DELIMITER;
String[] fieldNames = "Column 1,Column 2,Column
3,Column4".split(fieldDelim);

Character quoteCharacter = '"';
boolean ignoreFirstLine = Boolean.TRUE;
String ignoreComments = null;
boolean lenient = Boolean.FALSE;

TypeInformation[] fieldTypes = new TypeInformation[] {
BasicTypeInfo.STRING_TYPE_INFO,
BasicTypeInfo.INT_TYPE_INFO,
BasicTypeInfo.STRING_TYPE_INFO,
BasicTypeInfo.STRING_TYPE_INFO
};

ExecutionEnvironment env =
ExecutionEnvironment.getExecutionEnvironment();

CsvTableSource csvTableSource = new
CsvTableSource(path,fieldNames,fieldTypes,fieldDelim,
   rowDelim, quoteCharacter, ignoreFirstLine, ignoreComments, lenient);
DataSet csvDataSet = csvTableSource.getDataSet(env);
JDBCOutputFormat jdbcOutputFormat =
JDBCOutputFormat.buildJDBCOutputFormat()
.setDrivername("org.postgresql.Driver")
.setDBUrl("jdbc:postgresql://localhost/test?user=xxx=xxx")
.setQuery("insert into %s (id, title, author, price, qty) values
(?,?,?,?,?)") .finish(); csvDataSet.output(jdbcOutputFormat);
}

Best,
Flavio

On Fri, Sep 23, 2016 at 12:52 AM, Jagan  wrote:

> Hi Guys,
>
> We have requirement like – loading data from local CSV file to Postgress
> database using Flink Scala…
>
> Do you have any sample Flink scala code for this?
>
> We have tried and searched in Google/Flinkweb website for data load, we
> haven’t found any sample code for this requisite.
>
> Code: Flink Scala.
>
> Load: from CSV to local postgres database.
>
> Thanks
>
> Jagan
>
> On 22 September 2016 at 20:37, Jagan  wrote:
>
> > Hi Team,
> >
> >   Will you be able to guide me on this? Is this a known issue that we
> > can't implement  dataload in flink scala ?
> >
> >data load from csv to postgress or any relational database in
> Flink
> > Scala
> >
> > Thanks
> >
> > Jagan.
> >
> > On 22 September 2016 at 20:15, Jagan  wrote:
> >
> >> Thanks Suneel,
> >>
> >> but client want to implement the data load in Flink Scala..
> >>
> >>
> >> On 22 September 2016 at 20:07, Suneel Marthi 
> wrote:
> >>
> >>> Couldn't u use SQLLoader or something for doing that?
> >>>
> >>> http://stackoverflow.com/questions/2987433/how-to-import-csv
> >>> -file-data-into-a-postgresql-table
> >>>
> >>>
> >>>
> >>> On Thu, Sep 22, 2016 at 3:01 PM, Jagan  wrote:
> >>>
> >>> > Hi Guys,
> >>> >
> >>> > We have a requirement like – loading data from local CSV file to
> >>> Postgress
> >>> > database using Flink Scala…We have tried number of ways all failed
> >>> >
> >>> > Do you have any example for this? With dependency libraries to
> >>> understand
> >>> > how to load data from CSV to postgres
> >>> >
> >>> > We have tried and searched in Google/Flinkweb website for data load,
> we
> >>> > haven’t found any sample code for this requisite.
> >>> >
> >>> > Code: Flink Scala.
> >>> >
> >>> > Load: from CSV to local postgres database.
> >>> >
> >>> > Thanks
> >>> >
> >>> > Jagan
> >>> >
> >>> > 0044-7411239688
> >>> >
> >>> >
> >>> >
> >>> > --
> >>> > Regards.
> >>> > Jagan.
> >>> >
> >>> > 
> >>> > -
> >>> > *  The Good You Do Today, People will often forget tomorrow: Do
> >>> Good
> >>> > Anyway 
> >>> >
> >>>
> >>
> >>
> >>
> >> --
> >> Regards.
> >> Jagan.
> >>
> >> 
> >> -
> >> *  The Good You Do Today, People will often forget tomorrow: Do Good
> >> Anyway 
> >>
> >
> >
> >
> > --
> > Regards.
> > Jagan.
> >
> > 
> > -
> > *  The Good You Do Today, People will often forget tomorrow: Do Good
> > Anyway 
> >
>
>
>
> --
> Regards.
> Jagan.
>
> 
> -
> *  The Good You Do Today, People will often forget tomorrow: Do Good
> Anyway 
>

Re: Flink Scala - data load from CSV to postgress database.

[Jagan]

Hi Guys,

We have requirement like – loading data from local CSV file to Postgress
database using Flink Scala…

Do you have any sample Flink scala code for this?

We have tried and searched in Google/Flinkweb website for data load, we
haven’t found any sample code for this requisite.

Code: Flink Scala.

Load: from CSV to local postgres database.

Thanks

Jagan

On 22 September 2016 at 20:37, Jagan  wrote:

> Hi Team,
>
>   Will you be able to guide me on this? Is this a known issue that we
> can't implement  dataload in flink scala ?
>
>data load from csv to postgress or any relational database in Flink
> Scala
>
> Thanks
>
> Jagan.
>
> On 22 September 2016 at 20:15, Jagan  wrote:
>
>> Thanks Suneel,
>>
>> but client want to implement the data load in Flink Scala..
>>
>>
>> On 22 September 2016 at 20:07, Suneel Marthi  wrote:
>>
>>> Couldn't u use SQLLoader or something for doing that?
>>>
>>> http://stackoverflow.com/questions/2987433/how-to-import-csv
>>> -file-data-into-a-postgresql-table
>>>
>>>
>>>
>>> On Thu, Sep 22, 2016 at 3:01 PM, Jagan  wrote:
>>>
>>> > Hi Guys,
>>> >
>>> > We have a requirement like – loading data from local CSV file to
>>> Postgress
>>> > database using Flink Scala…We have tried number of ways all failed
>>> >
>>> > Do you have any example for this? With dependency libraries to
>>> understand
>>> > how to load data from CSV to postgres
>>> >
>>> > We have tried and searched in Google/Flinkweb website for data load, we
>>> > haven’t found any sample code for this requisite.
>>> >
>>> > Code: Flink Scala.
>>> >
>>> > Load: from CSV to local postgres database.
>>> >
>>> > Thanks
>>> >
>>> > Jagan
>>> >
>>> > 0044-7411239688
>>> >
>>> >
>>> >
>>> > --
>>> > Regards.
>>> > Jagan.
>>> >
>>> > 
>>> > -
>>> > *  The Good You Do Today, People will often forget tomorrow: Do
>>> Good
>>> > Anyway 
>>> >
>>>
>>
>>
>>
>> --
>> Regards.
>> Jagan.
>>
>> 
>> -
>> *  The Good You Do Today, People will often forget tomorrow: Do Good
>> Anyway 
>>
>
>
>
> --
> Regards.
> Jagan.
>
> 
> -
> *  The Good You Do Today, People will often forget tomorrow: Do Good
> Anyway 
>



-- 
Regards.
Jagan.

-
*  The Good You Do Today, People will often forget tomorrow: Do Good
Anyway 

Re: Flink Scala - data load from CSV to postgress database.

[Jagan]

Hi Team,

  Will you be able to guide me on this? Is this a known issue that we
can't implement  dataload in flink scala ?

   data load from csv to postgress or any relational database in Flink
Scala

Thanks

Jagan.

On 22 September 2016 at 20:15, Jagan  wrote:

> Thanks Suneel,
>
> but client want to implement the data load in Flink Scala..
>
>
> On 22 September 2016 at 20:07, Suneel Marthi  wrote:
>
>> Couldn't u use SQLLoader or something for doing that?
>>
>> http://stackoverflow.com/questions/2987433/how-to-import-
>> csv-file-data-into-a-postgresql-table
>>
>>
>>
>> On Thu, Sep 22, 2016 at 3:01 PM, Jagan  wrote:
>>
>> > Hi Guys,
>> >
>> > We have a requirement like – loading data from local CSV file to
>> Postgress
>> > database using Flink Scala…We have tried number of ways all failed
>> >
>> > Do you have any example for this? With dependency libraries to
>> understand
>> > how to load data from CSV to postgres
>> >
>> > We have tried and searched in Google/Flinkweb website for data load, we
>> > haven’t found any sample code for this requisite.
>> >
>> > Code: Flink Scala.
>> >
>> > Load: from CSV to local postgres database.
>> >
>> > Thanks
>> >
>> > Jagan
>> >
>> > 0044-7411239688
>> >
>> >
>> >
>> > --
>> > Regards.
>> > Jagan.
>> >
>> > 
>> > -
>> > *  The Good You Do Today, People will often forget tomorrow: Do Good
>> > Anyway 
>> >
>>
>
>
>
> --
> Regards.
> Jagan.
>
> 
> -
> *  The Good You Do Today, People will often forget tomorrow: Do Good
> Anyway 
>



-- 
Regards.
Jagan.

-
*  The Good You Do Today, People will often forget tomorrow: Do Good
Anyway 

Re: Flink Scala - data load from CSV to postgress database.

[Jagan]

Thanks Suneel,

but client want to implement the data load in Flink Scala..


On 22 September 2016 at 20:07, Suneel Marthi  wrote:

> Couldn't u use SQLLoader or something for doing that?
>
> http://stackoverflow.com/questions/2987433/how-to-
> import-csv-file-data-into-a-postgresql-table
>
>
>
> On Thu, Sep 22, 2016 at 3:01 PM, Jagan  wrote:
>
> > Hi Guys,
> >
> > We have a requirement like – loading data from local CSV file to
> Postgress
> > database using Flink Scala…We have tried number of ways all failed
> >
> > Do you have any example for this? With dependency libraries to understand
> > how to load data from CSV to postgres
> >
> > We have tried and searched in Google/Flinkweb website for data load, we
> > haven’t found any sample code for this requisite.
> >
> > Code: Flink Scala.
> >
> > Load: from CSV to local postgres database.
> >
> > Thanks
> >
> > Jagan
> >
> > 0044-7411239688
> >
> >
> >
> > --
> > Regards.
> > Jagan.
> >
> > 
> > -
> > *  The Good You Do Today, People will often forget tomorrow: Do Good
> > Anyway 
> >
>



-- 
Regards.
Jagan.

-
*  The Good You Do Today, People will often forget tomorrow: Do Good
Anyway 

Re: Flink Scala - data load from CSV to postgress database.

[Suneel Marthi]

Couldn't u use SQLLoader or something for doing that?

http://stackoverflow.com/questions/2987433/how-to-import-csv-file-data-into-a-postgresql-table



On Thu, Sep 22, 2016 at 3:01 PM, Jagan  wrote:

> Hi Guys,
>
> We have a requirement like – loading data from local CSV file to Postgress
> database using Flink Scala…We have tried number of ways all failed
>
> Do you have any example for this? With dependency libraries to understand
> how to load data from CSV to postgres
>
> We have tried and searched in Google/Flinkweb website for data load, we
> haven’t found any sample code for this requisite.
>
> Code: Flink Scala.
>
> Load: from CSV to local postgres database.
>
> Thanks
>
> Jagan
>
> 0044-7411239688
>
>
>
> --
> Regards.
> Jagan.
>
> 
> -
> *  The Good You Do Today, People will often forget tomorrow: Do Good
> Anyway 
>