Re: (APEXMALHAR-2340) Initialize the list of JdbcFieldInfo in JdbcPOJOInsertOutput operator from properties.xml

[Hitesh Kapoor]

Thank you for the inputs.
I will go ahead with JSON based input for mapping.
For e.g. the user will initialize the list of JdbcFieldInfos from
properties.xml file as follows:


dt.operator.JdbcOutput.fieldInfosItem[0]

{
"sqlType": 0,
"coumnName":"customerName",
"pojoFieldExpression": "customerName",
"type":"STRING"
}


  

  
dt.operator.JdbcOutput.fieldInfosItem[1]

{
"sqlType": 0,
"coumnName":"customerPhone",
"pojoFieldExpression": "customerPhone",
"type":"STRING"
}


  


Regards,
Hitesh Kapoor


On Tue, Nov 15, 2016 at 11:41 PM, Sanjay Pujare 
wrote:

> +1 for standardized JSON based mapping/schema definitions
>
> On Mon, Nov 14, 2016 at 11:15 PM, Priyanka Gugale 
> wrote:
>
> > +1 for having json based input for mappings.
> >
> > -Priyanka
> >
> > On Mon, Nov 14, 2016 at 11:21 PM, Devendra Tagare <
> > devend...@datatorrent.com
> > > wrote:
> >
> > > Hi,
> > >
> > > CSV schemas formats are based on delimited schemas which are meant to
> be
> > > sequence sensitive ref : DelimitedSchema
> > >  > > contrib/src/main/java/com/datatorrent/contrib/parser/
> > DelimitedSchema.java>
> > > and
> > > don't have a notion of input to output field mappings.
> > >
> > > Field info mappings for output operators are typically are a of the
> form
> > -
> > > destFieldName:pojoFieldName:type/supportType and are not intended to
> be
> > > sequence sensitive.
> > >
> > > We can go with a JSON based structure which maps sources to
> destinations
> > > with their respective types,
> > >
> > > {
> > >   "destinationFieldName": "destination field name",
> > >   "destType" : "support type, type",
> > >   "srcFieldName" : "source pojo field name",
> > >   "srcType" : "support type, type",
> > >   "constraints" : "constraint expression"
> > > }
> > >
> > > Thanks,
> > > Dev
> > >
> > >
> > >
> > >
> > >
> > > Thanks,
> > > Dev
> > >
> > > On Mon, Nov 14, 2016 at 9:27 AM, Ashwin Chandra Putta <
> > > ashwinchand...@gmail.com> wrote:
> > >
> > > > Hitesh,
> > > >
> > > > We should standardize the schema definition across apex for
> individual
> > > > operators and tuple classes.
> > > >
> > > > I think you should be able to use schema definition for CSV parser
> > > without
> > > > the delimiter.
> > > >
> > > > Regards,
> > > > Ashwin.
> > > >
> > > > On Nov 14, 2016 2:45 AM, "Hitesh Kapoor" 
> > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > Currently in JdbcPOJOInsertOuput operator we cannot configure
> > > > JdbcFieldInfo
> > > > > via properties.xml and the user has to do the necessary coding in
> his
> > > > > application.
> > > > >
> > > > > To implement this improvement, the approach mentioned in
> > > > > http://docs.datatorrent.com/application_packages/#
> > operator-properties
> > > > > could
> > > > > be followed.
> > > > > Now we need to provide the user a format for specifying the value
> of
> > > > > fieldInfo.
> > > > >
> > > > > Kindly let me know which of the following is the best format to be
> > used
> > > > for
> > > > > this
> > > > > 1) CSV string (or any delimited string) with values for data
> members
> > of
> > > > > JdbcFieldInfo in a fixed sequence.
> > > > > 2) JSON format with appropriate mapping.
> > > > > 3) XML format with appropriate name tags and values.
> > > > >
> > > > > Regards,
> > > > > Hitesh
> > > > >
> > > >
> > >
> >
>

Re: (APEXMALHAR-2340) Initialize the list of JdbcFieldInfo in JdbcPOJOInsertOutput operator from properties.xml

[Sanjay Pujare]

+1 for standardized JSON based mapping/schema definitions

On Mon, Nov 14, 2016 at 11:15 PM, Priyanka Gugale  wrote:

> +1 for having json based input for mappings.
>
> -Priyanka
>
> On Mon, Nov 14, 2016 at 11:21 PM, Devendra Tagare <
> devend...@datatorrent.com
> > wrote:
>
> > Hi,
> >
> > CSV schemas formats are based on delimited schemas which are meant to be
> > sequence sensitive ref : DelimitedSchema
> >  > contrib/src/main/java/com/datatorrent/contrib/parser/
> DelimitedSchema.java>
> > and
> > don't have a notion of input to output field mappings.
> >
> > Field info mappings for output operators are typically are a of the form
> -
> > destFieldName:pojoFieldName:type/supportType and are not intended to be
> > sequence sensitive.
> >
> > We can go with a JSON based structure which maps sources to destinations
> > with their respective types,
> >
> > {
> >   "destinationFieldName": "destination field name",
> >   "destType" : "support type, type",
> >   "srcFieldName" : "source pojo field name",
> >   "srcType" : "support type, type",
> >   "constraints" : "constraint expression"
> > }
> >
> > Thanks,
> > Dev
> >
> >
> >
> >
> >
> > Thanks,
> > Dev
> >
> > On Mon, Nov 14, 2016 at 9:27 AM, Ashwin Chandra Putta <
> > ashwinchand...@gmail.com> wrote:
> >
> > > Hitesh,
> > >
> > > We should standardize the schema definition across apex for individual
> > > operators and tuple classes.
> > >
> > > I think you should be able to use schema definition for CSV parser
> > without
> > > the delimiter.
> > >
> > > Regards,
> > > Ashwin.
> > >
> > > On Nov 14, 2016 2:45 AM, "Hitesh Kapoor" 
> wrote:
> > >
> > > > Hi All,
> > > >
> > > > Currently in JdbcPOJOInsertOuput operator we cannot configure
> > > JdbcFieldInfo
> > > > via properties.xml and the user has to do the necessary coding in his
> > > > application.
> > > >
> > > > To implement this improvement, the approach mentioned in
> > > > http://docs.datatorrent.com/application_packages/#
> operator-properties
> > > > could
> > > > be followed.
> > > > Now we need to provide the user a format for specifying the value of
> > > > fieldInfo.
> > > >
> > > > Kindly let me know which of the following is the best format to be
> used
> > > for
> > > > this
> > > > 1) CSV string (or any delimited string) with values for data members
> of
> > > > JdbcFieldInfo in a fixed sequence.
> > > > 2) JSON format with appropriate mapping.
> > > > 3) XML format with appropriate name tags and values.
> > > >
> > > > Regards,
> > > > Hitesh
> > > >
> > >
> >
>

Re: (APEXMALHAR-2340) Initialize the list of JdbcFieldInfo in JdbcPOJOInsertOutput operator from properties.xml

[Priyanka Gugale]

+1 for having json based input for mappings.

-Priyanka

On Mon, Nov 14, 2016 at 11:21 PM, Devendra Tagare  wrote:

> Hi,
>
> CSV schemas formats are based on delimited schemas which are meant to be
> sequence sensitive ref : DelimitedSchema
>  contrib/src/main/java/com/datatorrent/contrib/parser/DelimitedSchema.java>
> and
> don't have a notion of input to output field mappings.
>
> Field info mappings for output operators are typically are a of the form -
> destFieldName:pojoFieldName:type/supportType and are not intended to be
> sequence sensitive.
>
> We can go with a JSON based structure which maps sources to destinations
> with their respective types,
>
> {
>   "destinationFieldName": "destination field name",
>   "destType" : "support type, type",
>   "srcFieldName" : "source pojo field name",
>   "srcType" : "support type, type",
>   "constraints" : "constraint expression"
> }
>
> Thanks,
> Dev
>
>
>
>
>
> Thanks,
> Dev
>
> On Mon, Nov 14, 2016 at 9:27 AM, Ashwin Chandra Putta <
> ashwinchand...@gmail.com> wrote:
>
> > Hitesh,
> >
> > We should standardize the schema definition across apex for individual
> > operators and tuple classes.
> >
> > I think you should be able to use schema definition for CSV parser
> without
> > the delimiter.
> >
> > Regards,
> > Ashwin.
> >
> > On Nov 14, 2016 2:45 AM, "Hitesh Kapoor"  wrote:
> >
> > > Hi All,
> > >
> > > Currently in JdbcPOJOInsertOuput operator we cannot configure
> > JdbcFieldInfo
> > > via properties.xml and the user has to do the necessary coding in his
> > > application.
> > >
> > > To implement this improvement, the approach mentioned in
> > > http://docs.datatorrent.com/application_packages/#operator-properties
> > > could
> > > be followed.
> > > Now we need to provide the user a format for specifying the value of
> > > fieldInfo.
> > >
> > > Kindly let me know which of the following is the best format to be used
> > for
> > > this
> > > 1) CSV string (or any delimited string) with values for data members of
> > > JdbcFieldInfo in a fixed sequence.
> > > 2) JSON format with appropriate mapping.
> > > 3) XML format with appropriate name tags and values.
> > >
> > > Regards,
> > > Hitesh
> > >
> >
>

Re: (APEXMALHAR-2340) Initialize the list of JdbcFieldInfo in JdbcPOJOInsertOutput operator from properties.xml

[Devendra Tagare]

Hi,

CSV schemas formats are based on delimited schemas which are meant to be
sequence sensitive ref : DelimitedSchema

and
don't have a notion of input to output field mappings.

Field info mappings for output operators are typically are a of the form -
destFieldName:pojoFieldName:type/supportType and are not intended to be
sequence sensitive.

We can go with a JSON based structure which maps sources to destinations
with their respective types,

{
  "destinationFieldName": "destination field name",
  "destType" : "support type, type",
  "srcFieldName" : "source pojo field name",
  "srcType" : "support type, type",
  "constraints" : "constraint expression"
}

Thanks,
Dev





Thanks,
Dev

On Mon, Nov 14, 2016 at 9:27 AM, Ashwin Chandra Putta <
ashwinchand...@gmail.com> wrote:

> Hitesh,
>
> We should standardize the schema definition across apex for individual
> operators and tuple classes.
>
> I think you should be able to use schema definition for CSV parser without
> the delimiter.
>
> Regards,
> Ashwin.
>
> On Nov 14, 2016 2:45 AM, "Hitesh Kapoor"  wrote:
>
> > Hi All,
> >
> > Currently in JdbcPOJOInsertOuput operator we cannot configure
> JdbcFieldInfo
> > via properties.xml and the user has to do the necessary coding in his
> > application.
> >
> > To implement this improvement, the approach mentioned in
> > http://docs.datatorrent.com/application_packages/#operator-properties
> > could
> > be followed.
> > Now we need to provide the user a format for specifying the value of
> > fieldInfo.
> >
> > Kindly let me know which of the following is the best format to be used
> for
> > this
> > 1) CSV string (or any delimited string) with values for data members of
> > JdbcFieldInfo in a fixed sequence.
> > 2) JSON format with appropriate mapping.
> > 3) XML format with appropriate name tags and values.
> >
> > Regards,
> > Hitesh
> >
>

Re: (APEXMALHAR-2340) Initialize the list of JdbcFieldInfo in JdbcPOJOInsertOutput operator from properties.xml

[Ashwin Chandra Putta]

Hitesh,

We should standardize the schema definition across apex for individual
operators and tuple classes.

I think you should be able to use schema definition for CSV parser without
the delimiter.

Regards,
Ashwin.

On Nov 14, 2016 2:45 AM, "Hitesh Kapoor"  wrote:

> Hi All,
>
> Currently in JdbcPOJOInsertOuput operator we cannot configure JdbcFieldInfo
> via properties.xml and the user has to do the necessary coding in his
> application.
>
> To implement this improvement, the approach mentioned in
> http://docs.datatorrent.com/application_packages/#operator-properties
> could
> be followed.
> Now we need to provide the user a format for specifying the value of
> fieldInfo.
>
> Kindly let me know which of the following is the best format to be used for
> this
> 1) CSV string (or any delimited string) with values for data members of
> JdbcFieldInfo in a fixed sequence.
> 2) JSON format with appropriate mapping.
> 3) XML format with appropriate name tags and values.
>
> Regards,
> Hitesh
>

(APEXMALHAR-2340) Initialize the list of JdbcFieldInfo in JdbcPOJOInsertOutput operator from properties.xml

[Hitesh Kapoor]

Hi All,

Currently in JdbcPOJOInsertOuput operator we cannot configure JdbcFieldInfo
via properties.xml and the user has to do the necessary coding in his
application.

To implement this improvement, the approach mentioned in
http://docs.datatorrent.com/application_packages/#operator-properties could
be followed.
Now we need to provide the user a format for specifying the value of
fieldInfo.

Kindly let me know which of the following is the best format to be used for
this
1) CSV string (or any delimited string) with values for data members of
JdbcFieldInfo in a fixed sequence.
2) JSON format with appropriate mapping.
3) XML format with appropriate name tags and values.

Regards,
Hitesh