Re: APEXMALHAR-2382 User needs to create dt_meta table while using JdbcPOJOInsertOutputOperator

[AJAY GUPTA]

Since the query can be different for different databases, the user will
have to provide query to the operator. Rather than this, I believe it's
easier for user to directly execute create table query on DB.

Also, the create table script won't be that heavy that we create script for
it. Probably adding a generic type of query in the docs itself should
suffice.


Ajay

On Sat, 14 Jan 2017 at 10:27 AM, Yogi Devendra 
wrote:

> As Aniruddha pointed out, table creation should be done by dbadmin.
>
> In that case, utility script will be helpful.
>
>
>
> If we embed this code inside operator or application; then it will be
>
> difficult for dbadmin to use it.
>
>
>
> ~ Yogi
>
>
>
> On 14 January 2017 at 03:43, Thomas Weise  wrote:
>
>
>
> > -1 for automatic schema modification, unless the user asked for it. See
>
> > comment on JIRA.
>
> >
>
> >
>
> > On Fri, Jan 13, 2017 at 5:11 AM, Aniruddha Thombare <
>
> > anirud...@datatorrent.com> wrote:
>
> >
>
> > > The tables should be created / altered by dbadmin.
>
> > > We shouldn't worry about table creations as its one-time activity.
>
> > >
>
> > >
>
> > >
>
> > > Thanks,
>
> > >
>
> > > A
>
> > >
>
> > >
>
> > > _
>
> > > Sent with difficulty, I mean handheld ;)
>
> > >
>
> > > On 13 Jan 2017 6:37 pm, "Yogi Devendra" 
> wrote:
>
> > >
>
> > > I am not very keen on having utility script.
>
> > > But, "no side-effects without explicit ask by the end-user" is
> important.
>
> > >
>
> > > ~ Yogi
>
> > >
>
> > > On 13 January 2017 at 16:44, Priyanka Gugale 
> wrote:
>
> > >
>
> > > > IMO it's okay to create table in java code. We should document it in
>
> > > > operator guide as well as put a log message when we create table.
>
> > > > And in case you don't have privileges, the operator should throw
>
> > > meaningful
>
> > > > message.
>
> > > >
>
> > > > -Priyanka
>
> > > >
>
> > > > On Fri, Jan 13, 2017 at 4:07 PM, Yogi Devendra <
>
> > yogideven...@apache.org>
>
> > > > wrote:
>
> > > >
>
> > > > > My suggestions:
>
> > > > >
>
> > > > >1. Have a separate utility script for creating this table.
>
> > > > >2. Have README for the utility script
>
> > > > >3. Mention about the utility script in the operator javadocs.
>
> > > > >4. Mention  about the utility script in the application README.
>
> > > > >5. If at all, you wish to ease out the process; you can
> introduce
>
> > > flag
>
> > > > >like autoPopulateMetaTable. But. default value of this flag
> should
>
> > > to
>
> > > > be
>
> > > > >off.
>
> > > > >6. I would prefer to avoid side-effects unless explicitly asked
> by
>
> > > the
>
> > > > >end user.
>
> > > > >7. Relevant exceptions should be caught and should have a
> message
>
> > > > which
>
> > > > >can be understood by the end user.
>
> > > > >
>
> > > > > ~ Yogi
>
> > > > >
>
> > > > > On 13 January 2017 at 15:57, Hitesh Kapoor  >
>
> > > > wrote:
>
> > > > >
>
> > > > > > Hi All,
>
> > > > > >
>
> > > > > > Currently to use JdbcPOJOInsertOutputOperator, user needs to
> create
>
> > > > > > "dt_meta" table to enforce
>
> > > > > > exactly-once processing semantic. If the user fails to create
> this
>
> > > > table
>
> > > > > > before launching the application an exception is thrown.
>
> > > > > > To handle this scenario we can automate the process of creating
>
> > this
>
> > > > > table,
>
> > > > > > assuming the user has the appropriate privileges. The problem
> with
>
> > > this
>
> > > > > > approach is that it may not be a very good idea to modify user's
>
> > > > database
>
> > > > > > automatically , also if the user doesn't has the appropriate
>
> > > privileges
>
> > > > > it
>
> > > > > > will eventually throw an exception (however a different
> exception).
>
> > > > > > So I need your opinion if we should automate the creation of this
>
> > > > > internal
>
> > > > > > table (if it doesn't exists) or continue with the existing
>
> > behaviour
>
> > > or
>
> > > > > > anything else.
>
> > > > > >
>
> > > > > > Regards,
>
> > > > > > Hitesh
>
> > > > > >
>
> > > > >
>
> > > >
>
> > >
>
> >
>
>

Re: APEXMALHAR-2382 User needs to create dt_meta table while using JdbcPOJOInsertOutputOperator

[Yogi Devendra]

As Aniruddha pointed out, table creation should be done by dbadmin.
In that case, utility script will be helpful.

If we embed this code inside operator or application; then it will be
difficult for dbadmin to use it.

~ Yogi

On 14 January 2017 at 03:43, Thomas Weise  wrote:

> -1 for automatic schema modification, unless the user asked for it. See
> comment on JIRA.
>
>
> On Fri, Jan 13, 2017 at 5:11 AM, Aniruddha Thombare <
> anirud...@datatorrent.com> wrote:
>
> > The tables should be created / altered by dbadmin.
> > We shouldn't worry about table creations as its one-time activity.
> >
> >
> >
> > Thanks,
> >
> > A
> >
> >
> > _
> > Sent with difficulty, I mean handheld ;)
> >
> > On 13 Jan 2017 6:37 pm, "Yogi Devendra"  wrote:
> >
> > I am not very keen on having utility script.
> > But, "no side-effects without explicit ask by the end-user" is important.
> >
> > ~ Yogi
> >
> > On 13 January 2017 at 16:44, Priyanka Gugale  wrote:
> >
> > > IMO it's okay to create table in java code. We should document it in
> > > operator guide as well as put a log message when we create table.
> > > And in case you don't have privileges, the operator should throw
> > meaningful
> > > message.
> > >
> > > -Priyanka
> > >
> > > On Fri, Jan 13, 2017 at 4:07 PM, Yogi Devendra <
> yogideven...@apache.org>
> > > wrote:
> > >
> > > > My suggestions:
> > > >
> > > >1. Have a separate utility script for creating this table.
> > > >2. Have README for the utility script
> > > >3. Mention about the utility script in the operator javadocs.
> > > >4. Mention  about the utility script in the application README.
> > > >5. If at all, you wish to ease out the process; you can introduce
> > flag
> > > >like autoPopulateMetaTable. But. default value of this flag should
> > to
> > > be
> > > >off.
> > > >6. I would prefer to avoid side-effects unless explicitly asked by
> > the
> > > >end user.
> > > >7. Relevant exceptions should be caught and should have a message
> > > which
> > > >can be understood by the end user.
> > > >
> > > > ~ Yogi
> > > >
> > > > On 13 January 2017 at 15:57, Hitesh Kapoor 
> > > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > Currently to use JdbcPOJOInsertOutputOperator, user needs to create
> > > > > "dt_meta" table to enforce
> > > > > exactly-once processing semantic. If the user fails to create this
> > > table
> > > > > before launching the application an exception is thrown.
> > > > > To handle this scenario we can automate the process of creating
> this
> > > > table,
> > > > > assuming the user has the appropriate privileges. The problem with
> > this
> > > > > approach is that it may not be a very good idea to modify user's
> > > database
> > > > > automatically , also if the user doesn't has the appropriate
> > privileges
> > > > it
> > > > > will eventually throw an exception (however a different exception).
> > > > > So I need your opinion if we should automate the creation of this
> > > > internal
> > > > > table (if it doesn't exists) or continue with the existing
> behaviour
> > or
> > > > > anything else.
> > > > >
> > > > > Regards,
> > > > > Hitesh
> > > > >
> > > >
> > >
> >
>

[GitHub] apex-malhar pull request #518: APEXMALHAR-2359 #resolve #comment Optimise fi...

[brightchen]

Github user brightchen closed the pull request at:

https://github.com/apache/apex-malhar/pull/518


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-malhar pull request #518: APEXMALHAR-2359 #resolve #comment Optimise fi...

[brightchen]

GitHub user brightchen reopened a pull request:

https://github.com/apache/apex-malhar/pull/518

APEXMALHAR-2359 #resolve #comment Optimise fire trigger to avoid go t…

…hrough all data

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/brightchen/apex-malhar APEXMALHAR-2359

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-malhar/pull/518.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #518


commit 5cea599d5a7965b3771c840d6399790c2ad8
Author: brightchen 
Date:   2016-11-29T23:05:09Z

APEXMALHAR-2359 #resolve #comment Optimise fire trigger to avoid go through 
all data




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: APEXMALHAR-2382 User needs to create dt_meta table while using JdbcPOJOInsertOutputOperator

[Thomas Weise]

-1 for automatic schema modification, unless the user asked for it. See
comment on JIRA.


On Fri, Jan 13, 2017 at 5:11 AM, Aniruddha Thombare <
anirud...@datatorrent.com> wrote:

> The tables should be created / altered by dbadmin.
> We shouldn't worry about table creations as its one-time activity.
>
>
>
> Thanks,
>
> A
>
>
> _
> Sent with difficulty, I mean handheld ;)
>
> On 13 Jan 2017 6:37 pm, "Yogi Devendra"  wrote:
>
> I am not very keen on having utility script.
> But, "no side-effects without explicit ask by the end-user" is important.
>
> ~ Yogi
>
> On 13 January 2017 at 16:44, Priyanka Gugale  wrote:
>
> > IMO it's okay to create table in java code. We should document it in
> > operator guide as well as put a log message when we create table.
> > And in case you don't have privileges, the operator should throw
> meaningful
> > message.
> >
> > -Priyanka
> >
> > On Fri, Jan 13, 2017 at 4:07 PM, Yogi Devendra 
> > wrote:
> >
> > > My suggestions:
> > >
> > >1. Have a separate utility script for creating this table.
> > >2. Have README for the utility script
> > >3. Mention about the utility script in the operator javadocs.
> > >4. Mention  about the utility script in the application README.
> > >5. If at all, you wish to ease out the process; you can introduce
> flag
> > >like autoPopulateMetaTable. But. default value of this flag should
> to
> > be
> > >off.
> > >6. I would prefer to avoid side-effects unless explicitly asked by
> the
> > >end user.
> > >7. Relevant exceptions should be caught and should have a message
> > which
> > >can be understood by the end user.
> > >
> > > ~ Yogi
> > >
> > > On 13 January 2017 at 15:57, Hitesh Kapoor 
> > wrote:
> > >
> > > > Hi All,
> > > >
> > > > Currently to use JdbcPOJOInsertOutputOperator, user needs to create
> > > > "dt_meta" table to enforce
> > > > exactly-once processing semantic. If the user fails to create this
> > table
> > > > before launching the application an exception is thrown.
> > > > To handle this scenario we can automate the process of creating this
> > > table,
> > > > assuming the user has the appropriate privileges. The problem with
> this
> > > > approach is that it may not be a very good idea to modify user's
> > database
> > > > automatically , also if the user doesn't has the appropriate
> privileges
> > > it
> > > > will eventually throw an exception (however a different exception).
> > > > So I need your opinion if we should automate the creation of this
> > > internal
> > > > table (if it doesn't exists) or continue with the existing behaviour
> or
> > > > anything else.
> > > >
> > > > Regards,
> > > > Hitesh
> > > >
> > >
> >
>

Re: APEXMALHAR-2382 User needs to create dt_meta table while using JdbcPOJOInsertOutputOperator

[Aniruddha Thombare]

The tables should be created / altered by dbadmin.
We shouldn't worry about table creations as its one-time activity.



Thanks,

A


_
Sent with difficulty, I mean handheld ;)

On 13 Jan 2017 6:37 pm, "Yogi Devendra"  wrote:

I am not very keen on having utility script.
But, "no side-effects without explicit ask by the end-user" is important.

~ Yogi

On 13 January 2017 at 16:44, Priyanka Gugale  wrote:

> IMO it's okay to create table in java code. We should document it in
> operator guide as well as put a log message when we create table.
> And in case you don't have privileges, the operator should throw
meaningful
> message.
>
> -Priyanka
>
> On Fri, Jan 13, 2017 at 4:07 PM, Yogi Devendra 
> wrote:
>
> > My suggestions:
> >
> >1. Have a separate utility script for creating this table.
> >2. Have README for the utility script
> >3. Mention about the utility script in the operator javadocs.
> >4. Mention  about the utility script in the application README.
> >5. If at all, you wish to ease out the process; you can introduce
flag
> >like autoPopulateMetaTable. But. default value of this flag should to
> be
> >off.
> >6. I would prefer to avoid side-effects unless explicitly asked by
the
> >end user.
> >7. Relevant exceptions should be caught and should have a message
> which
> >can be understood by the end user.
> >
> > ~ Yogi
> >
> > On 13 January 2017 at 15:57, Hitesh Kapoor 
> wrote:
> >
> > > Hi All,
> > >
> > > Currently to use JdbcPOJOInsertOutputOperator, user needs to create
> > > "dt_meta" table to enforce
> > > exactly-once processing semantic. If the user fails to create this
> table
> > > before launching the application an exception is thrown.
> > > To handle this scenario we can automate the process of creating this
> > table,
> > > assuming the user has the appropriate privileges. The problem with
this
> > > approach is that it may not be a very good idea to modify user's
> database
> > > automatically , also if the user doesn't has the appropriate
privileges
> > it
> > > will eventually throw an exception (however a different exception).
> > > So I need your opinion if we should automate the creation of this
> > internal
> > > table (if it doesn't exists) or continue with the existing behaviour
or
> > > anything else.
> > >
> > > Regards,
> > > Hitesh
> > >
> >
>

Re: APEXMALHAR-2382 User needs to create dt_meta table while using JdbcPOJOInsertOutputOperator

[Yogi Devendra]

I am not very keen on having utility script.
But, "no side-effects without explicit ask by the end-user" is important.

~ Yogi

On 13 January 2017 at 16:44, Priyanka Gugale  wrote:

> IMO it's okay to create table in java code. We should document it in
> operator guide as well as put a log message when we create table.
> And in case you don't have privileges, the operator should throw meaningful
> message.
>
> -Priyanka
>
> On Fri, Jan 13, 2017 at 4:07 PM, Yogi Devendra 
> wrote:
>
> > My suggestions:
> >
> >1. Have a separate utility script for creating this table.
> >2. Have README for the utility script
> >3. Mention about the utility script in the operator javadocs.
> >4. Mention  about the utility script in the application README.
> >5. If at all, you wish to ease out the process; you can introduce flag
> >like autoPopulateMetaTable. But. default value of this flag should to
> be
> >off.
> >6. I would prefer to avoid side-effects unless explicitly asked by the
> >end user.
> >7. Relevant exceptions should be caught and should have a message
> which
> >can be understood by the end user.
> >
> > ~ Yogi
> >
> > On 13 January 2017 at 15:57, Hitesh Kapoor 
> wrote:
> >
> > > Hi All,
> > >
> > > Currently to use JdbcPOJOInsertOutputOperator, user needs to create
> > > "dt_meta" table to enforce
> > > exactly-once processing semantic. If the user fails to create this
> table
> > > before launching the application an exception is thrown.
> > > To handle this scenario we can automate the process of creating this
> > table,
> > > assuming the user has the appropriate privileges. The problem with this
> > > approach is that it may not be a very good idea to modify user's
> database
> > > automatically , also if the user doesn't has the appropriate privileges
> > it
> > > will eventually throw an exception (however a different exception).
> > > So I need your opinion if we should automate the creation of this
> > internal
> > > table (if it doesn't exists) or continue with the existing behaviour or
> > > anything else.
> > >
> > > Regards,
> > > Hitesh
> > >
> >
>

Re: APEXMALHAR-2382 User needs to create dt_meta table while using JdbcPOJOInsertOutputOperator

[Priyanka Gugale]

IMO it's okay to create table in java code. We should document it in
operator guide as well as put a log message when we create table.
And in case you don't have privileges, the operator should throw meaningful
message.

-Priyanka

On Fri, Jan 13, 2017 at 4:07 PM, Yogi Devendra 
wrote:

> My suggestions:
>
>1. Have a separate utility script for creating this table.
>2. Have README for the utility script
>3. Mention about the utility script in the operator javadocs.
>4. Mention  about the utility script in the application README.
>5. If at all, you wish to ease out the process; you can introduce flag
>like autoPopulateMetaTable. But. default value of this flag should to be
>off.
>6. I would prefer to avoid side-effects unless explicitly asked by the
>end user.
>7. Relevant exceptions should be caught and should have a message which
>can be understood by the end user.
>
> ~ Yogi
>
> On 13 January 2017 at 15:57, Hitesh Kapoor  wrote:
>
> > Hi All,
> >
> > Currently to use JdbcPOJOInsertOutputOperator, user needs to create
> > "dt_meta" table to enforce
> > exactly-once processing semantic. If the user fails to create this table
> > before launching the application an exception is thrown.
> > To handle this scenario we can automate the process of creating this
> table,
> > assuming the user has the appropriate privileges. The problem with this
> > approach is that it may not be a very good idea to modify user's database
> > automatically , also if the user doesn't has the appropriate privileges
> it
> > will eventually throw an exception (however a different exception).
> > So I need your opinion if we should automate the creation of this
> internal
> > table (if it doesn't exists) or continue with the existing behaviour or
> > anything else.
> >
> > Regards,
> > Hitesh
> >
>

[GitHub] apex-malhar pull request #528: APEXMALHAR-2377-Move LopParser to org.apache....

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/apex-malhar/pull/528


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: APEXMALHAR-2382 User needs to create dt_meta table while using JdbcPOJOInsertOutputOperator

[Yogi Devendra]

My suggestions:

   1. Have a separate utility script for creating this table.
   2. Have README for the utility script
   3. Mention about the utility script in the operator javadocs.
   4. Mention  about the utility script in the application README.
   5. If at all, you wish to ease out the process; you can introduce flag
   like autoPopulateMetaTable. But. default value of this flag should to be
   off.
   6. I would prefer to avoid side-effects unless explicitly asked by the
   end user.
   7. Relevant exceptions should be caught and should have a message which
   can be understood by the end user.

~ Yogi

On 13 January 2017 at 15:57, Hitesh Kapoor  wrote:

> Hi All,
>
> Currently to use JdbcPOJOInsertOutputOperator, user needs to create
> "dt_meta" table to enforce
> exactly-once processing semantic. If the user fails to create this table
> before launching the application an exception is thrown.
> To handle this scenario we can automate the process of creating this table,
> assuming the user has the appropriate privileges. The problem with this
> approach is that it may not be a very good idea to modify user's database
> automatically , also if the user doesn't has the appropriate privileges it
> will eventually throw an exception (however a different exception).
> So I need your opinion if we should automate the creation of this internal
> table (if it doesn't exists) or continue with the existing behaviour or
> anything else.
>
> Regards,
> Hitesh
>

APEXMALHAR-2382 User needs to create dt_meta table while using JdbcPOJOInsertOutputOperator

[Hitesh Kapoor]

Hi All,

Currently to use JdbcPOJOInsertOutputOperator, user needs to create
"dt_meta" table to enforce
exactly-once processing semantic. If the user fails to create this table
before launching the application an exception is thrown.
To handle this scenario we can automate the process of creating this table,
assuming the user has the appropriate privileges. The problem with this
approach is that it may not be a very good idea to modify user's database
automatically , also if the user doesn't has the appropriate privileges it
will eventually throw an exception (however a different exception).
So I need your opinion if we should automate the creation of this internal
table (if it doesn't exists) or continue with the existing behaviour or
anything else.

Regards,
Hitesh