APEXMALHAR-2261 Python Binding for HighLevel APIs

[vikram patil]

Hi All,

I would like to take up development for python binding implementation for
highlevel APIs (APEXMALHAR-2261 ). I went over High-Level APIs from Apache
Malhar Stream API project. It can be initiated as separated project in the
Apache Malhar project just like sql or stream project.

In first phase I would like to focus on providing python binding for
following APIs:

1) StreamFactory.fromFolder
2) StreamFactory.fromKafka*
3) StreamFactory.fromLocal
4) StreamFactory.fromInput
5) ApexStream.map
6) ApexStream.flatMap
7) ApexStream.filter
9) ApexStream.endWith
11) ApexStream.setGlobalAttribute
12) Custom functions in python .


Rest of the Apex HighLevel APIs such as addStream, addOperator can be
implemented as part of phase II .


For integration of this purpose,I would like to use py4j as python-java
binding due to wide acceptance and very good community support.  Also py4j
also allows call backs to python code from java which can make certain
functionalities easier to implement.

Py4j Version: 0.10.4

Please share your suggestions about this implementation.

Thanks & Regards,
Vikram

[jira] [Resolved] (APEXCORE-611) Stram Event Log Levels

[Vlad Rozov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXCORE-611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vlad Rozov resolved APEXCORE-611.
-
   Resolution: Implemented
Fix Version/s: 3.6.0

> Stram Event Log Levels
> --
>
> Key: APEXCORE-611
> URL: https://issues.apache.org/jira/browse/APEXCORE-611
> Project: Apache Apex Core
>  Issue Type: Improvement
>Reporter: Ajay Gupta
>Assignee: Ajay Gupta
>Priority: Minor
> Fix For: 3.6.0
>
>
> Provide log levels for Stram events. Such as INFO, WARN, ERROR
> Eg: 
> 1. Start Container, Start Operator are INFO level events
> 2. OperatorError is ERROR level event
> 3. Stop Container, Stop Operator are WARN level events
> Log level for events can help in user experience when showing the event list 
> in a GUI. eg: In datatorrent gateway UI, we want to provide color-coded log 
> levels so that user can focus more on ERROR and WARN events.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (APEXCORE-611) Stram Event Log Levels

[Vlad Rozov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXCORE-611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vlad Rozov updated APEXCORE-611:

Assignee: Ajay Gupta

> Stram Event Log Levels
> --
>
> Key: APEXCORE-611
> URL: https://issues.apache.org/jira/browse/APEXCORE-611
> Project: Apache Apex Core
>  Issue Type: Improvement
>Reporter: Ajay Gupta
>Assignee: Ajay Gupta
>
> Provide log levels for Stram events. Such as INFO, WARN, ERROR
> Eg: 
> 1. Start Container, Start Operator are INFO level events
> 2. OperatorError is ERROR level event
> 3. Stop Container, Stop Operator are WARN level events
> Log level for events can help in user experience when showing the event list 
> in a GUI. eg: In datatorrent gateway UI, we want to provide color-coded log 
> levels so that user can focus more on ERROR and WARN events.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (APEXCORE-611) Stram Event Log Levels

[Vlad Rozov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXCORE-611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vlad Rozov updated APEXCORE-611:

Priority: Minor  (was: Major)

> Stram Event Log Levels
> --
>
> Key: APEXCORE-611
> URL: https://issues.apache.org/jira/browse/APEXCORE-611
> Project: Apache Apex Core
>  Issue Type: Improvement
>Reporter: Ajay Gupta
>Assignee: Ajay Gupta
>Priority: Minor
>
> Provide log levels for Stram events. Such as INFO, WARN, ERROR
> Eg: 
> 1. Start Container, Start Operator are INFO level events
> 2. OperatorError is ERROR level event
> 3. Stop Container, Stop Operator are WARN level events
> Log level for events can help in user experience when showing the event list 
> in a GUI. eg: In datatorrent gateway UI, we want to provide color-coded log 
> levels so that user can focus more on ERROR and WARN events.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXCORE-611) Stram Event Log Levels

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15843872#comment-15843872
 ] 

ASF GitHub Bot commented on APEXCORE-611:
-

Github user asfgit closed the pull request at:

https://github.com/apache/apex-core/pull/454


> Stram Event Log Levels
> --
>
> Key: APEXCORE-611
> URL: https://issues.apache.org/jira/browse/APEXCORE-611
> Project: Apache Apex Core
>  Issue Type: Improvement
>Reporter: Ajay Gupta
>
> Provide log levels for Stram events. Such as INFO, WARN, ERROR
> Eg: 
> 1. Start Container, Start Operator are INFO level events
> 2. OperatorError is ERROR level event
> 3. Stop Container, Stop Operator are WARN level events
> Log level for events can help in user experience when showing the event list 
> in a GUI. eg: In datatorrent gateway UI, we want to provide color-coded log 
> levels so that user can focus more on ERROR and WARN events.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] apex-core pull request #454: APEXCORE-611 Added log levels for Stram Events

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/apex-core/pull/454


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (APEXCORE-627) Unit test AtMostOnceTest intermittently fails

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15843654#comment-15843654
 ] 

ASF GitHub Bot commented on APEXCORE-627:
-

GitHub user sgolovko opened a pull request:

https://github.com/apache/apex-core/pull/460

APEXCORE-627 : Unit test AtMostOnceTest intermittently fails

Fixed a race problem for calling checkpointed and committed in 
RecoverableInputOperator. The original implementation of the class used the 
method checkpointed() of the interface CheckpointListener to refresh a value of 
one of the criteria variables checkpointedWindowId. The fix update uses the 
method beforeCheckpoint() of the interface CheckpointNotificationListener. It 
guaranties that the update of the variable checkpointedWindowId will be done 
before the call of the method committed().

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sgolovko/apex-core APEXCORE-627

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-core/pull/460.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #460


commit 0c7e091e1bede9b70768bc91b5ee86a6b15f1a9a
Author: Sergey Golovko 
Date:   2017-01-27T23:15:09Z

APEXCORE-627 : Unit test AtMostOnceTest intermittently fails

Fixed a race problem for calling checkpointed and committed in 
RecoverableInputOperator. The original implementation of the class used the 
method checkpointed() of the interface CheckpointListener to refresh a value of 
one of the criteria variables checkpointedWindowId. The fix update uses the 
method beforeCheckpoint() of the interface CheckpointNotificationListener. It 
guaranties that the update of the variable checkpointedWindowId will be done 
before the call of the method committed().




> Unit test AtMostOnceTest intermittently fails
> -
>
> Key: APEXCORE-627
> URL: https://issues.apache.org/jira/browse/APEXCORE-627
> Project: Apache Apex Core
>  Issue Type: Bug
> Environment: The test is reproducible on macOS Sierra, Processor 2.2 
> GHz Intel Core i7, Memory 16GB 1600 MHz DDR3.
>Reporter: Sergey Golovko
>Assignee: Sergey Golovko
>Priority: Minor
>
> The test AtMostOnceTest is not able to reach the criteria to stop the test. 
> And it continue to recover an input operator and rerun the test in a loop.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] apex-core pull request #460: APEXCORE-627 : Unit test AtMostOnceTest intermi...

[sgolovko]

GitHub user sgolovko opened a pull request:

https://github.com/apache/apex-core/pull/460

APEXCORE-627 : Unit test AtMostOnceTest intermittently fails

Fixed a race problem for calling checkpointed and committed in 
RecoverableInputOperator. The original implementation of the class used the 
method checkpointed() of the interface CheckpointListener to refresh a value of 
one of the criteria variables checkpointedWindowId. The fix update uses the 
method beforeCheckpoint() of the interface CheckpointNotificationListener. It 
guaranties that the update of the variable checkpointedWindowId will be done 
before the call of the method committed().

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sgolovko/apex-core APEXCORE-627

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-core/pull/460.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #460


commit 0c7e091e1bede9b70768bc91b5ee86a6b15f1a9a
Author: Sergey Golovko 
Date:   2017-01-27T23:15:09Z

APEXCORE-627 : Unit test AtMostOnceTest intermittently fails

Fixed a race problem for calling checkpointed and committed in 
RecoverableInputOperator. The original implementation of the class used the 
method checkpointed() of the interface CheckpointListener to refresh a value of 
one of the criteria variables checkpointedWindowId. The fix update uses the 
method beforeCheckpoint() of the interface CheckpointNotificationListener. It 
guaranties that the update of the variable checkpointedWindowId will be done 
before the call of the method committed().




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #459: Fix download link in README.md

[tweise]

GitHub user tweise opened a pull request:

https://github.com/apache/apex-core/pull/459

Fix download link in README.md

@davidyan74 review

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tweise/apex-core master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-core/pull/459.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #459


commit 895633241640f2e29ca7b772f11d0e43eadb2519
Author: Thomas Weise 
Date:   2017-01-27T22:24:16Z

Fix download link in README.md




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (APEXCORE-627) Unit test AtMostOnceTest intermittently fails

[Sergey Golovko (JIRA)]

Sergey Golovko created APEXCORE-627:
---

 Summary: Unit test AtMostOnceTest intermittently fails
 Key: APEXCORE-627
 URL: https://issues.apache.org/jira/browse/APEXCORE-627
 Project: Apache Apex Core
  Issue Type: Bug
 Environment: The test is reproducible on macOS Sierra, Processor 2.2 
GHz Intel Core i7, Memory 16GB 1600 MHz DDR3.

Reporter: Sergey Golovko
Assignee: Sergey Golovko
Priority: Minor


The test AtMostOnceTest is not able to reach the criteria to stop the test. And 
it continue to recover an input operator and rerun the test in a loop.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXCORE-606) Suggestion: Optimise Kryo Output

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15843358#comment-15843358
 ] 

ASF GitHub Bot commented on APEXCORE-606:
-

Github user brightchen closed the pull request at:

https://github.com/apache/apex-core/pull/455


> Suggestion: Optimise Kryo Output
> 
>
> Key: APEXCORE-606
> URL: https://issues.apache.org/jira/browse/APEXCORE-606
> Project: Apache Apex Core
>  Issue Type: New Feature
>Reporter: bright chen
>Assignee: bright chen
>
> The kryo Output has some limitation
>   - The size of the data is limited. kryo write data to the buffer, it will 
> throw the overflow exception if the data exceed the size
>   - The Output.toBytes() will copy the data to temporary buffer and output, 
> it will  decrease the performance and introduce garbage collection.
> When I was tuning Spillable Data structure and Manage State. I create a 
> mechanism to share and reuse the memory to avoid above problem.  And it can 
> be reused in core serialization with small change. Please see jira: 
> https://issues.apache.org/jira/browse/APEXMALHAR-2190



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] apex-core pull request #455: APEXCORE-606-RV: review only: optimise kryo out...

[brightchen]

Github user brightchen closed the pull request at:

https://github.com/apache/apex-core/pull/455


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] Policy for patches

[Vlad Rozov]

I prefer to go with the second approach as well.

My preference is to go not with a strict end of life policy, but by 
severity of an issue and complexity of providing fixes for all 
subsequent releases. In a case a contributor decides to fix a bug in
an old release, she will need to provide the fix for many branches. It 
is unlikely that such work will be done without justification.


I am strongly against deleting old branches:
- they preserve history.
- I am not 100% sure, but it is likely against ASF policy. Any 
contribution to a project needs to be preserved (including author of a 
commit).
- It does not cost much to have branches in remote git repository and it 
does not affect git operations

- It is not necessary to load all branches into local repository

Thank you,

Vlad

On 1/27/17 10:16, Sanjay Pujare wrote:

A strong +1 for the second approach for the reasons Pramod mentioned.

Is it also possible to “prune” branches so that we have less of this activity 
of merging fixes across branches? If we can ascertain that a certain branch is 
not used by any user/customer (by asking in the community) we should be able to 
remove it. For example, apex-malhar has release-3.6 which is definitely 
required but 3 year old branches like release-0.8.5, release-0.9.0, … telecom 
most probably are not being used by anybody.

On 1/27/17, 8:43 AM, "Pramod Immaneni"  wrote:

 Hi,
 
 I wanted to bring up the topic of patches for issues discovered in older

 releases and start a discussion to come up with a policy on how to apply
 them.
 
 One approach is the patch gets only applied to the release it was

 discovered in and master. Another approach is it gets applied to all
 release branches >= discovered release and master. There may be other
 approaches as well which can come up in this discussion.
 
 The advantage of the first approach is that the immediate work is limited

 to a single fix and merge. The second approach requires more work initially
 as the patch needs to get applied to more one or more places.
 
 I am tending towards the second approach of applying the fix to all release

 branches >= discovered release, while also having some sort of an end of
 life policy for releases otherwise it might become difficult to manage the
 fixes. The end of life policy would mean that beyond a time period
 (something reasonable period between 1 - 3 years) after the release is made
 the release branch become frozen and no more fixes are applied to it.
 Consider the following two problematic scenarios if we apply the fix only
 to the one discovered release.
 
- Let's say tomorrow a new fix needs to be made to a newer release

branch (which had existed at the time of the fix) that is dependent on 
the
current fix. Before applying the new fix one would need to first know to
cherry-pick the old fix (may not be trivial to know this) and second and
more important there could be merge conflicts when cherry-picking. At 
this
point, you may be trying to resolve the conflicts where you may not be 
the
primary author of the fix and/or the knowledge of the fix is no longer
fresh in folks minds. This is prone to errors.
 
 
- Let's call this fix a. Let's say there was another fix b. requiring a.

to work correctly also on the same release branch but no compile
dependencies on a. If in future we have a fix c. that depends on b. that
needs to be applied on a different release branch both b. and a. would 
need
to be cherry picked. It might be easy to figure out b. is needed, as c. 
has
a direct dependency to it but would not be easy to determine that a. is
also needed since that is old knowledge. It will not be easy to catch 
this
omission because of no compile errors if a. is not included.
 
 Thanks
 

[GitHub] apex-core pull request #458: fix the issue by decrementing unallocated conta...

[sanjaypujare]

GitHub user sanjaypujare opened a pull request:

https://github.com/apache/apex-core/pull/458

fix the issue by decrementing unallocated containers and also released 
containers so that the exit condition for the shutdown check is satisfied.

@sandeshh  Pls review and merge. If okay I will also merge this into master 
and all the later branches.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sanjaypujare/apex-core 
APEXCORE-624.branch-3.2.sanjay

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-core/pull/458.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #458


commit b45825ae1aa635fecaf50063567700a8a5d9b7ee
Author: Sanjay Pujare 
Date:   2017-01-27T18:35:09Z

fix the issue by decrementing unallocated containers and also released 
containers so that the exit condition for the shutdown check is satisfied.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] Policy for patches

[Matt Zhang]

+1 for 2nd approach. We can omit those frozen branches and maintain only
active branches.

Thanks,
Matt

On Fri, Jan 27, 2017 at 10:16 AM, Sanjay Pujare 
wrote:

> A strong +1 for the second approach for the reasons Pramod mentioned.
>
> Is it also possible to “prune” branches so that we have less of this
> activity of merging fixes across branches? If we can ascertain that a
> certain branch is not used by any user/customer (by asking in the
> community) we should be able to remove it. For example, apex-malhar has
> release-3.6 which is definitely required but 3 year old branches like
> release-0.8.5, release-0.9.0, … telecom most probably are not being used by
> anybody.
>
> On 1/27/17, 8:43 AM, "Pramod Immaneni"  wrote:
>
> Hi,
>
> I wanted to bring up the topic of patches for issues discovered in
> older
> releases and start a discussion to come up with a policy on how to
> apply
> them.
>
> One approach is the patch gets only applied to the release it was
> discovered in and master. Another approach is it gets applied to all
> release branches >= discovered release and master. There may be other
> approaches as well which can come up in this discussion.
>
> The advantage of the first approach is that the immediate work is
> limited
> to a single fix and merge. The second approach requires more work
> initially
> as the patch needs to get applied to more one or more places.
>
> I am tending towards the second approach of applying the fix to all
> release
> branches >= discovered release, while also having some sort of an end
> of
> life policy for releases otherwise it might become difficult to manage
> the
> fixes. The end of life policy would mean that beyond a time period
> (something reasonable period between 1 - 3 years) after the release is
> made
> the release branch become frozen and no more fixes are applied to it.
> Consider the following two problematic scenarios if we apply the fix
> only
> to the one discovered release.
>
>- Let's say tomorrow a new fix needs to be made to a newer release
>branch (which had existed at the time of the fix) that is dependent
> on the
>current fix. Before applying the new fix one would need to first
> know to
>cherry-pick the old fix (may not be trivial to know this) and
> second and
>more important there could be merge conflicts when cherry-picking.
> At this
>point, you may be trying to resolve the conflicts where you may not
> be the
>primary author of the fix and/or the knowledge of the fix is no
> longer
>fresh in folks minds. This is prone to errors.
>
>
>- Let's call this fix a. Let's say there was another fix b.
> requiring a.
>to work correctly also on the same release branch but no compile
>dependencies on a. If in future we have a fix c. that depends on b.
> that
>needs to be applied on a different release branch both b. and a.
> would need
>to be cherry picked. It might be easy to figure out b. is needed,
> as c. has
>a direct dependency to it but would not be easy to determine that
> a. is
>also needed since that is old knowledge. It will not be easy to
> catch this
>omission because of no compile errors if a. is not included.
>
> Thanks
>
>
>
>

Re: [DISCUSS] Policy for patches

[Sanjay Pujare]

A strong +1 for the second approach for the reasons Pramod mentioned.

Is it also possible to “prune” branches so that we have less of this activity 
of merging fixes across branches? If we can ascertain that a certain branch is 
not used by any user/customer (by asking in the community) we should be able to 
remove it. For example, apex-malhar has release-3.6 which is definitely 
required but 3 year old branches like release-0.8.5, release-0.9.0, … telecom 
most probably are not being used by anybody.

On 1/27/17, 8:43 AM, "Pramod Immaneni"  wrote:

Hi,

I wanted to bring up the topic of patches for issues discovered in older
releases and start a discussion to come up with a policy on how to apply
them.

One approach is the patch gets only applied to the release it was
discovered in and master. Another approach is it gets applied to all
release branches >= discovered release and master. There may be other
approaches as well which can come up in this discussion.

The advantage of the first approach is that the immediate work is limited
to a single fix and merge. The second approach requires more work initially
as the patch needs to get applied to more one or more places.

I am tending towards the second approach of applying the fix to all release
branches >= discovered release, while also having some sort of an end of
life policy for releases otherwise it might become difficult to manage the
fixes. The end of life policy would mean that beyond a time period
(something reasonable period between 1 - 3 years) after the release is made
the release branch become frozen and no more fixes are applied to it.
Consider the following two problematic scenarios if we apply the fix only
to the one discovered release.

   - Let's say tomorrow a new fix needs to be made to a newer release
   branch (which had existed at the time of the fix) that is dependent on 
the
   current fix. Before applying the new fix one would need to first know to
   cherry-pick the old fix (may not be trivial to know this) and second and
   more important there could be merge conflicts when cherry-picking. At 
this
   point, you may be trying to resolve the conflicts where you may not be 
the
   primary author of the fix and/or the knowledge of the fix is no longer
   fresh in folks minds. This is prone to errors.


   - Let's call this fix a. Let's say there was another fix b. requiring a.
   to work correctly also on the same release branch but no compile
   dependencies on a. If in future we have a fix c. that depends on b. that
   needs to be applied on a different release branch both b. and a. would 
need
   to be cherry picked. It might be easy to figure out b. is needed, as c. 
has
   a direct dependency to it but would not be easy to determine that a. is
   also needed since that is old knowledge. It will not be easy to catch 
this
   omission because of no compile errors if a. is not included.

Thanks

[jira] [Commented] (APEXMALHAR-2239) Null pointer exception in JDBCPojoInputOperator when the field names do not match in Pojo, table

[Dongming Liang (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15843119#comment-15843119
 ] 

Dongming Liang commented on APEXMALHAR-2239:


[~Hitesh_]I do need some help. I'm not sure if the following will reproduce the 
problem. I tried to change the column name
in JdbcPojoOperatorTest.java
from STARTDATE to STARTDATE1
fieldInfos.add(new FieldInfo("STARTDATE1", "startDate1", null));

It gives a different error though:

java.lang.RuntimeException: java.sql.SQLSyntaxErrorException: user lacks 
privilege or object not found: STARTDATE1

at 
com.datatorrent.lib.db.jdbc.JdbcPOJOInputOperator.setup(JdbcPOJOInputOperator.java:147)
at 
com.datatorrent.lib.db.jdbc.JdbcPojoOperatorTest.testJdbcPojoInputOperator(JdbcPojoOperatorTest.java:635)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
at 
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
at 
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:234)
at 
com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
Caused by: java.sql.SQLSyntaxErrorException: user lacks privilege or object not 
found: STARTDATE1
at org.hsqldb.jdbc.JDBCUtil.sqlException(Unknown Source)
at org.hsqldb.jdbc.JDBCUtil.sqlException(Unknown Source)
at org.hsqldb.jdbc.JDBCPreparedStatement.(Unknown Source)
at org.hsqldb.jdbc.JDBCConnection.prepareStatement(Unknown Source)
at 
com.datatorrent.lib.db.jdbc.JdbcPOJOInputOperator.setup(JdbcPOJOInputOperator.java:142)
... 28 more

Could you advise if this the right way to investigate?

> Null pointer exception in JDBCPojoInputOperator when the field names do not 
> match in Pojo, table
> 
>
> Key: APEXMALHAR-2239
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2239
> Project: Apache Apex Malhar
>  Issue Type: Bug
>Affects Versions: 3.5.0
>Reporter: Venkatesh Kottapalli
>Assignee: Dongming Liang
>Priority: Minor
>
> The following exception is thrown in JDBCPojoInputOperator when the field 
> names for the table provided in the FieldInfo mapping don't match with the 
> column names of the table.
> Exception: 
> Abandoning deployment due to setup failure. java.lang.NullPointerException
>  at 
> com.datatorrent.lib.db.jdbc.JdbcPOJOInputOperator.activate(JdbcPOJOInputOperator.java:366)
>  at 
> com.datatorrent.lib.db.jdbc.JdbcPOJOInputOperator.activate(JdbcPOJOInputOperator.java:67)
>  at com.datatorrent.stram.engine.Node.activate(Node.java:619)
>  at 
> com.datatorrent.stram.engine.StreamingContainer.setupNode(StreamingContainer.java:1336)
>  at 
> com.datatorrent.stram.engine.StreamingContainer.access$100(StreamingContainer.java:130)
>  at 
> 

[DISCUSS] Policy for patches

[Pramod Immaneni]

Hi,

I wanted to bring up the topic of patches for issues discovered in older
releases and start a discussion to come up with a policy on how to apply
them.

One approach is the patch gets only applied to the release it was
discovered in and master. Another approach is it gets applied to all
release branches >= discovered release and master. There may be other
approaches as well which can come up in this discussion.

The advantage of the first approach is that the immediate work is limited
to a single fix and merge. The second approach requires more work initially
as the patch needs to get applied to more one or more places.

I am tending towards the second approach of applying the fix to all release
branches >= discovered release, while also having some sort of an end of
life policy for releases otherwise it might become difficult to manage the
fixes. The end of life policy would mean that beyond a time period
(something reasonable period between 1 - 3 years) after the release is made
the release branch become frozen and no more fixes are applied to it.
Consider the following two problematic scenarios if we apply the fix only
to the one discovered release.

   - Let's say tomorrow a new fix needs to be made to a newer release
   branch (which had existed at the time of the fix) that is dependent on the
   current fix. Before applying the new fix one would need to first know to
   cherry-pick the old fix (may not be trivial to know this) and second and
   more important there could be merge conflicts when cherry-picking. At this
   point, you may be trying to resolve the conflicts where you may not be the
   primary author of the fix and/or the knowledge of the fix is no longer
   fresh in folks minds. This is prone to errors.


   - Let's call this fix a. Let's say there was another fix b. requiring a.
   to work correctly also on the same release branch but no compile
   dependencies on a. If in future we have a fix c. that depends on b. that
   needs to be applied on a different release branch both b. and a. would need
   to be cherry picked. It might be easy to figure out b. is needed, as c. has
   a direct dependency to it but would not be easy to determine that a. is
   also needed since that is old knowledge. It will not be easy to catch this
   omission because of no compile errors if a. is not included.

Thanks

[jira] [Commented] (APEXCORE-625) Exception from StreamingAppMasterService.execute is lost if eventloop.stop thrown an exception.

[Vlad Rozov (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15843095#comment-15843095
 ] 

Vlad Rozov commented on APEXCORE-625:
-

Please provide details (exception stack trace) for the exception raised in the 
eventloop.stop(); eventloop.stop() is asynchronous and actual execution is done 
on the event loop thread. I don't see why eventloop.stop() will be raising an 
exception.

> Exception from StreamingAppMasterService.execute is lost if eventloop.stop 
> thrown an exception.
> ---
>
> Key: APEXCORE-625
> URL: https://issues.apache.org/jira/browse/APEXCORE-625
> Project: Apache Apex Core
>  Issue Type: Bug
>Reporter: Tushar Gosavi
>Assignee: Tushar Gosavi
>
> This was observed while debugging an application master crash. The 
> application master main loop terminated because of an exception, but this 
> exception was masked by exception thrown from eventloop stop in finally 
> block, causing difficulties in debugging the original issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXCORE-620) Support Fat Jar without POM, allowing Gradle and other build systems

[Jose Casillas (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15843007#comment-15843007
 ] 

Jose Casillas commented on APEXCORE-620:


Thank you for looking at this.  I'd suggest that Apex with the ignorepom 
option, auto include all jars inside the apa (or jar file?) as dependencies.  A 
fat jar from the gradle spring boot plugin has a format like this:

app class files
resource files
lib/dependency jars

Thanks!
Jose


> Support Fat Jar without POM, allowing Gradle and other build systems
> 
>
> Key: APEXCORE-620
> URL: https://issues.apache.org/jira/browse/APEXCORE-620
> Project: Apache Apex Core
>  Issue Type: New Feature
>Reporter: Jose Casillas
>
> For a project that is built with Gradle to be used in Apex, we must convert 
> our suite of libraries to Maven or resort to:
> apex> launch -ignorepom -libjars x,y,z,...
> Can the launcher simply accept a jar and include all of the dependency jars 
> within it, without requiring a POM or explicit declaration with -libjars?
> As a stop-gap, allowing -libjars to support a directory of dependencies would 
> free us from specifying each one in the launch statement.
> The fat jar support would allow build systems other than Maven to power Apex 
> applications.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: APEXCORE-294 Clean application shutdown

[Tushar Gosavi]

I have updated the documentation for kill v/s shutdown. let me know if
anything more is required.

- Tushar.


On Mon, Jan 23, 2017 at 6:09 PM, Tushar Gosavi  wrote:
> Hi Thomas,
>
> I have updated the pull request as well as design document, will take
> care of apex documentation shortly.
>
> - Tushar.
>
>
> On Fri, Jan 20, 2017 at 8:57 PM, Thomas Weise  wrote:
>> As discussed please also look at appropriate user documentation and clarity
>> WRT the CLI commands.
>>
>> Thanks,
>> Thomas
>>
>>
>> On Thu, Jan 19, 2017 at 10:46 PM, Tushar Gosavi 
>> wrote:
>>
>>> Hi Thomas,
>>>
>>> As discussed I will remote the old behaviour of shutdown.
>>>
>>> - Tushar.
>>>
>>> On Fri, Jan 20, 2017 at 10:03 AM, Tushar Gosavi 
>>> wrote:
>>> > Hi Thomas,
>>> >
>>> > I agree with you on the current behavior. The effect of shutdown can
>>> > be achieve with kill. I was more worried about backward compatibility
>>> > issue
>>> > while relaunching the shutdown app, hence kept the default as same as
>>> > before (cli and rest api default).  If we don't want to retain the
>>> > relaunch
>>> > behaviour of shutdown app then the code will be more simpler :).  Let
>>> > me know, I will update the PR accordinlgy.
>>> >
>>> > - Tushar.
>>> >
>>> >
>>> > On Thu, Jan 19, 2017 at 9:22 PM, Thomas Weise  wrote:
>>> >> Tushar,
>>> >>
>>> >> I would like to see the use case for "hard shutdown" vs kill. The
>>> current
>>> >> shutdown behavior isn't useful, confusing and sometimes does not even
>>> end
>>> >> the app (based on feedback received in the past from users). I would
>>> prefer
>>> >> we don't retain it without clear understanding of use case and how it
>>> will
>>> >> really work.
>>> >>
>>> >> Thanks,
>>> >> Thomas
>>> >>
>>> >>
>>> >> On Wed, Jan 18, 2017 at 1:41 AM, Tushar Gosavi 
>>> >> wrote:
>>> >>
>>> >>> Dear Community,
>>> >>>
>>> >>> I have changed the implementaion plan based on PR comments. Following
>>> >>> is the new proposal for this feature.
>>> >>>
>>> >>> https://docs.google.com/a/datatorrent.com/document/d/
>>> 1hSLH4xi_15OWwW4KY7--
>>> >>> LU3e2iHHhfthOpATicnC5eE/edit?usp=sharing
>>> >>>
>>> >>> Please provide the feedback. I will update the PR accrodinly.
>>> >>>
>>> >>> - Tushar.
>>> >>>
>>> >>>
>>> >>> On Fri, Nov 25, 2016 at 4:40 PM, Tushar Gosavi >> >
>>> >>> wrote:
>>> >>> > Dear Community,
>>> >>> >
>>> >>> > I have open an pull request for shutting down application by sending
>>> >>> > END_STREAM control tuples from all input operator. This is similar to
>>> >>> > all input operator have stopped after raising ShutdownException.
>>> >>> >
>>> >>> > On receiving shutdown request, master will prepare OPERATOR_STOP
>>> >>> > command for all input partitions, and send it to container as part of
>>> >>> > heartbeat response to container. Container will shutdown the operator
>>> >>> > thread after receiving this command.
>>> >>> >
>>> >>> > https://github.com/apache/apex-core/pull/424
>>> >>> > Please review.
>>> >>> >
>>> >>> > Thanks,
>>> >>> > - Tushar.
>>> >>>
>>>

[jira] [Created] (APEXCORE-626) shutdown-app should accept application similar to kill-app command.

[Tushar Gosavi (JIRA)]

Tushar Gosavi created APEXCORE-626:
--

 Summary: shutdown-app should accept application similar to 
kill-app command.
 Key: APEXCORE-626
 URL: https://issues.apache.org/jira/browse/APEXCORE-626
 Project: Apache Apex Core
  Issue Type: Bug
Reporter: Tushar Gosavi
Priority: Trivial


kill-app command support killing application through application name instead 
of application id.

shutdown-app command should also support the same.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (APEXCORE-626) shutdown-app should accept application name similar to kill-app command.

[Tushar Gosavi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXCORE-626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tushar Gosavi updated APEXCORE-626:
---
Summary: shutdown-app should accept application name similar to kill-app 
command.  (was: shutdown-app should accept application similar to kill-app 
command.)

> shutdown-app should accept application name similar to kill-app command.
> 
>
> Key: APEXCORE-626
> URL: https://issues.apache.org/jira/browse/APEXCORE-626
> Project: Apache Apex Core
>  Issue Type: Bug
>Reporter: Tushar Gosavi
>Priority: Trivial
>
> kill-app command support killing application through application name instead 
> of application id.
> shutdown-app command should also support the same.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXCORE-625) Exception from StreamingAppMasterService.execute is lost if eventloop.stop thrown an exception.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15842387#comment-15842387
 ] 

ASF GitHub Bot commented on APEXCORE-625:
-

GitHub user tushargosavi opened a pull request:

https://github.com/apache/apex-core/pull/457

APEXCORE-625 log exception from eventloop.stop, and let original exception 
to propagate to caller.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tushargosavi/apex-core APEXCORE-625

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-core/pull/457.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #457


commit c82e222aacb04d97aed7e4fc1d1ef455e904dd27
Author: Tushar R. Gosavi 
Date:   2017-01-27T08:23:14Z

APEXCORE-625 log exception from eventloop.stop, and let original
exception to propagate to caller.




> Exception from StreamingAppMasterService.execute is lost if eventloop.stop 
> thrown an exception.
> ---
>
> Key: APEXCORE-625
> URL: https://issues.apache.org/jira/browse/APEXCORE-625
> Project: Apache Apex Core
>  Issue Type: Bug
>Reporter: Tushar Gosavi
>Assignee: Tushar Gosavi
>
> This was observed while debugging an application master crash. The 
> application master main loop terminated because of an exception, but this 
> exception was masked by exception thrown from eventloop stop in finally 
> block, causing difficulties in debugging the original issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] apex-core pull request #457: APEXCORE-625 log exception from eventloop.stop,...

[tushargosavi]

GitHub user tushargosavi opened a pull request:

https://github.com/apache/apex-core/pull/457

APEXCORE-625 log exception from eventloop.stop, and let original exception 
to propagate to caller.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tushargosavi/apex-core APEXCORE-625

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-core/pull/457.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #457


commit c82e222aacb04d97aed7e4fc1d1ef455e904dd27
Author: Tushar R. Gosavi 
Date:   2017-01-27T08:23:14Z

APEXCORE-625 log exception from eventloop.stop, and let original
exception to propagate to caller.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (APEXCORE-625) Exception from StreamingAppMasterService.execute is lost if eventloop.stop thrown an exception.

[Tushar Gosavi (JIRA)]

Tushar Gosavi created APEXCORE-625:
--

 Summary: Exception from StreamingAppMasterService.execute is lost 
if eventloop.stop thrown an exception.
 Key: APEXCORE-625
 URL: https://issues.apache.org/jira/browse/APEXCORE-625
 Project: Apache Apex Core
  Issue Type: Bug
Reporter: Tushar Gosavi
Assignee: Tushar Gosavi


This was observed while debugging an application master crash. The application 
master main loop terminated because of an exception, but this exception was 
masked by exception thrown from eventloop stop in finally block, causing 
difficulties in debugging the original issue.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)