[jira] [Commented] (APEXCORE-759) Topology validation needs to happen correctly for operators connected using THREAD_LOCAL stream locality

2017-07-15 Thread Vlad Rozov (JIRA) 

[ 
https://issues.apache.org/jira/browse/APEXCORE-759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16088816#comment-16088816
 ] 

Vlad Rozov commented on APEXCORE-759:
-

THREAD_LOCAL, CONTAINER_LOCAL and other stream localities are optimization 
hints for the platform. Should engine use supported stream locality when 
partitioning topology can't be supported by the specified stream locality? In 
this case, should platform use CONTAINER_LOCAL stream locality instead of 
THREAD_LOCAL with a proper warning message?

> Topology validation needs to happen correctly for operators connected using 
> THREAD_LOCAL stream locality
> 
>
> Key: APEXCORE-759
> URL: https://issues.apache.org/jira/browse/APEXCORE-759
> Project: Apache Apex Core
>  Issue Type: Bug
>Reporter: Vinay Bangalore Srikanth
> Attachments: apex (1).log, apex.log
>
>
> In my application - 
> http://node0.morado.com:9090/static/#/ops/apps/application_1499808956620_0190 
> , I have set the locality to be THREAD_LOCAL.
> Upstream operator - Random string generator (with 4 partitions)
> Downstream operator - Custom console operator (with 5 partitions)
> The containers are getting killed. Exceptions from container-launch are seen.
> Logical message has to be displayed by handling exceptions.
> Logs are attached for one of the containers - Id: 01_000126  that was killed. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (APEXCORE-762) Gitbox PR integration test

2017-07-15 Thread Vlad Rozov (JIRA) 

 [ 
https://issues.apache.org/jira/browse/APEXCORE-762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vlad Rozov closed APEXCORE-762.
---
Resolution: Done

> Gitbox PR integration test
> --
>
> Key: APEXCORE-762
> URL: https://issues.apache.org/jira/browse/APEXCORE-762
> Project: Apache Apex Core
>  Issue Type: GitBox Request
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
>Priority: Trivial
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (APEXCORE-762) Gitbox PR integration test

2017-07-15 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/APEXCORE-762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16088810#comment-16088810
 ] 

ASF GitHub Bot commented on APEXCORE-762:
-

vrozov closed pull request #557: APEXCORE-762 Gitbox PR integration test (do 
not merge)
URL: https://github.com/apache/apex-core/pull/557
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Gitbox PR integration test
> --
>
> Key: APEXCORE-762
> URL: https://issues.apache.org/jira/browse/APEXCORE-762
> Project: Apache Apex Core
>  Issue Type: GitBox Request
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
>Priority: Trivial
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (APEXCORE-762) Gitbox PR integration test

2017-07-15 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/APEXCORE-762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16088809#comment-16088809
 ] 

ASF GitHub Bot commented on APEXCORE-762:
-

vrozov opened a new pull request #557: APEXCORE-762 Gitbox PR integration test 
(do not merge)
URL: https://github.com/apache/apex-core/pull/557
 
 
   Testing JIRA 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Gitbox PR integration test
> --
>
> Key: APEXCORE-762
> URL: https://issues.apache.org/jira/browse/APEXCORE-762
> Project: Apache Apex Core
>  Issue Type: GitBox Request
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
>Priority: Trivial
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (APEXCORE-762) Gitbox PR integration test

2017-07-15 Thread Vlad Rozov (JIRA) 

 [ 
https://issues.apache.org/jira/browse/APEXCORE-762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vlad Rozov reassigned APEXCORE-762:
---

Assignee: Vlad Rozov

> Gitbox PR integration test
> --
>
> Key: APEXCORE-762
> URL: https://issues.apache.org/jira/browse/APEXCORE-762
> Project: Apache Apex Core
>  Issue Type: GitBox Request
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
>Priority: Trivial
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (APEXCORE-762) Gitbox PR integration test

2017-07-15 Thread Vlad Rozov (JIRA) 
Vlad Rozov created APEXCORE-762:
---

 Summary: Gitbox PR integration test
 Key: APEXCORE-762
 URL: https://issues.apache.org/jira/browse/APEXCORE-762
 Project: Apache Apex Core
  Issue Type: GitBox Request
Reporter: Vlad Rozov
Priority: Trivial






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Malhar CI builds fail

2017-07-15 Thread Vlad Rozov 

Anyone to RCA why Malhar CI builds fail?

Thank you,

Vlad

Re: Java packages: legacy -> org.apache.apex

2017-07-15 Thread Sandesh Hegde 
Why is there a urgency, why cant this go into 4.0 Malhar with possibly
other breaking changes?
On Sat, Jul 15, 2017 at 7:57 AM Thomas Weise  wrote:

> Discussing what in the future might become stable needs to be a separate
> thread, it will be a much bigger discussion.
>
> The topic here is to relocate the packages. With a few exceptions
> relocation won't affect the semantic versioning. Semantic versioning is
> essentially not effective for Malhar because almost everything is @Evolving
> (and there are reasons for that.. -> separate topic)
>
> I don't really like the idea of creating bw compatibility stubs for the
> follow up PR. It creates even more clutter in the source tree than there
> already is and so here is an alternative suggestion:
>
>
> https://github.com/tweise/apex-malhar/blob/malhar37-compat/shaded-malhar37/pom.xml
>
> Create a shaded artifact that provides the old com.datatorrent.* classes as
> of release 3.7. Users can include that artifact if they don't want to
> change import statements. At the same time they have an incentive to switch
> to the relocated classes to take advantage of bug fixes and new
> functionality.
>
> I will work on the first PR that does the relocate. In the meantime, we can
> finalize what backward compatibility support we want to provide and how.
>
> Thanks,
> Thomas
>
>
>
>
> On Fri, Jul 14, 2017 at 11:33 AM, Pramod Immaneni 
> wrote:
>
> > How about coming up with a list of @Evolving operators that we would like
> > to see in the eventual stable list and move those along with the not
> > @Evolving ones in org.apache.apex with b/w stubs and leave the rest as
> they
> > are. Then have a follow up JIRA for the rest to be moved over to contrib
> > and be deprecated.
> >
> > Thanks
> >
> > On Fri, Jul 14, 2017 at 10:37 AM, Thomas Weise 
> > wrote:
> >
> > > We need to keep the discussion here on topic, if other things are piled
> > on
> > > top then nothing gets done.
> > >
> > > Most operators today are not stable, they are @Evolving. So perhaps it
> is
> > > necessary to have a separate discussion about that, outside of this
> > thread.
> > > My guess is that there are only a few operators that we could declare
> > > stable. Specifically, under contrib the closest one might have been
> > Kafka,
> > > but that is already superseded by the newer versions.
> > >
> > > Thomas
> > >
> > >
> > > On Fri, Jul 14, 2017 at 10:21 AM, Pramod Immaneni <
> > pra...@datatorrent.com>
> > > wrote:
> > >
> > > > We had a discussion a while back, agreed to relegate non-stable and
> > > > experimental operators to contrib and also added this to contribution
> > > > guidelines. We aexecuted on this and cleaned up the repo by moving
> > > > operators deemed non-suitable for production use at that time to
> > contrib.
> > > > So I wouldn't say the operators that are at the top level today or
> the
> > > ones
> > > > in library are 0.x.x quality. Granted, we may need to do one more
> pass
> > as
> > > > some of the operators may no longer be considered the right
> > > implementations
> > > > with the advent of the windowed operator.
> > > >
> > > > Since contrib used to be the place that housed operators that
> required
> > > > third party libraries, there are some production quality operators in
> > > there
> > > > that need to be pulled out into top level modules.
> > > >
> > > > I think we are in agreement that for operators that we consider
> stable,
> > > we
> > > > should provide a b/w stub. I would add that we strongly consider
> > creating
> > > > the org.apache.apex counterparts of any stable operators that are in
> > > > contrib out in top level modules and have the com.datatorrent stubs
> in
> > > > contrib.
> > > >
> > > > For the operators not considered stable, I would prefer we either
> leave
> > > the
> > > > current package path as is or if we change the package path then
> create
> > > the
> > > > b/w stub as I am not sure how widely they are in use (so, in essence,
> > > > preserve semantic versioning). It would be good if there is a
> followup
> > > JIRA
> > > > that takes a look at what other operators can be moved to contrib
> with
> > > the
> > > > advent of the newer frameworks and understanding.
> > > >
> > > > Thanks
> > > >
> > > > On Fri, Jul 14, 2017 at 9:44 AM, Thomas Weise 
> wrote:
> > > >
> > > > > Most of the operators evolve, as is quite clearly indicated by
> > > @Evolving
> > > > > annotations. So while perhaps 0.x.x would be a more appropriate
> > version
> > > > > number, I don't think you can change that.
> > > > >
> > > > > Thomas
> > > > >
> > > > > On Fri, Jul 14, 2017 at 9:35 AM, Vlad Rozov <
> v.ro...@datatorrent.com
> > >
> > > > > wrote:
> > > > >
> > > > > > If entire library is not stable, its version should be 0.x.x and
> > > there
> > > > > > should not be any semantic versioning enabled or implied. It
> > evolves.
> > > > If
> > > > > > some operators are

[jira] [Resolved] (APEXCORE-756) Fix ConcurrentModificationException in GroupingManager

2017-07-15 Thread Vlad Rozov (JIRA) 

 [ 
https://issues.apache.org/jira/browse/APEXCORE-756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vlad Rozov resolved APEXCORE-756.
-
   Resolution: Fixed
Fix Version/s: 3.7.0

> Fix ConcurrentModificationException in GroupingManager
> --
>
> Key: APEXCORE-756
> URL: https://issues.apache.org/jira/browse/APEXCORE-756
> Project: Apache Apex Core
>  Issue Type: Bug
>Reporter: Priyanka Gugale
>Assignee: Priyanka Gugale
>Priority: Minor
> Fix For: 3.7.0
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> At times the StreamingContainerManager test fails as:
> {noformat}
> com.datatorrent.stram.StreamingContainerManagerTest
> testShutdownOperatorRecovery(com.datatorrent.stram.StreamingContainerManagerTest)
>   Time elapsed: 0.046 sec  <<< ERROR!
> java.util.ConcurrentModificationException
>   at 
> com.datatorrent.stram.StreamingContainerManagerTest.testShutdownOperatorRecovery(StreamingContainerManagerTest.java:464)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (APEXCORE-756) Fix ConcurrentModificationException in GroupingManager

2017-07-15 Thread Vlad Rozov (JIRA) 

 [ 
https://issues.apache.org/jira/browse/APEXCORE-756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vlad Rozov updated APEXCORE-756:

Description: 
At times the StreamingContainerManager test fails as:
{noformat}
com.datatorrent.stram.StreamingContainerManagerTest
testShutdownOperatorRecovery(com.datatorrent.stram.StreamingContainerManagerTest)
  Time elapsed: 0.046 sec  <<< ERROR!
java.util.ConcurrentModificationException
at 
com.datatorrent.stram.StreamingContainerManagerTest.testShutdownOperatorRecovery(StreamingContainerManagerTest.java:464)
{noformat}

  was:
At times the StreamingContainerManager test fails as:

com.datatorrent.stram.StreamingContainerManagerTest
testShutdownOperatorRecovery(com.datatorrent.stram.StreamingContainerManagerTest)
  Time elapsed: 0.046 sec  <<< ERROR!
java.util.ConcurrentModificationException
at 
com.datatorrent.stram.StreamingContainerManagerTest.testShutdownOperatorRecovery(StreamingContainerManagerTest.java:464)



> Fix ConcurrentModificationException in GroupingManager
> --
>
> Key: APEXCORE-756
> URL: https://issues.apache.org/jira/browse/APEXCORE-756
> Project: Apache Apex Core
>  Issue Type: Bug
>Reporter: Priyanka Gugale
>Assignee: Priyanka Gugale
>Priority: Minor
> Fix For: 3.7.0
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> At times the StreamingContainerManager test fails as:
> {noformat}
> com.datatorrent.stram.StreamingContainerManagerTest
> testShutdownOperatorRecovery(com.datatorrent.stram.StreamingContainerManagerTest)
>   Time elapsed: 0.046 sec  <<< ERROR!
> java.util.ConcurrentModificationException
>   at 
> com.datatorrent.stram.StreamingContainerManagerTest.testShutdownOperatorRecovery(StreamingContainerManagerTest.java:464)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Java packages: legacy -> org.apache.apex

2017-07-15 Thread Thomas Weise 
Discussing what in the future might become stable needs to be a separate
thread, it will be a much bigger discussion.

The topic here is to relocate the packages. With a few exceptions
relocation won't affect the semantic versioning. Semantic versioning is
essentially not effective for Malhar because almost everything is @Evolving
(and there are reasons for that.. -> separate topic)

I don't really like the idea of creating bw compatibility stubs for the
follow up PR. It creates even more clutter in the source tree than there
already is and so here is an alternative suggestion:

https://github.com/tweise/apex-malhar/blob/malhar37-compat/shaded-malhar37/pom.xml

Create a shaded artifact that provides the old com.datatorrent.* classes as
of release 3.7. Users can include that artifact if they don't want to
change import statements. At the same time they have an incentive to switch
to the relocated classes to take advantage of bug fixes and new
functionality.

I will work on the first PR that does the relocate. In the meantime, we can
finalize what backward compatibility support we want to provide and how.

Thanks,
Thomas




On Fri, Jul 14, 2017 at 11:33 AM, Pramod Immaneni 
wrote:

> How about coming up with a list of @Evolving operators that we would like
> to see in the eventual stable list and move those along with the not
> @Evolving ones in org.apache.apex with b/w stubs and leave the rest as they
> are. Then have a follow up JIRA for the rest to be moved over to contrib
> and be deprecated.
>
> Thanks
>
> On Fri, Jul 14, 2017 at 10:37 AM, Thomas Weise 
> wrote:
>
> > We need to keep the discussion here on topic, if other things are piled
> on
> > top then nothing gets done.
> >
> > Most operators today are not stable, they are @Evolving. So perhaps it is
> > necessary to have a separate discussion about that, outside of this
> thread.
> > My guess is that there are only a few operators that we could declare
> > stable. Specifically, under contrib the closest one might have been
> Kafka,
> > but that is already superseded by the newer versions.
> >
> > Thomas
> >
> >
> > On Fri, Jul 14, 2017 at 10:21 AM, Pramod Immaneni <
> pra...@datatorrent.com>
> > wrote:
> >
> > > We had a discussion a while back, agreed to relegate non-stable and
> > > experimental operators to contrib and also added this to contribution
> > > guidelines. We aexecuted on this and cleaned up the repo by moving
> > > operators deemed non-suitable for production use at that time to
> contrib.
> > > So I wouldn't say the operators that are at the top level today or the
> > ones
> > > in library are 0.x.x quality. Granted, we may need to do one more pass
> as
> > > some of the operators may no longer be considered the right
> > implementations
> > > with the advent of the windowed operator.
> > >
> > > Since contrib used to be the place that housed operators that required
> > > third party libraries, there are some production quality operators in
> > there
> > > that need to be pulled out into top level modules.
> > >
> > > I think we are in agreement that for operators that we consider stable,
> > we
> > > should provide a b/w stub. I would add that we strongly consider
> creating
> > > the org.apache.apex counterparts of any stable operators that are in
> > > contrib out in top level modules and have the com.datatorrent stubs in
> > > contrib.
> > >
> > > For the operators not considered stable, I would prefer we either leave
> > the
> > > current package path as is or if we change the package path then create
> > the
> > > b/w stub as I am not sure how widely they are in use (so, in essence,
> > > preserve semantic versioning). It would be good if there is a followup
> > JIRA
> > > that takes a look at what other operators can be moved to contrib with
> > the
> > > advent of the newer frameworks and understanding.
> > >
> > > Thanks
> > >
> > > On Fri, Jul 14, 2017 at 9:44 AM, Thomas Weise  wrote:
> > >
> > > > Most of the operators evolve, as is quite clearly indicated by
> > @Evolving
> > > > annotations. So while perhaps 0.x.x would be a more appropriate
> version
> > > > number, I don't think you can change that.
> > > >
> > > > Thomas
> > > >
> > > > On Fri, Jul 14, 2017 at 9:35 AM, Vlad Rozov  >
> > > > wrote:
> > > >
> > > > > If entire library is not stable, its version should be 0.x.x and
> > there
> > > > > should not be any semantic versioning enabled or implied. It
> evolves.
> > > If
> > > > > some operators are stable as 3.8.x version suggests, the library
> > should
> > > > > follow semantic versioning and it is not OK to make a major change
> > that
> > > > > breaks semantic versioning across the entire library. It is not a
> > > finding
> > > > > an excuse not to make a change. For me semantic versioning and
> > backward
> > > > > compatibility is more important than a proper package name.
> > > > >
> > > > > Thank you,
> > >