Re: Adding new log4j appender to Apex core

[AJAY GUPTA]

+1 for approach 2.


Regards,
Ajay

On Wed, Nov 23, 2016 at 12:16 PM, David Yan  wrote:

> The goal of this log4j appender is to provide a log offset and the fixed
> name of the container log file (instead of apex.log becoming apex.log.1 and
> then apex.log.2, etc due to rotation) as part of an error STRAM event so
> users can easily locate the log entries around the error.
>
> The user can override the appender, but in that case, the engine detects
> that and will not include the log location as part of the STRAM event.
>
> David
>
> On Tue, Nov 22, 2016 at 7:10 PM, Priyanka Gugale  >
> wrote:
>
> > Hi,
> >
> > Thomas,
> > Yes log4j is ultimately owned by user, and they should be able to
> override
> > it. What I am trying to do is provide a default behavior for Apex. In
> case
> > user isn't using any logger of their own we should use this new appender
> of
> > Apex rather than using standard log4j appender as per hadoop config.
> >
> > Sanjay,
> > Archetype is the good place to put this and I will add it there, but many
> > time people won't use it. So I wanted to keep it at ~/.dt as well. Is
> there
> > any other default config folder for Apex?
> >
> > Also I am not relying on anything. If we fail to find config in app jar
> or
> > ~/.dt we are going to skip usage of this new appender.
> >
> > -Priyanka
> >
> > On Wed, Nov 23, 2016 at 5:58 AM, Sanjay Pujare 
> > wrote:
> >
> > > The only way to “enforce” this new appender is to update the archetypes
> > > (apex-app-archetype and apex-conf-archetype under apex-core/ )  to use
> > the
> > > new ones as default. But there does not seem to be a way to enforce
> this
> > > for anyone not using the archetypes.
> > >
> > > I agree with not relying on ~/.dt in apex-core.
> > >
> > > On 11/22/16, 1:08 PM, "Thomas Weise"  wrote:
> > >
> > > The log4j configuration is ultimately owned by the user, so how do
> > you
> > > want
> > > to enforce a custom appender?
> > >
> > > I don't think that this should rely on anything in ~/.dt either
> > >
> > > Thomas
> > >
> > > On Tue, Nov 22, 2016 at 10:00 AM, Priyanka Gugale <
> > > priya...@datatorrent.com>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > I am working on APEXCORE-563
> > > > 
> > > > As per this Jira we should put log file name in
> container/operator
> > > events.
> > > > The problem is current RollingFileAppender keeps renaming files
> > from
> > > 1 to 2
> > > > to ... n as files reach maximum allowed file size. Because of
> > > constant
> > > > renaming of files we can't put a fixed file name in stram event.
> > > >
> > > > To overcome this I would like to add a new log4j appender to
> > > ApexCore.
> > > > There are two ways I can implement this:
> > > > 1. Have Daily rolling file appender. The current file will be
> > > recognized
> > > > based on timestamp in file name. Also to control max file size,
> we
> > > need to
> > > > keep rolling files based on size as well.
> > > > 2. Have Rolling File Appender but do not rename files. When max
> > file
> > > size
> > > > is reached create new file with next number e.g. crate log file
> > > dt.log.2
> > > > after dt.log.1 is full. Also to recognize the latest file keep
> the
> > > softlink
> > > > named dt.log pointing to current log file.
> > > >
> > > > I would prefer to implement approach 2. Please provide your
> > > > comments/feedback if you feel otherwise.
> > > >
> > > > Also to use this new appender we need to use our custom
> > > log4j.properties
> > > > file instead of one present in hadoop conf. For that we need to
> set
> > > jvm
> > > > option -Dlog4j.configuration. I am planning to update file
> > > dt-site.xml in
> > > > folder ~/.dt  and default properties file available in apex
> > > archetype to
> > > > set jvm options as follows:
> > > >  
> > > >dt.attr.CONTAINER_JVM_OPTIONS
> > > >-Dlog4j.configuration=log4j.props
> > > >  
> > > >
> > > > And I will copy log4j.props file in ~/.dt folder as well as
> > > > apex-archetypes.
> > > >
> > > > Lastly if someone still miss this new log4j properties file or
> jvm
> > > option
> > > > to set -Dlog4j.configuration we will not put log file name in
> event
> > > raised
> > > > by container or operator.
> > > >
> > > > Please provide your feedback on this approach.
> > > >
> > > > -Priyanka
> > > >
> > >
> > >
> > >
> > >
> >
>

[jira] [Commented] (APEXMALHAR-2351) Exception while fetching properties for Operators using JdbcStore

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15689143#comment-15689143
 ] 

ASF GitHub Bot commented on APEXMALHAR-2351:


Github user asfgit closed the pull request at:

https://github.com/apache/apex-malhar/pull/505


> Exception while fetching properties for Operators using JdbcStore 
> --
>
> Key: APEXMALHAR-2351
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2351
> Project: Apache Apex Malhar
>  Issue Type: Bug
>Reporter: Hitesh Kapoor
>Assignee: Hitesh Kapoor
>
> apex (application_1476346787877_0157) > get-operator-properties JdbcInput
> com.datatorrent.stram.cli.ApexCli$CliException: Failed web service request 
> for appid application_1476346787877_0157
>   at 
> com.datatorrent.stram.cli.ApexCli$GetOperatorPropertiesCommand.execute(ApexCli.java:2641)
>   at com.datatorrent.stram.cli.ApexCli$3.run(ApexCli.java:1468)
> Caused by: com.datatorrent.stram.cli.ApexCli$CliException: Failed to request 
> web service for appid application_1476346787877_0157
>   at com.datatorrent.stram.cli.ApexCli.getResource(ApexCli.java:1681)
>   at com.datatorrent.stram.cli.ApexCli.getResource(ApexCli.java:1658)
>   at com.datatorrent.stram.cli.ApexCli.access$9400(ApexCli.java:153)
>   at 
> com.datatorrent.stram.cli.ApexCli$GetOperatorPropertiesCommand.execute(ApexCli.java:2638)
>   ... 1 more
> Caused by: com.sun.jersey.api.client.UniformInterfaceException: GET 
> http://node32.morado.com:39573/ws/v2/stram/logicalPlan/operators/JdbcInput/properties
>  returned a response status of 404 Not Found
>   at com.sun.jersey.api.client.WebResource.handle(WebResource.java:676)
>   at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
>   at 
> com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:503)
>   at 
> com.datatorrent.stram.util.WebServicesClient$GetWebServicesHandler.process(WebServicesClient.java:259)
>   at 
> com.datatorrent.stram.util.WebServicesClient$2.run(WebServicesClient.java:201)
>   at 
> com.datatorrent.stram.util.SecureExecutor.execute(SecureExecutor.java:46)
>   at 
> com.datatorrent.stram.util.WebServicesClient.process(WebServicesClient.java:196)
>   at 
> com.datatorrent.stram.client.StramAgent.issueStramWebRequest(StramAgent.java:285)
>   at 
> com.datatorrent.stram.client.StramAgent.issueStramWebRequest(StramAgent.java:305)
>   at com.datatorrent.stram.cli.ApexCli.getResource(ApexCli.java:1675)
>   ... 4 more
> 16/10/28 02:35:15 ERROR cli.ApexCli: Exception caught: 
> com.datatorrent.stram.cli.ApexCli$CliException: Failed web service request 
> for appid application_1476346787877_0157
>   at 
> com.datatorrent.stram.cli.ApexCli$GetOperatorPropertiesCommand.execute(ApexCli.java:2641)
>   at com.datatorrent.stram.cli.ApexCli$3.run(ApexCli.java:1468)
> Caused by: com.datatorrent.stram.cli.ApexCli$CliException: Failed to request 
> web service for appid application_1476346787877_0157
>   at com.datatorrent.stram.cli.ApexCli.getResource(ApexCli.java:1681)
>   at com.datatorrent.stram.cli.ApexCli.getResource(ApexCli.java:1658)
>   at com.datatorrent.stram.cli.ApexCli.access$9400(ApexCli.java:153)
>   at 
> com.datatorrent.stram.cli.ApexCli$GetOperatorPropertiesCommand.execute(ApexCli.java:2638)
>   ... 1 more
> Caused by: com.sun.jersey.api.client.UniformInterfaceException: GET 
> http://node32.morado.com:39573/ws/v2/stram/logicalPlan/operators/JdbcInput/properties
>  returned a response status of 404 Not Found
>   at com.sun.jersey.api.client.WebResource.handle(WebResource.java:676)
>   at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
>   at 
> com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:503)
>   at 
> com.datatorrent.stram.util.WebServicesClient$GetWebServicesHandler.process(WebServicesClient.java:259)
>   at 
> com.datatorrent.stram.util.WebServicesClient$2.run(WebServicesClient.java:201)
>   at 
> com.datatorrent.stram.util.SecureExecutor.execute(SecureExecutor.java:46)
>   at 
> com.datatorrent.stram.util.WebServicesClient.process(WebServicesClient.java:196)
>   at 
> com.datatorrent.stram.client.StramAgent.issueStramWebRequest(StramAgent.java:285)
>   at 
> com.datatorrent.stram.client.StramAgent.issueStramWebRequest(StramAgent.java:305)
>   at com.datatorrent.stram.cli.ApexCli.getResource(ApexCli.java:1675)
>   ... 4 more



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] apex-malhar pull request #505: APEXMALHAR-2351 fix to extract operator prope...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/apex-malhar/pull/505


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Adding new log4j appender to Apex core

[David Yan]

The goal of this log4j appender is to provide a log offset and the fixed
name of the container log file (instead of apex.log becoming apex.log.1 and
then apex.log.2, etc due to rotation) as part of an error STRAM event so
users can easily locate the log entries around the error.

The user can override the appender, but in that case, the engine detects
that and will not include the log location as part of the STRAM event.

David

On Tue, Nov 22, 2016 at 7:10 PM, Priyanka Gugale 
wrote:

> Hi,
>
> Thomas,
> Yes log4j is ultimately owned by user, and they should be able to override
> it. What I am trying to do is provide a default behavior for Apex. In case
> user isn't using any logger of their own we should use this new appender of
> Apex rather than using standard log4j appender as per hadoop config.
>
> Sanjay,
> Archetype is the good place to put this and I will add it there, but many
> time people won't use it. So I wanted to keep it at ~/.dt as well. Is there
> any other default config folder for Apex?
>
> Also I am not relying on anything. If we fail to find config in app jar or
> ~/.dt we are going to skip usage of this new appender.
>
> -Priyanka
>
> On Wed, Nov 23, 2016 at 5:58 AM, Sanjay Pujare 
> wrote:
>
> > The only way to “enforce” this new appender is to update the archetypes
> > (apex-app-archetype and apex-conf-archetype under apex-core/ )  to use
> the
> > new ones as default. But there does not seem to be a way to enforce this
> > for anyone not using the archetypes.
> >
> > I agree with not relying on ~/.dt in apex-core.
> >
> > On 11/22/16, 1:08 PM, "Thomas Weise"  wrote:
> >
> > The log4j configuration is ultimately owned by the user, so how do
> you
> > want
> > to enforce a custom appender?
> >
> > I don't think that this should rely on anything in ~/.dt either
> >
> > Thomas
> >
> > On Tue, Nov 22, 2016 at 10:00 AM, Priyanka Gugale <
> > priya...@datatorrent.com>
> > wrote:
> >
> > > Hi,
> > >
> > > I am working on APEXCORE-563
> > > 
> > > As per this Jira we should put log file name in container/operator
> > events.
> > > The problem is current RollingFileAppender keeps renaming files
> from
> > 1 to 2
> > > to ... n as files reach maximum allowed file size. Because of
> > constant
> > > renaming of files we can't put a fixed file name in stram event.
> > >
> > > To overcome this I would like to add a new log4j appender to
> > ApexCore.
> > > There are two ways I can implement this:
> > > 1. Have Daily rolling file appender. The current file will be
> > recognized
> > > based on timestamp in file name. Also to control max file size, we
> > need to
> > > keep rolling files based on size as well.
> > > 2. Have Rolling File Appender but do not rename files. When max
> file
> > size
> > > is reached create new file with next number e.g. crate log file
> > dt.log.2
> > > after dt.log.1 is full. Also to recognize the latest file keep the
> > softlink
> > > named dt.log pointing to current log file.
> > >
> > > I would prefer to implement approach 2. Please provide your
> > > comments/feedback if you feel otherwise.
> > >
> > > Also to use this new appender we need to use our custom
> > log4j.properties
> > > file instead of one present in hadoop conf. For that we need to set
> > jvm
> > > option -Dlog4j.configuration. I am planning to update file
> > dt-site.xml in
> > > folder ~/.dt  and default properties file available in apex
> > archetype to
> > > set jvm options as follows:
> > >  
> > >dt.attr.CONTAINER_JVM_OPTIONS
> > >-Dlog4j.configuration=log4j.props
> > >  
> > >
> > > And I will copy log4j.props file in ~/.dt folder as well as
> > > apex-archetypes.
> > >
> > > Lastly if someone still miss this new log4j properties file or jvm
> > option
> > > to set -Dlog4j.configuration we will not put log file name in event
> > raised
> > > by container or operator.
> > >
> > > Please provide your feedback on this approach.
> > >
> > > -Priyanka
> > >
> >
> >
> >
> >
>

[jira] [Resolved] (APEXCORE-379) Latency is stuck when an operator is stuck (before the 1000 window threshold)

[Thomas Weise (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXCORE-379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Weise resolved APEXCORE-379.
---
   Resolution: Fixed
Fix Version/s: 3.5.0

> Latency is stuck when an operator is stuck (before the 1000 window threshold)
> -
>
> Key: APEXCORE-379
> URL: https://issues.apache.org/jira/browse/APEXCORE-379
> Project: Apache Apex Core
>  Issue Type: Improvement
>Reporter: David Yan
>Assignee: David Yan
> Fix For: 3.5.0
>
>
> When the operator suddenly gets stuck, the latency value will also gets 
> stuck, until it is more than THROUGHPUT_CALCULATION_MAX_SAMPLES (default 
> 1000) windows behind, at which point STRAM estimates the latency by window 
> width * windowsBehind.  But for 1000 windows, the latency for that operator 
> is stuck and hence the application latency is also based on stale value.
> The way to fix this is for STRAM to estimate the latency when STRAM has not 
> received the end window stats for that operator for a certain amount of time.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (APEXCORE-544) Latency values are out of whack immediately after app is launched

[Thomas Weise (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXCORE-544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Weise resolved APEXCORE-544.
---
   Resolution: Fixed
Fix Version/s: 3.5.0

> Latency values are out of whack immediately after app is launched
> -
>
> Key: APEXCORE-544
> URL: https://issues.apache.org/jira/browse/APEXCORE-544
> Project: Apache Apex Core
>  Issue Type: Bug
>Reporter: David Yan
>Assignee: David Yan
> Fix For: 3.5.0
>
>
> The latency values of some operators are out of whack immediately after the 
> app is launched and they get back to normal after a few seconds.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXCORE-544) Latency values are out of whack immediately after app is launched

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15689070#comment-15689070
 ] 

ASF GitHub Bot commented on APEXCORE-544:
-

Github user asfgit closed the pull request at:

https://github.com/apache/apex-core/pull/401


> Latency values are out of whack immediately after app is launched
> -
>
> Key: APEXCORE-544
> URL: https://issues.apache.org/jira/browse/APEXCORE-544
> Project: Apache Apex Core
>  Issue Type: Bug
>Reporter: David Yan
>Assignee: David Yan
>
> The latency values of some operators are out of whack immediately after the 
> app is launched and they get back to normal after a few seconds.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] apex-core pull request #401: APEXCORE-544 APEXCORE-379 #resolve fixed latenc...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/apex-core/pull/401


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-malhar pull request #504: APEXMALHAR-2340 code changes to initialize th...

[Hitesh-Scorpio]

Github user Hitesh-Scorpio closed the pull request at:

https://github.com/apache/apex-malhar/pull/504


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (APEXMALHAR-2316) Cannot register tuple class in XmlParser Operator

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15689019#comment-15689019
 ] 

ASF GitHub Bot commented on APEXMALHAR-2316:


GitHub user Hitesh-Scorpio opened a pull request:

https://github.com/apache/apex-malhar/pull/506

APEXMALHAR-2316 APEXMALHAR-2346 Moved suitable code to activate() for…

… initializing tuple class attribute and changed the argument type of 
DocumentBuilder.parse() to InputSource. @bhupeshchawda please review

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Hitesh-Scorpio/apex-malhar 
APEXMALHAR-2316_XMLParser

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-malhar/pull/506.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #506


commit ded9c03a63e9104db41e1ef8296f1ed77ceb6577
Author: Hitesh-Scorpio 
Date:   2016-11-07T09:20:13Z

APEXMALHAR-2316 APEXMALHAR-2346 Moved suitable code to activate() for 
initializing tuple class attribute and changed the argument type of 
DocumentBuilder.parse() to InputSource.




> Cannot register tuple class in XmlParser Operator
> -
>
> Key: APEXMALHAR-2316
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2316
> Project: Apache Apex Malhar
>  Issue Type: Bug
>Reporter: Hitesh Kapoor
>Assignee: Hitesh Kapoor
>
> Unable to set the tuple class attribute, getting the following stack trace 
> Abandoning deployment due to setup failure. java.lang.IllegalArgumentException
>   at javax.xml.bind.JAXBContext.newInstance(JAXBContext.java:635)
>   at javax.xml.bind.JAXBContext.newInstance(JAXBContext.java:584)
>   at com.datatorrent.lib.parser.XmlParser.setup(XmlParser.java:135)
>   at com.datatorrent.lib.parser.XmlParser.setup(XmlParser.java:63)
>   at com.datatorrent.stram.engine.Node.setup(Node.java:187)
>   at 
> com.datatorrent.stram.engine.StreamingContainer.setupNode(StreamingContainer.java:1309)
>   at 
> com.datatorrent.stram.engine.StreamingContainer.access$100(StreamingContainer.java:130)
>   at 
> com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1388)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] apex-malhar pull request #506: APEXMALHAR-2316 APEXMALHAR-2346 Moved suitabl...

[Hitesh-Scorpio]

GitHub user Hitesh-Scorpio opened a pull request:

https://github.com/apache/apex-malhar/pull/506

APEXMALHAR-2316 APEXMALHAR-2346 Moved suitable code to activate() for…

… initializing tuple class attribute and changed the argument type of 
DocumentBuilder.parse() to InputSource. @bhupeshchawda please review

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Hitesh-Scorpio/apex-malhar 
APEXMALHAR-2316_XMLParser

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-malhar/pull/506.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #506


commit ded9c03a63e9104db41e1ef8296f1ed77ceb6577
Author: Hitesh-Scorpio 
Date:   2016-11-07T09:20:13Z

APEXMALHAR-2316 APEXMALHAR-2346 Moved suitable code to activate() for 
initializing tuple class attribute and changed the argument type of 
DocumentBuilder.parse() to InputSource.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-malhar pull request #484: APEXMALHAR-2316 Moved suitable code to activa...

[Hitesh-Scorpio]

Github user Hitesh-Scorpio closed the pull request at:

https://github.com/apache/apex-malhar/pull/484


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (APEXCORE-498) Named Checkpoints - Checkpoint the DAG with a name/tag and start the app from that point

[David Yan (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688960#comment-15688960
 ] 

David Yan commented on APEXCORE-498:


In addition to this, there are many components and operators in Malhar that 
assume you don't ever see a given window ID after it's been committed. Managed 
state is one of them, and it is the foundation of multiple operators. Having 
this feature will mean the assumption is wrong and things can go horribly wrong.

> Named Checkpoints - Checkpoint the DAG with a name/tag and start the app from 
> that point
> 
>
> Key: APEXCORE-498
> URL: https://issues.apache.org/jira/browse/APEXCORE-498
> Project: Apache Apex Core
>  Issue Type: Improvement
>Reporter: Sandesh
>Assignee: Sandesh
>  Labels: roadmap
>
> Named Checkpoints 
> 1. Ability to tag/name the checkpoints
> 2. On demand - checkpoint the DAG
> 3. Start the app from the named checkpoints
> All checkpoints that happened before the committed window is deleted but the 
> named checkpoints won't be deleted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Adding new log4j appender to Apex core

[Priyanka Gugale]

Hi,

Thomas,
Yes log4j is ultimately owned by user, and they should be able to override
it. What I am trying to do is provide a default behavior for Apex. In case
user isn't using any logger of their own we should use this new appender of
Apex rather than using standard log4j appender as per hadoop config.

Sanjay,
Archetype is the good place to put this and I will add it there, but many
time people won't use it. So I wanted to keep it at ~/.dt as well. Is there
any other default config folder for Apex?

Also I am not relying on anything. If we fail to find config in app jar or
~/.dt we are going to skip usage of this new appender.

-Priyanka

On Wed, Nov 23, 2016 at 5:58 AM, Sanjay Pujare 
wrote:

> The only way to “enforce” this new appender is to update the archetypes
> (apex-app-archetype and apex-conf-archetype under apex-core/ )  to use the
> new ones as default. But there does not seem to be a way to enforce this
> for anyone not using the archetypes.
>
> I agree with not relying on ~/.dt in apex-core.
>
> On 11/22/16, 1:08 PM, "Thomas Weise"  wrote:
>
> The log4j configuration is ultimately owned by the user, so how do you
> want
> to enforce a custom appender?
>
> I don't think that this should rely on anything in ~/.dt either
>
> Thomas
>
> On Tue, Nov 22, 2016 at 10:00 AM, Priyanka Gugale <
> priya...@datatorrent.com>
> wrote:
>
> > Hi,
> >
> > I am working on APEXCORE-563
> > 
> > As per this Jira we should put log file name in container/operator
> events.
> > The problem is current RollingFileAppender keeps renaming files from
> 1 to 2
> > to ... n as files reach maximum allowed file size. Because of
> constant
> > renaming of files we can't put a fixed file name in stram event.
> >
> > To overcome this I would like to add a new log4j appender to
> ApexCore.
> > There are two ways I can implement this:
> > 1. Have Daily rolling file appender. The current file will be
> recognized
> > based on timestamp in file name. Also to control max file size, we
> need to
> > keep rolling files based on size as well.
> > 2. Have Rolling File Appender but do not rename files. When max file
> size
> > is reached create new file with next number e.g. crate log file
> dt.log.2
> > after dt.log.1 is full. Also to recognize the latest file keep the
> softlink
> > named dt.log pointing to current log file.
> >
> > I would prefer to implement approach 2. Please provide your
> > comments/feedback if you feel otherwise.
> >
> > Also to use this new appender we need to use our custom
> log4j.properties
> > file instead of one present in hadoop conf. For that we need to set
> jvm
> > option -Dlog4j.configuration. I am planning to update file
> dt-site.xml in
> > folder ~/.dt  and default properties file available in apex
> archetype to
> > set jvm options as follows:
> >  
> >dt.attr.CONTAINER_JVM_OPTIONS
> >-Dlog4j.configuration=log4j.props
> >  
> >
> > And I will copy log4j.props file in ~/.dt folder as well as
> > apex-archetypes.
> >
> > Lastly if someone still miss this new log4j properties file or jvm
> option
> > to set -Dlog4j.configuration we will not put log file name in event
> raised
> > by container or operator.
> >
> > Please provide your feedback on this approach.
> >
> > -Priyanka
> >
>
>
>
>

Re: Core 3.5.0 release

[Vlad Rozov]

Thomas,

Agreed, both issues are not critical. I don't expect that APEXCORE-456 
(and related JIRAs) require API changes, so they may be released as part 
of the next minor release. At the same time, APEXCORE-456 is an 
enhancement/improvement and does not qualify for the minor bug fix release.


Thank you,

Vlad

On 11/22/16 14:31, Thomas Weise wrote:

Vlad,

I'm hoping to get the release done to fix a dependency issue in Beam. IMO
the issues above aren't critical for the release, they can go either into a
patch or the next minor version?

Thanks

On Tue, Nov 22, 2016 at 2:17 PM, Vlad Rozov  wrote:


Can the release be postponed by a week or two? I suggest to include
https://issues.apache.org/jira/browse/APEXCORE-539 and
https://issues.apache.org/jira/browse/APEXCORE-456. With APEXCORE-456 and
few related JIRAs (to be filed) I expect 2x performance improvement sending
tuples over the wire.

Thank you,

Vlad


On 11/22/16 12:32, David Yan wrote:


Hi Thomas,

Would like to get this merged for Apex Core 3.5.0 release if possible:

https://github.com/apache/apex-core/pull/401

Thanks,

David

On Tue, Nov 22, 2016 at 11:59 AM, Thomas Weise  wrote:

Hi,

I would like to propose the 3.5.0 release of Apex core:

https://issues.apache.org/jira/issues/?jql=project%20%3D%
20APEXCORE%20AND%
20fixVersion%20%3D%203.5.0%20ORDER%20BY%20status%20ASC%
2C%20updated%20DESC%2C%20priority%20DESC%2C%20created%20ASC

There are several enhancements that we should make available as release,
and the previous 3.4.0 release was on  2016-05-10

There is one outstanding issue, maybe that can it be moved to next
release?

Thanks,
Thomas

[jira] [Updated] (APEXCORE-461) Hidden ports in extended operators appear in the application package

[Vlad Rozov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXCORE-461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vlad Rozov updated APEXCORE-461:

Assignee: Bhupesh Chawda

> Hidden ports in extended operators appear in the application package
> 
>
> Key: APEXCORE-461
> URL: https://issues.apache.org/jira/browse/APEXCORE-461
> Project: Apache Apex Core
>  Issue Type: Bug
>Reporter: Bhupesh Chawda
>Assignee: Bhupesh Chawda
> Fix For: 3.5.0
>
>
> Ports which are hidden due to overriding in the sub classes of the operator 
> should not be shown in the final app package.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXCORE-498) Named Checkpoints - Checkpoint the DAG with a name/tag and start the app from that point

[Sanjay M Pujare (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688437#comment-15688437
 ] 

Sanjay M Pujare commented on APEXCORE-498:
--

If a user decides to use savepoints (named checkpoints), the platform cannot 
guarantee end-to-end exactly once or should not be expected to guarantee, isn't 
that right? And especially if the use-case is reprocessing. In such a case, if 
some of these guarantees are relaxed, doesn't the feature boil down to just the 
ability to name checkpoints and restore from these checkpoints (and related 
book-keeping).

> Named Checkpoints - Checkpoint the DAG with a name/tag and start the app from 
> that point
> 
>
> Key: APEXCORE-498
> URL: https://issues.apache.org/jira/browse/APEXCORE-498
> Project: Apache Apex Core
>  Issue Type: Improvement
>Reporter: Sandesh
>Assignee: Sandesh
>  Labels: roadmap
>
> Named Checkpoints 
> 1. Ability to tag/name the checkpoints
> 2. On demand - checkpoint the DAG
> 3. Start the app from the named checkpoints
> All checkpoints that happened before the committed window is deleted but the 
> named checkpoints won't be deleted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Adding new log4j appender to Apex core

[Sanjay Pujare]

The only way to “enforce” this new appender is to update the archetypes 
(apex-app-archetype and apex-conf-archetype under apex-core/ )  to use the new 
ones as default. But there does not seem to be a way to enforce this for anyone 
not using the archetypes.

I agree with not relying on ~/.dt in apex-core. 

On 11/22/16, 1:08 PM, "Thomas Weise"  wrote:

The log4j configuration is ultimately owned by the user, so how do you want
to enforce a custom appender?

I don't think that this should rely on anything in ~/.dt either

Thomas

On Tue, Nov 22, 2016 at 10:00 AM, Priyanka Gugale 
wrote:

> Hi,
>
> I am working on APEXCORE-563
> 
> As per this Jira we should put log file name in container/operator events.
> The problem is current RollingFileAppender keeps renaming files from 1 to 
2
> to ... n as files reach maximum allowed file size. Because of constant
> renaming of files we can't put a fixed file name in stram event.
>
> To overcome this I would like to add a new log4j appender to ApexCore.
> There are two ways I can implement this:
> 1. Have Daily rolling file appender. The current file will be recognized
> based on timestamp in file name. Also to control max file size, we need to
> keep rolling files based on size as well.
> 2. Have Rolling File Appender but do not rename files. When max file size
> is reached create new file with next number e.g. crate log file dt.log.2
> after dt.log.1 is full. Also to recognize the latest file keep the 
softlink
> named dt.log pointing to current log file.
>
> I would prefer to implement approach 2. Please provide your
> comments/feedback if you feel otherwise.
>
> Also to use this new appender we need to use our custom log4j.properties
> file instead of one present in hadoop conf. For that we need to set jvm
> option -Dlog4j.configuration. I am planning to update file dt-site.xml in
> folder ~/.dt  and default properties file available in apex archetype to
> set jvm options as follows:
>  
>dt.attr.CONTAINER_JVM_OPTIONS
>-Dlog4j.configuration=log4j.props
>  
>
> And I will copy log4j.props file in ~/.dt folder as well as
> apex-archetypes.
>
> Lastly if someone still miss this new log4j properties file or jvm option
> to set -Dlog4j.configuration we will not put log file name in event raised
> by container or operator.
>
> Please provide your feedback on this approach.
>
> -Priyanka
>

[jira] [Commented] (APEXMALHAR-2352) Improve performance of keyed windowed operators

[bright chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688259#comment-15688259
 ] 

bright chen commented on APEXMALHAR-2352:
-

This is an issue causes memory leak: 
https://issues.apache.org/jira/browse/APEXMALHAR-2350

> Improve performance of keyed windowed operators
> ---
>
> Key: APEXMALHAR-2352
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2352
> Project: Apache Apex Malhar
>  Issue Type: Task
>Reporter: bright chen
>Assignee: bright chen
>
> refer to https://issues.apache.org/jira/browse/APEXMALHAR-2339 to the keyed 
> windowed operator benchmark



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (APEXMALHAR-2352) Improve performance of keyed windowed operators

[bright chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

bright chen updated APEXMALHAR-2352:

Description: 
refer to https://issues.apache.org/jira/browse/APEXMALHAR-2339 to the keyed 
windowed operator benchmark


> Improve performance of keyed windowed operators
> ---
>
> Key: APEXMALHAR-2352
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2352
> Project: Apache Apex Malhar
>  Issue Type: Task
>Reporter: bright chen
>Assignee: bright chen
>
> refer to https://issues.apache.org/jira/browse/APEXMALHAR-2339 to the keyed 
> windowed operator benchmark



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (APEXMALHAR-2352) Improve performance of keyed windowed operators

[bright chen (JIRA)]

bright chen created APEXMALHAR-2352:
---

 Summary: Improve performance of keyed windowed operators
 Key: APEXMALHAR-2352
 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2352
 Project: Apache Apex Malhar
  Issue Type: Task
Reporter: bright chen
Assignee: bright chen






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] apex-core pull request #347: [APEXCORE-332][WIP] Improve partitioning to sup...

[ilganeli]

Github user ilganeli closed the pull request at:

https://github.com/apache/apex-core/pull/347


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #348: [APEXMALHAR-1978] Replaced usages of {groupId} ...

[ilganeli]

Github user ilganeli closed the pull request at:

https://github.com/apache/apex-core/pull/348


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (APEXMALHAR-1978) Replace ${groupId} with ${project.groupId} in modules and project pom

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-1978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688203#comment-15688203
 ] 

ASF GitHub Bot commented on APEXMALHAR-1978:


Github user ilganeli closed the pull request at:

https://github.com/apache/apex-core/pull/348


> Replace ${groupId} with ${project.groupId} in modules and project pom
> -
>
> Key: APEXMALHAR-1978
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-1978
> Project: Apache Apex Malhar
>  Issue Type: Task
>Reporter: Vlad Rozov
>Priority: Minor
> Fix For: 3.4.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXCORE-332) Support Even Distribution Of Tuples To A Non Power Of 2 Number Of Partitions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688204#comment-15688204
 ] 

ASF GitHub Bot commented on APEXCORE-332:
-

Github user ilganeli closed the pull request at:

https://github.com/apache/apex-core/pull/347


> Support Even Distribution Of Tuples To A Non Power Of 2 Number Of Partitions
> 
>
> Key: APEXCORE-332
> URL: https://issues.apache.org/jira/browse/APEXCORE-332
> Project: Apache Apex Core
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Ilya Ganelin
>Priority: Minor
>
> Currently partitions masks must be defined as a binary mask. As a result the 
> number of partitions must be a power of 2, otherwise the distribution of 
> tuples will be uneven. If we support the modulus operation instead of a 
> binary mask, we could support an even distribution of tuples to a non-power 
> of 2 number of partitions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Core 3.5.0 release

[Thomas Weise]

Vlad,

I'm hoping to get the release done to fix a dependency issue in Beam. IMO
the issues above aren't critical for the release, they can go either into a
patch or the next minor version?

Thanks

On Tue, Nov 22, 2016 at 2:17 PM, Vlad Rozov  wrote:

> Can the release be postponed by a week or two? I suggest to include
> https://issues.apache.org/jira/browse/APEXCORE-539 and
> https://issues.apache.org/jira/browse/APEXCORE-456. With APEXCORE-456 and
> few related JIRAs (to be filed) I expect 2x performance improvement sending
> tuples over the wire.
>
> Thank you,
>
> Vlad
>
>
> On 11/22/16 12:32, David Yan wrote:
>
>> Hi Thomas,
>>
>> Would like to get this merged for Apex Core 3.5.0 release if possible:
>>
>> https://github.com/apache/apex-core/pull/401
>>
>> Thanks,
>>
>> David
>>
>> On Tue, Nov 22, 2016 at 11:59 AM, Thomas Weise  wrote:
>>
>> Hi,
>>>
>>> I would like to propose the 3.5.0 release of Apex core:
>>>
>>> https://issues.apache.org/jira/issues/?jql=project%20%3D%
>>> 20APEXCORE%20AND%
>>> 20fixVersion%20%3D%203.5.0%20ORDER%20BY%20status%20ASC%
>>> 2C%20updated%20DESC%2C%20priority%20DESC%2C%20created%20ASC
>>>
>>> There are several enhancements that we should make available as release,
>>> and the previous 3.4.0 release was on  2016-05-10
>>>
>>> There is one outstanding issue, maybe that can it be moved to next
>>> release?
>>>
>>> Thanks,
>>> Thomas
>>>
>>>
>

[jira] [Commented] (APEXCORE-526) Publish javadoc for releases on ASF infrastructure

[Thomas Weise (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688112#comment-15688112
 ] 

Thomas Weise commented on APEXCORE-526:
---

It would be good to document what we have for Malhar so that others can add the 
branches when needed. I would like to make this part of the upcoming 3.6 
release. Would it be (later) possible to generate the user docs with this 
process also?
 

> Publish javadoc for releases on ASF infrastructure 
> ---
>
> Key: APEXCORE-526
> URL: https://issues.apache.org/jira/browse/APEXCORE-526
> Project: Apache Apex Core
>  Issue Type: Documentation
>Reporter: Thomas Weise
>
> Every release should have the javadocs published and we should have it linked 
> from the download page, as is the case with user docs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Core 3.5.0 release

[Vlad Rozov]

Can the release be postponed by a week or two? I suggest to include 
https://issues.apache.org/jira/browse/APEXCORE-539 and 
https://issues.apache.org/jira/browse/APEXCORE-456. With APEXCORE-456 
and few related JIRAs (to be filed) I expect 2x performance improvement 
sending tuples over the wire.


Thank you,

Vlad

On 11/22/16 12:32, David Yan wrote:

Hi Thomas,

Would like to get this merged for Apex Core 3.5.0 release if possible:

https://github.com/apache/apex-core/pull/401

Thanks,

David

On Tue, Nov 22, 2016 at 11:59 AM, Thomas Weise  wrote:


Hi,

I would like to propose the 3.5.0 release of Apex core:

https://issues.apache.org/jira/issues/?jql=project%20%3D%20APEXCORE%20AND%
20fixVersion%20%3D%203.5.0%20ORDER%20BY%20status%20ASC%
2C%20updated%20DESC%2C%20priority%20DESC%2C%20created%20ASC

There are several enhancements that we should make available as release,
and the previous 3.4.0 release was on  2016-05-10

There is one outstanding issue, maybe that can it be moved to next release?

Thanks,
Thomas

Re: Malhar release 3.6

[Thomas Weise]

IMO we should not delay the release any longer. I will cut the RC tomorrow
unless there is an objection.

Thomas


On Tue, Nov 22, 2016 at 2:05 PM, David Yan  wrote:

> Thomas,
>
> We need more time for the timebucket change and the purging feature, and we
> need to do benchmarking after those features have been implemented.
> Is it possible to postpone the release? If not, we can shoot for completing
> the features for the 3.7.0 release.
>
> David
>
> On Tue, Nov 22, 2016 at 12:08 PM, Thomas Weise  wrote:
>
> > David, Siyuan,
> >
> > Any update, I would like to cut the RC.
> >
> > Thanks
> >
> > On Sun, Nov 20, 2016 at 10:23 PM, Siyuan Hua 
> > wrote:
> >
> > > Yes yes,
> > > We need to put the clear plan upfront
> > >
> > > This ticket was thought and scheduled to be done after the
> > APEXMALHAR-2271
> > > SpillableSetMultimap which bucketize elements on the time from values.
> > > But per discussion with David,  we realized this ticket is more
> important
> > > and affect the general purging functionality so I just jumped to this
> > issue
> > > late last week.
> > > Now the status is the pull request is there. I feel it can be merged
> > soon.
> > > https://github.com/apache/apex-malhar/pull/503
> > >
> > > Regards,
> > > Siyuan
> > >
> > > On Sat, Nov 19, 2016 at 12:56 AM, Thomas Weise  wrote:
> > >
> > > > David,
> > > >
> > > > APEXMALHAR-2301 is still open. Are we on track to complete the JIRAs
> by
> > > > Monday or should we move to next release?
> > > >
> > > > Siyuan,
> > > >
> > > > There is a question on the ticket from a month ago that wasn't
> > > addressed. I
> > > > think there needs to be more visibility on the thought process and
> > > > decisions made, otherwise it is very difficult for others to follow.
> > Open
> > > > collaboration is key.
> > > >
> > > > Thanks
> > > >
> > > >
> > > > On Thu, Nov 17, 2016 at 8:29 PM, David Yan 
> > > wrote:
> > > >
> > > > > Hi Thomas,
> > > > >
> > > > > We would like to finish the following tickets if possible for the
> > 3.6.0
> > > > > release:
> > > > >
> > > > > https://issues.apache.org/jira/browse/APEXMALHAR-2301 (ETA: 11/18)
> > > > > https://issues.apache.org/jira/browse/APEXMALHAR-2339 (ETA: 11/21)
> > > > > https://issues.apache.org/jira/browse/APEXMALHAR-2345 (ETA: 11/21)
> > > > >
> > > > > For https://issues.apache.org/jira/browse/APEXMALHAR-2271, since
> > this
> > > > only
> > > > > affects Session Windows and still quite some work needs to be done
> > > there,
> > > > > we would like to defer that to the next release.
> > > > >
> > > > > Siyuan and Bright, please also chime in if I'm missing anything.
> > > > >
> > > > > David
> > > > >
> > > > > On Thu, Nov 17, 2016 at 2:49 AM, Thomas Weise 
> > wrote:
> > > > >
> > > > > > David,
> > > > > >
> > > > > > Any update WRT APEXMALHAR-2130
> > > > > >  /
> > > > > APEXMALHAR-2301
> > > > > >  ?
> > > > > >
> > > > > > I would like to cut first RC by end of the week.
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > On Wed, Nov 16, 2016 at 12:44 AM, Thomas Weise 
> > > wrote:
> > > > > >
> > > > > > > It has been a while and the issues that we were waiting for are
> > > still
> > > > > > > unresolved.
> > > > > > >
> > > > > > > https://issues.apache.org/jira/issues/?jql=fixVersion%
> > > > > > > 20%3D%203.6.0%20AND%20project%20%3D%20APEXMALHAR%20and%
> > > > > > 20resolution%20%3D%
> > > > > > > 20unresolved%20ORDER%20BY%20status%20ASC
> > > > > > >
> > > > > > > I would suggest:
> > > > > > >
> > > > > > > APEXMALHAR-2300  > > > jira/browse/APEXMALHAR-2300
> > > > > >
> > > > > > -
> > > > > > > defer
> > > > > > > APEXMALHAR-2130  > > > jira/browse/APEXMALHAR-2130
> > > > > >
> > > > > > /
> > > > > > > APEXMALHAR-2301  > > > jira/browse/APEXMALHAR-2301
> > > > > >
> > > > > > -
> > > > > > > David/Siyuan can you please give an update and recommendation.
> > > > > > > APEXMALHAR-2284  > > > jira/browse/APEXMALHAR-2284
> > > > > >
> > > > > > -
> > > > > > > see discussion on the PR and JIRA. Also discussed it with
> Bhupesh
> > > and
> > > > > > > Chinmay today and we think that this join operator should
> > probably
> > > be
> > > > > > > replaced with the new operator that is based on WindowOperator.
> > > > > > > APEXMALHAR-2203  > > > jira/browse/APEXMALHAR-2203
> > > > > >
> > > > > > -
> > > > > > > defer
> > > > > > >
> > > > > > > Please provide any feedback you may have within a day so that
> we
> > > can
> > > > > > > unblock the release.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Thomas
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > 

[jira] [Commented] (APEXCORE-148) Auto metric: metric aggregations test is failing

[Vlad Rozov (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688055#comment-15688055
 ] 

Vlad Rozov commented on APEXCORE-148:
-

Any update on the JIRA? I suggest lowering the priority.

> Auto metric: metric aggregations test is failing
> 
>
> Key: APEXCORE-148
> URL: https://issues.apache.org/jira/browse/APEXCORE-148
> Project: Apache Apex Core
>  Issue Type: Bug
>Reporter: Chandni Singh
>Priority: Critical
>
> From the build log :
> AutoMetricTest.testMetricsAggregations:279 progress expected:<2> but was:<1>



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Malhar release 3.6

[David Yan]

Thomas,

We need more time for the timebucket change and the purging feature, and we
need to do benchmarking after those features have been implemented.
Is it possible to postpone the release? If not, we can shoot for completing
the features for the 3.7.0 release.

David

On Tue, Nov 22, 2016 at 12:08 PM, Thomas Weise  wrote:

> David, Siyuan,
>
> Any update, I would like to cut the RC.
>
> Thanks
>
> On Sun, Nov 20, 2016 at 10:23 PM, Siyuan Hua 
> wrote:
>
> > Yes yes,
> > We need to put the clear plan upfront
> >
> > This ticket was thought and scheduled to be done after the
> APEXMALHAR-2271
> > SpillableSetMultimap which bucketize elements on the time from values.
> > But per discussion with David,  we realized this ticket is more important
> > and affect the general purging functionality so I just jumped to this
> issue
> > late last week.
> > Now the status is the pull request is there. I feel it can be merged
> soon.
> > https://github.com/apache/apex-malhar/pull/503
> >
> > Regards,
> > Siyuan
> >
> > On Sat, Nov 19, 2016 at 12:56 AM, Thomas Weise  wrote:
> >
> > > David,
> > >
> > > APEXMALHAR-2301 is still open. Are we on track to complete the JIRAs by
> > > Monday or should we move to next release?
> > >
> > > Siyuan,
> > >
> > > There is a question on the ticket from a month ago that wasn't
> > addressed. I
> > > think there needs to be more visibility on the thought process and
> > > decisions made, otherwise it is very difficult for others to follow.
> Open
> > > collaboration is key.
> > >
> > > Thanks
> > >
> > >
> > > On Thu, Nov 17, 2016 at 8:29 PM, David Yan 
> > wrote:
> > >
> > > > Hi Thomas,
> > > >
> > > > We would like to finish the following tickets if possible for the
> 3.6.0
> > > > release:
> > > >
> > > > https://issues.apache.org/jira/browse/APEXMALHAR-2301 (ETA: 11/18)
> > > > https://issues.apache.org/jira/browse/APEXMALHAR-2339 (ETA: 11/21)
> > > > https://issues.apache.org/jira/browse/APEXMALHAR-2345 (ETA: 11/21)
> > > >
> > > > For https://issues.apache.org/jira/browse/APEXMALHAR-2271, since
> this
> > > only
> > > > affects Session Windows and still quite some work needs to be done
> > there,
> > > > we would like to defer that to the next release.
> > > >
> > > > Siyuan and Bright, please also chime in if I'm missing anything.
> > > >
> > > > David
> > > >
> > > > On Thu, Nov 17, 2016 at 2:49 AM, Thomas Weise 
> wrote:
> > > >
> > > > > David,
> > > > >
> > > > > Any update WRT APEXMALHAR-2130
> > > > >  /
> > > > APEXMALHAR-2301
> > > > >  ?
> > > > >
> > > > > I would like to cut first RC by end of the week.
> > > > >
> > > > > Thanks
> > > > >
> > > > > On Wed, Nov 16, 2016 at 12:44 AM, Thomas Weise 
> > wrote:
> > > > >
> > > > > > It has been a while and the issues that we were waiting for are
> > still
> > > > > > unresolved.
> > > > > >
> > > > > > https://issues.apache.org/jira/issues/?jql=fixVersion%
> > > > > > 20%3D%203.6.0%20AND%20project%20%3D%20APEXMALHAR%20and%
> > > > > 20resolution%20%3D%
> > > > > > 20unresolved%20ORDER%20BY%20status%20ASC
> > > > > >
> > > > > > I would suggest:
> > > > > >
> > > > > > APEXMALHAR-2300  > > jira/browse/APEXMALHAR-2300
> > > > >
> > > > > -
> > > > > > defer
> > > > > > APEXMALHAR-2130  > > jira/browse/APEXMALHAR-2130
> > > > >
> > > > > /
> > > > > > APEXMALHAR-2301  > > jira/browse/APEXMALHAR-2301
> > > > >
> > > > > -
> > > > > > David/Siyuan can you please give an update and recommendation.
> > > > > > APEXMALHAR-2284  > > jira/browse/APEXMALHAR-2284
> > > > >
> > > > > -
> > > > > > see discussion on the PR and JIRA. Also discussed it with Bhupesh
> > and
> > > > > > Chinmay today and we think that this join operator should
> probably
> > be
> > > > > > replaced with the new operator that is based on WindowOperator.
> > > > > > APEXMALHAR-2203  > > jira/browse/APEXMALHAR-2203
> > > > >
> > > > > -
> > > > > > defer
> > > > > >
> > > > > > Please provide any feedback you may have within a day so that we
> > can
> > > > > > unblock the release.
> > > > > >
> > > > > > Thanks,
> > > > > > Thomas
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Nov 7, 2016 at 6:36 AM, Chaitanya Chebolu <
> > > > > > chaita...@datatorrent.com> wrote:
> > > > > >
> > > > > >> Hi Thomas,
> > > > > >>
> > > > > >>I am working on APEXMALHAR-2284 and will open a PR in couple
> of
> > > > days.
> > > > > >>
> > > > > >> Regards,
> > > > > >> Chaitanya
> > > > > >>
> > > > > >> On Sun, Nov 6, 2016 at 10:51 PM, Thomas Weise 
> > > wrote:
> > > > > >>
> > > > > >> > Is anyone working on 

Re: Adding new log4j appender to Apex core

[Thomas Weise]

The log4j configuration is ultimately owned by the user, so how do you want
to enforce a custom appender?

I don't think that this should rely on anything in ~/.dt either

Thomas

On Tue, Nov 22, 2016 at 10:00 AM, Priyanka Gugale 
wrote:

> Hi,
>
> I am working on APEXCORE-563
> 
> As per this Jira we should put log file name in container/operator events.
> The problem is current RollingFileAppender keeps renaming files from 1 to 2
> to ... n as files reach maximum allowed file size. Because of constant
> renaming of files we can't put a fixed file name in stram event.
>
> To overcome this I would like to add a new log4j appender to ApexCore.
> There are two ways I can implement this:
> 1. Have Daily rolling file appender. The current file will be recognized
> based on timestamp in file name. Also to control max file size, we need to
> keep rolling files based on size as well.
> 2. Have Rolling File Appender but do not rename files. When max file size
> is reached create new file with next number e.g. crate log file dt.log.2
> after dt.log.1 is full. Also to recognize the latest file keep the softlink
> named dt.log pointing to current log file.
>
> I would prefer to implement approach 2. Please provide your
> comments/feedback if you feel otherwise.
>
> Also to use this new appender we need to use our custom log4j.properties
> file instead of one present in hadoop conf. For that we need to set jvm
> option -Dlog4j.configuration. I am planning to update file dt-site.xml in
> folder ~/.dt  and default properties file available in apex archetype to
> set jvm options as follows:
>  
>dt.attr.CONTAINER_JVM_OPTIONS
>-Dlog4j.configuration=log4j.props
>  
>
> And I will copy log4j.props file in ~/.dt folder as well as
> apex-archetypes.
>
> Lastly if someone still miss this new log4j properties file or jvm option
> to set -Dlog4j.configuration we will not put log file name in event raised
> by container or operator.
>
> Please provide your feedback on this approach.
>
> -Priyanka
>

Re: Core 3.5.0 release

[David Yan]

Hi Thomas,

Would like to get this merged for Apex Core 3.5.0 release if possible:

https://github.com/apache/apex-core/pull/401

Thanks,

David

On Tue, Nov 22, 2016 at 11:59 AM, Thomas Weise  wrote:

> Hi,
>
> I would like to propose the 3.5.0 release of Apex core:
>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20APEXCORE%20AND%
> 20fixVersion%20%3D%203.5.0%20ORDER%20BY%20status%20ASC%
> 2C%20updated%20DESC%2C%20priority%20DESC%2C%20created%20ASC
>
> There are several enhancements that we should make available as release,
> and the previous 3.4.0 release was on  2016-05-10
>
> There is one outstanding issue, maybe that can it be moved to next release?
>
> Thanks,
> Thomas
>

Re: Malhar release 3.6

[Thomas Weise]

David, Siyuan,

Any update, I would like to cut the RC.

Thanks

On Sun, Nov 20, 2016 at 10:23 PM, Siyuan Hua  wrote:

> Yes yes,
> We need to put the clear plan upfront
>
> This ticket was thought and scheduled to be done after the APEXMALHAR-2271
> SpillableSetMultimap which bucketize elements on the time from values.
> But per discussion with David,  we realized this ticket is more important
> and affect the general purging functionality so I just jumped to this issue
> late last week.
> Now the status is the pull request is there. I feel it can be merged soon.
> https://github.com/apache/apex-malhar/pull/503
>
> Regards,
> Siyuan
>
> On Sat, Nov 19, 2016 at 12:56 AM, Thomas Weise  wrote:
>
> > David,
> >
> > APEXMALHAR-2301 is still open. Are we on track to complete the JIRAs by
> > Monday or should we move to next release?
> >
> > Siyuan,
> >
> > There is a question on the ticket from a month ago that wasn't
> addressed. I
> > think there needs to be more visibility on the thought process and
> > decisions made, otherwise it is very difficult for others to follow. Open
> > collaboration is key.
> >
> > Thanks
> >
> >
> > On Thu, Nov 17, 2016 at 8:29 PM, David Yan 
> wrote:
> >
> > > Hi Thomas,
> > >
> > > We would like to finish the following tickets if possible for the 3.6.0
> > > release:
> > >
> > > https://issues.apache.org/jira/browse/APEXMALHAR-2301 (ETA: 11/18)
> > > https://issues.apache.org/jira/browse/APEXMALHAR-2339 (ETA: 11/21)
> > > https://issues.apache.org/jira/browse/APEXMALHAR-2345 (ETA: 11/21)
> > >
> > > For https://issues.apache.org/jira/browse/APEXMALHAR-2271, since this
> > only
> > > affects Session Windows and still quite some work needs to be done
> there,
> > > we would like to defer that to the next release.
> > >
> > > Siyuan and Bright, please also chime in if I'm missing anything.
> > >
> > > David
> > >
> > > On Thu, Nov 17, 2016 at 2:49 AM, Thomas Weise  wrote:
> > >
> > > > David,
> > > >
> > > > Any update WRT APEXMALHAR-2130
> > > >  /
> > > APEXMALHAR-2301
> > > >  ?
> > > >
> > > > I would like to cut first RC by end of the week.
> > > >
> > > > Thanks
> > > >
> > > > On Wed, Nov 16, 2016 at 12:44 AM, Thomas Weise 
> wrote:
> > > >
> > > > > It has been a while and the issues that we were waiting for are
> still
> > > > > unresolved.
> > > > >
> > > > > https://issues.apache.org/jira/issues/?jql=fixVersion%
> > > > > 20%3D%203.6.0%20AND%20project%20%3D%20APEXMALHAR%20and%
> > > > 20resolution%20%3D%
> > > > > 20unresolved%20ORDER%20BY%20status%20ASC
> > > > >
> > > > > I would suggest:
> > > > >
> > > > > APEXMALHAR-2300  > jira/browse/APEXMALHAR-2300
> > > >
> > > > -
> > > > > defer
> > > > > APEXMALHAR-2130  > jira/browse/APEXMALHAR-2130
> > > >
> > > > /
> > > > > APEXMALHAR-2301  > jira/browse/APEXMALHAR-2301
> > > >
> > > > -
> > > > > David/Siyuan can you please give an update and recommendation.
> > > > > APEXMALHAR-2284  > jira/browse/APEXMALHAR-2284
> > > >
> > > > -
> > > > > see discussion on the PR and JIRA. Also discussed it with Bhupesh
> and
> > > > > Chinmay today and we think that this join operator should probably
> be
> > > > > replaced with the new operator that is based on WindowOperator.
> > > > > APEXMALHAR-2203  > jira/browse/APEXMALHAR-2203
> > > >
> > > > -
> > > > > defer
> > > > >
> > > > > Please provide any feedback you may have within a day so that we
> can
> > > > > unblock the release.
> > > > >
> > > > > Thanks,
> > > > > Thomas
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Nov 7, 2016 at 6:36 AM, Chaitanya Chebolu <
> > > > > chaita...@datatorrent.com> wrote:
> > > > >
> > > > >> Hi Thomas,
> > > > >>
> > > > >>I am working on APEXMALHAR-2284 and will open a PR in couple of
> > > days.
> > > > >>
> > > > >> Regards,
> > > > >> Chaitanya
> > > > >>
> > > > >> On Sun, Nov 6, 2016 at 10:51 PM, Thomas Weise 
> > wrote:
> > > > >>
> > > > >> > Is anyone working on APEXMALHAR-2284 ?
> > > > >> >
> > > > >> > On Fri, Oct 28, 2016 at 11:00 AM, Bhupesh Chawda <
> > > > >> bhup...@datatorrent.com>
> > > > >> > wrote:
> > > > >> >
> > > > >> > > +1
> > > > >> > >
> > > > >> > > ~ Bhupesh
> > > > >> > >
> > > > >> > > On Fri, Oct 28, 2016 at 2:29 PM, Tushar Gosavi <
> > > > >> tus...@datatorrent.com>
> > > > >> > > wrote:
> > > > >> > >
> > > > >> > > > +1
> > > > >> > > >
> > > > >> > > > - Tushar.
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > On Fri, Oct 28, 2016 at 12:52 PM, Aniruddha Thombare
> > > > >> > > >  wrote:
> > > > >> > > > > +1
> > > > >> > > > >
> > > > >> > > > > Thanks,
> > 

Re: Core 3.5.0 release

[Pramod Immaneni]

+1

On Tue, Nov 22, 2016 at 11:59 AM, Thomas Weise  wrote:

> Hi,
>
> I would like to propose the 3.5.0 release of Apex core:
>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20APEXCORE%20AND%
> 20fixVersion%20%3D%203.5.0%20ORDER%20BY%20status%20ASC%
> 2C%20updated%20DESC%2C%20priority%20DESC%2C%20created%20ASC
>
> There are several enhancements that we should make available as release,
> and the previous 3.4.0 release was on  2016-05-10
>
> There is one outstanding issue, maybe that can it be moved to next release?
>
> Thanks,
> Thomas
>

Core 3.5.0 release

[Thomas Weise]

Hi,

I would like to propose the 3.5.0 release of Apex core:

https://issues.apache.org/jira/issues/?jql=project%20%3D%20APEXCORE%20AND%20fixVersion%20%3D%203.5.0%20ORDER%20BY%20status%20ASC%2C%20updated%20DESC%2C%20priority%20DESC%2C%20created%20ASC

There are several enhancements that we should make available as release,
and the previous 3.4.0 release was on  2016-05-10

There is one outstanding issue, maybe that can it be moved to next release?

Thanks,
Thomas

[jira] [Resolved] (APEXCORE-532) New dynamically added operator does not start with correct windowId.

[Thomas Weise (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXCORE-532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Weise resolved APEXCORE-532.
---
Resolution: Fixed

> New dynamically added operator does not start with correct windowId.
> 
>
> Key: APEXCORE-532
> URL: https://issues.apache.org/jira/browse/APEXCORE-532
> Project: Apache Apex Core
>  Issue Type: Bug
>Reporter: Tushar Gosavi
>Assignee: Tushar Gosavi
>Priority: Critical
> Fix For: 3.5.0
>
>
> During dynamic DAG change, If new operator is added and connected to existing 
> operator, it does not starts with correct windowId. The baseSeconds is set to 
> 0 causing windowId management problems at master effectively halting purge 
> from buffer server.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXMALHAR-2339) Windowed Operator benchmarking

[bright chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15687670#comment-15687670
 ] 

bright chen commented on APEXMALHAR-2339:
-

The performance of KeyedWindowed operator decrease rapidly with the increase of 
key size (number of keys);
  - key size is 10K, rate is around 50K/second
  - key size is 100K, rate would be less than 1k/second

> Windowed Operator benchmarking
> --
>
> Key: APEXMALHAR-2339
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2339
> Project: Apache Apex Malhar
>  Issue Type: Task
>Reporter: bright chen
>Assignee: bright chen
> Attachments: Screen Shot 2016-11-21 at 10.34.38 AM.png
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Adding new log4j appender to Apex core

[Priyanka Gugale]

Hi,

I am working on APEXCORE-563

As per this Jira we should put log file name in container/operator events.
The problem is current RollingFileAppender keeps renaming files from 1 to 2
to ... n as files reach maximum allowed file size. Because of constant
renaming of files we can't put a fixed file name in stram event.

To overcome this I would like to add a new log4j appender to ApexCore.
There are two ways I can implement this:
1. Have Daily rolling file appender. The current file will be recognized
based on timestamp in file name. Also to control max file size, we need to
keep rolling files based on size as well.
2. Have Rolling File Appender but do not rename files. When max file size
is reached create new file with next number e.g. crate log file dt.log.2
after dt.log.1 is full. Also to recognize the latest file keep the softlink
named dt.log pointing to current log file.

I would prefer to implement approach 2. Please provide your
comments/feedback if you feel otherwise.

Also to use this new appender we need to use our custom log4j.properties
file instead of one present in hadoop conf. For that we need to set jvm
option -Dlog4j.configuration. I am planning to update file dt-site.xml in
folder ~/.dt  and default properties file available in apex archetype to
set jvm options as follows:
 
   dt.attr.CONTAINER_JVM_OPTIONS
   -Dlog4j.configuration=log4j.props
 

And I will copy log4j.props file in ~/.dt folder as well as apex-archetypes.

Lastly if someone still miss this new log4j properties file or jvm option
to set -Dlog4j.configuration we will not put log file name in event raised
by container or operator.

Please provide your feedback on this approach.

-Priyanka

[jira] [Commented] (APEXMALHAR-2351) Exception while fetching properties for Operators using JdbcStore

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15686273#comment-15686273
 ] 

ASF GitHub Bot commented on APEXMALHAR-2351:


GitHub user Hitesh-Scorpio opened a pull request:

https://github.com/apache/apex-malhar/pull/505

APEXMALHAR-2351 fix to extract operator properties for Jdbc Poll i/p …

…operator @bhupeshchawda please review

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Hitesh-Scorpio/apex-malhar 
APEXMALHAR-2351_JdbcInput

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-malhar/pull/505.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #505


commit b55d3ade5929189c6b6e65d93dbd0855d455c36e
Author: Hitesh-Scorpio 
Date:   2016-11-22T10:02:20Z

APEXMALHAR-2351 fix to extract operator properties for Jdbc Poll i/p 
operator




> Exception while fetching properties for Operators using JdbcStore 
> --
>
> Key: APEXMALHAR-2351
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2351
> Project: Apache Apex Malhar
>  Issue Type: Bug
>Reporter: Hitesh Kapoor
>Assignee: Hitesh Kapoor
>
> apex (application_1476346787877_0157) > get-operator-properties JdbcInput
> com.datatorrent.stram.cli.ApexCli$CliException: Failed web service request 
> for appid application_1476346787877_0157
>   at 
> com.datatorrent.stram.cli.ApexCli$GetOperatorPropertiesCommand.execute(ApexCli.java:2641)
>   at com.datatorrent.stram.cli.ApexCli$3.run(ApexCli.java:1468)
> Caused by: com.datatorrent.stram.cli.ApexCli$CliException: Failed to request 
> web service for appid application_1476346787877_0157
>   at com.datatorrent.stram.cli.ApexCli.getResource(ApexCli.java:1681)
>   at com.datatorrent.stram.cli.ApexCli.getResource(ApexCli.java:1658)
>   at com.datatorrent.stram.cli.ApexCli.access$9400(ApexCli.java:153)
>   at 
> com.datatorrent.stram.cli.ApexCli$GetOperatorPropertiesCommand.execute(ApexCli.java:2638)
>   ... 1 more
> Caused by: com.sun.jersey.api.client.UniformInterfaceException: GET 
> http://node32.morado.com:39573/ws/v2/stram/logicalPlan/operators/JdbcInput/properties
>  returned a response status of 404 Not Found
>   at com.sun.jersey.api.client.WebResource.handle(WebResource.java:676)
>   at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
>   at 
> com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:503)
>   at 
> com.datatorrent.stram.util.WebServicesClient$GetWebServicesHandler.process(WebServicesClient.java:259)
>   at 
> com.datatorrent.stram.util.WebServicesClient$2.run(WebServicesClient.java:201)
>   at 
> com.datatorrent.stram.util.SecureExecutor.execute(SecureExecutor.java:46)
>   at 
> com.datatorrent.stram.util.WebServicesClient.process(WebServicesClient.java:196)
>   at 
> com.datatorrent.stram.client.StramAgent.issueStramWebRequest(StramAgent.java:285)
>   at 
> com.datatorrent.stram.client.StramAgent.issueStramWebRequest(StramAgent.java:305)
>   at com.datatorrent.stram.cli.ApexCli.getResource(ApexCli.java:1675)
>   ... 4 more
> 16/10/28 02:35:15 ERROR cli.ApexCli: Exception caught: 
> com.datatorrent.stram.cli.ApexCli$CliException: Failed web service request 
> for appid application_1476346787877_0157
>   at 
> com.datatorrent.stram.cli.ApexCli$GetOperatorPropertiesCommand.execute(ApexCli.java:2641)
>   at com.datatorrent.stram.cli.ApexCli$3.run(ApexCli.java:1468)
> Caused by: com.datatorrent.stram.cli.ApexCli$CliException: Failed to request 
> web service for appid application_1476346787877_0157
>   at com.datatorrent.stram.cli.ApexCli.getResource(ApexCli.java:1681)
>   at com.datatorrent.stram.cli.ApexCli.getResource(ApexCli.java:1658)
>   at com.datatorrent.stram.cli.ApexCli.access$9400(ApexCli.java:153)
>   at 
> com.datatorrent.stram.cli.ApexCli$GetOperatorPropertiesCommand.execute(ApexCli.java:2638)
>   ... 1 more
> Caused by: com.sun.jersey.api.client.UniformInterfaceException: GET 
> http://node32.morado.com:39573/ws/v2/stram/logicalPlan/operators/JdbcInput/properties
>  returned a response status of 404 Not Found
>   at com.sun.jersey.api.client.WebResource.handle(WebResource.java:676)
>   at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
>   at 
> com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:503)
>   at 
> com.datatorrent.stram.util.WebServicesClient$GetWebServicesHandler.process(WebServicesClient.java:259)
>   at 
> 

[GitHub] apex-malhar pull request #505: APEXMALHAR-2351 fix to extract operator prope...

[Hitesh-Scorpio]

GitHub user Hitesh-Scorpio opened a pull request:

https://github.com/apache/apex-malhar/pull/505

APEXMALHAR-2351 fix to extract operator properties for Jdbc Poll i/p …

…operator @bhupeshchawda please review

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Hitesh-Scorpio/apex-malhar 
APEXMALHAR-2351_JdbcInput

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-malhar/pull/505.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #505


commit b55d3ade5929189c6b6e65d93dbd0855d455c36e
Author: Hitesh-Scorpio 
Date:   2016-11-22T10:02:20Z

APEXMALHAR-2351 fix to extract operator properties for Jdbc Poll i/p 
operator




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---