Re: Core release 3.7.0

+1 On Mon, Apr 9, 2018 at 11:31 AM Pramod Immaneni wrote: > Also, since we will be changing min jdk version to 8 for next release, I am > fine with thomas's suggestion to change next version to 4.0. If there are > no objections I will update the version on master. > >

Re: [RESULT] [VOTE] Major version change for Apex Library (Malhar)

Using all the technicalities and loop holes, we can declare many votes invalid. What purpose does it solve? This thread is dividing the community, instead of recognizing the difference if we move forward with this, there is a chance that Apex will alienate many contributors. What's the end game

Re: -1 or veto voting

Today, I saw the below -1 by Thomas, https://github.com/apache/apex-malhar/pull/666 without the technical justification. Saumya, PR Author, has created a mail thread to discuss the justification, but there was no comment in the mail thread. So should we consider this as invalid -1? On Thu, Aug

Re: Request to close/update progress of JIRA's.

Ambarish, those operators still need some work to make it user friendly. GenericRecord is not Kryo serializable, so AvroReader and GenericToPojo converter needs to be CONTAINER/THREAD local. This combo of 2 operators is best suitable for creation of Avro Module. On Mon, Aug 7, 2017 at 7:58 AM

Re: Java packages: legacy -> org.apache.apex

Why is there a urgency, why cant this go into 4.0 Malhar with possibly other breaking changes? On Sat, Jul 15, 2017 at 7:57 AM Thomas Weise wrote: > Discussing what in the future might become stable needs to be a separate > thread, it will be a much bigger discussion. > > The

Re: anyone else seeing a 404

+1 On Thu, Jul 13, 2017 at 2:00 PM Amol Kekre wrote: > me too > > Thks > Amol > > > E:a...@datatorrent.com | M: 510-449-2606 <(510)%20449-2606> | Twitter: > @*amolhkekre* > > www.datatorrent.com > > > On Thu, Jul 13, 2017 at 1:31 PM, Pramod Immaneni

Re: impersonation and application path

My vote is to make the new proposal as the default behavior. Is there a use case for the current behavior? If not then no need to add the configuration setting. On Thu, May 18, 2017 at 3:47 PM Pramod Immaneni wrote: > Sorry typo in sentence "as we are not asking for

Re: NO_LOCAL_WRITE Error from Stram

Please check the Hadoop dependency version in your POM. Also we need to move these discussions to users@ On Fri, Mar 3, 2017 at 2:14 PM Ganelin, Ilya wrote: > Minor amendment: hadoop-2.6.0+cdh5.8.0+1592 (2.6 vs 2.7) > > > > > > - Ilya Ganelin > > [image:

Re: APEXCORE-619 Recovery windowId in future during application relaunch.

Instead of treating the stateless operator in a special way and missing corner cases, just have a dummy checkpoint, then there is no need to handle corner cases. There is a name for this solution, https://en.wikipedia.org/wiki/Null_Object_pattern On Wed, Mar 1, 2017 at 2:52 PM Pramod Immaneni

Re: At-least once semantics for Kafka to Cassndra ingest

Settings mentioned by Sanjay, will only guarantee exactly once for Windows, but not for partial window processed by the operator, in a way that setting is a misnomer. To achieve Exactly once, there are some precoditions that need to be met along with the support in the output operator. Here is a

Re: Setting JAVA Serializer to be used at App Level.

Java serializer comes with a big performance cost, so it is better to reduce it's usage. Can you please give more detail about your use case? On Sun, Feb 5, 2017 at 10:05 PM Hitesh Kapoor wrote: Hi Ambarish, Yes you can plug in your own serializer. You will have to set

Re: Upgrade Apache Bigtop to Apex Core 3.5.0

Prefer it to be made part of Apex core release. No need for another vote. On Fri, Jan 20, 2017 at 1:53 PM Amol Kekre wrote: > +1 > > Thks > Amol > > > On Thu, Jan 19, 2017 at 8:55 PM, Priyanka Gugale > > wrote: > > > +1 > > > > On Fri, Jan 20,

Re: Suggestion on optimise kryo Output

Kryo is used in a default implementation of the StreamCodec interface. Ideally, if the StreamCodec interface itself allows the buffer to be passed then we can also send the buffer from the BufferServer in future. On Mon, Jan 9, 2017 at 4:10 PM Bright Chen wrote: > Hi, >

Re: [VOTE] Apache Apex Core Release 3.5.0 (RC1)

+1 Followed the steps in mentioned in http://apex.apache.org/verification.html Verified the launch of an application. On Tue, Dec 6, 2016 at 10:55 PM Thomas Weise wrote: > Dear Community, > > Please vote on the following Apache Apex Core 3.5.0 release candidate. > > This is a

Re: "ExcludeNodes" for an Apex application

Yarn allows the AppMaster to run on the selected node, Apex shouldn't select the blacklisted nodes, so it is possible to achieve not running the Apex containers on certain nodes. http://stackoverflow.com/questions/29302659/run-my-own-application-master-on-a-specific-node-in-a-yarn-cluster On

Re: "ExcludeNodes" for an Apex application

rn and request for other resources. > > Regards, > Mohit > > On Thu, Dec 1, 2016 at 2:34 AM, Sandesh Hegde <sand...@datatorrent.com > > > wrote: > > > Apex has automatic blacklisting of the troublesome nodes, please > take a > > loo

Re: [DISCUSSION] Custom Control Tuples

ation soon. > > Thanks! > > David > > On Mon, Nov 28, 2016 at 5:22 PM, Bhupesh Chawda <bhup...@datatorrent.com> > wrote: > > > +1 for the plan. > > > > I would be interested in contributing to this feature. > > > > ~ Bhupesh > > &g

Re: "ExcludeNodes" for an Apex application

Apex has automatic blacklisting of the troublesome nodes, please take a look at the following attributes, MAX_CONSECUTIVE_CONTAINER_FAILURES_FOR_BLACKLIST https://www.datatorrent.com/docs/apidocs/com/datatorrent/api/Context.DAGContext.html#MAX_CONSECUTIVE_CONTAINER_FAILURES_FOR_BLACKLIST

Re: [DISCUSSION] Custom Control Tuples

I am interested in contributing to this feature. On Mon, Nov 28, 2016 at 1:54 PM David Yan wrote: > I think we should probably go ahead with option 1 since this works with > most use cases and prevents developers from shooting themselves in the foot > in terms of

Re: Proposal for apex/malhar extensions

Do we have any projects today that can benefit from this setup? Earlier in this mail thread, we discussed "contrib (low bar) & graduation" in Malhar, that is not sufficient? On Wed, Nov 16, 2016 at 11:19 AM Chinmay Kolhatkar wrote: > @sanjay, yes we can define the process

Re: Malhar release 3.6

I have the PR open for the following issue, https://issues.apache.org/jira/browse/APEXMALHAR-2298 https://github.com/apache/apex-malhar/pull/492 This change was done after a user feedback. Should we get this in for 3.6? Thanks On Tue, Nov 15, 2016 at 3:45 PM Thomas Weise

Re: Integration with Apache Samoa

Good work Bhupesh. On Mon, Nov 7, 2016 at 11:17 AM David Yan wrote: > It took perseverance to get this merged, Good work Bhupesh! > > On Mon, Nov 7, 2016 at 1:25 AM, Bhupesh Chawda > wrote: > > > Hi All, > > > > The PR for making Apex a runner

Re: Malhar release 3.6

Here is a one more jira for 3.6, I need few more days to open the PR. https://issues.apache.org/jira/browse/APEXMALHAR-2298?filter=-1 Thanks On Thu, Oct 27, 2016 at 10:28 AM Vlad Rozov wrote: +1. It will be nice to have

Better error message for java version mismatch

Hi All, When an app package is compiled with Java 1.8 and Apex is using 1.7, we see the following behaviour with Apex CLI 1. launch "No applications in Application Package", ideally it should point out the exact error message 2. get-app-package-operators Throws the following exception

Re: [VOTE] Hadoop upgrade

+1 for 2.6 On Mon, Oct 3, 2016, 2:06 PM Sasha Parfenov wrote: > +1 for Hadoop 2.6 upgrade. > > Thanks, > Sasha > > On Monday, October 3, 2016, Thomas Weise wrote: > > > +1 for 2.6 upgrade > > > > > > On Mon, Oct 3, 2016 at 1:47 PM, David Yan

Re: Apex Core PR

Travis is lot faster than the Apache Jenkins, should we just stick with that? On Sat, Oct 1, 2016 at 9:46 AM Chinmay Kolhatkar wrote: > Hi Vlad, > > In one of the PR the travis build is passing but Jenkins build is failing. >

Re: Automated changes git author

BigFoot? On Wed, Sep 28, 2016 at 1:20 PM Thomas Weise wrote: > What about the name "CI Support"? Does not look like best fit either. Any > better ideas or keep it? > > I will document the outcome in the contributor guidelines. > > On Wed, Sep 28, 2016 at 11:13 AM, Pramod

Re: checkpoint statistics

Say it takes x MB size and y seconds to do the checkpoint. What does the user do with that information? On Sun, Sep 25, 2016, 6:51 AM Tushar Gosavi wrote: > +1 > > -Tushar > > On Sun, Sep 25, 2016, 8:54 AM Sanjay Pujare > wrote: > > > +1 > > > >

Re: Improving Apex relaunch time.

lure > analysis. Also the problem remains the same when we want to start from > savepoint, where we will need to copy state from > savepoint to application. > > -Tushar. > > > > On Tue, Sep 20, 2016 at 8:34 PM, Sandesh Hegde <sand...@datatorrent.com> > wrote: > &

Re: Improving Apex relaunch time.

How about re-launching the app from the same location? If at all they want to store the state we can provide savepoint feature. On Tue, Sep 20, 2016 at 4:39 AM Tushar Gosavi wrote: > We have observed that application relaunch takes long time. > The one major reason for

Re: [proposal] Application tags

One point to my previous mail, Yarn tags are supported from Hadoop 2.4 release onwards. Apex supports Hadoop 2.2, that is why Yarn feature cannot be leveraged. On Tue, Sep 13, 2016 at 1:24 PM Sandesh Hegde <sand...@datatorrent.com> wrote: > Hi All, > > I am proposing a new

[proposal] Application tags

Hi All, I am proposing a new attribute “Tags”, on the similar lines as Yarn application tags. (https://issues.apache.org/jira/browse/YARN-1461) This is useful in a case where an Admin wants to attach an extra information to Apex applications launched by various users/departments in the company.

Re: [ANNOUNCE] New Apache Apex PMC Member: Chandni Singh

Congratulations Chandni On Mon, Sep 12, 2016 at 10:46 AM Shubham Pathak wrote: > Congratulations Chandni !! > > Thanks, > Shubham > > On Mon, Sep 12, 2016 at 10:29 AM, Amol Kekre wrote: > > > Chandni, > > Congrats > > > > Thks > > Amol > > > > On

Re: [VOTE] Apache Apex Malhar Release 3.5.0 (RC2)

+1 Did the tests mentioned below http://apex.apache.org/verification.html On Wed, Aug 31, 2016 at 4:06 PM David Yan wrote: > +1 (binding) > > Downloaded the source, built with "mvn clean apache-rat:check verify > -Dlicense.skip=false -Pall-modules install" successfully

Re: Malhar 3.5.0 release

We can have categorized highlights, individuals can update it in the proper categories. New Operators New Features Major Bug Fixes Miscellaneous On Fri, Aug 26, 2016 at 2:04 PM Thomas Weise wrote: > There are 2 issues left which unless they get resolved today I

Re: [Proposal] DAG listener

t “major > activities in the DAG” will be available thru this listener? > > > > On 8/10/16, 8:20 AM, "Sandesh Hegde" <sand...@datatorrent.com> wrote: > > Any operators can subscribe to Stram Events affecting the DAG. > > Implementation will most

Re: [Proposal] DAG listener

fferent containers. > Are you suggesting an RPC interface or operator request for sending > this information > from Stram to all partitions of the interested operator? > > - Tushar. > > > > On Wed, Aug 10, 2016 at 11:28 AM, Sandesh Hegde <sand...@datatorrent.com> &

[Proposal] DAG listener

Hi All, As we add more features to support batch use cases, there will be a need to access more information about the DAG from an operator. One example is the need to access the operator name while using Batched StatsListener. The idea here is to implement DAG Listener ( similar to StatsListener

Re: [Proposal] Named Checkpoints

(a.k.a committed and dropping > > >> committed states and buffer data) while still having the ability to > > >> replay > > >>from that point from the input source and providing a way for > > >> operators (at > > >>first look input op

Re: [Proposal] Named Checkpoints

he lines of logrotate / general backup strategies. > > > Thanks, > > A > > _ > Sent with difficulty, I mean handheld ;) > On 4 Aug 2016 8:03 pm, "Munagala Ramanath" <r...@datatorrent.com> wrote: > > > +1

Re: Operator name in BatchedOperatorStats interface.

How about making the mapping of "OperatorId" to "OperatorName" ( and other extra information) as a part of the DAG context? On Mon, Jul 18, 2016 at 10:53 PM Tushar Gosavi wrote: > Hi All, > > We support shared stats listener, But user do not have any way to > identify

Re: [Proposal] Support storing apps in a Configuration Package

pp Package Apps? Or > something else? > > Thanks, > Sasha > > > > On Tue, Jul 19, 2016 at 5:37 PM, Sandesh Hegde <sand...@datatorrent.com> > wrote: > > > Hi All, > > > > Apex supports configuration package, separates application package

Re: Bleeding edge branch ?

nk to point > them to. BTW, High-level API can definitely benefit from java 8. :) > > Regards, > Siyuan > > On Wed, Jul 20, 2016 at 8:30 AM, Sandesh Hegde <sand...@datatorrent.com> > wrote: > > > Our current model of supporting the oldest supported Hadoop,

Re: Bleeding edge branch ?

Our current model of supporting the oldest supported Hadoop, penalizes the users of latest Hadoop versions by favoring the slow movers. Also, we won't benefit from the increased maturity of the Hadoop platform, as we will be working on the many years old version of Hadoop. We also need to

Re: Dynamic partition is not working in Kafka Input Operator

Was this resolved? My understanding is that, Kafka Input operator doesn't support the changes in Kafka partitions after the initial launch. On Mon, Jul 18, 2016 at 1:54 AM Chaitanya Chebolu wrote: > Hi All, > >I am facing dynamic partition issues in 0.8 version

Re: Bleeding edge branch ?

+1 with some variation Support next version, compared to one supported by the Apex main, of the Hadoop instead of the latest Hadoop. This makes moving the Apex main to next version of the Hadoop easy. On Mon, Jul 11, 2016 at 10:33 AM Sanjay Pujare wrote: > strong +1

Re: [Proposal] Make the default Unifier ThreadLocal with the downstream operator

Created a jira for this issue, https://issues.apache.org/jira/browse/APEXCORE-482 On Thu, Jul 7, 2016 at 9:22 PM Amol Kekre <a...@datatorrent.com> wrote: > +1. Makes sense. Do need to allow users to override if they want. > > Thks > Amol > > > On Thu, Jul 7, 2016 at 6

[Proposal] Make the default Unifier ThreadLocal with the downstream operator

Hi All, Unifier's are deployed as CONTAINER_LOCAL with the downstream operator( except in the corner case of Mx1 ). Default Unifiers are essentially doing buffer to buffer copy, so instead it should be THREAD_LOCAL to improve the performance. Let me know your thoughts on this. Thanks

Re: Getting various statitstics from StreamingContainers

://issues.apache.org/jira/browse/STORM-1157 On Mon, May 23, 2016 at 11:31 AM Sandesh Hegde <sand...@datatorrent.com> wrote: > After connecting to the app user will run the following command. > > Users will select the container id, jdk tool and the arguments to the tool. > > Apex C

Re: [DISCUSSION] Custom Control Tuples

Why restrict the control tuples to input operators? On Sat, Jun 25, 2016 at 9:07 AM Amol Kekre wrote: > David, > We should avoid control tuple within the window by simply restricting it > through API. This can be done by calling something like "sendControlTuple" > between

Re: APEXCORE-408 : Ability to schedule Sub-DAG from running application

For the usecase 1, is it possible to avoid changing the Context? Can we have something along the lines of "StramToNodeRequest" ? On Tue, Jun 21, 2016 at 11:09 AM Tushar Gosavi wrote: > Hi All, > > We have seen few use cases in field which require Apex application >

Re: Proposal : DAG - SetOperatorAttribute

Hi All, This change has been merged, will be part of 3.5. New API in the DAG, setOperatorAttribute(Operator operator, Attribute key, T value) ) Another API was deprecated, "setAttribute(Operator ... " Thanks On Tue, Jun 7, 2016 at 1:08 PM Sandesh Hegde <sand...@datatorrent.com

Proposal : Last Window Store for Output Operators

Hi All, Our current design pattern for using Window Data Manager is to purge the Window Data in the committed window call back. In the case of Output operators, all we care is the Last (/window) snapshot, that is because we have already written the tuples before that snapshot so those can be

Re: [ANNOUNCE] New Apache Apex PMC Member: Siyuan Hua

Congratulations!!! On Thu, Jun 16, 2016 at 10:06 AM Bright Chen wrote: > Siyuan, Congratulations! > > thanks > Bright > > > On Jun 16, 2016, at 9:27 AM, Pradeep A. Dalvi wrote: > > > > Congrats Siyuan! > > > > -prad > > > > On Thursday, June 16, 2016,

Purging the checkpoints from the StreamingContainers

Hello Team, Purging of the Checkpoints is done in Stram. Why not do that from the StreamingContainers? Committed window information is already available in StreamingContainers and it will also distribute the computation across the containers. Corner cases can still be handled in Stram. Example:

Re: Proposal : DAG - SetOperatorAttribute

easonable > to add *setOperatorAttribute*. > > Ram > > On Mon, Jun 6, 2016 at 1:39 PM, Sandesh Hegde <sand...@datatorrent.com> > wrote: > > > Currently, *setAttribute* is used to set the operator attributes. Other 2 > > Attribute setting APIs are specific to in

Re: A proposal for Malhar

+1 for removing the not-used operators. So we are creating a process for operator writers who don't want to understand the platform, yet wants to contribute? How big is that set? If we tell the app-user, here is the code which has not passed all the checklist, will they be ready to use that in

Re: Kafka Exactly once output operator

nd application has to be changed if it’s logic > depended on key. > > thanks > Bright > > > On May 13, 2016, at 12:37 PM, Sandesh Hegde <sand...@datatorrent.com> > wrote: > > > > Hi All, > > > > I am working on Kafka 0.9 output operator and one o