Re: Contribution and committer guidelines

Justin, I agree with your thoughts. Vetos are not rare in Apex. We are trying to figure a way to get there. Amol On Tue, Jan 29, 2019 at 3:01 PM Justin Mclean wrote: > Hi, > > If someone submits what you think is poor quality code just point it out > to them and ask them to fix it or even bette

Re: Contribution and committer guidelines

viewer > > > from my peer reviewers in the PR. I can dig into the archives and find > > > those if needed. > > > > > > Thanks > > > > > > On Mon, Jan 28, 2019 at 8:43 AM Vlad Rozov wrote: > > > > > >> Is there an example from prior PR

Re: Contribution and committer guidelines

n example from prior PRs where it was not accepted/merged due > to > >> a disagreement between a contributor and a committer on the amount of > >> refactoring or code quality? > >> > >> Thank you, > >> > >> Vlad > >> > >>

Re: Contribution and committer guidelines

+1 for this proposal. The only caveat I have is -> "acceptable performance and resolving logical flaws identified during the review process" is subjective. Functionally working should cover any logical issues. Performance should be applicable only to bug fixes and small enhancements to current fea

Re: [DISCUSS] Time for attic?

Vlad, Would you want to continue to be involved in the project, even if this involvement is itself causing community folks to stay away? If the issue is cultural, things will not improve. Doing the same thing again and expecting different result will not work. Why not change the policies that enfor

Re: [DISCUSS] Time for attic?

In the past we have made it difficult for code to be committed. Declining contributions is normal outcome of that. We should take a look at relaxing the threshold for commits, welcome more folks including into PMC before looking at taking the project to attic. Amol On Mon, Jan 7, 2019 at 8:24 PM

Re: [ANNOUNCE] New Apache Apex PMC Member: Chinmay Kolhatkar

Congrats Chinmay Amol On Thu, May 24, 2018 at 10:34 AM, Hitesh Kapoor wrote: > Congratulations Chinmay!! > > Regards, > Hitesh Kapoor > > On Thu 24 May, 2018, 11:02 PM Ilya Ganelin, wrote: > > > Congrats! > > > > On Thu, May 24, 2018, 10:09 AM Pramod Immaneni > wrote: > > > > > Congratulation

Re: Core release 3.7.0

+1 Thks, Amol E:a...@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre* www.datatorrent.com On Wed, Jan 17, 2018 at 1:55 PM, Pramod Immaneni wrote: > +1 > > > On Jan 17, 2018, at 7:25 AM, Thomas Weise wrote: > > > > Last release was 3.6.0 in May and following issues are ready for r

Re: Malhar release 3.8.0

+1 on a new malhar release. Thks, Amol E:a...@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre* www.datatorrent.com On Tue, Oct 24, 2017 at 9:12 PM, Tushar Gosavi wrote: > +1 on creating a new malhar release. > > - Tushar. > > > On Wed, Oct 25, 2017 at 4:39 AM, Pramod Immaneni > w

Re: [DISCUSS] inactive PR

Vlad, I am +1. Do proceed. I am not sure what the process is, i.e wait a day or so to get folks to give final opinion, or just proceed. Either way, your call. Thks Amol E:a...@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre* www.datatorrent.com On Wed, Sep 27, 2017 at 8:05 PM, Pram

Re: [DISCUSS] inactive PR

I am +1 on closing inactive PRs. Time wise 1 month looks short, 3 months looks long to me. In either case I do not have strong opinion on the time side; so I am 0+ on either. Thks, Amol E:a...@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre* www.datatorrent.com On Sun, Sep 24, 2017

Re: checking dependencies for known vulnerabilities

Vlad, Assuming you are in agreement that vulnerabilities should not be shown in public way; how would failing the build help. The reasons for failure will have be noted in public to be worked on. Anyway, IMO Apex may be better off exposing CVE as we are better off knowing these. But if folks want t

Re: following committer guideline when merging PR

Make sense. +1 Thks Amol E:a...@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre* www.datatorrent.com On Fri, Sep 8, 2017 at 9:16 AM, Vlad Rozov wrote: > Committers, > > Please make sure to follow Apex community guideline when merging PR > http://apex.apache.org/contributing.html.

Re: following committer guideline when merging PR

Make sense. +1 Thks Amol E:a...@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre* www.datatorrent.com On Fri, Sep 8, 2017 at 9:16 AM, Vlad Rozov wrote: > Committers, > > Please make sure to follow Apex community guideline when merging PR > http://apex.apache.org/contributing.html.

Re: [DISCUSS] Major version change for Apex Library (Malhar)

gt; > > > > vote argument can be used here for 4.x given that it involves > > > > modifications > > > > > to existing code. I would say we should drive towards getting a > > > consensus > > > > > by addressi

Re: [DISCUSS] changing project name to apex-library

gt;> On Thu, Aug 24, 2017 at 9:42 AM, Amol Kekre wrote: >> >> In terms of rebasing versions, there is no urgency in mimic-ing some of >>> the >>> other projects. Apex has already be been versioned. >>> >> >> There is an expectation users have f

Re: [DISCUSS] changing project name to apex-library

In terms of rebasing versions, there is no urgency in mimic-ing some of the other projects. Apex has already be been versioned. What functional gain do we have by changing versions, names? Functionality wise Apex users do not gain anything. With regards to bumping to 4.X, we should wait for a propo

Re: -1 or veto voting

* www.datatorrent.com On Wed, Aug 23, 2017 at 7:03 AM, Thomas Weise wrote: > The earlier discussion had concerns about making changes in 3.x and the > expressed preference was major version change. Accordingly the vote is for > major version change. > > > On Wed, Aug 23, 2017 at 6:56 AM,

Re: -1 or veto voting

The earlier discussion had concerns about this vote and the need to brand to 4.x right now. IMO they were not sufficiently addressed. Thks Amol E:a...@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre* www.datatorrent.com On Wed, Aug 23, 2017 at 6:54 AM, Thomas Weise wrote: > The di

Re: [VOTE] Major version change for Apex Library (Malhar)

On just voting part, I remain -1 on both options Thks Amol E:a...@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre* www.datatorrent.com On Tue, Aug 22, 2017 at 4:35 PM, Pramod Immaneni wrote: > I think we should take this discussion to a separate thread as it is a vote > thread. I

Re: [VOTE] Major version change for Apex Library (Malhar)

I am -1 on option 2. There is no need to do so, as going back on versions at this stage has consequences to Apex users. I am for option 1, but I want to propose explicit change to the text. Based on verbatim text, I am voting -1 on option 1. I believe in the original discussion thread there was ta

Re: Java packages: legacy -> org.apache.apex

This following pull request should be taken up in 4.0.0. See my comments in https://github.com/apache/apex-malhar/pull/664 https://github.com/

Re: anyone else seeing a 404

me too Thks Amol E:a...@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre* www.datatorrent.com On Thu, Jul 13, 2017 at 1:31 PM, Pramod Immaneni wrote: > https://git-wip-us.apache.org/repos/asf?p=apex-core.git >

Re: Backward compatibility issue in 3.6.0 release

I agree with Vlad; I would suggest option 1 too, but not a patch release. Documentation should suffice. Thks, Amol E:a...@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre* www.datatorrent.com On Mon, May 15, 2017 at 11:13 AM, Vlad Rozov wrote: > I considered proposing 3.6.1 patch r

Re: Proposal to upgrade Apex Core dependency in Malhar to 3.6

+1 Thks Amol E:a...@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre* www.datatorrent.com On Fri, May 12, 2017 at 3:26 PM, Pramod Immaneni wrote: > +1 > > On Thu, May 11, 2017 at 5:51 AM, AJAY GUPTA wrote: > > > Hi all, > > > > Apex Malhar currently depends on core 3.4. Custom tup

Re: binary releases

We should, it is good for the community. Thks Amol E:a...@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre* www.datatorrent.com On Fri, May 5, 2017 at 5:03 PM, Pramod Immaneni wrote: > I wanted to see how the community felt about also publishing binaries as > part of the release. H

Re: [VOTE] Apache Apex Core Release 3.6.0 (RC1)

+1 (binding) - Signatures, checksums ok - Build successfully - README.md, LICENSE, NOTICE, & CHANGELOG.md files Thks, Amol E:a...@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre* www.datatorrent.com On Tue, May 2, 2017 at 11:16 AM, Tushar Gosavi wrote: > Hi Thomas, > > I had pus

Re: PR merge policy

o respect the > community and policies it established. > > I would disagree that rolling back should be used only in case of a > disaster unless PR merge policy violation is a disaster :-) (and it > actually is). > > Thank you, > > Vlad > > On 4/28/17 14:21, Amol Kekre

Re: PR merge policy

Strongly agree with Ilya. Lets take these events as learning opportunities for folks to learn and improve. There can always be second commit to fix in case there is code issue. If it is a policy issue, we learn and improve. Rolling back, should be used rarely and it does need to be a disaster. We n

Re: Specialized operator recovery

+1. We should not change the default, as this needs explicit user involvement. The downstream operator has accomodate this behavior. Thks Amol E:a...@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre* www.datatorrent.com On Mon, Apr 24, 2017 at 10:56 AM, Sanjay Pujare wrote: > +1.

Re: Towards Apache Apex 3.6.0 release

+1 to cut a release Thks Amol E:a...@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre* www.datatorrent.com On Thu, Apr 13, 2017 at 9:22 AM, Pramod Immaneni wrote: > +1 > > I would like to see 699 and 700 addressed as well. > > On Wed, Apr 12, 2017 at 10:16 PM, Tushar Gosavi > wrot

Re: Why does emit require current thread to be the operator thread?

dy and > emitting it. Has anyone built a solution where it emits from the parent as > soon as it’s ready in the child (assuming I don’t care about order). > > - Ilya Ganelin > > > On 4/10/17, 12:45 PM, "Amol Kekre" wrote: > > Ilya, > This constraint

Re: Why does emit require current thread to be the operator thread?

Ilya, This constraint was introduced as allowing two threads to emit data creates lots of bad situations 1. The emit is triggered between end_window and begin_window. This was a critical blocker 2. Order no longer guaranteed, upon replay getting wrong order of events within a window. This was somet

Re: Apex Sandbox in Apex Dev Setup documentation

We should just point it to apex download page in the docs -> http://apex.apache.org/downloads.html; I will add my comments to the jira Thks Amol E:a...@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre* www.datatorrent.com On Fri, Apr 7, 2017 at 1:33 PM, Dean Lockgaard wrote: > Than

Re: [Design] - Kudu Output Operator

Ananth, This is good proposal. We will work with you. Thks Amol E:a...@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre* www.datatorrent.com On Sat, Apr 1, 2017 at 4:29 PM, ananth wrote: > Hello All, > > I would like to the community's opinion on the implementation of Kudu > output

Re: open/close ports and active/inactive streams

+1. This has come up with Apex customers on batch use cases. This will make batch use cases easy and robust. Today a lot of external tooling is needed. For Apex, this means reduction in stack of tech used. Thks Amol E:a...@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre* www.datatorr

Re: [VOTE] Apache Apex Malhar Release 3.7.0 (RC1)

+1 (binding) Verified signatures and file integrity Verified build successfully Verified presence of README.md, CHANGELOG.md, LICENSE, NOTICE files Thks, Amol E:a...@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre* www.datatorrent.com *Join us at Apex Big Data World Mt View

Re: Dependencies on libraries licensed as Category X

+1, need to take this as a way to reduce cost of submitting code by new contributors, along with the fact that this is a must do. Thks Amol E:a...@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre* www.datatorrent.com *Join us at Apex Big Data World Mt View

Re: PR merge policy

Pramod, That is a good idea. A timeout will help PR be more efficient. Thks Amol E:a...@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre* www.datatorrent.com *Join us at Apex Big Data World Mt View , April 4, 2017!* [image: http://www.ape

Re: Operator Node Affinity

ely. > > - Ilya Ganelin > > > On 3/2/17, 8:34 AM, "Amol Kekre" wrote: > > Ilya, > As Thomas says, attaching a JVM to an operator is do-able, but is > against > the norm in a distributed cluster. A distributed OS cannot guarantee a > nod

Re: Operator Node Affinity

Ilya, As Thomas says, attaching a JVM to an operator is do-able, but is against the norm in a distributed cluster. A distributed OS cannot guarantee a node. It could be down or not have resources, ZK way or any other way to discover post deployment is the way to go. I think a webservice call t

Re: APEXCORE-619 Recovery windowId in future during application relaunch.

computation > and compute the committed window to be the lowest common checkpoint. If > they are all stateless and you end up with long max you can start with > window id that reflects the current timestamp. > > Thanks > > On Wed, Mar 1, 2017 at 1:09 PM, Amol Kekre wrote: > &

Re: APEXCORE-619 Recovery windowId in future during application relaunch.

CommitWindowId could be computed from the existing checkpoints. That solution still needs purge to be done after commitWindowId is confirmed to be saved in Stram state. Without ths the commitWindowId computed from the checkpoints may have some checkpoints missing. Thks Amol E:a...@datatorrent.c

Re: APEXCORE-619 Recovery windowId in future during application relaunch.

The third option should be it. 1. On relaunch the DAG should start at commitWindowId 2. Pruning of checkpoints should only happen after committedWindowId is written by Stram state Thks Amol E:a...@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre* www.datatorrent.com | apex.apache.or

Re: example applications in malhar

endencies are compatible license > > > > >> > * maybe extract common boilerplate code from pom.xml > > > > >> > > > > > >> > etc. > > > > >> > > > > > >> > Existing examples are under http

Re: example applications in malhar

t; > >> > > > > >> > Existing examples are under https://github.com/apache/ > > > >> > apex-malhar/tree/master/demos > > > >> > > > > >> > Perhaps we should rename it to "examples" &g

Re: Redshift Output Operator

Chaitanya, This is good first cut. Post this work, do take a look at loading data before file rotation. Thks Amol *Follow @amolhkekre* *Join us at Apex Big Data World-San Jose , April 4, 2017!* [image: http://www.apexbigdata.com/san-jose-register.html] <

Re: [DISCUSS] Proposal for adapting Malhar operators for batch use cases

t; > ___ > > Bhupesh Chawda > > Software Engineer > > E: bhup...@datatorrent.com | Twitter: @bhupeshsc > > www.datatorrent.com | apex.apache.org > > > > On Sat, Feb 18, 2017 at 12:26 PM, Amol Kekre wrote: >

Re: [DISCUSS] Proposal for adapting Malhar operators for batch use cases

Thomas, I believe Bhupesh's proposal is to have a monotonically increasing watermark and filename as extra information. The usage of "file start" may have caused confusion. I agree, we do not need explicit "file start" watermark. I am at loss of words, maybe "start "->"end "; and then a "final-all

Re: [DISCUSS] Proposal for adapting Malhar operators for batch use cases

Thomas, The watermarks we have in Apex (start-window and end-window) are working good. It is fine to take a look at event time, but basic file I/O does not need anything more than start and end. Lets say they are start-something, end-something. The main difference here is that the tuples are user g

Re: PojoInnerJoin Accumulation emitting Map

yes it should be POJO Thks Amol *Join us at Apex Big Data World-San Jose , April 4, 2017!* [image: http://www.apexbigdata.com/san-jose-register.html] On Wed, Feb 15, 2017 at 7:34 AM, AJAY GUPTA wrote:

Re: [DISCUSS] Custom Control Tuples Design

is the > > > operator > > > > >> developer. I think the clear way for the operator developer to > > > override > > > > >> the > > > > >> propagation behavior is in code and if that is possible there is > no > > >

Re: [DISCUSS] Tweets from ApacheApex handle

commercial interests. > > Thomas > > > On Thu, Feb 2, 2017 at 12:22 PM, Pramod Immaneni > wrote: > > > I prefer option 2 at least till there is a critical mass of social > activity > > and awareness happening outside around apex that we don't need to tweet >

Re: [DISCUSS] Tweets from ApacheApex handle

dividual handles. I like the idea of retweet suggestions on the dev@ > list. > > Thomas > > > On Wed, Feb 1, 2017 at 4:43 PM, Amol Kekre wrote: > > > I want to see what the community feels about posting tweets on ApacheApex > > handle. My thoughts are that the committer

Re: relevant conferences and meetups on website

t; > On Wed, Feb 1, 2017 at 10:33 AM, Amol Kekre wrote: > > > I agree with Thomas on not putting meetup events. meetup.com does "index > > by > > Apex", and the search would simply pick up anyone putting Apex in the > event > > and at times they are not relevan

[DISCUSS] Tweets from ApacheApex handle

I want to see what the community feels about posting tweets on ApacheApex handle. My thoughts are that the committers should have the right to post a tweek on ApacheApex. For contributors and anyone else, we could have a mechanism where they request it on dev@apex.apache.org and a committer can hel

Re: relevant conferences and meetups on website

I agree with Thomas on not putting meetup events. meetup.com does "index by Apex", and the search would simply pick up anyone putting Apex in the event and at times they are not relevant to Apex. We get too much clutter. So the only way out is to manually select the events, and then take them out a

Re: Upgrade Apache Bigtop to Apex Core 3.5.0

+1 Thks Amol On Thu, Jan 19, 2017 at 8:55 PM, Priyanka Gugale wrote: > +1 > > On Fri, Jan 20, 2017 at 9:25 AM, Chinmay Kolhatkar < > chin...@datatorrent.com> > wrote: > > > Sanjay, > > Its not a lot of work. Just a version change, but primarily follwing > apache > > process for bigtop. > > Pow

Re: Shutdown of an Apex app

own the DAG and shutdown > operators as and when they encounter the control tuple. > > ~ Bhupesh > > On Wed, Jan 18, 2017 at 1:10 PM, Amol Kekre wrote: > > > Can be done by sending "shutdown" message via heartbeat to Stram. Then on > > stram can shutdown

Re: Shutdown of an Apex app

Can be done by sending "shutdown" message via heartbeat to Stram. Then on stram can shutdown the entire app Thks Amol On Tue, Jan 17, 2017 at 11:05 PM, Bhupesh Chawda wrote: > Yes Ajay, for a graceful shutdown, the data sent out should be processed. > > On Wed, Jan 18, 2017 at 12:19 PM, AJAY G

[jira] [Commented] (APEXCORE-503) support KillException

[ https://issues.apache.org/jira/browse/APEXCORE-503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15827550#comment-15827550 ] Amol Kekre commented on APEXCORE-503: - This is a good idea. We could have a

Re: Sharing jars among different Apex apps on cluster

Bhupesh, If stram changes the logical dag (runs only the sub-dag) on each runs, it should be ok. Thks Amol On Mon, Jan 16, 2017 at 9:18 AM, Bhupesh Chawda wrote: > Yes, I thought of that. That way it will be a single application > throughout. > Just wanted to see if there can be other options.

Re: Contribution Process before PR

I do see folks discussing issues for most part now. For me this does not look like an issue that is hurting Apex community. I do however want to discuss cases where code gets blocked and get spilled into larger context. As a culture and a process, we have a very high overhead that deters contributi

Re: [DISCUSS] Custom Control Tuples Design

I agree that by default the propagation must be implicit, i.e. if the operator does nothing, the control tuple propagates. I do think users should have control on deciding to "not propagate" or "create new" and in these cases they would need to do something explicit (override)? The following cases

Re: [DISCUSS] Custom Control Tuples Design

Yes, there is a chance that two output ports will have different send requirements. Thks Amol On Wed, Jan 4, 2017 at 10:59 AM, Bhupesh Chawda wrote: > Wouldn't having this with output ports give a finer control on the > propagation of control tuples? > We might have an operator with two output

Re: "ExcludeNodes" for an Apex application

listed ? > > Ram > > On Thu, Dec 1, 2016 at 11:52 PM, Amol Kekre wrote: > > > Yarn will deploy AM (Stram) on a node of its choice, therey rendering any > > attribute within the app un-enforceable in terms of not deploying master > on > > a node. > > > >

Re: "ExcludeNodes" for an Apex application

Yarn will deploy AM (Stram) on a node of its choice, therey rendering any attribute within the app un-enforceable in terms of not deploying master on a node. Thks Amol On Thu, Dec 1, 2016 at 11:19 PM, Milind Barve wrote: > Additionally, this would apply to Stram as well i.e. the master should

Re: "ExcludeNodes" for an Apex application

lable in Yarn itself. > > > > > > > > > > > > > > Second use case is also typical anti-affinity use case > which > > > > > ideally > > > > > > > should be implemented in Yarn – Milind’s example can also > > ap

Re: "ExcludeNodes" for an Apex application

with Amol that apps should be node neutral. Resource > management in > > Yarn together with fault tolerance in Apex should minimize the need > for > > this feature although I am sure one can find use cases. > > > > > > On 11/29/16, 10:41 PM, &q

Re: "ExcludeNodes" for an Apex application

We do have this feature in Yarn, but that applies to all applications. I am not sure if Yarn has anti-affinity. This feature may be used, but in general there is danger is an application taking over resource allocation. Another quirk is that big data apps should ideally be node-neutral. This is a g

Re: Apex internal documentation.

+1 Thks Amol On Tue, Nov 29, 2016 at 2:31 AM, Mohit Jotwani wrote: > +1 > > Regards, > Mohit > > On Tue, Nov 29, 2016 at 12:30 PM, Aniruddha Thombare < > anirud...@datatorrent.com> wrote: > > > +1 > > I would also request graphical representation for ease of understanding. > > I could not find

Re: JSON License and Apache Projects

repository.com/artifact/org.codehaus.jettison/jettison. > From the dependency tree in apex-core, I don't see anything that says > json.org. > > David > > On Fri, Nov 25, 2016 at 10:22 AM, Amol Kekre wrote: > > > Chinmay > > +1, Do you want to drive :) > >

Re: Megh operator library

> >> > > >> wrote: > >> > >> > Hi, > >> > > >> > Here is the initial proposal. Please go through it and you can comment > >> > right on the document. Regarding the discussions around Dimensional > >> > operators

Re: StramEvents for AppDataTracker for debugging and monitoring.

Deepak, This is great idea. Given that AppDataTracker is not part of Apex; dev@ may not be a place to take up this thread. Thks Amol On Mon, Nov 28, 2016 at 8:53 AM, Deepak Narkhede wrote: > Dear Community, > > Is it worth for addition of StramEvents for AppDataTracker for debugging > and moni

Re: JSON License and Apache Projects

we take a look at it and do the needful from our end too. > > -Chinmay. > > > On Fri, Nov 25, 2016 at 10:15 AM, Amol Kekre wrote: > > > Chinmay, > > Is this the thread you were looking for? > > > > Thks > > Amol > > > > -- Forward

Fwd: JSON License and Apache Projects

Chinmay, Is this the thread you were looking for? Thks Amol -- Forwarded message -- From: Ted Dunning Date: Thu, Nov 24, 2016 at 2:28 PM Subject: Re: JSON License and Apache Projects To: "gene...@incubator.apache.org" Stephan, What you suggest should work (if you add another

Re: Proposing a new feature to persist logical and physical plan snapshots in HDFS

p with when the plan changes took place inside the app. So saving > the plan JSON files on events described previously seems to be the most > efficient thing to do (as discussed with @Ashwin Putta) but if there are > use cases I think it is a good idea to do it on demand as well. >

Re: Proposing a new feature to persist logical and physical plan snapshots in HDFS

Good idea. Stram does save state, and maybe a script that translates may work. But explicit plan saving is also a good idea. Could this be "on demand"? a rest call that writes out the plan(s) to specifid hdfs files? We could do both (write on any change/set call) and/or on-demand. Thks Amol On

Re: Visitor API for DAG

codec could be written using asm library for > >> serialisation and use that as stream codec instead of default one. > >> > >> -Tushar. > >> > >> > >> On Thu, Nov 17, 2016 at 11:35 PM, Sanjay Pujare > > >> wrote: > >>

Re: Proposing an operator for log parsing.

Good addition Thks Amol On Thu, Nov 17, 2016 at 11:00 AM, Pradeep A. Dalvi wrote: > +1 for the feature > > On Thu, 17 Nov 2016 at 16:56 Shraddha Jog > wrote: > > > Dear community, > > > > We would like to add operator in malhar for parsing different types of > > logs. > > Idea of this operator

Re: Visitor API for DAG

+1. Opening up the API for users to put in their own code is good. In general we should enable users to register their code in a lot of scenerios. Thks Amol On Thu, Nov 17, 2016 at 9:06 AM, Tushar Gosavi wrote: > Yes, It could happen after current DAG validation and before the > application mas

Re: Proposal for apex/malhar extensions

+1 Thks Amol On Wed, Nov 16, 2016 at 5:37 AM, AJAY GUPTA wrote: > +1 > This is a good idea. > > Ajay > > On Wed, Nov 16, 2016 at 4:47 PM, Chinmay Kolhatkar > wrote: > > > Dear Community, > > > > This is in relation to malhar cleanup work that is ongoing. > > > > In one of the talks during Apac

Re: [DISCUSSION] Custom Control Tuples

after the custom control tuple till the next window > > begins. > > > > > > IMO, that too restrictive and in a case where input operator uses a > push > > > instead of a poll (for example, it provides an end point where remote > > > agents may connect a

Re: Malhar release 3.6

+1 Thks Amol On Wed, Oct 26, 2016 at 11:22 PM, Milind Barve wrote: > +1 > > On Thu, Oct 27, 2016 at 11:21 AM, Chinmay Kolhatkar > wrote: > > > +1. > > > > On Thu, Oct 27, 2016 at 1:41 AM, Thomas Weise wrote: > > > > > Hi, > > > > > > I'm proposing another release of Malhar in November. There

Re: can operators emit on a different from the operator itself thread?

t;> machine. This corresponds to 3% degradation at 30 million > events/second. I > >> think we can move forward with the check ON by default. Do we need an > >> ability to turn OFF check for a specific operator and/or port? My > thought > >> is that such abilit

Re: can operators emit on a different from the operator itself thread?

> Vlad > > > > > > On 8/11/16 10:21, Sanjay Pujare wrote: > > > > Thinking more about this – all of the “operator” threads are > > created > > > by the Stram engine with appropriate names. So we can put checks in >

Re: google analytics for apex website?

Yes, it should be accessible to project. How do we restrict it to dev@ only? Is there a convention to share this data open outside dev@? Thks Amol On Mon, Oct 10, 2016 at 6:59 PM, Thomas Weise wrote: > +1 this will be very valuable > > How would the data be accessible to the project? > > > On

Re: Kudu store operators

Ananth, This would be great to have. +1 Thks Amol On Sun, Oct 2, 2016 at 8:38 AM, Munagala Ramanath wrote: > +1 > > Kudu looks impressive from the overview, though it seems to still be > maturing. > > Ram > > > On Sat, Oct 1, 2016 at 11:42 PM, ananth wrote: > > > Hello All, > > > > I was wonde

Re: checkpoint statistics

+1. Very important stat for deciding a crucial question -> "Whether to checkpoint an operator?". It affects SLA, design, ... Thks Amol On Sat, Sep 24, 2016 at 10:01 AM, Vlad Rozov wrote: > IMO, it may be useful to provide checkpoint statistics for example, total > size of checkpoint for partic

Re: [ANNOUNCE] New Apache Apex PMC Member: Chandni Singh

Chandni, Congrats Thks Amol On Mon, Sep 12, 2016 at 10:22 AM, Chanchal Singh wrote: > Congratulations Chandni .. All the best > > On Sep 12, 2016 10:03 PM, "Thomas Weise" wrote: > > > The Apache Apex PMC is pleased to announce that Chandni Singh is now a > PMC > > member. We appreciate all her

Re: Megh operator library

Thomas, IMHO we should also look at the cost to users on keeping code in a github (even if under ASF 2.0 license) outside Malhar. There is value to deprecating code in Megh, and moving it to Malhar. Volunteers in this effort could decide on how much overlap means "mark as overlapping", My suggessti

Re: example applications in malhar

Good idea to consolidate them into Malhar. We should bring in as many from this gitHub as possible. Thks Amol On Tue, Sep 6, 2016 at 6:02 PM, Thomas Weise wrote: > I'm also for consolidating these different example locations. We should > also look if all of it is still relevant. > > The stuff

Re: anti-affinity - parameter to control the containers of an operator on a node

Eventually in long run Yarn (or other distributed OS) should handle isolation for all resources. Currently I/O is not isolated by Yarn, and will need Apex to help out for a year or two. Secondly if Apex works on another OS, we will also need to check if isolation is available for CPU, Memory. Overa

Re: can operators emit on a different from the operator itself thread?

+1 on debug proposal. Even if tuples lands up within the window, it breaks all guarantees. A rerun (after restart from a checkpoint) can have tuples in different windows from this thread. A separate thread simply exposes users to unwarranted risks. Thks Amol On Wed, Aug 10, 2016 at 6:05 PM, Vlad

Re: can operators emit on a different from the operator itself thread?

Sanjay just reminded me of my typo -> I meant between end_window and start_window :) Thks Amol On Wed, Aug 10, 2016 at 2:36 PM, Sanjay Pujare wrote: > If the goal is to do this validation through static analysis of operator > code, I guess it is possible but is going to be non-trivial. And ther

Re: can operators emit on a different from the operator itself thread?

Send too soon. A quicker way would be to catch emit happening between start_window and end_window and flag an error. Catching "another thread" for every tuple may have a huge performance hit. Thks Amol On Wed, Aug 10, 2016 at 2:31 PM, Amol Kekre wrote: > > Currently user can

Re: can operators emit on a different from the operator itself thread?

Currently user can code it that way. IMHO Apex should catch this and flag error. Thks Amol On Wed, Aug 10, 2016 at 2:04 PM, Ashwin Chandra Putta < ashwinchand...@gmail.com> wrote: > In a separate thread I mean. > > Regards, > Ashwin. > > On Wed, Aug 10, 2016 at 2:01 PM, Ashwin Chandra Putta < >

Re: [ANNOUNCE] New Apache Apex Committer: Devendra Tagare

Dev, Welcome aboard Thks Amol On Wed, Aug 10, 2016 at 1:13 PM, Siyuan Hua wrote: > Welcome, Devendra! > > On Wed, Aug 10, 2016 at 12:28 PM, Thomas Weise wrote: > > > The Project Management Committee (PMC) for Apache Apex has asked Devendra > > Tagare to become a committer and we are pleased t

Re: anti-affinity - parameter to control the containers of an operator on a node

Good idea. Thks Amol On Wed, Aug 10, 2016 at 11:27 AM, Venkatesh Kottapalli < venkat...@datatorrent.com> wrote: > Hi team, > > On the anti-affinity rules while deploying containers, do we have a > feature which can control the number of containers of the same operator > that get deployed on the

Re: empty operator/stream/module names

tomatic DAG generation by > higher level API or an execution plan generator, it will be responsibility > of the corresponding system to generate meaningful names. > > Vlad > > > On 8/5/16 09:04, Amol Kekre wrote: > >> Pradeep, >> The clash is if an user expli

  1   2   >