Re: [VOTE] Major version change for Apex Library (Malhar)

[Milind Barve]

-1 for option 2 as well. I do not see any reason why we should do so. This
will just confuse the users.

On Fri, Sep 1, 2017 at 5:48 PM, Yogi Devendra <yogideven...@apache.org>
wrote:

> One thing which is not clear to me is: if someone has any contributions
> planned for 3.x; will that contribution need 2 different PRs?
> If yes, then can we avoid this by giving sufficient time for the community
> to prepare for this change?
>
> -1 for immediate release.
> -1 for immediate separation for 3.x, 4.x.
> +0 for doing this after announcing some cooling period.
>
> ~ Yogi
>
> On 1 September 2017 at 17:32, Priyanka Gugale <pri...@apache.org> wrote:
>
> > Apologies for being late in discussions. I wanted to understand one
> thing.
> > As Thomas mentioned some of our operators are not matured enought or
> lacks
> > operability. Do we know if such operators need any backword incompatible
> > changes? e.g. modification to api etc? Do we have plan to promote
> operators
> > from Evolving to Stable during major version change? I think we should
> > think it through. We should list down all possible backword incompatible
> > changes, and then cut the release. Let's give sometime to developers to
> > come up with such issues in a given time window. A sudden change in major
> > version doesn't give us enough time to identify and address all such
> issues
> > and we may warrent one more major version change shortly.
> >
> > I will propose let's keep this open for sometime and we focus on
> > identifying changes which should go in next major version and then go for
> > it.
> >
> > -1 for immediate release or even making release branch now.
> >
> > -Priyanka
> >
> > On Fri, Sep 1, 2017 at 11:41 AM, Milind Barve <mili...@gmail.com> wrote:
> >
> > > Hi
> > >
> > > First of all my apologies for voting late. However, I will still do it
> > > since the mail says the vote would remain open for *at least* 72 hours
> :)
> > > I believe the objective is to do the right things rightly.
> > >
> > > Moving to a new version is something that is a part of any product
> > > lifecycle. While doing so, what is taken into account is -
> > >
> > > 1. What value the proposed changes are adding to the product.
> > > 2. Are the changes big enough to warrant a major version change
> > > 3. How big a disruption would it cause to the existing users or
> dependent
> > > products
> > > 4. Is enough of a heads up and time given to the users/dependent
> products
> > > to plan for the changes being introduced
> > >
> > > There could be other factors as well, but I think the above are the
> most
> > > critical ones.
> > >
> > > As regards the proposed changes, I don't think they satisfy either of
> the
> > > above criteria (expect may be #2 - which is a purely engg. decision)
> > >
> > > Given the changes proposed,
> > >
> > > 1. It is going to break the backward compatibility.
> > > 2. This is a disruptive change which is going to have existing users
> plan
> > > for and make changes in their product(s). They cannot move to 4.0
> without
> > > making these changes.
> > > 3. Don't think enough of a heads up has been giving to achieve what the
> > > users will have to do. As an industry standard, a heads up for at
> least 2
> > > releases is given before the change happens.
> > >
> > > Due to the above reasons, I am voting a -1
> > >
> > > Regards,
> > >
> > > On Sat, Aug 26, 2017 at 10:35 PM, Thomas Weise <t...@apache.org> wrote:
> > >
> > > > Being against something is not a valid reason. I also want to repeat
> a
> > > > point made earlier regarding discussion style:
> > > >
> > > > To facilitate a constructive, continuous discussion and make
> progress,
> > it
> > > > is necessary that participants take into account what was already
> > > > addressed. A few folks that never participated in the discussions
> > leading
> > > > to this vote don't make or don't want to make that effort.
> > > >
> > > > The list of functional features added by a release is determined when
> > the
> > > > release comes out, not before work starts. Furthermore, unless you
> own
> > a
> > > > crystal ball, you won't know what contributors decide to take up in
> the
> > > > future, as that's entirely up to them.
> > > >
> > 

Re: [VOTE] Major version change for Apex Library (Malhar)

[Milind Barve]

Hi

First of all my apologies for voting late. However, I will still do it
since the mail says the vote would remain open for *at least* 72 hours :)
I believe the objective is to do the right things rightly.

Moving to a new version is something that is a part of any product
lifecycle. While doing so, what is taken into account is -

1. What value the proposed changes are adding to the product.
2. Are the changes big enough to warrant a major version change
3. How big a disruption would it cause to the existing users or dependent
products
4. Is enough of a heads up and time given to the users/dependent products
to plan for the changes being introduced

There could be other factors as well, but I think the above are the most
critical ones.

As regards the proposed changes, I don't think they satisfy either of the
above criteria (expect may be #2 - which is a purely engg. decision)

Given the changes proposed,

1. It is going to break the backward compatibility.
2. This is a disruptive change which is going to have existing users plan
for and make changes in their product(s). They cannot move to 4.0 without
making these changes.
3. Don't think enough of a heads up has been giving to achieve what the
users will have to do. As an industry standard, a heads up for at least 2
releases is given before the change happens.

Due to the above reasons, I am voting a -1

Regards,

On Sat, Aug 26, 2017 at 10:35 PM, Thomas Weise  wrote:

> Being against something is not a valid reason. I also want to repeat a
> point made earlier regarding discussion style:
>
> To facilitate a constructive, continuous discussion and make progress, it
> is necessary that participants take into account what was already
> addressed. A few folks that never participated in the discussions leading
> to this vote don't make or don't want to make that effort.
>
> The list of functional features added by a release is determined when the
> release comes out, not before work starts. Furthermore, unless you own a
> crystal ball, you won't know what contributors decide to take up in the
> future, as that's entirely up to them.
>
> The reason to start a major release line is to enable breaking changes, as
> stated in the previous response. The top issues on my list don't include
> functional feature, but overall lack of consistency, modularity, too many
> operators that lack operability and maturity, that have not been designed
> to be production ready and those issues make it just very hard for users to
> find what is useful.
>
> New features are added from time to time. You can look at past release
> notes and you can also get some forward looking idea by following
> discussions and pull requests. One of the things I would hope to see added
> is the Python support, Kudu connectors and I would also hope to see changes
> to the high level Java API, SQL (windowing) and support for watermarks and
> batch demarcation in the existing operators. However, discussion of these
> isn't relevant to this vote thread.
>
>
> On Fri, Aug 25, 2017 at 6:12 PM, Ashwin Chandra Putta <
> ashwinchand...@gmail.com> wrote:
>
> > I am not against a new major version. I am against the reason for it.
> > Please include a list of functional feature set in the vote for 4.0, not
> > just package name changes.
> >
> > Regards,
> > Ashwin.
> >
> > On Aug 25, 2017 5:02 PM, "Thomas Weise"  wrote:
> >
> > > Always welcome. But before you go.. What do you mean by rigged? Your
> > vote?
> > > ;)
> > >
> > > Why do I get the impression someone could be pulling strings to stop a
> > new
> > > major version after the changes were discussed for such a long time?
> > >
> > > Why not let the community continue development, it does not prevent
> > anyone
> > > from continuing 3.x line. That's what versioning and branching are
> there
> > > for.
> > >
> > > Thomas
> > >
> > >
> > > On Fri, Aug 25, 2017 at 4:42 PM, Ashwin Chandra Putta <
> > > ashwinchand...@gmail.com> wrote:
> > >
> > > > Haha I missed your personal attacks Thomas. Love it! This voting
> thread
> > > is
> > > > rigged, I am out :)
> > > >
> > > > Anyway - as a user when a project announces a new major version - I
> > would
> > > > expect it to include some major features. I stand by my vote until we
> > > > associate and list some major features with the major version
> upgrade.
> > > >
> > > > Regards,
> > > > Ashwin.
> > > >
> > > > On Fri, Aug 25, 2017 at 3:54 PM, Thomas Weise 
> wrote:
> > > >
> > > > > Welcome back to the mailing list (it has been 6 months according to
> > the
> > > > > archive?). The topic seems to be important enough, so hopefully you
> > had
> > > > the
> > > > > chance to review the related discussion threads as these already
> > > address
> > > > > some of what repeats here?
> > > > >
> > > > > On Fri, Aug 25, 2017 at 2:00 PM, Ashwin Chandra Putta <
> > > > > ashwinchand...@gmail.com> wrote:
> > > > >
> > > > > > I agree that it makes the project 

Re: "ExcludeNodes" for an Apex application

[Milind Barve]

So all Apex will need to do is - to make sure as a part of the initial
configuration validations that the node selected to run the master is not a
part of the "excludeNode" list.

On Fri, Dec 2, 2016 at 1:47 PM, Sandesh Hegde <sand...@datatorrent.com>
wrote:

> Yarn allows the AppMaster to run on the selected node, Apex shouldn't
> select the blacklisted nodes, so it is possible to achieve not running the
> Apex containers on certain nodes.
>
> http://stackoverflow.com/questions/29302659/run-my-own-
> application-master-on-a-specific-node-in-a-yarn-cluster
>
>
> On Thu, Dec 1, 2016 at 11:52 PM Amol Kekre <a...@datatorrent.com> wrote:
>
> > Yarn will deploy AM (Stram) on a node of its choice, therey rendering any
> > attribute within the app un-enforceable in terms of not deploying master
> on
> > a node.
> >
> > Thks
> > Amol
> >
> >
> > On Thu, Dec 1, 2016 at 11:19 PM, Milind Barve <mili...@gmail.com> wrote:
> >
> > > Additionally, this would apply to Stram as well i.e. the master should
> > also
> > > not be deployed on these nodes. Not sure if anti-affinity goes beyond
> > > operators.
> > >
> > > On Fri, Dec 2, 2016 at 12:47 PM, Milind Barve <mili...@gmail.com>
> wrote:
> > >
> > > > My previous mail explains it, but just forgot to add : -1 to cover
> this
> > > > under anti affinity.
> > > >
> > > > On Fri, Dec 2, 2016 at 12:46 PM, Milind Barve <mili...@gmail.com>
> > wrote:
> > > >
> > > >> While it is possible to extend anti-affinity to take care of this, I
> > > feel
> > > >> it will cause confusion from a user perspective. As a user, when I
> > think
> > > >> about anti-affinity, what comes to mind right away is a relative
> > > relation
> > > >> between operators.
> > > >>
> > > >> On the other hand, the current ask is not that, but a relation at an
> > > >> application level w.r.t. a node. (Further, we might even think of
> > > extending
> > > >> this at an operator level - which would mean do not deploy an
> operator
> > > on a
> > > >> particular node)
> > > >>
> > > >> We would be better off clearly articulating and allowing users to
> > > >> configure it seperately as against using anti-affinity.
> > > >>
> > > >> On Fri, Dec 2, 2016 at 10:03 AM, Bhupesh Chawda <
> > > bhup...@datatorrent.com>
> > > >> wrote:
> > > >>
> > > >>> Okay, I think that serves an alternate purpose of detecting any
> newly
> > > >>> gone
> > > >>> bad node and excluding it.
> > > >>>
> > > >>> +1 for covering the original scenario under anti-affinity.
> > > >>>
> > > >>> ~ Bhupesh
> > > >>>
> > > >>> On Fri, Dec 2, 2016 at 9:14 AM, Munagala Ramanath <
> > r...@datatorrent.com
> > > >
> > > >>> wrote:
> > > >>>
> > > >>> > It only takes effect after failures -- no way to exclude from the
> > > >>> get-go.
> > > >>> >
> > > >>> > Ram
> > > >>> >
> > > >>> > On Dec 1, 2016 7:15 PM, "Bhupesh Chawda" <
> bhup...@datatorrent.com>
> > > >>> wrote:
> > > >>> >
> > > >>> > > As suggested by Sandesh, the parameter
> > > >>> > > MAX_CONSECUTIVE_CONTAINER_FAILURES_FOR_BLACKLIST seems to do
> > > exactly
> > > >>> > what
> > > >>> > > is needed.
> > > >>> > > Why would this not work?
> > > >>> > >
> > > >>> > > ~ Bhupesh
> > > >>> > >
> > > >>> >
> > > >>>
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> ~Milind bee at gee mail dot com
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > > ~Milind bee at gee mail dot com
> > > >
> > >
> > >
> > >
> > > --
> > > ~Milind bee at gee mail dot com
> > >
> >
>



-- 
~Milind bee at gee mail dot com

[jira] [Commented] (APEXCORE-584) User specified blacklist of nodes for resource allocation

[Milind Barve (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15714319#comment-15714319
 ] 

Milind Barve commented on APEXCORE-584:
---

Please note that this is at an application level. May be the ticket description 
should mention that.

> User specified blacklist of nodes for resource allocation
> -
>
> Key: APEXCORE-584
> URL: https://issues.apache.org/jira/browse/APEXCORE-584
> Project: Apache Apex Core
>  Issue Type: Improvement
>Reporter: Sandesh
>Assignee: Sandesh
>
> Based on the discussion in the mailing list, there is a need for user 
> specified blacklist of nodes.
> http://mail-archives.apache.org/mod_mbox/apex-dev/201611.mbox/browser



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: "ExcludeNodes" for an Apex application

[Milind Barve]

While it is possible to extend anti-affinity to take care of this, I feel
it will cause confusion from a user perspective. As a user, when I think
about anti-affinity, what comes to mind right away is a relative relation
between operators.

On the other hand, the current ask is not that, but a relation at an
application level w.r.t. a node. (Further, we might even think of extending
this at an operator level - which would mean do not deploy an operator on a
particular node)

We would be better off clearly articulating and allowing users to configure
it seperately as against using anti-affinity.

On Fri, Dec 2, 2016 at 10:03 AM, Bhupesh Chawda 
wrote:

> Okay, I think that serves an alternate purpose of detecting any newly gone
> bad node and excluding it.
>
> +1 for covering the original scenario under anti-affinity.
>
> ~ Bhupesh
>
> On Fri, Dec 2, 2016 at 9:14 AM, Munagala Ramanath 
> wrote:
>
> > It only takes effect after failures -- no way to exclude from the get-go.
> >
> > Ram
> >
> > On Dec 1, 2016 7:15 PM, "Bhupesh Chawda" 
> wrote:
> >
> > > As suggested by Sandesh, the parameter
> > > MAX_CONSECUTIVE_CONTAINER_FAILURES_FOR_BLACKLIST seems to do exactly
> > what
> > > is needed.
> > > Why would this not work?
> > >
> > > ~ Bhupesh
> > >
> >
>



-- 
~Milind bee at gee mail dot com

"ExcludeNodes" for an Apex application

[Milind Barve]

We have seen 2 cases mentioned below, where, it would have been nice if
Apex allowed us to exclude a node from the cluster for an application.

1. A node in the cluster had gone bad (was randomly rebooting) and so an
Apex app should not use it - other apps can use it as they were batch jobs.
2. A node is being used for a mission critical app (Could be an Apex app
itself), but another Apex app which is mission critical should not be using
resources on that node.

Can we have a way in which, Stram and YARN can coordinate between each
other to not use a set of nodes for the application. It an be done in 2 way
s-

1. Have a list of "exclude" nodes with Stram- when YARN allcates resources
on either of these, STRAM rejects and gets resources allocated again frm
YARN
2. Have a list of nodes that can be used for an app - This can be a part of
config. Hwever, I don't think this would be a right way to do so as we will
need support from YARN as well. Further, this might be difficult to change
at runtim if need be.

Any thoughts?


-- 
~Milind bee at gee mail dot com

Re: Malhar release 3.6

[Milind Barve]

+1

On Thu, Oct 27, 2016 at 11:21 AM, Chinmay Kolhatkar 
wrote:

> +1.
>
> On Thu, Oct 27, 2016 at 1:41 AM, Thomas Weise  wrote:
>
> > Hi,
> >
> > I'm proposing another release of Malhar in November. There are 49 issues
> > marked for the release, including important bug fixes, new documentation,
> > SQL support and the work for windowed operator state management:
> >
> > https://issues.apache.org/jira/issues/?jql=fixVersion%
> > 20%3D%203.6.0%20AND%20project%20%3D%20APEXMALHAR%20ORDER%
> > 20BY%20status%20ASC
> >
> > Currently there is at least one blocker, the join operator is broken
> after
> > change in managed state. It also affects the SQL feature.
> >
> > Thanks,
> > Thomas
> >
>



-- 
~Milind bee at gee mail dot com