Re: Moving to Apache

[Sanjeev Kulkarni]

Thanks Jerry for taking the lead on this!
I second the proposal to just transfer the github account to Apache. Could
someone from Twitter follow up internally?


On Tue, Feb 27, 2018 at 3:20 PM, Jerry Peng 
wrote:

> Hello all,
>
> I just want to start an email thread discussing moving Heron to
> Apache.  There are some items we need to figure out for this:
>
> 1. Moving the code to Apache github
>
> I was told that an repo can be transferred to another account and
> people have done this in the past to move to the Apache github
> account.  This is the best way to move the code to be under apache
> since with this method heron will keep all its stars and forks.
>
> We need to start converting heron packages from com.twitter -> org.apache.
>
> Ideally this whole process of migrating to Apache will not be a
> blocker for development and releases.
>
> Thus, if mentors or people with experience in this area want to chime
> in on the exact details (step by step) of what needs to be do for
> heron to be completely migrated to Apache that would be great!
>
> 2. Moving website to heron.apache.org
>
> What do we want to do here?  Migrate the whole website to
> heron.apache.org? And Have heron.io forward to heron.apache.org?
>
> 3.  How can we use apache infra
>
> I think committers/mentors need to file some tickets to apache infra for
> this.
>
> How can we use the apache infra to do apache release for heron?
>
> Lets get the discussion going!
>
> Thanks!
>
> Jerry
>

Re: Release Today/Tomorrow

[Sanjeev Kulkarni]

+1

On Mon, Feb 26, 2018 at 8:38 PM, Jerry Peng 
wrote:

> We haven't done a release in a while.  Should we do a release today or
> tomorrow?
>
> Best,
>
> Jerry
>

Re: About stream manager's quitting logic on connection failures

[Sanjeev Kulkarni]

This sounds good to me!

On Mon, Feb 5, 2018 at 1:08 AM, Ning Wang <wangnin...@gmail.com> wrote:

> Yeah. That is an option too. In fact it was my first try:
> https://github.com/twitter/heron/pull/2693 (just an initiative, not
> completed, a count map should be used instead of a single total count)
>
> In most cases, I think both solutions should have the same result. A few
> reasons I changed to a tmaster check:
> - with tmaster, there is only one source of truth and tmaster is more
> critical anyway. If the tmaster link is not healthy, stmgrs won't work
> correctly: topology may have created replacement nodes but the disconnected
> nodes could keep going by themselves.
> - it is more straightforward. The logic is the same as the current one. One
> the other side, if we use an array for all remote stmgrs, we could have a
> smarter logic (which is good) but it could make stmgrs more complicated and
> less straightforward (bad). I left the stmgr counters there so if in future
> we decide to add this feature, it should be easy to add. There is a gap
> between "errors from all" and "errors from a few" and this is not a
> simple/quick question.
>
>
>
>
> On Sun, Feb 4, 2018 at 6:48 PM, Sanjeev Kulkarni <sanjee...@gmail.com>
> wrote:
>
> > I could't add comments to the document, thus am posting my comments to
> the
> > mailing list
> > One more approach could be to do the current measurement as it is, but
> > instead of leaving the quitting decision to the stmgtclient, have
> > stmgrclientmgr do the decision. Thus everytime a stmgr client detects
> > connection issues, inform that to stmgrclientmgr which keeps a map of
> > peerstmgrid to error count. Thus it is able to decide things like am i
> > seeing connection errors from all stmgrs or if only a few of them are
> > having issues. Then it can take the decisions better.
> >
> > On Sat, Feb 3, 2018 at 8:11 PM, Ning Wang <wangnin...@gmail.com> wrote:
> >
> > > Hi, heron devs~
> > >
> > > I think the current stream manager's quitting logic on connection
> > failures
> > > is problematic. We saw a few internal cases in Twitter that this logic
> > > could cause extra issue.
> > >
> > > Here is a doc with more details:
> > >
> > > https://docs.google.com/document/d/1WHNc2NEp2gVL9ge2QVKp9t4Hpd4U9
> > > sAbzBqCu4-iDUM/edit#
> > >
> > > Comments and feedbacks are welcome!
> > >
> > > Thanks.
> > > --ning
> > >
> >
>

Re: Proposal : build single linux binary

[Sanjeev Kulkarni]

sounds good

On Thu, Dec 7, 2017 at 8:45 PM, Karthik Ramasamy  wrote:

> Sounds good.
>
> On Thu, Dec 7, 2017 at 8:06 PM Ali Ahmed  wrote:
>
> > I am considering changing bazel to  build a single binary for linux as
> > opposed to split among platforms for ubuntu and centos do people have
> > concerns or technical objections to this.
> >
> >
>

Release 0.17.1

[Sanjeev Kulkarni]

Hi all,
In light of the critical bug fix(https://github.com/twitter/heron/pull/2551),
might make sense to make a new release today. Any objections?

Re: Release 0.17.0

[Sanjeev Kulkarni]

Agreed on the freeze part with Karthik.

On Mon, Nov 13, 2017 at 2:22 PM, Karthik Ramasamy <kart...@streaml.io>
wrote:

> Sounds good to me. There is no need to freeze since the release tag
> essentially snapshots the code at that point in time. Users can choose the
> appropriate release that they are interested in.
>
> On Mon, Nov 13, 2017 at 1:51 PM FatJ Love <huijun.wu.2...@gmail.com>
> wrote:
>
> > looks good to me.
> >
> > btw. do we freeze code for the coming holiday season?
> >
> > On Mon, Nov 13, 2017 at 1:45 PM, Ali Ahmed <a.ah...@streaml.io> wrote:
> >
> > > Looks good to me.
> > >
> > >
> > > > On Nov 13, 2017, at 1:22 PM, Sanjeev Kulkarni <sanjee...@gmail.com>
> > > wrote:
> > > >
> > > > Hi all,
> > > > Continuing with our release cadence, we are planning a release today.
> > > > Please let me know if you have something that you want to be
> included.
> > > > Thanks!
> > >
> > >
> >
>

Release 0.17.0

[Sanjeev Kulkarni]

Hi all,
Continuing with our release cadence, we are planning a release today.
Please let me know if you have something that you want to be included.
Thanks!

Re: Release 0.16.5

[Sanjeev Kulkarni]

My bad! Will follow in the existing thread.

On Thu, Nov 2, 2017 at 2:32 PM Karthik Ramasamy <kart...@streaml.io> wrote:

> Can you use the previous thread instead of starting a new one?
>
> On Thu, Nov 2, 2017 at 2:31 PM Sanjeev Kulkarni <sanjee...@gmail.com>
> wrote:
>
> > Hi folks,
> > Planning a new release tomorrow. Anyone has any particular prs that they
> > absolutely need to go in this release?
> >
>

Re: Release 0.16.5

[Sanjeev Kulkarni]

Also looks like 2450 hasn’t yet gone in yet. Any idea if it will be ready
by today?

On Thu, Nov 2, 2017 at 2:35 PM Sanjeev Kulkarni <sanjee...@gmail.com> wrote:

> Just following up on this thread. We are planning to do this tomorrow. Any
> other prs?
>
> On Tue, Oct 31, 2017 at 1:27 PM Chris Kellogg <cckell...@gmail.com> wrote:
>
>> I would like to get the kubernetes scheduler fix in
>> https://github.com/twitter/heron/pull/2450.
>>
>> On Tue, Oct 31, 2017 at 1:24 PM, Karthik Ramasamy <kart...@streaml.io>
>> wrote:
>>
>> > Team - Any concerns on this, please let us know. Otherwise, we will go
>> > ahead and make the release this weekend.
>> >
>> > On Sat, Oct 28, 2017 at 1:22 PM, FatJ Love <huijun.wu.2...@gmail.com>
>> > wrote:
>> >
>> > > looks good to me
>> > >
>> > > On Sat, Oct 28, 2017 at 12:05 PM, Karthik Ramasamy <
>> kart...@streaml.io>
>> > > wrote:
>> > >
>> > > > All -
>> > > >
>> > > > While 0.16.4 release is out, we are interested in following up with
>> a
>> > > quick
>> > > > 0.16.5 release - since some of the necessary bug fixes needs to be
>> > > pushed.
>> > > >
>> > > > I propose releasing 0.16.5 this week. Thoughts and feedbacks are
>> > welcome!
>> > > >
>> > > > cheers
>> > > > /karthik
>> > > >
>> > >
>> >
>>
>

Re: Release 0.16.5

[Sanjeev Kulkarni]

Just following up on this thread. We are planning to do this tomorrow. Any
other prs?

On Tue, Oct 31, 2017 at 1:27 PM Chris Kellogg  wrote:

> I would like to get the kubernetes scheduler fix in
> https://github.com/twitter/heron/pull/2450.
>
> On Tue, Oct 31, 2017 at 1:24 PM, Karthik Ramasamy 
> wrote:
>
> > Team - Any concerns on this, please let us know. Otherwise, we will go
> > ahead and make the release this weekend.
> >
> > On Sat, Oct 28, 2017 at 1:22 PM, FatJ Love 
> > wrote:
> >
> > > looks good to me
> > >
> > > On Sat, Oct 28, 2017 at 12:05 PM, Karthik Ramasamy  >
> > > wrote:
> > >
> > > > All -
> > > >
> > > > While 0.16.4 release is out, we are interested in following up with a
> > > quick
> > > > 0.16.5 release - since some of the necessary bug fixes needs to be
> > > pushed.
> > > >
> > > > I propose releasing 0.16.5 this week. Thoughts and feedbacks are
> > welcome!
> > > >
> > > > cheers
> > > > /karthik
> > > >
> > >
> >
>

Release 0.16.5

[Sanjeev Kulkarni]

Hi folks,
Planning a new release tomorrow. Anyone has any particular prs that they
absolutely need to go in this release?

Re: Hello heron community

[Sanjeev Kulkarni]

My initial hunch is that these are different goals and therefore concur
with your thought that you could do this in phase 2.

On Mon, Oct 30, 2017 at 3:03 PM, Saikat Kanjilal <sxk1...@hotmail.com>
wrote:

> The goal is to tie together spark streaming and heron, I could very easily
> see an architecture where we build topologies in heron to apply business
> logic and pump them into time windows with spark streaming where can apply
> machine learning algorithms inside these windows.
>
>
> Detailed Example:
>
>
> Stock Trading or gamin use case:
>
>
> Heron is used to extract real time data from a gaming or trading client,
>
> Heron will apply a set of business rules to cleanup this data to fit into
> a set of machine learning models
>
> We need to then figure out how to connect spark streaming to heron, in the
> case of spark streaming it listens to some tcp or http endpoint so the
> objective would be to figure out the bridge between the two worlds.
>
>
> My point here is that if I am going through and building out a scala API
> it might make sense to also build a connector to a tech stack entirely
> written using scala as a first class citizen.
>
>
>
> Let me know your thoughts, we can also push this to phase 2 once the heron
> scala api is readily available.
>
>
> 
> From: Sanjeev Kulkarni <sanjee...@gmail.com>
> Sent: Monday, October 30, 2017 2:54 PM
> To: dev@heron.incubator.apache.org
> Subject: Re: Hello heron community
>
> I didn't really get what you meant by 'stream endpoint would be coming from
> heron'. Could you please elaborate? Preferably with an example?
>
> On Mon, Oct 30, 2017 at 2:26 PM, Saikat Kanjilal <sxk1...@hotmail.com>
> wrote:
>
> > I'm conflicted a bit on 1, here's why I feel like most of our users would
> > want to use the spouts/bolts API who are coming over from Storm or using
> > heron for the first time, however the Streamlets interface seems like it
> > has the right level of abstraction, I think I will start with the
> Streamlet
> > interface and add in the spout/bolt API as needed.   One other thought I
> > had was to figure out an integration plan with spark, to this end the fit
> > between heron and spark would be in using spark streaming where the
> stream
> > endpoint would be coming from heron, not sure if that should be part of
> > this effort or not, what do you guys think?
> >
> >
> > 
> > From: Sanjeev Kulkarni <sanjee...@gmail.com>
> > Sent: Monday, October 30, 2017 2:17 PM
> > To: dev@heron.incubator.apache.org
> > Subject: Re: Hello heron community
> >
> > I have a couple of questions
> > 1. Do you plan on exposing both the low level(spouts/bolts) and the
> > streamlet api? Or are you preferring one over another?
> > 2. My suggestion would be to start with the Streamlet interface.
> Primarily
> > because a) Most of scala ml libraries that I've seen tend to operate on
> the
> > streamlet kind of interface and b) It has a far smaller surface area(i.e.
> > number of interfaces) so might be easy to get something up quickly for
> > testing.
> > Thanks!
> >
> > On Mon, Oct 30, 2017 at 2:13 PM, Saikat Kanjilal <sxk1...@hotmail.com>
> > wrote:
> >
> > > Thanks Sanjeev,  I had an initial idea that I wanted to float on the
> > list,
> > > I was thinking that as part of the initial scala port of the API I'd
> like
> > > to propose that we use this tool: http://javatoscala.com/
> Java to Scala converter<http://javatoscala.com/>
> javatoscala.com
> How does it work? Java to Scala converter is created with Play framework
> and Scalagen library. I don't want you to see my code. No problem. Java to
> Scala converter ...
>
>
>
> > Java to Scala converter<http://javatoscala.com/>
> Java to Scala converter<http://javatoscala.com/>
> javatoscala.com
> How does it work? Java to Scala converter is created with Play framework
> and Scalagen library. I don't want you to see my code. No problem. Java to
> Scala converter ...
>
>
>
> > javatoscala.com
> > How does it work? Java to Scala converter is created with Play framework
> > and Scalagen library. I don't want you to see my code. No problem. Java
> to
> > Scala converter ...
> >
> >
> >
> > >
> > > Java to Scala converter<http://javatoscala.com/>
> Java to Scala converter<http://javatoscala.com/>
> javatoscala.com
> How does it work? Java to Scala converter is created with Play framework
> and Scalagen library. I don'

Re: Hello heron community

[Sanjeev Kulkarni]

I didn't really get what you meant by 'stream endpoint would be coming from
heron'. Could you please elaborate? Preferably with an example?

On Mon, Oct 30, 2017 at 2:26 PM, Saikat Kanjilal <sxk1...@hotmail.com>
wrote:

> I'm conflicted a bit on 1, here's why I feel like most of our users would
> want to use the spouts/bolts API who are coming over from Storm or using
> heron for the first time, however the Streamlets interface seems like it
> has the right level of abstraction, I think I will start with the Streamlet
> interface and add in the spout/bolt API as needed.   One other thought I
> had was to figure out an integration plan with spark, to this end the fit
> between heron and spark would be in using spark streaming where the stream
> endpoint would be coming from heron, not sure if that should be part of
> this effort or not, what do you guys think?
>
>
> ____
> From: Sanjeev Kulkarni <sanjee...@gmail.com>
> Sent: Monday, October 30, 2017 2:17 PM
> To: dev@heron.incubator.apache.org
> Subject: Re: Hello heron community
>
> I have a couple of questions
> 1. Do you plan on exposing both the low level(spouts/bolts) and the
> streamlet api? Or are you preferring one over another?
> 2. My suggestion would be to start with the Streamlet interface. Primarily
> because a) Most of scala ml libraries that I've seen tend to operate on the
> streamlet kind of interface and b) It has a far smaller surface area(i.e.
> number of interfaces) so might be easy to get something up quickly for
> testing.
> Thanks!
>
> On Mon, Oct 30, 2017 at 2:13 PM, Saikat Kanjilal <sxk1...@hotmail.com>
> wrote:
>
> > Thanks Sanjeev,  I had an initial idea that I wanted to float on the
> list,
> > I was thinking that as part of the initial scala port of the API I'd like
> > to propose that we use this tool: http://javatoscala.com/
> Java to Scala converter<http://javatoscala.com/>
> javatoscala.com
> How does it work? Java to Scala converter is created with Play framework
> and Scalagen library. I don't want you to see my code. No problem. Java to
> Scala converter ...
>
>
>
> >
> > Java to Scala converter<http://javatoscala.com/>
> Java to Scala converter<http://javatoscala.com/>
> javatoscala.com
> How does it work? Java to Scala converter is created with Play framework
> and Scalagen library. I don't want you to see my code. No problem. Java to
> Scala converter ...
>
>
>
> > javatoscala.com
> > How does it work? Java to Scala converter is created with Play framework
> > and Scalagen library. I don't want you to see my code. No problem. Java
> to
> > Scala converter ...
> >
> >
> > I propose that we use the above tool to do an initial conversion of all
> > the Java functional and oo interfaces and then plug that into the initial
> > design doc as well as the codebase to get things off the ground.   I have
> > already forked the heron repo and will be doing this off of a child
> branch
> > based on my fork.
> >
> >
> > How does that sound to folks, anyone has strong objections, of course I
> > fully realize that I will still need to do a lot of work around
> refactoring
> > the code even when using the converter so I'm prepared for that, the goal
> > in using the converters is to save a bit of time so that manual
> > intervention isnt necessarily needed in designing every interface.
> >
> > Thoughts.
> >
> > 
> > From: Sanjeev Kulkarni <sanjee...@gmail.com>
> > Sent: Sunday, October 29, 2017 10:18 PM
> > To: dev@heron.incubator.apache.org
> > Subject: Re: Hello heron community
> >
> > Hi Saikat,
> > Welcome to the Heron users group. Your plan of action sounds good to me.
> A
> > native scala interface would be great for all those scala fans. Eagerly
> > awaiting design doc.
> > Thanks!
> >
> > On Sun, Oct 29, 2017 at 6:17 PM, Saikat Kanjilal <sxk1...@hotmail.com>
> > wrote:
> >
> > > Hello Folks,
> > >
> > > I'm interested in taking and driving the following github component
> (this
> > > means design/development/architecture and more).
> > >
> > >
> > > https://github.com/twitter/heron/issues/668
> > [https://avatars1.githubusercontent.com/u/367684?s=400=4]<https://
> > github.com/twitter/heron/issues/668>
> >
> > Ability to write native scala topologies · Issue #668 · twitter/heron<
> > https://github.com/twitter/heron/issues/668>
> [https://avatars1.githubusercontent.com/u/367684?s=400=4]<https://
> github.com/twi

Re: Hello heron community

[Sanjeev Kulkarni]

I have a couple of questions
1. Do you plan on exposing both the low level(spouts/bolts) and the
streamlet api? Or are you preferring one over another?
2. My suggestion would be to start with the Streamlet interface. Primarily
because a) Most of scala ml libraries that I've seen tend to operate on the
streamlet kind of interface and b) It has a far smaller surface area(i.e.
number of interfaces) so might be easy to get something up quickly for
testing.
Thanks!

On Mon, Oct 30, 2017 at 2:13 PM, Saikat Kanjilal <sxk1...@hotmail.com>
wrote:

> Thanks Sanjeev,  I had an initial idea that I wanted to float on the list,
> I was thinking that as part of the initial scala port of the API I'd like
> to propose that we use this tool: http://javatoscala.com/
>
> Java to Scala converter<http://javatoscala.com/>
> javatoscala.com
> How does it work? Java to Scala converter is created with Play framework
> and Scalagen library. I don't want you to see my code. No problem. Java to
> Scala converter ...
>
>
> I propose that we use the above tool to do an initial conversion of all
> the Java functional and oo interfaces and then plug that into the initial
> design doc as well as the codebase to get things off the ground.   I have
> already forked the heron repo and will be doing this off of a child branch
> based on my fork.
>
>
> How does that sound to folks, anyone has strong objections, of course I
> fully realize that I will still need to do a lot of work around refactoring
> the code even when using the converter so I'm prepared for that, the goal
> in using the converters is to save a bit of time so that manual
> intervention isnt necessarily needed in designing every interface.
>
> Thoughts.
>
> 
> From: Sanjeev Kulkarni <sanjee...@gmail.com>
> Sent: Sunday, October 29, 2017 10:18 PM
> To: dev@heron.incubator.apache.org
> Subject: Re: Hello heron community
>
> Hi Saikat,
> Welcome to the Heron users group. Your plan of action sounds good to me. A
> native scala interface would be great for all those scala fans. Eagerly
> awaiting design doc.
> Thanks!
>
> On Sun, Oct 29, 2017 at 6:17 PM, Saikat Kanjilal <sxk1...@hotmail.com>
> wrote:
>
> > Hello Folks,
> >
> > I'm interested in taking and driving the following github component (this
> > means design/development/architecture and more).
> >
> >
> > https://github.com/twitter/heron/issues/668
> [https://avatars1.githubusercontent.com/u/367684?s=400=4]<https://
> github.com/twitter/heron/issues/668>
>
> Ability to write native scala topologies · Issue #668 · twitter/heron<
> https://github.com/twitter/heron/issues/668>
> github.com
> Support for Scala Instance and APIs.
>
>
>
> >
> >
> > [https://avatars1.githubusercontent.com/u/367684?v=4=400]<https://
> > github.com/twitter/heron/issues/668>
> >
> > Ability to write native scala topologies · Issue #668 ...<
> > https://github.com/twitter/heron/issues/668>
> [https://avatars1.githubusercontent.com/u/367684?s=400=4]<https://
> github.com/twitter/heron/issues/668>
>
> Ability to write native scala topologies · Issue #668 · twitter/heron<
> https://github.com/twitter/heron/issues/668>
> github.com
> Support for Scala Instance and APIs.
>
>
>
> > github.com
> > heron - Heron is a realtime, distributed, fault-tolerant stream
> processing
> > engine from Twitter
> >
> >
> >
> >
> > I've joined the heron slack channel as recommended and am ready to start
> > by creating a master JIRA issue to track this if not already done, I
> wanted
> > some guidance from a committer or two to help guide me a bit take care of
> > at he administrative details and how to get this kicked off the ground.
> My
> > background includes working on apache reef codebase as well as dabbling
> in
> > apache mahout and apache flume as well as storm and various nosql/graph
> > databases.  Looking forward to being part of the community.
> >
> >
> >
> > The rough plan of action in my mind so you know:
> >
> > 1) Review heron architecture and codebase in detail
> >
> > 2) Create an umbrella JIRA with the first series of tasks
> >
> > 3) Come up with a design doc
> >
> > 4) Build the interfaces
> >
> > 5) Add more dev tasks to JIRA
> >
> > 6) Drive the implementation
> >
> >
> > Thanks in advance for your help.
> >
> >
>

Re: Hello heron community

[Sanjeev Kulkarni]

Hi Saikat,
Welcome to the Heron users group. Your plan of action sounds good to me. A
native scala interface would be great for all those scala fans. Eagerly
awaiting design doc.
Thanks!

On Sun, Oct 29, 2017 at 6:17 PM, Saikat Kanjilal 
wrote:

> Hello Folks,
>
> I'm interested in taking and driving the following github component (this
> means design/development/architecture and more).
>
>
> https://github.com/twitter/heron/issues/668
>
>
> [https://avatars1.githubusercontent.com/u/367684?v=4=400] github.com/twitter/heron/issues/668>
>
> Ability to write native scala topologies · Issue #668 ...<
> https://github.com/twitter/heron/issues/668>
> github.com
> heron - Heron is a realtime, distributed, fault-tolerant stream processing
> engine from Twitter
>
>
>
>
> I've joined the heron slack channel as recommended and am ready to start
> by creating a master JIRA issue to track this if not already done, I wanted
> some guidance from a committer or two to help guide me a bit take care of
> at he administrative details and how to get this kicked off the ground.  My
> background includes working on apache reef codebase as well as dabbling in
> apache mahout and apache flume as well as storm and various nosql/graph
> databases.  Looking forward to being part of the community.
>
>
>
> The rough plan of action in my mind so you know:
>
> 1) Review heron architecture and codebase in detail
>
> 2) Create an umbrella JIRA with the first series of tasks
>
> 3) Come up with a design doc
>
> 4) Build the interfaces
>
> 5) Add more dev tasks to JIRA
>
> 6) Drive the implementation
>
>
> Thanks in advance for your help.
>
>

Re: Heron Release

[Sanjeev Kulkarni]

Hi all,
Status update. Currently https://github.com/twitter/heron/issues/2328 is
blocking the release.
Thanks!

On Tue, Sep 19, 2017 at 5:54 PM, Sanjeev Kulkarni <sanjee...@gmail.com>
wrote:

> Howdy,
> How is the timeline for this week's release looking?
> I believe we merged the changes requested by Neng, as well as 2 out of 4
> issues mentioned by Maosong. The other two bugs have Twitter related
> internal dependencies.
> Anything else we want to really go in for this one?
> Thanks!
>
> On Mon, Sep 18, 2017 at 11:57 AM, Sanjeev Kulkarni <sanjee...@gmail.com>
> wrote:
>
>> Neng, I left some comments on 2291. Thanks!
>>
>> On Mon, Sep 18, 2017 at 11:50 AM, Karthik Ramasamy <kart...@streaml.io>
>> wrote:
>>
>>> +1 - let us do a release this week.
>>>
>>> On Mon, Sep 18, 2017 at 11:31 AM, Neng Lu <freen...@gmail.com> wrote:
>>>
>>> > Hi all,
>>> >
>>> > Bring up the release again, user is requesting the fix:
>>> > https://github.com/twitter/heron/pull/2291
>>> >
>>> > And I think we already have a bunch of new thing merged since last
>>> release.
>>> > Could we schedule a release by the end of this week or early next week?
>>> >
>>> > On Thu, Sep 14, 2017 at 4:57 PM, Fu Maosong <maoson...@gmail.com>
>>> wrote:
>>> >
>>> > > I just mentioned some heron-ui issues and it seems we are planning to
>>> > work
>>> > > on it. Do you guys have any ideas how long it will take? If we can
>>> fix
>>> > them
>>> > > fast, it is also good to contain them.
>>> > >
>>> > > 2017-09-14 15:42 GMT-07:00 Sanjeev Kulkarni <sanjee...@gmail.com>:
>>> > >
>>> > > > My suggestion would be to let them fall into the next release.
>>> Would
>>> > love
>>> > > > to keep the cadence a release every 2 weeks.
>>> > > >
>>> > > > On Thu, Sep 14, 2017 at 3:40 PM, Karthik Ramasamy <
>>> kart...@streaml.io>
>>> > > > wrote:
>>> > > >
>>> > > > > Few days IMO, perhaps this weekend?
>>> > > > >
>>> > > > > On Thu, Sep 14, 2017 at 3:29 PM, Sanjeev Kulkarni <
>>> > sanjee...@gmail.com
>>> > > >
>>> > > > > wrote:
>>> > > > >
>>> > > > > > Do you have an eta for these?
>>> > > > > >
>>> > > > > > On Thu, Sep 14, 2017 at 3:24 PM Karthik Ramasamy <
>>> > kart...@streaml.io
>>> > > >
>>> > > > > > wrote:
>>> > > > > >
>>> > > > > > > I would like to get a couple of items in as well
>>> > > > > > >
>>> > > > > > > Ability to set some simple configs on the CLI tool
>>> > > > > > > Ability to run examples easily.
>>> > > > > > >
>>> > > > > > > It is not a lot of work but it will be nice to get them in.
>>> > > > > > >
>>> > > > > > > cheers
>>> > > > > > > /karthk
>>> > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > > On Thu, Sep 14, 2017 at 2:21 PM, FatJ Love <
>>> > > huijun.wu.2...@gmail.com
>>> > > > >
>>> > > > > > > wrote:
>>> > > > > > >
>>> > > > > > > > 0.15.3
>>> > > > > > > > LGTM
>>> > > > > > > >
>>> > > > > > > > On Thu, Sep 14, 2017 at 1:27 PM, Runhang Li <
>>> > > obj.runh...@gmail.com
>>> > > > >
>>> > > > > > > wrote:
>>> > > > > > > >
>>> > > > > > > > > Hi Sanjeev,
>>> > > > > > > > >
>>> > > > > > > > > I'd like to fix this https://github.com/twitter/
>>> > > > heron/issues/2272.
>>> > > > > I
>>> > > > > > > > > almost
>>> > > > > > > > > finished my patch and will do the PR today.
>>> > > > > > > > >
>>> > > > > > > > > > On Sep 14, 2017, at 1:10 PM, Sanjeev Kulkarni <
>>> > > > > sanjee...@gmail.com
>>> > > > > > >
>>> > > > > > > > > wrote:
>>> > > > > > > > > >
>>> > > > > > > > > > Hi folks,
>>> > > > > > > > > > Wondering what ppl feel about a release this week. Some
>>> > > > important
>>> > > > > > > > python
>>> > > > > > > > > > bug fixes have gone in this week, would be nice to cut
>>> a
>>> > > > 0.15.3.
>>> > > > > > > > > > Of-course, this is a non-apache release since the SGA
>>> is
>>> > not
>>> > > > > there
>>> > > > > > > yet.
>>> > > > > > > > > > Thanks!
>>> > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> > >
>>> > >
>>> > > --
>>> > > With my best Regards
>>> > > --
>>> > > Fu Maosong
>>> > > Twitter Inc.
>>> > > Mobile: +001-415-244-7520
>>> > >
>>> >
>>>
>>
>>
>

Re: PEX upgrade to 1.2.11

[Sanjeev Kulkarni]

This looks good to me. One additional side benefit of this change is that
it removes a couple of third party dependencies, thereby smoothening our
way into Apache.

On Mon, Sep 18, 2017 at 11:16 AM, Neng Lu  wrote:

> Currently, Maintaining pex is a pain for us. If we can get rid of its
> source code. That would be really great.
>
> Also, I verified this change with our environment. It passed our building
> process.
>
> On Mon, Sep 18, 2017 at 10:16 AM, Karthik Ramasamy 
> wrote:
>
> > All -
> >
> > I would like to move the PEX used by Heron to the latest version. The big
> > advantage of this new version is that it allows to generate
> multi-platform
> > PEX files.
> >
> > Typically, we upgrade by checking in the PEX sources into the Heron repo.
> > This is not desirable since the code needs to be maintained and some
> > changes to make it work in the repo. Instead a desirable approach is to
> > download the PEX code and dependencies and build them on the fly. This
> > makes it easier to maintain and upgrade quickly. We use bazel to do heavy
> > lifting for the download and building PEX using a wrapper script as
> > specified in
> >
> > https://github.com/benley/bazel_rules_pex
> >
> > Using the later approach, I have completed the update in the following
> PR:
> >
> > https://github.com/twitter/heron/pull/2314
> >
> > The PR does the following
> >
> > - Upgrade PEX and its dependencies to 1.2.11
> > - It uses bazel http file to download the repo and compile
> > - It loads the pex rules and java_tests apriori using
> > tools/build_rules/prelude_bazel (this is convenient so that you don't
> need
> > to include them in every files)
> >
> > Let me know your thoughts.
> >
> > cheers
> > /karthik
> >
>

Re: Issues of Heron tooling (Heron-ui)

[Sanjeev Kulkarni]

Have you guys tried on the latest release(0.15.2)?

On Thu, Sep 14, 2017 at 4:51 PM, Fu Maosong <maoson...@gmail.com> wrote:

> Hi Chris,
>
> I updated the issue context for 1 here:
> https://github.com/twitter/heron/issues/2307
>
> Opening the log file works, but download link there doesn't work. There are
> some cases where we can't scroll up all the way and search on the log page
> itself, and the only way is to download the file but the download link
> doesn't work:
>
> The downloaded file doesn't contain data, it contains only an error
> message:
> {"status": "failure", "executiontime": 0.0030930042266845703, "message":
> "HTTP 599: Stream closed", "version": "0.14.11"}
>
> 2017-09-14 16:02 GMT-07:00 Fu Maosong <maoson...@gmail.com>:
>
> > Thanks Chris. I would be great if you can work on 2,3 and 4. We would
> take
> > more context on 1 and circle back to you.
> >
> > 2017-09-14 15:43 GMT-07:00 Sanjeev Kulkarni <sanjee...@gmail.com>:
> >
> >> Wrt 1. I believe that internally Twitter uses another wrapper on top of
> >> the
> >> oss heron tool. Have you checked in that part?
> >>
> >> On Thu, Sep 14, 2017 at 3:41 PM, Chris Kellogg <cckell...@gmail.com>
> >> wrote:
> >>
> >> > 2,3 and 4 should be easy fixes. 1 - needs some more investigating. I
> am
> >> > able to download log files locally and from a remote kubernetes
> cluster
> >> > without any issues. I can work on 2,3 and 4.
> >> >
> >> > On Thu, Sep 14, 2017 at 2:15 PM, Fu Maosong <maoson...@gmail.com>
> >> wrote:
> >> >
> >> > > Hi,
> >> > >
> >> > > Many things in Heron tooling, especially for heron-ui, are broken
> >> right
> >> > > now:
> >> > >
> >> > > 1. Download of log files doesn't work (
> >> > > https://github.com/twitter/heron/issues/2307)
> >> > > 2. Stack trace listing is truncated (
> >> > > https://github.com/twitter/heron/issues/2308)
> >> > > 3. Heap dump download instructions are in-correct (
> >> > > https://github.com/twitter/heron/issues/2309)
> >> > > 4. This one has been around forever: view dropdown keeps
> disappearing
> >> on
> >> > > trying to click on underlying options. It takes 2-3 clicks many
> >> times. (
> >> > > https://github.com/twitter/heron/issues/2310)
> >> > >
> >> > > Fixing these can make our life much easier. I have created issue for
> >> each
> >> > > on github. Do you guys think we should prioritize them?
> >> > > Thanks.
> >> > >
> >> > > --
> >> > > With my best Regards
> >> > > --
> >> > > Fu Maosong
> >> > > Twitter Inc.
> >> > > Mobile: +001-415-244-7520
> >> > >
> >> >
> >>
> >
> >
> >
> > --
> > With my best Regards
> > --
> > Fu Maosong
> > Twitter Inc.
> > Mobile: +001-415-244-7520
> >
>
>
>
> --
> With my best Regards
> --
> Fu Maosong
> Twitter Inc.
> Mobile: +001-415-244-7520
>

Re: Inter-Container Encryption in Heron

[Sanjeev Kulkarni]

At this point, the thought is towards having one key pair accross the
entire topology. I'm not sure one key pair per container will add that much
more security. Thoughts?

On Thu, Sep 14, 2017 at 4:30 PM, FatJ Love <huijun.wu.2...@gmail.com> wrote:

> do you plan to use one key-pair for all the contaienrs or
> each container has one-key-pair?
>
> On Thu, Sep 14, 2017 at 3:39 PM, Sanjeev Kulkarni <sanjee...@gmail.com>
> wrote:
>
> > The keys ideally should be different for every topology. And ideally they
> > should be created in a just in time manner while launching the topology.
> > Otherwise, the chance that keys are no longer secret increases.
> >
> >
> > On Thu, Sep 14, 2017 at 3:34 PM, Karthik Ramasamy <kart...@streaml.io>
> > wrote:
> >
> > > Another option is keep this configuration in heron cli. For example,
> > >
> > > heron config  set --private-key ""
> > > heron config  set --publick-key ""
> > >
> > > Once the user sets it, then we can use these keys?
> > >
> > > cheers
> > > /karthik
> > >
> > > On Thu, Sep 14, 2017 at 3:11 PM, Sanjeev Kulkarni <sanjee...@gmail.com
> >
> > > wrote:
> > >
> > > > Hi all,
> > > > I believe at-least one current user of Heron is interested in
> > encrypting
> > > > their inter-container data flow within a Heron topology.  Since
> > > > inter-container traffic is between stmgrs, and because stmgrs use
> > > libevent
> > > > bufferevents for their transport, adding ssl in transport layer
> between
> > > > stmgrs is fairly straightforward.
> > > > The bigger question is how credentials are
> managed/transferred/stored.
> > > One
> > > > approach would to pass the public/private key as an argument to the
> > heron
> > > > cli while submitting the job. These will be stored in the uploader
> > > > alongside topology jars and downloaded by the containers upon start.
> > The
> > > > one issue with this approach is that the keys need to be secured by
> the
> > > > uploader.
> > > > I believe Kubernetes has provision for keeping secrets for jobs, but
> it
> > > > might not be portable to other scheduler environments. A way around
> > this
> > > > would be to create an spi for keeping secrets in heron/spi, but not
> > sure
> > > > what others feel about this.
> > > > Any other ideas?
> > > >
> > >
> >
>

Re: Heron Release

[Sanjeev Kulkarni]

Do you have an eta for these?

On Thu, Sep 14, 2017 at 3:24 PM Karthik Ramasamy <kart...@streaml.io> wrote:

> I would like to get a couple of items in as well
>
> Ability to set some simple configs on the CLI tool
> Ability to run examples easily.
>
> It is not a lot of work but it will be nice to get them in.
>
> cheers
> /karthk
>
>
>
>
> On Thu, Sep 14, 2017 at 2:21 PM, FatJ Love <huijun.wu.2...@gmail.com>
> wrote:
>
> > 0.15.3
> > LGTM
> >
> > On Thu, Sep 14, 2017 at 1:27 PM, Runhang Li <obj.runh...@gmail.com>
> wrote:
> >
> > > Hi Sanjeev,
> > >
> > > I'd like to fix this https://github.com/twitter/heron/issues/2272. I
> > > almost
> > > finished my patch and will do the PR today.
> > >
> > > > On Sep 14, 2017, at 1:10 PM, Sanjeev Kulkarni <sanjee...@gmail.com>
> > > wrote:
> > > >
> > > > Hi folks,
> > > > Wondering what ppl feel about a release this week. Some important
> > python
> > > > bug fixes have gone in this week, would be nice to cut a 0.15.3.
> > > > Of-course, this is a non-apache release since the SGA is not there
> yet.
> > > > Thanks!
> > >
> > >
> >
>

Re: Copyrights on code copied from other projects

[Sanjeev Kulkarni]

Hi Taylor,
Thanks for catching this. We're working on making sure that the copyright
is reflected accurately. Expect the change in a few days.
Thanks!

On Wed, Sep 6, 2017 at 1:28 PM, P. Taylor Goetz  wrote:

> Hi Heron devs,
>
> I know the Apache Heron repository has yet to move to ASF infrastructure,
> but I wanted to point out something that applies regardless of whether or
> not a project is an ASF project.
>
> When pulling in code from other projects (e.g. [1]), it isn’t appropriate
> to assert Twitter’s copyright in the code. Regardless of whether the code
> includes a copyright notice, the copyright belongs to the original author,
> not the individual or organization copying the code. While in this case the
> fact that the code is ALv2-licensed allows you to include the code in
> Heron, it does not transfer copyright.
>
> -Taylor
>
> [1] https://github.com/twitter/heron/pull/2241

Re: [Discuss] HealthManager launch switch on container-0

[Sanjeev Kulkarni]

Having a setting enabling clusterwide is indeed one of the desired
properties, as mentioned by Ashvin's first email. The setting in
healthmgr.yaml would control that. It would be set to false as default.
Users interested in trying it out could change that and submit it.


On Fri, Aug 4, 2017 at 10:55 AM, Karthik Ramasamy <kart...@streaml.io>
wrote:

> If we enable at healthmgr.yaml, it becomes cluster wide - which is not the
> desired option. For cli, there are no changes
> in the code. The config property function is built-in already. All you have
> to do is determine - what should be the key and
> its value.
>
> cheers
> /karthik
>
> On Fri, Aug 4, 2017 at 10:51 AM, Sanjeev Kulkarni <sanjee...@gmail.com>
> wrote:
>
> > Enabling health manager doesn't sound like a API. Thus I agree that
> Config
> > is not the right place for a setting like this.
> > I also don't like overloading cli with this. IMO cli is already
> overloaded
> > with a bunch of things that it shouldn't be.
> > Why can't we make this part of healthmgr.yaml itself? Or maybe
> > heron_internals.yaml?
> >
> > On Fri, Aug 4, 2017 at 10:12 AM, Karthik Ramasamy <kart...@streaml.io>
> > wrote:
> >
> > > Ashvin -
> > >
> > > Instead of adding a Config API to enable self-healing per topology, an
> > > interested user can enable the config using --config-property during
> > heron
> > > submit. For example,
> > >
> > > heron submit  --config-property
> > > "heron.config.topology.healthmanager.mode=enable" 
> > >  
> > >
> > > The advantage of this approach is that there is no hard coded config in
> > the
> > > code that will require later removal. Thoughts?
> > >
> > > cheers
> > > /karthik
> > >
> > >
> > > On Fri, Aug 4, 2017 at 8:57 AM, Ashvin A <ash...@apache.org> wrote:
> > >
> > > > Hi,
> > > >
> > > > We are in the process of merging the core building blocks of the
> > topology
> > > > health manager (HM) based on Dhalion. This integration is still
> > > > experimental and needs to be tested thoroughly. So it is desired that
> > the
> > > > HM be activated on-demand and remain disabled by default. Accordingly
> > we
> > > > are proposing the following scheme to launch HM process.
> > > >
> > > > We are thinking of satisfying the following constraints:
> > > >
> > > >1. Launch on container-0, colocated with the scheduler and the
> > metrics
> > > >cache.
> > > >2. Initially HM will be disabled by default. This means HM process
> > > >should not be started to avoid any side-effects. Once HM is well
> > > > tested, a
> > > >system wide configuration would enable HM for all topologies
> > submitted
> > > >afterwards.
> > > >3. If topology explicitly configure, opt-in, HM will be started
> and
> > > take
> > > >actions as per the configuration, i.e. healthmgr.yaml
> > > >4. Like other Heron processes, executor should manage the HM's
> life
> > > > cycle
> > > >
> > > > Accordingly we propose the following.
> > > >
> > > >1. Add new Config api to enable self-healing per topology:
> > > >Config.enableHealthManager(Topology.HealthManagerMode mode).
> > Default
> > > >value will be "system" to indicate use the system wide
> > configuration.
> > > >2. Add a new config to heron_internal.yaml:
> > > >"heron.healthmgr.default.mode". The value will be "disabled".
> > > >3. The Scheduler will read the default value of HM mode from the
> > > >heron_internals config file, like done in
> SchedulerMain.setupLogging
> > > > [3].
> > > >It will provide the either the user configured mode value or the
> > > default
> > > >mode value to the executor as a command line argument.
> > > >4. Add HM mode to the command like arguments to heron_executor.py.
> > > This
> > > >is similar to the executor command line arguments for check
> pointing
> > > > [2].
> > > >5. The executor will launch HM if mode is not "disabled".
> > > >6. Later if the default HM mode value is set to "dryrun" or
> > > >"self-healing", HM will be launched for all newly submitted
> > > topologies.
> > > >
> > > >
> > > > What do you think about this approach?
> > > >
> > > > Thanks,
> > > > Ashvin
> > > >
> > > >
> > > > [1] https://github.com/twitter/heron/pull/2132
> > > > [2] https://github.com/twitter/heron/blob/master/heron/
> > > > executor/src/python/
> > > > heron_executor.py#L58
> > > > [3] https://github.com/twitter/heron/blob/master/
> > > > heron/scheduler-core/src/java/com/twitter/heron/scheduler/
> > > > SchedulerMain.java#L277
> > > >
> > >
> >
>

Re: [Discuss] HealthManager launch switch on container-0

[Sanjeev Kulkarni]

Enabling health manager doesn't sound like a API. Thus I agree that Config
is not the right place for a setting like this.
I also don't like overloading cli with this. IMO cli is already overloaded
with a bunch of things that it shouldn't be.
Why can't we make this part of healthmgr.yaml itself? Or maybe
heron_internals.yaml?

On Fri, Aug 4, 2017 at 10:12 AM, Karthik Ramasamy 
wrote:

> Ashvin -
>
> Instead of adding a Config API to enable self-healing per topology, an
> interested user can enable the config using --config-property during heron
> submit. For example,
>
> heron submit  --config-property
> "heron.config.topology.healthmanager.mode=enable" 
>  
>
> The advantage of this approach is that there is no hard coded config in the
> code that will require later removal. Thoughts?
>
> cheers
> /karthik
>
>
> On Fri, Aug 4, 2017 at 8:57 AM, Ashvin A  wrote:
>
> > Hi,
> >
> > We are in the process of merging the core building blocks of the topology
> > health manager (HM) based on Dhalion. This integration is still
> > experimental and needs to be tested thoroughly. So it is desired that the
> > HM be activated on-demand and remain disabled by default. Accordingly we
> > are proposing the following scheme to launch HM process.
> >
> > We are thinking of satisfying the following constraints:
> >
> >1. Launch on container-0, colocated with the scheduler and the metrics
> >cache.
> >2. Initially HM will be disabled by default. This means HM process
> >should not be started to avoid any side-effects. Once HM is well
> > tested, a
> >system wide configuration would enable HM for all topologies submitted
> >afterwards.
> >3. If topology explicitly configure, opt-in, HM will be started and
> take
> >actions as per the configuration, i.e. healthmgr.yaml
> >4. Like other Heron processes, executor should manage the HM's life
> > cycle
> >
> > Accordingly we propose the following.
> >
> >1. Add new Config api to enable self-healing per topology:
> >Config.enableHealthManager(Topology.HealthManagerMode mode). Default
> >value will be "system" to indicate use the system wide configuration.
> >2. Add a new config to heron_internal.yaml:
> >"heron.healthmgr.default.mode". The value will be "disabled".
> >3. The Scheduler will read the default value of HM mode from the
> >heron_internals config file, like done in SchedulerMain.setupLogging
> > [3].
> >It will provide the either the user configured mode value or the
> default
> >mode value to the executor as a command line argument.
> >4. Add HM mode to the command like arguments to heron_executor.py.
> This
> >is similar to the executor command line arguments for check pointing
> > [2].
> >5. The executor will launch HM if mode is not "disabled".
> >6. Later if the default HM mode value is set to "dryrun" or
> >"self-healing", HM will be launched for all newly submitted
> topologies.
> >
> >
> > What do you think about this approach?
> >
> > Thanks,
> > Ashvin
> >
> >
> > [1] https://github.com/twitter/heron/pull/2132
> > [2] https://github.com/twitter/heron/blob/master/heron/
> > executor/src/python/
> > heron_executor.py#L58
> > [3] https://github.com/twitter/heron/blob/master/
> > heron/scheduler-core/src/java/com/twitter/heron/scheduler/
> > SchedulerMain.java#L277
> >
>

Re: Proposal for Heron API Server

[Sanjeev Kulkarni]

One comment that I have wrt the API server is some sort of design note on
how you plan to accomodate authentication(atleast on some popular
schedulers like DCOS and Kubernetics). It need not be a full fledged
detailed document, but should have the basic contours.


On Tue, Jul 25, 2017 at 11:54 AM, Karthik Ramasamy 
wrote:

> Bill -
>
> The main driving factors for API server are the following -
>
> - How can heron jobs be managed using purely API instead of using CLI
> without any dependency?
>
> - A single place for maintaining config including keys (which you don’t
> want to expose to every client)
>
> - Reduce the installation steps needed
>
> - Provide authentication support
>
> Rest of my responses are inlined as k>
>
> > On Jul 25, 2017, at 10:15 AM, Bill Graham  wrote:
> >
> > It's not entirely accurate that Heron's deployment mode is only library
> > mode. The Heron scheduler could be implemented to either manage resource
> > scheduling from the client (i.e., Aurora Scheduler) or to run the
> > scheduling logic on the scheduler framework (i.e., Yarn).
> ISchedulerClient
> > has both LIbrarySchedulerClient and HttpServiceSchedulerClient for these
> > two use cases. These are the modes for scheduling single topologies
> > components though, and not about managing a centralized scheduling
> service
> > for multi-tenant usage which this proposal is about. Basically, we
> already
> > have Library and Service modes as terminology in the existing codebase,
> so
> > we shouldn't overload the concepts with a new definition of service mode.
>
> k>I agree I might be overloading the terminology here. Whether the
> scheduler is run
> in library mode vs in the schedule framework is independent of the API
> server. API
> server is not a scheduling service - it just a REST end point server that
> translates
> REST API into actions. Perhaps a change of terminology might make it
> easier to understand.
> Any suggestions?
>
> > If config distribution is the main issue, have we explored adding support
> > for fetching configs from a repository, just as we upload and fetch the
> > binary?
>
> k>We did explore this aspect of having a config in a central place.
> However, there are issues with this approach
>
> - Heron cli have to download every time it has to 
> submit/kill/activate/deactivate
> the topologies. Alternatively,
> the config can be cached but it require invalidation and refresh
> periodically at the client side - which could lead
> to issues.
>
> - All the keys and important stuff could be exposed on the client (if you
> are working with cloud environments)
>
> - If we have to manage the jobs programmatically including
> submission/killing/updating/activating/deactivating, it
> introduces a dependency - such as downloading config before submitting
> making it cumbersome for programmers.
>
> >
> > One concern about adding a scheduling service, is that it creates yet
> > another service to be maintained, and it increases the matrix of modes of
> > deployment available which adds complexity. For example today Aurora
> > topologies can be submitted in local mode only, but they can be updated
> in
> > local or service mode. YARN does both submit and update in service mode
> > today. With this additional service, we would need to support those
> modes,
> > plus those modes when run behind yet another service. The combination of
> > modes gets complex because we now anywhere from 0..2 potential layers of
> > services to go through.
>
> k>As pointed out above, this is not a scheduling service - it is just a
> rest end point. The API
> service will be deployed as yet another job similar to heron-ui and
> heron-tracker. This service
> will be stateless and hence it will be restarted by the scheduler if it
> dies - which means it is fault
> tolerant. We can run multiple instance of the service as well for
> scalability.
>
> Furthermore, the API server will preserve those deployment modes for
> Aurora and YARN - independent
> of whether you deploy using API server or directly from the dev machine
> (like we have now).
>
>
> > This approach also requires the design of a delegated auth mechanism. For
> > example if the deploy service is running as a shared account, how will it
> > delegate auth on behalf of the user who is deploying the topology? If we
> go
> > down this path, we'd need to design for this.
>
> k>As I mentioned earlier, one of the motivations for API server is to
> implement some kind of authentication
> - Kerberos/TLS/LDAP. However, the first phase will be providing the
> functionality followed by the 2nd phase
> which includes an authentication mechanism.
>
> > I also share Maosong's concern of merging the tracker into the api
> service.
> > The design of the system will be more clear and easy to maintain/manage
> if
> > each system could live independently. If the goal is to make it easier
> for
> > administrators to manage all at once, I'd suggest we handle 

Re: testing

[Sanjeev Kulkarni]

I got this one

On Tue, Jul 18, 2017 at 4:15 PM Bill Graham  wrote:

> Thanks Taylor, it seems you're the only one. :) Other project developers
> who have subscribe (including me) are reporting that they're not getting
> emails. I've opened https://issues.apache.org/jira/browse/INFRA-14636 for
> investigation. If anyone besides Taylor is getting these please chime in.
>
> On Thu, Jul 13, 2017 at 10:08 AM, P. Taylor Goetz 
> wrote:
>
>> Loud and clear. ;)
>>
>> > On Jul 13, 2017, at 1:03 PM, Bill Graham  wrote:
>> >
>> > 1, 2, 3
>>
>>
>