Re: Slides of exactly once/effectively once processing in Heron

[Karthik Ramasamy]

Thanks Sree.

On Mon, Mar 19, 2018 at 7:22 AM, Sree V 
wrote:

> Great work, Karthik & Sanjeev.
>
> Thanks
> Sree
>
> On Saturday, March 10, 2018, 11:58:21 AM PST, Karthik Ramasamy <
> kart...@streaml.io> wrote:
>
>  Thanks Dave - more is on the way!
>
> cheers
> /karthik
>
> On Sat, Mar 10, 2018 at 11:53 AM, Dave Fisher 
> wrote:
>
> > Hi Karthick,
> >
> > Nice. I like slide 37.
> >
> > Regards,
> > Dave
> >
> > > On Mar 9, 2018, at 8:22 AM, Karthik Ramasamy 
> wrote:
> > >
> > > Sanjeev and I presented at Strata Data Conference on how exactly once
> is
> > > implemented in Heron San Jose. Slides are available at
> > >
> > > https://www.slideshare.net/KarthikRamasamy3/exactly-once-
> > effectively-once-in-apache-heron
> > >
> > > cheers
> > > /karthik
> >
> >
>
>

Re: Heron OSS Sync

[Fu Maosong]

Hi Dave -

I added Ning Wang from Twitter with Admin permission and he has started
working on it.

2018-03-16 10:31 GMT-07:00 Dave Fisher :

> Hi Maosong -
>
> Infra is waiting.
>
> Daniel Takamori commented on INFRA-16117:
> -
>
> [~sreevaddi] [~wave]  Ping on adding someone from infra to the Github org
> so we can move repos into the apache organization.
>
>
> Let us know if you need any help or advice.
>
> Regards,
> Dave
>
>
> On Mar 13, 2018, at 3:20 PM, Ning Wang  wrote:
>
> And here are the brief notes for today's meeting:
>
>
> * - Maosong needs to contact with Apache admin about github to
> apache/github.- After code migration is done, we are going to follow apache
> release process- Sree is following up on the site and documentations- A few
> bug fixes on k8s support were merged and will be included in the new
> release.- Dhalion testing is ongoing in Microsoft- SQL API is ongoing,
> https://docs.google.com/document/d/1PxLCyR_H-
> mOgPjyFj3DhWXryKW21CH2zFWwzTnqjfEA/edit
>  mOgPjyFj3DhWXryKW21CH2zFWwzTnqjfEA/edit>-
> More Dhalion metrics are being added (Twitter internal feedback/requests)-
> Data rate limiting is on going. Will be hooked up with runtime config. In
> future some automation might be possible. It could also be related to auto
> scaling. To document after it is stabilized.- First meetup date/time. Late
> April, Sree is talking to some meeting groups such as
> http://www.sfbayacm.org/ - Slack/email
> integration is ongoing (almost ready)*
> Meeting notes are tracked here:
> https://docs.google.com/document/d/1cTIBq3jOVRTSR0Zd5OKK20OqwT2l9
> 0xXiY_HssVo8mE/edit?ts=5aa84932#
>
>
> On Tue, Mar 13, 2018 at 11:00 AM, Karthik Ramasamy 
> wrote:
>
> Will be happening today at 2.00 pm PST. Please use the following hangout
> link
>
> https://hangouts.google.com/hangouts/_/streaml.io/oss-
> heron-sync?authuser=0
>
> See you all then.
>
> cheers
> /karthik
>
>
>


-- 
With my best Regards
--
Fu Maosong
Twitter Inc.
Mobile: +001-415-244-7520

Re: Fault tolerance

[Ning Wang]

Two features I can think of:

Dhalion
https://blog.acolyer.org/2017/06/30/dhalion-self-regulating-stream-processing-in-heron/

Backpressure
Streammanager is responsible for all data transactions and it monitors the
data consumption. If a process runs slow and incoming data buffer is full,
it triggers backpressure to stop new data until the data in pipeline is
consumed.



On Tue, Mar 20, 2018 at 10:26 AM, Thuvarakan Tharmarajasingam <
thuva4@gmail.com> wrote:

> Hello,
> I am currently working on a distributed fault tolerance system for big data
> toolkit. Could you please give me an overview of how Heron achieved the
> fault tolerance?
>

Fault tolerance

[Thuvarakan Tharmarajasingam]

Hello,
I am currently working on a distributed fault tolerance system for big data
toolkit. Could you please give me an overview of how Heron achieved the
fault tolerance?

Re: Heron OSS Sync

[Dave Fisher]

Hi Ning Wang,

You can reach out to Infra on their HipChat at 
https://www.hipchat.com/gh6DJhKrN 

Regards,
Dave


> On Mar 20, 2018, at 2:55 PM, Fu Maosong  wrote:
> 
> Hi Dave -
> 
> I added Ning Wang from Twitter with Admin permission and he has started
> working on it.
> 
> 2018-03-16 10:31 GMT-07:00 Dave Fisher :
> 
>> Hi Maosong -
>> 
>> Infra is waiting.
>> 
>> Daniel Takamori commented on INFRA-16117:
>> -
>> 
>> [~sreevaddi] [~wave]  Ping on adding someone from infra to the Github org
>> so we can move repos into the apache organization.
>> 
>> 
>> Let us know if you need any help or advice.
>> 
>> Regards,
>> Dave
>> 
>> 
>> On Mar 13, 2018, at 3:20 PM, Ning Wang  wrote:
>> 
>> And here are the brief notes for today's meeting:
>> 
>> 
>> * - Maosong needs to contact with Apache admin about github to
>> apache/github.- After code migration is done, we are going to follow apache
>> release process- Sree is following up on the site and documentations- A few
>> bug fixes on k8s support were merged and will be included in the new
>> release.- Dhalion testing is ongoing in Microsoft- SQL API is ongoing,
>> https://docs.google.com/document/d/1PxLCyR_H-
>> mOgPjyFj3DhWXryKW21CH2zFWwzTnqjfEA/edit
>> > mOgPjyFj3DhWXryKW21CH2zFWwzTnqjfEA/edit>-
>> More Dhalion metrics are being added (Twitter internal feedback/requests)-
>> Data rate limiting is on going. Will be hooked up with runtime config. In
>> future some automation might be possible. It could also be related to auto
>> scaling. To document after it is stabilized.- First meetup date/time. Late
>> April, Sree is talking to some meeting groups such as
>> http://www.sfbayacm.org/ - Slack/email
>> integration is ongoing (almost ready)*
>> Meeting notes are tracked here:
>> https://docs.google.com/document/d/1cTIBq3jOVRTSR0Zd5OKK20OqwT2l9
>> 0xXiY_HssVo8mE/edit?ts=5aa84932#
>> 
>> 
>> On Tue, Mar 13, 2018 at 11:00 AM, Karthik Ramasamy 
>> wrote:
>> 
>> Will be happening today at 2.00 pm PST. Please use the following hangout
>> link
>> 
>> https://hangouts.google.com/hangouts/_/streaml.io/oss-
>> heron-sync?authuser=0
>> 
>> See you all then.
>> 
>> cheers
>> /karthik
>> 
>> 
>> 
> 
> 
> --
> With my best Regards
> --
> Fu Maosong
> Twitter Inc.
> Mobile: +001-415-244-7520



signature.asc
Description: Message signed with OpenPGP