[GitHub] huijunwu commented on issue #2884: move downloader registry map to yaml

[GitBox]

huijunwu commented on issue #2884: move downloader registry map to yaml
URL: https://github.com/apache/incubator-heron/pull/2884#issuecomment-387594464
 
 
   @jerrypeng updated to be compatible with `downloader  
`. could you review it again


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

Re: Heron OSS Sync

[Karthik Ramasamy]

+1

On Tue, May 8, 2018 at 4:50 PM, Ning Wang  wrote:

> Agreed. :)
>
> On Tue, May 8, 2018 at 4:49 PM, P. Taylor Goetz  wrote:
>
> > +1 to everything Dave said.
> >
> > Your first Apache release will likely be the hardest. At times it may
> even
> > seem like hazing. You may get a lot of feedback that seems like
> nitpicking.
> >
> > That is not the case. The goal is to make sure to make sure Apache Heron
> > knows how to make compliant releases and will continue to do so after
> > graduation.
> >
> > Don’t get discouraged. Lean on your mentors/advisors.
> >
> > -Taylor
> >
> > > On May 8, 2018, at 7:21 PM, Dave Fisher  wrote:
> > >
> > > Thanks for the quick update!
> > >
> > >> On May 8, 2018, at 4:08 PM, Ning Wang  wrote:
> > >>
> > >> And here is a brief notes:
> > >>
> > >>
> > >>
> > >> * - Our current focus is to have our first apache release by the end
> of
> > >> this week (we should be pretty much ready for it)
> > >
> > > If you mean have the first release candidate ready for voting by the
> end
> > of the week this is achievable. What’s not achievable is completing the
> > VOTE process and making the first release. You should consider this an
> > indeterminate process until the community gets things correct in the
> Apache
> > Way.
> > >
> > > (1) The project will need to take at least 72 hours to review the
> > candidate.
> > > (2) Then the IPMC needs to VOTE on general@. That takes another
> minimum
> > of 72 hours. Often more.
> > >
> > > To pass requires 3 +1 IPMC votes. We review the source and binaries to
> > make sure that all files have license text and that the NOTICE and
> LICENSE
> > is correct for the source and binary.
> > >
> > > Regards,
> > > Dave
> > >
> > >> .- Heron webpage needs
> > >> some update and reorg- Oracle can’t host the meet up next week.
> > >> Rescheduling.- We need more blogs. Karthik will send out some ideas.-
> > >> Please review Saikat’s machine learning support proposal.
> > >> https://docs.google.com/document/d/1LrO7XRcMxJoMM83wjRd-
> > Ov74VAaomA_mXOAhCStgGng/edit
> > >>  > Ov74VAaomA_mXOAhCStgGng/edit>-
> > >> Stateful processing is in progress. Found two issues and working on
> them
> > >> (state data removal and hadoop config)- Investigating stuck stmgr
> issue-
> > >> Working on the model to predict BP when traffic increases- Got a
> > streamlet
> > >> bug report from user about missing acks.- New ubunton 18.04 has
> python 3
> > >> only. We need to migrate- A hands-on session will be scheduled by the
> > end
> > >> of June (sree)- Security concern in downloader/extractor java code.
> Sree
> > >> and karthik to sync up.- CDCI *
> > >> The google doc is here:
> > >> https://docs.google.com/document/d/1cTIBq3jOVRTSR0Zd5OKK20OqwT2l9
> > 0xXiY_HssVo8mE/edit?ts=5aa84932#heading=h.nq0bjo3oqwfy.
> > >> Please feel free to reply/comment if anything is mission.
> > >>
> > >>
> > >>
> > >>
> > >>> On Tue, May 8, 2018 at 1:15 PM, Ning Wang 
> > wrote:
> > >>>
> > >>> Hi,
> > >>>
> > >>> The heron OSS sync meeting will be happening today at 2.00 pm PST.
> > Please
> > >>> use the following hangout link:
> > >>> https://hangouts.google.com/hangouts/_/streaml.io/oss-heron-
> > >>> sync?authuser=0
> > >>>
> > >>>
> > >>> See you all then.
> > >>>
> > >>>
> > >
> >
>

Re: Heron OSS Sync

[Ning Wang]

Agreed. :)

On Tue, May 8, 2018 at 4:49 PM, P. Taylor Goetz  wrote:

> +1 to everything Dave said.
>
> Your first Apache release will likely be the hardest. At times it may even
> seem like hazing. You may get a lot of feedback that seems like nitpicking.
>
> That is not the case. The goal is to make sure to make sure Apache Heron
> knows how to make compliant releases and will continue to do so after
> graduation.
>
> Don’t get discouraged. Lean on your mentors/advisors.
>
> -Taylor
>
> > On May 8, 2018, at 7:21 PM, Dave Fisher  wrote:
> >
> > Thanks for the quick update!
> >
> >> On May 8, 2018, at 4:08 PM, Ning Wang  wrote:
> >>
> >> And here is a brief notes:
> >>
> >>
> >>
> >> * - Our current focus is to have our first apache release by the end of
> >> this week (we should be pretty much ready for it)
> >
> > If you mean have the first release candidate ready for voting by the end
> of the week this is achievable. What’s not achievable is completing the
> VOTE process and making the first release. You should consider this an
> indeterminate process until the community gets things correct in the Apache
> Way.
> >
> > (1) The project will need to take at least 72 hours to review the
> candidate.
> > (2) Then the IPMC needs to VOTE on general@. That takes another minimum
> of 72 hours. Often more.
> >
> > To pass requires 3 +1 IPMC votes. We review the source and binaries to
> make sure that all files have license text and that the NOTICE and LICENSE
> is correct for the source and binary.
> >
> > Regards,
> > Dave
> >
> >> .- Heron webpage needs
> >> some update and reorg- Oracle can’t host the meet up next week.
> >> Rescheduling.- We need more blogs. Karthik will send out some ideas.-
> >> Please review Saikat’s machine learning support proposal.
> >> https://docs.google.com/document/d/1LrO7XRcMxJoMM83wjRd-
> Ov74VAaomA_mXOAhCStgGng/edit
> >>  Ov74VAaomA_mXOAhCStgGng/edit>-
> >> Stateful processing is in progress. Found two issues and working on them
> >> (state data removal and hadoop config)- Investigating stuck stmgr issue-
> >> Working on the model to predict BP when traffic increases- Got a
> streamlet
> >> bug report from user about missing acks.- New ubunton 18.04 has python 3
> >> only. We need to migrate- A hands-on session will be scheduled by the
> end
> >> of June (sree)- Security concern in downloader/extractor java code. Sree
> >> and karthik to sync up.- CDCI *
> >> The google doc is here:
> >> https://docs.google.com/document/d/1cTIBq3jOVRTSR0Zd5OKK20OqwT2l9
> 0xXiY_HssVo8mE/edit?ts=5aa84932#heading=h.nq0bjo3oqwfy.
> >> Please feel free to reply/comment if anything is mission.
> >>
> >>
> >>
> >>
> >>> On Tue, May 8, 2018 at 1:15 PM, Ning Wang 
> wrote:
> >>>
> >>> Hi,
> >>>
> >>> The heron OSS sync meeting will be happening today at 2.00 pm PST.
> Please
> >>> use the following hangout link:
> >>> https://hangouts.google.com/hangouts/_/streaml.io/oss-heron-
> >>> sync?authuser=0
> >>>
> >>>
> >>> See you all then.
> >>>
> >>>
> >
>

Re: Heron OSS Sync

[P. Taylor Goetz]

+1 to everything Dave said.

Your first Apache release will likely be the hardest. At times it may even seem 
like hazing. You may get a lot of feedback that seems like nitpicking.

That is not the case. The goal is to make sure to make sure Apache Heron knows 
how to make compliant releases and will continue to do so after graduation.

Don’t get discouraged. Lean on your mentors/advisors.

-Taylor

> On May 8, 2018, at 7:21 PM, Dave Fisher  wrote:
> 
> Thanks for the quick update!
> 
>> On May 8, 2018, at 4:08 PM, Ning Wang  wrote:
>> 
>> And here is a brief notes:
>> 
>> 
>> 
>> * - Our current focus is to have our first apache release by the end of
>> this week (we should be pretty much ready for it)
> 
> If you mean have the first release candidate ready for voting by the end of 
> the week this is achievable. What’s not achievable is completing the VOTE 
> process and making the first release. You should consider this an 
> indeterminate process until the community gets things correct in the Apache 
> Way.
> 
> (1) The project will need to take at least 72 hours to review the candidate.
> (2) Then the IPMC needs to VOTE on general@. That takes another minimum of 72 
> hours. Often more.
> 
> To pass requires 3 +1 IPMC votes. We review the source and binaries to make 
> sure that all files have license text and that the NOTICE and LICENSE is 
> correct for the source and binary.
> 
> Regards,
> Dave
> 
>> .- Heron webpage needs
>> some update and reorg- Oracle can’t host the meet up next week.
>> Rescheduling.- We need more blogs. Karthik will send out some ideas.-
>> Please review Saikat’s machine learning support proposal.
>> https://docs.google.com/document/d/1LrO7XRcMxJoMM83wjRd-Ov74VAaomA_mXOAhCStgGng/edit
>> -
>> Stateful processing is in progress. Found two issues and working on them
>> (state data removal and hadoop config)- Investigating stuck stmgr issue-
>> Working on the model to predict BP when traffic increases- Got a streamlet
>> bug report from user about missing acks.- New ubunton 18.04 has python 3
>> only. We need to migrate- A hands-on session will be scheduled by the end
>> of June (sree)- Security concern in downloader/extractor java code. Sree
>> and karthik to sync up.- CDCI *
>> The google doc is here:
>> https://docs.google.com/document/d/1cTIBq3jOVRTSR0Zd5OKK20OqwT2l90xXiY_HssVo8mE/edit?ts=5aa84932#heading=h.nq0bjo3oqwfy.
>> Please feel free to reply/comment if anything is mission.
>> 
>> 
>> 
>> 
>>> On Tue, May 8, 2018 at 1:15 PM, Ning Wang  wrote:
>>> 
>>> Hi,
>>> 
>>> The heron OSS sync meeting will be happening today at 2.00 pm PST. Please
>>> use the following hangout link:
>>> https://hangouts.google.com/hangouts/_/streaml.io/oss-heron-
>>> sync?authuser=0
>>> 
>>> 
>>> See you all then.
>>> 
>>> 
> 

Re: Heron OSS Sync

[Ning Wang]

Got it. Thanks!

Yeah. Cadidate is our goal.


On Tue, May 8, 2018 at 4:21 PM, Dave Fisher  wrote:

> Thanks for the quick update!
>
> > On May 8, 2018, at 4:08 PM, Ning Wang  wrote:
> >
> > And here is a brief notes:
> >
> >
> >
> > * - Our current focus is to have our first apache release by the end of
> > this week (we should be pretty much ready for it)
>
> If you mean have the first release candidate ready for voting by the end
> of the week this is achievable. What’s not achievable is completing the
> VOTE process and making the first release. You should consider this an
> indeterminate process until the community gets things correct in the Apache
> Way.
>
> (1) The project will need to take at least 72 hours to review the
> candidate.
> (2) Then the IPMC needs to VOTE on general@. That takes another minimum
> of 72 hours. Often more.
>
> To pass requires 3 +1 IPMC votes. We review the source and binaries to
> make sure that all files have license text and that the NOTICE and LICENSE
> is correct for the source and binary.
>
> Regards,
> Dave
>
> > .- Heron webpage needs
> > some update and reorg- Oracle can’t host the meet up next week.
> > Rescheduling.- We need more blogs. Karthik will send out some ideas.-
> > Please review Saikat’s machine learning support proposal.
> > https://docs.google.com/document/d/1LrO7XRcMxJoMM83wjRd-
> Ov74VAaomA_mXOAhCStgGng/edit
> >  Ov74VAaomA_mXOAhCStgGng/edit>-
> > Stateful processing is in progress. Found two issues and working on them
> > (state data removal and hadoop config)- Investigating stuck stmgr issue-
> > Working on the model to predict BP when traffic increases- Got a
> streamlet
> > bug report from user about missing acks.- New ubunton 18.04 has python 3
> > only. We need to migrate- A hands-on session will be scheduled by the end
> > of June (sree)- Security concern in downloader/extractor java code. Sree
> > and karthik to sync up.- CDCI *
> > The google doc is here:
> > https://docs.google.com/document/d/1cTIBq3jOVRTSR0Zd5OKK20OqwT2l9
> 0xXiY_HssVo8mE/edit?ts=5aa84932#heading=h.nq0bjo3oqwfy.
> > Please feel free to reply/comment if anything is mission.
> >
> >
> >
> >
> > On Tue, May 8, 2018 at 1:15 PM, Ning Wang  wrote:
> >
> >> Hi,
> >>
> >> The heron OSS sync meeting will be happening today at 2.00 pm PST.
> Please
> >> use the following hangout link:
> >> https://hangouts.google.com/hangouts/_/streaml.io/oss-heron-
> >> sync?authuser=0
> >>
> >>
> >> See you all then.
> >>
> >>
>
>

Re: Heron OSS Sync

[Karthik Ramasamy]

Thanks Dave for the feedback. If we can get the release candidate out for
review - it will be a good first step.

It will give mentors and others to review the source, NOTICE, LICENSE and
any other items. Based on the
feedback, we can iterate on a few release candidates.

cheers
/karthik

On Tue, May 8, 2018 at 4:21 PM, Dave Fisher  wrote:

> Thanks for the quick update!
>
> > On May 8, 2018, at 4:08 PM, Ning Wang  wrote:
> >
> > And here is a brief notes:
> >
> >
> >
> > * - Our current focus is to have our first apache release by the end of
> > this week (we should be pretty much ready for it)
>
> If you mean have the first release candidate ready for voting by the end
> of the week this is achievable. What’s not achievable is completing the
> VOTE process and making the first release. You should consider this an
> indeterminate process until the community gets things correct in the Apache
> Way.
>
> (1) The project will need to take at least 72 hours to review the
> candidate.
> (2) Then the IPMC needs to VOTE on general@. That takes another minimum
> of 72 hours. Often more.
>
> To pass requires 3 +1 IPMC votes. We review the source and binaries to
> make sure that all files have license text and that the NOTICE and LICENSE
> is correct for the source and binary.
>
> Regards,
> Dave
>
> > .- Heron webpage needs
> > some update and reorg- Oracle can’t host the meet up next week.
> > Rescheduling.- We need more blogs. Karthik will send out some ideas.-
> > Please review Saikat’s machine learning support proposal.
> > https://docs.google.com/document/d/1LrO7XRcMxJoMM83wjRd-
> Ov74VAaomA_mXOAhCStgGng/edit
> >  Ov74VAaomA_mXOAhCStgGng/edit>-
> > Stateful processing is in progress. Found two issues and working on them
> > (state data removal and hadoop config)- Investigating stuck stmgr issue-
> > Working on the model to predict BP when traffic increases- Got a
> streamlet
> > bug report from user about missing acks.- New ubunton 18.04 has python 3
> > only. We need to migrate- A hands-on session will be scheduled by the end
> > of June (sree)- Security concern in downloader/extractor java code. Sree
> > and karthik to sync up.- CDCI *
> > The google doc is here:
> > https://docs.google.com/document/d/1cTIBq3jOVRTSR0Zd5OKK20OqwT2l9
> 0xXiY_HssVo8mE/edit?ts=5aa84932#heading=h.nq0bjo3oqwfy.
> > Please feel free to reply/comment if anything is mission.
> >
> >
> >
> >
> > On Tue, May 8, 2018 at 1:15 PM, Ning Wang  wrote:
> >
> >> Hi,
> >>
> >> The heron OSS sync meeting will be happening today at 2.00 pm PST.
> Please
> >> use the following hangout link:
> >> https://hangouts.google.com/hangouts/_/streaml.io/oss-heron-
> >> sync?authuser=0
> >>
> >>
> >> See you all then.
> >>
> >>
>
>

Re: Heron OSS Sync

[Dave Fisher]

Thanks for the quick update!

> On May 8, 2018, at 4:08 PM, Ning Wang  wrote:
> 
> And here is a brief notes:
> 
> 
> 
> * - Our current focus is to have our first apache release by the end of
> this week (we should be pretty much ready for it)

If you mean have the first release candidate ready for voting by the end of the 
week this is achievable. What’s not achievable is completing the VOTE process 
and making the first release. You should consider this an indeterminate process 
until the community gets things correct in the Apache Way.

(1) The project will need to take at least 72 hours to review the candidate.
(2) Then the IPMC needs to VOTE on general@. That takes another minimum of 72 
hours. Often more.

To pass requires 3 +1 IPMC votes. We review the source and binaries to make 
sure that all files have license text and that the NOTICE and LICENSE is 
correct for the source and binary.

Regards,
Dave

> .- Heron webpage needs
> some update and reorg- Oracle can’t host the meet up next week.
> Rescheduling.- We need more blogs. Karthik will send out some ideas.-
> Please review Saikat’s machine learning support proposal.
> https://docs.google.com/document/d/1LrO7XRcMxJoMM83wjRd-Ov74VAaomA_mXOAhCStgGng/edit
> -
> Stateful processing is in progress. Found two issues and working on them
> (state data removal and hadoop config)- Investigating stuck stmgr issue-
> Working on the model to predict BP when traffic increases- Got a streamlet
> bug report from user about missing acks.- New ubunton 18.04 has python 3
> only. We need to migrate- A hands-on session will be scheduled by the end
> of June (sree)- Security concern in downloader/extractor java code. Sree
> and karthik to sync up.- CDCI *
> The google doc is here:
> https://docs.google.com/document/d/1cTIBq3jOVRTSR0Zd5OKK20OqwT2l90xXiY_HssVo8mE/edit?ts=5aa84932#heading=h.nq0bjo3oqwfy.
> Please feel free to reply/comment if anything is mission.
> 
> 
> 
> 
> On Tue, May 8, 2018 at 1:15 PM, Ning Wang  wrote:
> 
>> Hi,
>> 
>> The heron OSS sync meeting will be happening today at 2.00 pm PST. Please
>> use the following hangout link:
>> https://hangouts.google.com/hangouts/_/streaml.io/oss-heron-
>> sync?authuser=0
>> 
>> 
>> See you all then.
>> 
>> 



signature.asc
Description: Message signed with OpenPGP

[GitHub] tomncooper commented on issue #2875: Duration reporting inconsistency

[GitBox]

tomncooper commented on issue #2875: Duration reporting inconsistency
URL: 
https://github.com/apache/incubator-heron/issues/2875#issuecomment-387559852
 
 
   Duplicate of #2874 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nlu90 commented on a change in pull request #2852: first of the connectors - twitter spout

[GitBox]

nlu90 commented on a change in pull request #2852: first of the connectors - 
twitter spout
URL: https://github.com/apache/incubator-heron/pull/2852#discussion_r186883119
 
 

 ##
 File path: third_party/cereal/BUILD
 ##
 @@ -0,0 +1,126 @@
+licenses(["notice"])
 
 Review comment:
   We already have a cereal lib in the third_party dir 
[here](https://github.com/apache/incubator-heron/tree/master/third_party/cereal).
   
   So please either remove the files here or send out another separate PR for 
changing this lib


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nlu90 commented on a change in pull request #2852: first of the connectors - twitter spout

[GitBox]

nlu90 commented on a change in pull request #2852: first of the connectors - 
twitter spout
URL: https://github.com/apache/incubator-heron/pull/2852#discussion_r186883787
 
 

 ##
 File path: 
connectors/heron-twitter/src/java/com/twitter/heron/twitter/spout/Twitter.java
 ##
 @@ -0,0 +1,122 @@
+package com.streamlio.connectors.twitter;
 
 Review comment:
   the package should be `org.apache.heron.connectors.twitter` now


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nlu90 closed issue #2877: Compilation error in a container

[GitBox]

nlu90 closed issue #2877: Compilation error in a container
URL: https://github.com/apache/incubator-heron/issues/2877
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nlu90 commented on issue #2877: Compilation error in a container

[GitBox]

nlu90 commented on issue #2877: Compilation error in a container
URL: 
https://github.com/apache/incubator-heron/issues/2877#issuecomment-387558212
 
 
   @comes5 In short, Yes. You'll need to build the whole project to get the 
core jars for testing. But since bazel is smart enough to do the incremental 
building, it will only build targets affected by your changes after the initial 
full building.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

Heron OSS Sync

[Ning Wang]

Hi,

The heron OSS sync meeting will be happening today at 2.00 pm PST. Please
use the following hangout link:
https://hangouts.google.com/hangouts/_/streaml.io/oss-heron-sync?authuser=0


See you all then.

[GitHub] ajorgensen opened a new issue #2897: Potential double free crash in BaseConnection destructor

[GitBox]

ajorgensen opened a new issue #2897: Potential double free crash in 
BaseConnection destructor
URL: https://github.com/apache/incubator-heron/issues/2897
 
 
   On version `0.17.8` it appears that there may be a double free error that 
causes a stream manager to crash sometimes in the destructor for the 
BaseConnection
   
   ```
   [Thread debugging using libthread_db enabled]
   Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
   Core was generated by `./heron-core/bin/heron-stmgr 
--topology_name=insights-speed-android --topology_'.
   Program terminated with signal SIGABRT, Aborted.
   #0  0x7ff4f8033c37 in __GI_raise (sig=sig@entry=6) at 
../nptl/sysdeps/unix/sysv/linux/raise.c:56
   56../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
   (gdb) bt
   #0  0x7ff4f8033c37 in __GI_raise (sig=sig@entry=6) at 
../nptl/sysdeps/unix/sysv/linux/raise.c:56
   #1  0x7ff4f8037028 in __GI_abort () at abort.c:89
   #2  0x00513481 in tcmalloc::Log (mode=mode@entry=tcmalloc::kCrash, 
filename=filename@entry=0x6109c2 "src/tcmalloc.cc", line=line@entry=278, a=..., 
b=..., c=..., d=...) at src/internal_logging.cc:120
   #3  0x0050ef35 in (anonymous namespace)::InvalidFree 
(ptr=ptr@entry=0x6f72646e412e6e6f) at src/tcmalloc.cc:278
   #4  0x0040a3e2 in free_null_or_invalid (invalid_free_fn=0x50eee0 
<(anonymous namespace)::InvalidFree(void*)>, ptr=ptr@entry=0x6f72646e412e6e6f) 
at src/tcmalloc.cc:1137
   #5  (anonymous namespace)::do_free_helper (ptr=ptr@entry=0x6f72646e412e6e6f, 
heap_must_be_valid=true, heap=0x21720a0, invalid_free_fn=0x50eee0 <(anonymous 
namespace)::InvalidFree(void*)>) at src/tcmalloc.cc:1181
   #6  0x00609cdc in do_free_helper (invalid_free_fn=0x50eee0 
<(anonymous namespace)::InvalidFree(void*)>, heap_must_be_valid=true, 
heap=0x21720a0, ptr=0x6f72646e412e6e6f) at src/thread_cache.h:381
   #7  do_free_with_callback (invalid_free_fn=0x50eee0 <(anonymous 
namespace)::InvalidFree(void*)>, ptr=0x6f72646e412e6e6f) at src/tcmalloc.cc:1221
   #8  do_free (ptr=0x6f72646e412e6e6f) at src/tcmalloc.cc:1230
   #9  tc_free (ptr=0x6f72646e412e6e6f) at src/tcmalloc.cc:1581
   #10 0x004c1d35 in BaseConnection::disableRateLimit() ()
   #11 0x004c1d71 in BaseConnection::~BaseConnection() ()
   #12 0x004ca4b1 in Connection::~Connection() ()
   #13 0x004cdccc in BaseClient::OnClose(NetworkErrorCode) ()
   #14 0x004c1e7a in BaseConnection::internalClose(NetworkErrorCode) ()
   #15 0x004c2131 in BaseConnection::handleEvent(short) ()
   #16 0x004d643d in bufferevent_run_deferred_callbacks_locked 
(cb=, arg=0x87d6000) at bufferevent.c:161
   #17 0x004deb10 in event_process_active_single_queue 
(base=base@entry=0x28dc000, activeq=0x28883f0, 
max_to_process=max_to_process@entry=2147483647, endtime=endtime@entry=0x0) at 
event.c:1675
   #18 0x004df597 in event_process_active (base=0x28dc000) at 
event.c:1738
   #19 event_base_loop (base=0x28dc000, flags=0) at event.c:1961
   #20 0x0040b07c in main ()```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] skanjila opened a new issue #2896: Design of streaming machine learning algorithms in heron

[GitBox]

skanjila opened a new issue #2896: Design of streaming machine learning 
algorithms in heron
URL: https://github.com/apache/incubator-heron/issues/2896
 
 
   The goal of this effort is to design and document streaming real time 
machine learning algorithms that can fit into a storm topology and eventually 
into a heron streamlet based architecture


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] kramasamy closed pull request #2895: Nwang/update license in other files

[GitBox]

kramasamy closed pull request #2895: Nwang/update license in other files
URL: https://github.com/apache/incubator-heron/pull/2895
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/config/empty.cc b/config/empty.cc
index 3305740970..8c25c1ea4d 100644
--- a/config/empty.cc
+++ b/config/empty.cc
@@ -1,17 +1,20 @@
-/*
- * Copyright 2015 Twitter, Inc.
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
  *
  *   http://www.apache.org/licenses/LICENSE-2.0
  *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
  */
 
 // TODO(kramasamy): Can we get rid of this file?
diff --git 
a/eco-heron-examples/src/java/org/apache/heron/examples/eco/sample.properties 
b/eco-heron-examples/src/java/org/apache/heron/examples/eco/sample.properties
index 65a85c4df6..67da8b194d 100644
--- 
a/eco-heron-examples/src/java/org/apache/heron/examples/eco/sample.properties
+++ 
b/eco-heron-examples/src/java/org/apache/heron/examples/eco/sample.properties
@@ -1,16 +1,19 @@
-# // Copyright 2018 Twitter. All rights reserved.
-# //
-# // Licensed under the Apache License, Version 2.0 (the "License");
-# // you may not use this file except in compliance with the License.
-# // You may obtain a copy of the License at
-# //
-# // http://www.apache.org/licenses/LICENSE-2.0
-# //
-# // Unless required by applicable law or agreed to in writing, software
-# // distributed under the License is distributed on an "AS IS" BASIS,
-# // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# // See the License for the specific language governing permissions and
-# // limitations under the License.
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
 
 ecoPropertyOne=thisValueWasSetFromAPropertiesFile
 
diff --git 
a/eco-storm-examples/src/java/org/apache/heron/examples/eco/sample.properties 
b/eco-storm-examples/src/java/org/apache/heron/examples/eco/sample.properties
index 65a85c4df6..e2cd8ad7b9 100644
--- 
a/eco-storm-examples/src/java/org/apache/heron/examples/eco/sample.properties
+++ 
b/eco-storm-examples/src/java/org/apache/heron/examples/eco/sample.properties
@@ -1,16 +1,19 @@
-# // Copyright 2018 Twitter. All rights reserved.
-# //
-# // Licensed under the Apache License, Version 2.0 (the "License");
-# // you may not use this file except in compliance with the License.
-# // You may obtain a copy of the License at
-# //
-# // http://www.apache.org/licenses/LICENSE-2.0
-# //
-# // Unless required by applicable law or agreed to in writing, software
-# // distributed under the License is distributed on an "AS IS" BASIS,
-# // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# // See the License for the specific language governing permissions and
-# // limitations under the License.
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license 

[GitHub] nwangtw commented on a change in pull request #2884: move downloader registry map to yaml

[GitBox]

nwangtw commented on a change in pull request #2884: move downloader registry 
map to yaml
URL: https://github.com/apache/incubator-heron/pull/2884#discussion_r186623390
 
 

 ##
 File path: 
heron/downloaders/src/java/org/apache/heron/downloader/DownloadRunner.java
 ##
 @@ -18,17 +18,160 @@
 import java.nio.file.Path;
 import java.nio.file.Paths;
 
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.CommandLineParser;
+import org.apache.commons.cli.DefaultParser;
+import org.apache.commons.cli.HelpFormatter;
+import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
+import org.apache.commons.cli.ParseException;
+import org.apache.heron.spi.common.Config;
+import org.apache.heron.spi.common.ConfigLoader;
+
 public final class DownloadRunner {
 
+  public enum DownloaderMode {
+cluster,
+local
+  }
+
+  private enum CliArgs {
+HERON_HOME("heron_home"),
+CONFIG_PATH("config_path"),
+MODE("mode"),
+TOPOLOGY_PACKAGE_URI("topology_package_uri"),
+EXTRACT_DESTINATION("extract_destination");
+
+private String text;
+
+CliArgs(String name) {
+  this.text = name;
+}
+  }
+
+  // Print usage options
+  private static void usage(Options options) {
+HelpFormatter formatter = new HelpFormatter();
+formatter.printHelp(DownloadRunner.class.getSimpleName(), options);
+  }
+
+  // construct command line help options
+  private static Options constructHelpOptions() {
+Options options = new Options();
+Option help = Option.builder("h")
+.desc("List all options and their description")
+.longOpt("help")
+.build();
+
+options.addOption(help);
+return options;
+  }
+
+  // Construct all required command line options
+  private static Options constructCliOptions() {
+Options options = new Options();
+
+Option packageUri = Option.builder("u")
+.desc("Uri indicating from where to download the file")
+.longOpt(CliArgs.TOPOLOGY_PACKAGE_URI.text)
+.hasArgs()
+.argName(CliArgs.TOPOLOGY_PACKAGE_URI.text)
+.required()
+.build();
+
+Option destination = Option.builder("f")
+.desc("Destination to store the downloaded file")
+.longOpt(CliArgs.EXTRACT_DESTINATION.text)
+.hasArgs()
+.argName(CliArgs.EXTRACT_DESTINATION.text)
+.required()
+.build();
+
+Option heronHome = Option.builder("d")
+.desc("Directory where heron is installed")
+.longOpt(CliArgs.HERON_HOME.text)
+.hasArgs()
+.argName("heron home dir")
+.build();
+
+Option configFile = Option.builder("p")
+.desc("Path of the config files")
+.longOpt(CliArgs.CONFIG_PATH.text)
+.hasArgs()
+.argName("config path")
+.build();
+
+// candidates:
+// local: download to client local machine
+// cluster: download into the container in the cloud
+Option mode = Option.builder("m")
+.desc("download mode, cluster or local")
+.longOpt(CliArgs.MODE.text)
+.hasArg()
+.argName("download mode")
+.build();
+
+options.addOption(packageUri);
+options.addOption(destination);
+options.addOption(heronHome);
+options.addOption(configFile);
+options.addOption(mode);
+
+return options;
+  }
+
+
   // takes topology package URI and extracts it to a directory
   public static void main(String[] args) throws Exception {
-if (args.length != 2) {
-  System.err.println("Usage: downloader  
");
+CommandLineParser parser = new DefaultParser();
+Options slaManagerCliOptions = constructCliOptions();
+
+// parse the help options first.
+Options helpOptions = constructHelpOptions();
+CommandLine cmd = parser.parse(helpOptions, args, true);
+if (cmd.hasOption("h")) {
+  usage(slaManagerCliOptions);
   return;
 }
 
-final String uri = args[0];
-final String destination = args[1];
+try {
+  cmd = parser.parse(slaManagerCliOptions, args);
+} catch (ParseException e) {
+  usage(slaManagerCliOptions);
+  throw new RuntimeException("Error parsing command line options: ", e);
+}
+
+DownloaderMode mode = DownloaderMode.cluster;
+if (cmd.hasOption(CliArgs.MODE.text)) {
+  mode = DownloaderMode.valueOf(cmd.getOptionValue(CliArgs.MODE.text, 
null));
+}
+
+Config config;
+switch (mode) {
+  case cluster:
+config = Config.toClusterMode(Config.newBuilder()
+.putAll(ConfigLoader.loadClusterConfig())
+.build());
+break;
+
+  case local:
+if (!cmd.hasOption(CliArgs.HERON_HOME.text) || 
!cmd.hasOption(CliArgs.CONFIG_PATH.text)) {
+  throw new IllegalArgumentException("Missing heron_home or 
config_path argument");
+}
+String heronHome = cmd.getOptionValue(CliArgs.HERON_HOME.text, null);
+String configPath =