Re: [VOTE] Heron Release 0.20.3-incubating Release Candidate 2

[Simon Weng]

+1

On Sun, Apr 19, 2020 at 12:03 PM thinker0  wrote:

> +1
>
> 2020년 4월 20일 (월) 오전 12:23, Josh Fischer 님이 작성:
>
> > I looked at the other files and you are right. This one is named
> > differently.  The other ones have an "incubator" in the beginning of the
> > file name.   I can change this and create a new release candidate.
> >
> > As for the binaries.  I figured we could vote on source first, once that
> > passes then we could have another vote for convenience binaries. It will
> > require an additional call for votes, but it's smaller chunks of work to
> be
> > done at one time.
> >
> > Let me know!
> >
> > On Sat, Apr 18, 2020 at 8:37 PM Ning Wang  wrote:
> >
> > > P lease rename the tar.gz file in the
> > > dist.apache.org to
> apache_heron-0.2.3-incubating-candidate-2-src.tar.gz.
> > >
> > > So we won't have binary this time? I am fine with it.
> > >
> > >
> > > On Sat, Apr 18, 2020 at 9:43 AM Nicholas Nezis <
> nicholas.ne...@gmail.com
> > >
> > > wrote:
> > >
> > > > +1 approve
> > > >
> > > > On Sat, Apr 18, 2020 at 10:42 AM Josh Fischer 
> > > wrote:
> > > >
> > > > > Hello Heron Community,
> > > > >
> > > > > This is a call for a vote to the 2cnd release candidate for Apache
> > > Heron,
> > > > > version 0.20.3-incubating. We request project mentors (binded) as
> > well
> > > as
> > > > > all contributors (unbinded) and users to review and vote on this
> > > > incubator
> > > > > release.
> > > > >
> > > > > *  The tag to be voted upon:
> > > > > 0.20.3-incubating-rc2 (2a4f112fc567897711ee7117689e636534cb6fb4)
> > > > > The full list of changes and release notes are available
> > > > > at:
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-heron/releases/tag/0.20.3-incubating-rc2
> > > > >
> > > > > Files can be found in dist.apache.org
> > > > > Source:
> > > > >
> > > > >
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/incubator/heron/heron-0.20.3-incubating-candidate-2/
> > > > >
> > > > > SHA-512
> > > > > checksums:
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> 84a94d0c68581e230033a04c4539e8dabbd14461f75319c9caaa1bcbec288599fffc98c2928b520eb320deb9184b594d0e23eeb0521452cd014f21dda1776413
> > > > >  heron-0.20.3-incubating-rc2.tar.gz
> > > > >
> > > > >
> > > > > Please download the source package, and follow the compiling guide
> > > > > https://heron.incubator.apache.org/docs/next/compiling-overview/
> > > > > to setup the environment, build and run locally.
> > > > >
> > > > > Note that currently Bazel 3.0 is required to build this version.
> > After
> > > > > Bazel is set up and works correctly, you can use these release
> check
> > > > > scripts
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-heron/blob/master/scripts/release_check/README.md#run-individual-release-checks
> > > > > to
> > > > > verify the build.  The vote will be open for at least 72 hours or
> > until
> > > > the
> > > > > necessary number of votes are reached.
> > > > >
> > > > > Please vote accordingly:
> > > > > [ ] +1 approve
> > > > > [ ] +0 no opinion
> > > > > [ ] -1 disapprove with the reason
> > > > >
> > > > > Thank you,
> > > > >
> > > > > The Apache Heron (Incubating) Team
> > > > >
> > > >
> > >
> >
>
-- 
Sent from Gmail Mobile

Re: error on heron submit local heron-kafka-spout-java-sample.jar

[Simon Weng]

Are you building the project from master branch or a specific version/tag?
I’ll try to reproduce it here.

On Sat, Feb 29, 2020 at 1:26 PM dttlg  wrote:

> I just pull the newest codes and compiled it by myself last week, but I
> found contrib Kafka example can not be run well. The error is below:
>
> heron submit local heron-kafka-spout-java-sample.jar
> org.apache.heron.spouts.kafka.sample.HeronKafkaSpoutSampleTopology
> HeronKafkaSpoutSampleTopology
> [2020-02-22 19:38:04 +0800] [INFO]: Using cluster definition in
> /Users/guoxinghua/.heron/conf/local
> Error: A JNI error has occurred, please check your installation and try
> again
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/heron/spouts/kafka/KafkaConsumerFactory
> at java.lang.Class.getDeclaredMethods0(Native Method)
> at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
> at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
> at java.lang.Class.getMethod0(Class.java:3018)
> at java.lang.Class.getMethod(Class.java:1784)
> at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
> at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.heron.spouts.kafka.KafkaConsumerFactory
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 7 more
> [2020-02-22 19:38:04 +0800] [ERROR]: Failed to create topology definition
> file when executing class
> 'org.apache.heron.spouts.kafka.sample.HeronKafkaSpoutSampleTopology' of
> file 'heron-kafka-spout-java-sample.jar'

-- 
Sent from Gmail Mobile

Re: [DISCUSSION] Questions about docker images support

[Simon Weng]

Do we want to try alpine base image to minimize the image size? Or we want
to stay with one of our existings?

On Tue, Dec 10, 2019 at 6:47 PM Simon Weng  wrote:

> System process, in the form of container, can still work side by side with
> the spout and bolt container, because they are in the same pod, sharing the
> same Linux kernel and networking space, they’re seeing each other on
> localhost. They can even share the same file system as long as they share
> the same volume.
>
> On Tue, Dec 10, 2019 at 6:37 PM Ning Wang  wrote:
>
>> One image sounds good to me.
>>
>> The two ideas are interesting. Heron system processes are designed to work
>> in the same container of worker instances so is might be tricky to
>> separate. That being said, there are definitely things to improve.
>>
>> On Tue, Dec 10, 2019 at 1:07 PM SiMing Weng 
>> wrote:
>>
>> > Hi, guys:
>> >
>> > I think it indeed makes sense to narrow down to a single base image.
>> > Regarding which variant to choose, it boils down to any
>> platform-dependent
>> > component in Heron, I can only think of native StreamManager as far as I
>> > know. I’m sure others on the team knows better than I.
>> >
>> > Also, it may sounds a long shot, but I’d like to throw the idea here.
>> > Currently, we containerize Heron as an all-in-one image, which is huge
>> and
>> > serves as something like a VM image. Ideally, we should containerize
>> each
>> > component and make them slim, then it becomes easier to contribute
>> > to/maintain/release.
>> >
>> > Then even more ambitiously, on k8s cluster, what if we make it possible
>> to
>> > deploy individual Spout/Bolt also as container inside each pod, with a
>> > bunch of side-car container, such as StreamManger, etc, rather than
>> > managing the individual process manually in our Executor. This k8s
>> native
>> > approach would open a whole lot possibility, such as easier log
>> aggregation
>> > by existing open source tools, auto scaling/self-tuning with k8s
>> operator
>> > pattern. I think the process-based stream processing nature of Heron
>> gives
>> > itself a unique advantage to be used in containerized environment,
>> compared
>> > to thread-based peers.
>> >
>> > Cheers,
>> >
>> > SiMing Weng
>> >
>> > > On Dec 10, 2019, at 2:11 PM, Josh Fischer 
>> wrote:
>> > >
>> > > I think we should settle on one "official" version for the community
>> to
>> > > support.  I would like to hear from other users on the  list as to
>> which
>> > > image it is we keep supporting moving forward.  I don't think we have
>> > > enough bandwidth or resources to support 5 different versions of
>> Heron
>> > in
>> > > a container.
>> > >
>> > > On Tue, Dec 10, 2019 at 1:53 AM Ning Wang 
>> wrote:
>> > >
>> > >> Hi, everyone,
>> > >>
>> > >> When I was working on this PR (
>> > >> https://github.com/apache/incubator-heron/pull/3411), Josh raised
>> up a
>> > >> question about the docker images supported by Heron. There are 5
>> images
>> > >> configured: centos7, debian9, ubuntu 14.04, 16.04 and 18.04.
>> > >>
>> > >> Questions to discuss:
>> > >> - It could be a burden to maintain all of them. Do you feel it is
>> > necessary
>> > >> to support all of them? Or is there another one we should have?
>> > >> - For ubuntu, do we really need 3 versions?
>> > >>
>> > >> Regards,
>> > >> --ning
>> > >>
>> >
>> >
>>
> --
> Sent from Gmail Mobile
>
-- 
Sent from Gmail Mobile

Re: [DISCUSSION] Questions about docker images support

[Simon Weng]

System process, in the form of container, can still work side by side with
the spout and bolt container, because they are in the same pod, sharing the
same Linux kernel and networking space, they’re seeing each other on
localhost. They can even share the same file system as long as they share
the same volume.

On Tue, Dec 10, 2019 at 6:37 PM Ning Wang  wrote:

> One image sounds good to me.
>
> The two ideas are interesting. Heron system processes are designed to work
> in the same container of worker instances so is might be tricky to
> separate. That being said, there are definitely things to improve.
>
> On Tue, Dec 10, 2019 at 1:07 PM SiMing Weng  wrote:
>
> > Hi, guys:
> >
> > I think it indeed makes sense to narrow down to a single base image.
> > Regarding which variant to choose, it boils down to any
> platform-dependent
> > component in Heron, I can only think of native StreamManager as far as I
> > know. I’m sure others on the team knows better than I.
> >
> > Also, it may sounds a long shot, but I’d like to throw the idea here.
> > Currently, we containerize Heron as an all-in-one image, which is huge
> and
> > serves as something like a VM image. Ideally, we should containerize each
> > component and make them slim, then it becomes easier to contribute
> > to/maintain/release.
> >
> > Then even more ambitiously, on k8s cluster, what if we make it possible
> to
> > deploy individual Spout/Bolt also as container inside each pod, with a
> > bunch of side-car container, such as StreamManger, etc, rather than
> > managing the individual process manually in our Executor. This k8s native
> > approach would open a whole lot possibility, such as easier log
> aggregation
> > by existing open source tools, auto scaling/self-tuning with k8s operator
> > pattern. I think the process-based stream processing nature of Heron
> gives
> > itself a unique advantage to be used in containerized environment,
> compared
> > to thread-based peers.
> >
> > Cheers,
> >
> > SiMing Weng
> >
> > > On Dec 10, 2019, at 2:11 PM, Josh Fischer  wrote:
> > >
> > > I think we should settle on one "official" version for the community to
> > > support.  I would like to hear from other users on the  list as to
> which
> > > image it is we keep supporting moving forward.  I don't think we have
> > > enough bandwidth or resources to support 5 different versions of  Heron
> > in
> > > a container.
> > >
> > > On Tue, Dec 10, 2019 at 1:53 AM Ning Wang 
> wrote:
> > >
> > >> Hi, everyone,
> > >>
> > >> When I was working on this PR (
> > >> https://github.com/apache/incubator-heron/pull/3411), Josh raised up
> a
> > >> question about the docker images supported by Heron. There are 5
> images
> > >> configured: centos7, debian9, ubuntu 14.04, 16.04 and 18.04.
> > >>
> > >> Questions to discuss:
> > >> - It could be a burden to maintain all of them. Do you feel it is
> > necessary
> > >> to support all of them? Or is there another one we should have?
> > >> - For ubuntu, do we really need 3 versions?
> > >>
> > >> Regards,
> > >> --ning
> > >>
> >
> >
>
-- 
Sent from Gmail Mobile

Re: [Mentors] Heron Docs Update

[Simon Weng]

Nice! Great work!

On Sun, Dec 8, 2019 at 12:11 AM Josh Fischer  wrote:

> Thanks Huijun.  I went ahead and published the site.  You can view it at
> https://heron.incubator.apache.org/.
>
> On Sat, Dec 7, 2019 at 7:12 PM Huijun Wu  wrote:
>
> > Good job. The staging website looks good
> >
> > Josh Fischer  于 2019年12月7日周六 12:57写道：
> >
> > > Hey All,
> > >
> > > I have a few more tweaks to the CI script now that I understand the
> > website
> > > publishing process.  You can view the new site in the staging
> environment
> > > here.  https://heron.staged.apache.org/.
> > >
> > >
> > >
> > > On Fri, Dec 6, 2019 at 2:31 PM Josh Fischer 
> wrote:
> > >
> > > > Hey All,
> > > >
> > > > I filed a Jira issue for this.  You can view the details here ->
> > > > https://issues.apache.org/jira/browse/INFRA-19537
> > > >
> > > > On Fri, Dec 6, 2019 at 9:52 AM Josh Fischer 
> > wrote:
> > > >
> > > >> Hi,
> > > >>
> > > >> The current site is hosted on Github pages and it needs to be
> > replaced.
> > > I
> > > >> recently managed to get a CI job to run, build the docs correctly
> and
> > > >> commit them to the asf-site branch under the folder "content" in the
> > > root
> > > >> directory.
> > > >>
> > > >> Does anyone know what the next steps to have these docs served by
> > Apache
> > > >> infrastructure are?
> > > >>
> > > >> I've looked at both of the below pages and haven't seen any details
> > that
> > > >> stick out to me on what to do next.
> > > >> https://www.apache.org/dev/infra-site.html
> > > >> https://www.apache.org/dev/project-site.html
> > > >>
> > > >> - Josh
> > > >>
> > > >> On Tue, Dec 3, 2019 at 9:39 PM Josh Fischer 
> > > wrote:
> > > >>
> > > >>> Hi,
> > > >>>
> > > >>> The current site is hosted on Github pages and it needs to be
> > replaced.
> > > >>> I recently managed to get a CI job to run, build the docs correctly
> > and
> > > >>> commit them to the asf-site branch under the folder "content" in
> the
> > > root
> > > >>> directory.
> > > >>>
> > > >>> I think we will  need to use is the gitpubsub to fully
> publish/serve
> > > our
> > > >>> new documentation.  Does anyone know what the next steps to have
> > these
> > > docs
> > > >>> served by Apache infrastructure are?
> > > >>>
> > > >>> I've looked at both of the below pages and haven't seen any details
> > > that
> > > >>> stick out to me on what to do next.
> > > >>> https://www.apache.org/dev/infra-site.html
> > > >>> https://www.apache.org/dev/project-site.html
> > > >>>
> > > >>> - Josh
> > > >>>
> > > >>
> > >
> >
>
-- 
Sent from Gmail Mobile

Re: New committer: Siming Weng

[Simon Weng]

Thank you, guys. Looking forward to making Heron better together.

On Fri, Oct 18, 2019 at 5:20 PM FatJ Love  wrote:

> +1
>
> On Fri, Oct 18, 2019 at 2:15 PM Ning Wang  wrote:
>
> > Great!
> >
> > On Fri, Oct 18, 2019 at 1:55 PM Karthik Ramasamy 
> > wrote:
> >
> > > +1
> > >
> > > On Fri, Oct 18, 2019 at 1:23 PM Neng Lu  wrote:
> > >
> > > > Hi all,
> > > >
> > > > The Project Management Committee (PMC) for Apache Heron-Incubator has
> > > > invited Siming Weng to become a committer and we are pleased to
> > announce
> > > > that he has accepted.
> > > >
> > > > Siming Weng has been involved in the community for a while. He is
> > active
> > > in
> > > > slack discussions, code contributions and code reviews.
> > > > His contributions include a wide variety of improvements and addition
> > of
> > > > features into Heron
> > > >
> > > > https://github.com/apache/incubator-heron/commits?author=simingweng
> > > >
> > > > It will be good to bring him as a committer based on his
> contributions
> > > and
> > > > his positive impact on the project.
> > > >
> > > > Being a committer enables easier contribution to the project since
> > there
> > > is
> > > > no need to go via the patch submission process. This should enable
> > better
> > > > productivity. Being a PMC member enables assistance with the
> management
> > > and
> > > > to guide the direction of the project.
> > > >
> > > > Best regards,
> > > > Neng Lu
> > > >
> > >
> >
>
-- 
Sent from Gmail Mobile

Re: [VOTE] Heron Release 0.20.1-incubating Release Candidate 3

[Simon Weng]

+1

Tested the locally built Heron Docker image.

On Sun, Jun 30, 2019 at 1:45 PM Josh Fischer  wrote:

> +1
>
> Rat checks passed
> Tag built successfully
> Topology launched successfully
> Topology killed successfully
>
> On Fri, Jun 28, 2019 at 5:32 PM thinker0  wrote:
>
> > +1
> >
> > 2019년 6월 29일 (토) 07:31, Ning Wang 님이 작성:
> >
> > > *Hello, Heron Community,This is a call for a vote to the 3rd release
> > > candidate for Apache Heron, version v0.20.1-incubating. We kindly
> request
> > > project mentors (binded) as well as all contributors (unbinded) and
> users
> > > to review and vote on this incubator release.*
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > *The tag to be voted upon:0.20.1-incubating-rc3
> > > (40ccec635b45191593ff984a9ae81fd4e0a5dadd)The full list of changes and
> > > release notes are available
> > > at:
> > >
> >
> https://github.com/apache/incubator-heron/releases/tag/0.20.1-incubating-rc3
> > > <
> > >
> >
> https://github.com/apache/incubator-heron/releases/tag/0.20.1-incubating-rc3
> > > >Source
> > > files can be found in dist.apache.org 
> > > site:
> > >
> >
> https://dist.apache.org/repos/dist/dev/incubator/heron/heron-0.20.1-incubating-candidate-3/
> > > <
> > >
> >
> https://dist.apache.org/repos/dist/dev/incubator/heron/heron-0.20.1-incubating-candidate-3/
> > > >Source
> > > SHA-512
> > >
> > >
> >
> checksums:68e2d4db598d73dc69e474dc82940402dc4689cc14dda14ad114cbaec79b1b3a8926625083e901f8234631f6345c6aa9e58fc0fec3176fce3e91e041e724d3b8
> > > incubator-heron-0.20.1-incubating-rc3.tar.gzPlease download the source
> > > package, and follow the compiling
> > > guide(
> > >
> >
> https://apache.github.io/incubator-heron/docs/developers/compiling/compiling/
> > > <
> > >
> >
> https://apache.github.io/incubator-heron/docs/developers/compiling/compiling/
> > > >)to
> > > setup environment, build and run locally. Note that currently Bazel
> > 0.26.0
> > > is required to build this version.After Bazel is set up and works
> > > correctly, you can use these release check scripts
> > > (
> > >
> >
> https://github.com/apache/incubator-heron/blob/master/scripts/release_check/README.md#run-individual-release-checks
> > > <
> > >
> >
> https://github.com/apache/incubator-heron/blob/master/scripts/release_check/README.md#run-individual-release-checks
> > > >)
> > > to verify the build.The vote will be open for at least 72 hours or
> until
> > > the necessary number of votes are reached.Please vote accordingly:[ ]
> +1
> > > approve[ ] +0 no opinion[ ] -1 disapprove with the reasonThanks,The
> > Apache
> > > Heron (Incubating) Team*
> > >
> >
>

Re: 6/18/2019 Bi-Weekly OSS Heron Sync-up

[Simon Weng]

Hi, All:

My update is:

   1. help answering question from the community on Slack
   2. have been working on the first version of a Kafka Bolt


On Mon, Jun 17, 2019 at 2:43 PM Josh Fischer  wrote:

> Hi All,
>
> My updates:
> - I had the start to new site with required apache links and text added
> into the site.. This has been merged in to master for anyone how like to
> help.  I have about the first half of the site link checked (to verify they
> work).  Still working on the other half.  This new site will allow us to
> version our docs.
>
> On Mon, Jun 17, 2019 at 12:54 PM Ning Wang  wrote:
>
> > Hi, all~,
> >
> > It has been two weeks since our last sync! Let's share our works for the
> > last two weeks in this thread.
> >
> > My updates:
> > - Experiment with the component rate limit feature after issues reported
> > internally (still investigating and trying to repro).
> > - Add config log in heron CLI.
> > - Disable verbose in integration tests to speed ci a little.
> >
> > Regards,
> > --ning
> >
>

Re: 6/4/2019 Bi-Weekly OSS Heron Sync-up

[Simon Weng]

Hi, all:


   1. bug fix for Kafka Spout
   2. restructuring the "contrib" folder to make "java_doc" bazel rule
   happy, and adjust bazel build files accordingly
   3. enable generating Maven artifacts (jar, javadoc, src, POM) in our CI
   scripts


On Mon, Jun 3, 2019 at 2:56 PM Dmitry Rusakov 
wrote:

> My updates:
> - Last week I released the new Heron 0.20.0.3 with the following release
> notes:
>
>1. UI improvements (for example, now MonViz button is available in heron
>ui).
>2. Memory leak in TMaster fixed.
>3. KRYO serializer support added into Heron API.
>
>
> On Mon, Jun 3, 2019 at 11:20 AM Josh Fischer  wrote:
>
> > Hi all,
> >
> > My updates
> > - Submitted a new PR for the Apache website
> >
> > On Mon, Jun 3, 2019 at 12:05 PM Ning Wang  wrote:
> >
> > > Hi, all~,
> > >
> > > It has been two weeks since our last sync and there has been quite a
> few
> > > works done! Let's share our works for the last two weeks in this
> thread.
> > >
> > > My updates:
> > > - Continue working on Apache Beam support
> > > - Added release check scripts
> > > - Make RoundRobin packing algo more forgivable when container resource
> is
> > > not set.
> > >
> > >
> > > Regards,
> > > --ning
> > >
> > --
> > Sent from A Mobile Device
> >
>
>
> --
> Dmitry Rusakov,
> Senior Software Engineer,
> Twitter CDM team
>

Re: Bazel version support

[Simon Weng]



On Tue, May 21, 2019 at 6:50 PM Rohan Agarwal  wrote:

> Hi all,
> I will be making some changes to make Heron compatible with the latest
> Bazel version.
>
> Currently compatible Bazel version - 0.23.2
> Latest Bazel version - 0.25.3
>
> The changes should be very small, and only involve changing a few build
> rules. Also, I expect the changes to be backwards compatible with Bazel
> 0.23.2.
>
> Tracking the changes in
> https://github.com/apache/incubator-heron/issues/3265
>
> --
> Rohan
>
-- 
Sent from Gmail Mobile

Re: [Mentors] Fixing Heron Docs

[Simon Weng]

Certainly I don’t know much about documentation tooling, but I heard good
things about Docusarus, and also Gitbook.

On Tue, May 21, 2019 at 8:16 AM Josh Fischer  wrote:

> Ok sounds good.  Thanks.
>
> On Tue, May 21, 2019 at 2:37 AM Ning Wang  wrote:
>
> > Looks good~
> >
> > For the move, I think we don't really need to wait for the release.
> > Especially we may need to have a "master" version and other versions for
> > different releases.
> >
> > How far back might depends on how easy it is to generate web pages. :)
> It
> > might be good enough to start having the pages after moving to Apache
> > package name (0.20.0) so it is only one version back.
> >
> >
> >
> > On Mon, May 20, 2019 at 8:24 PM Josh Fischer 
> wrote:
> >
> > > Good Evening,
> > >
> > > I've been looking around the Pulsar repo to see how they version their
> > > docs.. It looks like they are using
> > >  Docusarus -> https://docusaurus.io/docs/en/versioning.  This looks
> > like a
> > > nice little tool that could handle the versioning for us.
> > >
> > > A question I still have.. Is it more important to move where the static
> > > pages are served from with the added Apache information and then work
> on
> > > versioning the docs? I imagine we would have to manually trigger this
> > page
> > > deployment only once to get it moved. Or should wait to move the site
> > once
> > > a new release is created and add in all the automation before hand?
> > >
> > > Finally, how far back to do we go to try and match the documentation to
> > the
> > > version?
> > >
> > > On Fri, May 17, 2019 at 11:04 PM Ning Wang 
> wrote:
> > >
> > > > got it. that is small enough then :) thx
> > > >
> > > > On Fri, May 17, 2019 at 7:33 PM Josh Fischer 
> > > wrote:
> > > >
> > > > > I just built the static site on my local.  The zip file of the
> > website
> > > is
> > > > > 15.1 MB.
> > > > >
> > > > > On Fri, May 17, 2019 at 6:03 PM Ning Wang 
> > > wrote:
> > > > >
> > > > > > I think it makes sense for each release to have a separated
> > > > doc/website.
> > > > > It
> > > > > > might be a question though if the website should be included in
> the
> > > > > > release. I am also curious how big the zip file of the website
> is?
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Fri, May 17, 2019 at 6:48 AM Josh Fischer <
> j...@joshfischer.io>
> > > > > wrote:
> > > > > >
> > > > > > > My responses are Inlined
> > > > > > >
> > > > > > > On Wed, May 15, 2019 at 1:37 PM Dave Fisher <
> > dave2w...@comcast.net
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > A few thoughts about the website. There are two parts to
> > > consider.
> > > > > > > >
> > > > > > > > (1) The constant part that talks about the Heron project,
> where
> > > > > > resources
> > > > > > > > are and how to contribute to it. This part includes linkages
> to
> > > ASF
> > > > > > > > resources and also the proper download pages etc.
> > > > > > > >
> > > > > > > > This part should be published at any time and I suggest that
> > the
> > > > > > Jenkins
> > > > > > > > build be triggered by changes to the repository. This is why
> > many
> > > > > > > projects
> > > > > > > > have a separate repository for the website.
> > > > > > >
> > > > > > >
> > > > > > > *** I do have a fair amount of this merged into master.  I
> looked
> > > at
> > > > > > > another incubating project and reused a lot of what they have
> > done.
> > > > > The
> > > > > > > static site just hasn’t been updated yet.   If think this
> should
> > go
> > > > out
> > > > > > > before we move where the site is served from I think we can
> make
> > > that
> > > > > > > happen.  Please advise.
> > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > See the following for missing issues:
> > > > > > > > https://whimsy.apache.org/pods/project/heron - go to
> > > > > > > > https://whimsy.apache.org/pods/project to find another
> podling
> > > for
> > > > > > > > comparison.
> > > > > > >
> > > > > > >
> > > > > > > *** see above
> > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > (2) Heron docs ought to be tied together with product
> releases
> > > > along
> > > > > > with
> > > > > > > > master. It makes sense for a releases docs to be part of a
> > > release
> > > > > > build.
> > > > > > > > Some projects even include release docs with a binary
> > convenience
> > > > > > > release.
> > > > > > > >
> > > > > > >
> > > > > > > *** ok would have to think about this approach a little more.
> I
> > > > think
> > > > > I
> > > > > > > misunderstood the reasoning behind separating the website from
> > the
> > > > > > code.  I
> > > > > > > do this would be an easier to manage workflow.   I believe the
> > > > current
> > > > > > set
> > > > > > > up builds all static content and Java/Py docs from the same
> > > command.
> > > > > We
> > > > > > > may have to do a little more work to separate the two.
> > > > > > >
> > > > > > > *** Any one else have any thoughts?  Did I miss anything?
> > > > > > >
> > > 

Re: 5/21/2019 Bi-Weekly OSS Heron Sync-up

[Simon Weng]

Hi, all:

   1. convert the Java Kafka Spout sub project into a Bazel target, making
   it use in-tree Heron Java API and dependencies, instead of public maven
   artifacts
   2. adapt "setup-intellij.sh" to include the "contrib" module
   3. ensure source style compliance of Java Kafka Spout

Cheers,

Simon Weng

On Mon, May 20, 2019 at 2:51 PM Josh Fischer  wrote:

> Hey all,
>
> I’ve working on how to move the heron site over to Apache infra and a
> better workflow for versioning docs.
>
> On Mon, May 20, 2019 at 1:13 PM Dmitry Rusakov
> 
> wrote:
>
> > I continue working on Heron Stream Manager migration from manual memory
> > management to smart pointers (TMaster migrated previously).
> >
> > On Mon, May 20, 2019 at 10:17 AM Ning Wang  wrote:
> >
> > > Hi, all~,
> > >
> > > It has been two weeks since our last sync and there has been quite a
> few
> > > works done! Let's share our works for the last two weeks in this
> thread.
> > >
> > > My updates:
> > > - Continue working on Beam
> > > - Added Kryo support in Heron API as well as Kryo serializer
> > registration.
> > >
> > >
> > > Regards,
> > > --ning
> > >
> >
> >
> > --
> > Dmitry Rusakov,
> > Senior Software Engineer,
> > Twitter CDM team
> >
> --
> Sent from A Mobile Device
>

Re: 4/22/2019 Bi-Weekly OSS Heron Sync-up

[Simon Weng]

My updates:

Troubleshoot and fixed an issue that Heron ECO does not initialize Stateful
Windowed Bolt correctly.

On Tue, Apr 23, 2019 at 1:19 AM Xiaoyao Qian  wrote:

> My updates:
> 1. Made extra links in the Heron UI data-driven instead of hardcoded.
>
> Xiaoyao Qian
>
>
> On Mon, Apr 22, 2019 at 2:57 PM FatJ Love 
> wrote:
>
> > my updates:
> > 1. volunteer at https://apacheheronmeetup.splashthat.com/
> >
> >
> > On Mon, Apr 22, 2019 at 11:28 AM Ning Wang  wrote:
> >
> > > Correction: subject should be 4/23/2019 since our sync up schedule is
> on
> > > Tuesday.  Monday is hard. :(
> > >
> > >
> > > On Mon, Apr 22, 2019 at 11:18 AM Ning Wang 
> wrote:
> > >
> > > > Hi,
> > > >
> > > > I am sorry that I was in a business trip two weeks ago and couldn't
> > send
> > > > this sync up message. Let's share our works for the last *four* weeks
> > in
> > > > this thread.
> > > >
> > > > My updates:
> > > > - Code reviews
> > > > - Prepared 0.20.1-incubating-rc2 for vote. it was successful but Dave
> > had
> > > > some concerns and suggestions and discussed about it. Then the RC2
> was
> > > > cleaned up and sent out for another pass.
> > > > - Learn Apache Beam
> > > >
> > > > Regards,
> > > > --ning
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> >
>
-- 
Sent from Gmail Mobile

Re: [VOTE] Heron Release 0.20.1-incubating Release Candidate 2

[Simon Weng]

+1

On Fri, Mar 29, 2019 at 6:26 AM Josh Fischer  wrote:

> +1
>
> On Thu, Mar 28, 2019 at 11:27 PM Ning Wang  wrote:
>
> > Nice~ Thanks thinker0!
> >
> > On Thu, Mar 28, 2019 at 8:29 PM thinker0  wrote:
> >
> > > +1 We are using it well for production.
> > >
> > > 2019년 3월 29일 (금) 오전 8:00, Ning Wang 님이 작성:
> > >
> > > > *Hello, Heron Community,This is a call for vote to the 2nd release
> > > > candidate for Apache Heron, version v0.20.1-incubating.The tag to be
> > > voted
> > > > upon:0.20.1-incubating-rc2
> > (e6134da336fa290fa1b40972bc747a7507948d8a)The
> > > > full list of changes and release notes are available
> > > > at:
> > > >
> > >
> >
> https://github.com/apache/incubator-heron/releases/tag/0.20.1-incubating-rc2
> > > > <
> > > >
> > >
> >
> https://github.com/apache/incubator-heron/releases/tag/0.20.1-incubating-rc2
> > > > >Source
> > > > files can be found in dist.apache.org 
> > > > site:
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/incubator/heron/heron-0.20.1-incubating-candidate-2/
> > > > <
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/incubator/heron/heron-0.20.1-incubating-candidate-2/
> > > > >Docker
> > > > image is available at:
> https://hub.docker.com/r/apacheheron/heron/tags
> > > > The generated
> > packages,
> > > > including maven artifacts, installers and docker image are available
> > here
> > > > on GitHub:
> > > > <
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/incubator/heron/heron-0.20.0-incubating-candidate-5/
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-heron/releases/tag/0.20.1-incubating-rc2
> > > > <
> > > >
> > >
> >
> https://github.com/apache/incubator-heron/releases/tag/0.20.1-incubating-rc2
> > > > >Source
> > > > SHA-512
> > > >
> > > >
> > >
> >
> checksums:c47fc8c228b5543f94dcf8fb5eb0f8083e84602be4f3b5ca52402b6e3e0f893434f971c317f44c3a69e78e597b96642fd69b5bec63e9a8eb7456c816f8e118f3
> > > >  incubator-heron-0.20.1-incubating-rc2.tar.gzArtifacts are published
> > > > to:API:
> > > >
> > > >
> > >
> >
> https://repository.apache.org/content/repositories/staging/org/apache/heron/heron-api/0.20.1-incubating-rc2/
> > > > <
> > > >
> > >
> >
> https://repository.apache.org/content/repositories/staging/org/apache/heron/heron-api/0.20.1-incubating-rc2/
> > > > >SPI:
> > > >
> > > >
> > >
> >
> https://repository.apache.org/content/repositories/staging/org/apache/heron/heron-spi/0.20.1-incubating-rc2/
> > > > <
> > > >
> > >
> >
> https://repository.apache.org/content/repositories/staging/org/apache/heron/heron-spi/0.20.1-incubating-rc2/
> > > > >Storm
> > > > API:
> > > >
> > > >
> > >
> >
> https://repository.apache.org/content/repositories/staging/org/apache/heron/heron-storm/0.20.1-incubating-rc2/
> > > > <
> > > >
> > >
> >
> https://repository.apache.org/content/repositories/staging/org/apache/heron/heron-storm/0.20.1-incubating-rc2/
> > > > >Simulator:
> > > >
> > > >
> > >
> >
> https://repository.apache.org/content/repositories/staging/org/apache/heron/heron-simulator/0.20.1-incubating-rc2/
> > > > <
> > > >
> > >
> >
> https://repository.apache.org/content/repositories/staging/org/apache/heron/heron-simulator/0.20.1-incubating-rc2/
> > > > >The
> > > > artifacts are signed with PGP key 293DB72F865688D1, corresponding to
> > > > nw...@apache.org , that can be found in keys
> > > > file:https://dist.apache.org/repos/dist/release/incubator/heron/KEYS
> > > >  > >Please
> > > > download the source package, and follow the compiling
> > > > guide(
> > > >
> > >
> >
> https://apache.github.io/incubator-heron/docs/developers/compiling/compiling/
> > > > <
> > > >
> > >
> >
> https://apache.github.io/incubator-heron/docs/developers/compiling/compiling/
> > > > >)to
> > > > build and run the Heron locally. Note that currently Bazel 0.14.1 is
> > > > required to build this version.The vote will be open for at least 72
> > > hours
> > > > or until the necessary number of votes are reached.Please vote
> > > > accordingly:[ ] +1 approve[ ] +0 no opinion[ ] -1 disapprove with the
> > > > reasonThanks,The Apache Heron (Incubating) Team*
> > > >
> > >
> >
>

Re: 3/26/2019 Bi-Weekly OSS Heron Sync-up

[Simon Weng]

Hi, all:

My updates:

1. Update the KafkaSpout to support EFFECTIVE_ONCE mode and support
converting 1 ConsumerRecord to multiple output tuples
2. Upgrade to Apache Bookkeeper 4.7.3, so that Dlog-based stateful storage
can work
3. Fix bugs that prevents checkpoint protocol buffer message from being
persisted into/restore from Bookkeeper cluster
4. Add extra command line arguments to CheckpointManager process so that it
can be passed the overridden configuration that user specifies via Heron
API server on Kubernetes

Cheers,

SiMing Weng

On Mon, Mar 25, 2019 at 1:39 PM Ning Wang  wrote:

> Hi, all~,
>
> It has been two weeks since our last sync and there has been quite a few
> works done! Let's share our works for the last two weeks in this thread.
>
> My updates (mostly internal works, not much on Apache side):
> - Release 0.20.1-incubating-rc1 was sent out for vote and a license issue
> was found. Preparing 0.20.1-incubating-rc2 currently.
> - Small update in Heron UI.
> - Make cppcheck independent of pcre.
>
> Regards,
> --ning
>
-- 
Sent from Gmail Mobile

Re: 3/12/2019 Bi-Weekly OSS Heron Sync-up

[Simon Weng]

Hi:

Here're what I did:

   1. contributed a KafkaSpout to the Heron Spout effort
   2. fixed an issue that prevents "setup-intellij.sh" from generating
   correct project for IntelliJ
   3. troubleshooted and found the root cause why DistributedLog-based
   stateful storage, i.e. "org.apache.heron.statefulstorage.dlog.DlogStorage"
   didn't work, and a ticket was logged to track that
   4. troubleshooted and found the root cause why CheckpointManager could
   not be configured at deployment time on Kubernetes, a ticket was logged to
   track that
   5. help answering some Kafka related questions on Slack channel

Cheers,

Simon Weng

On Tue, Mar 12, 2019 at 7:58 PM Dave Fisher  wrote:

> Hi -
>
> > On Mar 12, 2019, at 4:22 PM, Josh Fischer  wrote:
> >
> > Not too much for me this week.
> >
> > I participated in a few conversations on the Bazel upgrade process and
> the
> > script that sets up IntelliJ.
> >
> > A thought/question  on my mind: How do we move conversations that happen
> in
> > slack to the mailing list?  I know this has been discussed a few times.
> > Maybe some time of job that pulls conversations from each channel and
> mails
> > them to the list on a nightly basis?
>
> Ask the Apache Pulsar team about slack jobs - Matteo has one.
>
> >
> > I can understand the ease of use of slack and why we have it.  However I
> > also understand the importance of managing this project’s discussions
> > through the mailing list.
>
> Dev list is preferred for long thoughtful asynchronous threaded
> discussions that you may want to find 15 years from now. I have gone that
> far back on projects where I’ve been on the PMC for 11 years. Slack my be
> gone in 10 years, but the Apache Archives will live.
>
> Regards,
> Dave
>
> >
> > - Josh
> >
> > On Tue, Mar 12, 2019 at 4:55 PM Xiaoyao Qian 
> wrote:
> >
> >> Hi all,
> >>
> >> My updates:
> >> 1. Modified the FirstFitDecreasingPacking algorithm so that now it takes
> >> CPU and RAM constraints into consideration when generating a packing
> plan.
> >> 2. Added static (cppcheck) and runtime (gperftool heapcheck) checks to
> >> native cpp modules.
> >>
> >> Xiaoyao Qian
> >>
> >>
> >> On Tue, Mar 12, 2019 at 2:48 PM Ning Wang  wrote:
> >>
> >>> Friendly ping~
> >>>
> >>> On Mon, Mar 11, 2019 at 11:42 AM Neng Lu  wrote:
> >>>
> >>>> Hi All,
> >>>>
> >>>> My updates:
> >>>> - I'm finishing the process of a security vulnerability issue.
> >>>>
> >>>> On Mon, Mar 11, 2019 at 11:19 AM FatJ Love 
> >>>> wrote:
> >>>>
> >>>>> My update
> >>>>> - Worked with OSS team about the March meetup
> >>>>>
> >>>>> On Mon, Mar 11, 2019 at 10:39 AM Ning Wang 
> >>> wrote:
> >>>>>
> >>>>>> Hi, all~,
> >>>>>>
> >>>>>> It has been two weeks since our last sync and it is time to share
> >> our
> >>>>>> progress again. Let's share our works for the last two weeks in
> >> this
> >>>>>> thread.
> >>>>>>
> >>>>>> My updates (mostly internal works, not much on Apache side):
> >>>>>> - Worked with OSS team about the April meetup
> >>>>>> - Maven artifacts for Apache release 0.20.1 rc1 is uploaded to
> >> Apache
> >>>>> maven
> >>>>>> repo:
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> https://repository.apache.org/content/repositories/staging/org/apache/heron/
> >>>>>> .
> >>>>>> Working on the binary packages now. The current discussion about
> >> file
> >>>>>> hosting can be found here:
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> http://mail-archives.apache.org/mod_mbox/heron-dev/201903.mbox/%3CCAHZ_pm6Dh0mtRdRZL07jMPKZitr%3DNmQLFq7ZuDzL_jGRR%2Btxxw%40mail.gmail.com%3E
> >>>>>>
> >>>>>>
> >>>>>> Regards,
> >>>>>> --ning
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> Best Regards,
> >>>> Neng
> >>>>
> >>>
> >>
> > --
> > Sent from A Mobile Device
>
>

Re: Heron Spouts Code

[Simon Weng]

This is a good question. Each version spout must maintain a compatibility
matrix {Spout Version, external SDK version, Heron API version}. It’s more
of a documentation effort so that user haves enough information to
determine which one to pick, isn’t it?

On Thu, Jan 17, 2019 at 7:48 AM Josh Fischer  wrote:

> If we were to go with a separate repo for the spouts how would we version
> it?  Would it be consistent with the Heron repo?  How would people know
> what version spout to use with the Heron version they are running?
>
>
> On Thu, Jan 17, 2019 at 1:26 AM Ning Wang  wrote:
>
>> This is an option. I have a few concerns about it:
>> - There will be a lot of repos and it will be messy to manage and it might
>> be harder for users to find it. I am expecting at least more than ten
>> (different services times different languages).
>> - There will be some duplicated code such as build/release configs,
>> scripts. etc.
>>
>> I think we should be able to achieve the first reason with a single repo.
>> Different spouts should likely be in different folders and they can evolve
>> separately.
>> The second reason is valid, but duplicated code is a side effect.
>> The third reason depends on building tool I feel. Bazel is powerful, but
>> it
>> is just changing time by time. :(
>>
>> Just my two cents.
>>
>>
>>
>>
>>
>> On Wed, Jan 16, 2019 at 8:09 PM Simon Weng  wrote:
>>
>> > Hi, all:
>> >
>> > Can it also be one of the options to even have separate repo for each
>> type
>> > of spouts? The reasons it is worth considering are:
>> >
>> > 1. Allow each spout to evolve and release in different pace because each
>> > is technically driven by external source software. For example, the
>> > community may need different versions of the Kafka Spout to be
>> compatible
>> > with their deployed Kafka cluster in production
>> > 2. Allow each spout project to use the de facto build tool that suits
>> the
>> > external SDK best. This will help to minimize the learning curve for
>> > constributors who specialize in different source software stack
>> > 3. Simply the maintainence of the build and CI
>> >
>> > I’m not familiar with the capability of Bazel, so certainly I’m not
>> > against it. If it can help to achieve some of the above, I guess one
>> single
>> > repo will also work then.
>> >
>> > SiMing
>> >
>> > On Wed, Jan 16, 2019 at 5:34 PM Ning Wang  wrote:
>> >
>> >> +Siming
>> >>
>> >> On Tue, Jan 15, 2019 at 11:35 PM Ning Wang 
>> wrote:
>> >>
>> >>> Hi, all,
>> >>>
>> >>> A few of us (Spencer, Saikat, Siming, Karthik, Josh, Sree) discussed
>> >>> today in our general slack channel that we should have spouts code
>> >>> somewhere so that people can reuse them (spouts are highly reusable in
>> >>> general) and contribute improvements. This is just a recap of the
>> idea and
>> >>> some updates.
>> >>>
>> >>> We have two options:
>> >>> 1. add a spouts/ dir in heron project.
>> >>> 2. create a new project in github.
>> >>>
>> >>> For option 1, it is easy to start. But the iteration and release will
>> be
>> >>> coupled with Heron project itself. It is likely there will be quite
>> some
>> >>> activities around spouts time by time when new spouts are added. Also,
>> >>> Heron itself is basically the engine itself plus APIs and tooling,
>> while
>> >>> there could be quite some spouts in future with many new dependencies
>> like
>> >>> Kafka, pubsub, neo4j and neptune, etc. It is debatable to have spout
>> >>> implementations in Heron project, and these extra dependencies could
>> add
>> >>> some unnecessary complexity.
>> >>>
>> >>> For option 2, there will be some work up front. but it will be much
>> >>> easier to manage and evolve. And here will be less concerns about new
>> >>> spouts (in different languages) and dependencies because spouts are
>> >>> relatively independent to each other and we may generate artifacts per
>> >>> spout.
>> >>>
>> >>> Overall most people prefer option 2 for its cleanness.
>> >>>
>> >>> I talked with Twitter OSS team. They are happy to support the
>> initiative
>> >>> and suggest us to check with Apache team and see what is the best
>> process.
>> >>> First question is that should this new side project be under Apache
>> or not?
>> >>> This might be a question to mentors. What do you think/suggest?
>> >>>
>> >>> Another topic being discussed is the build tool in case we decide to
>> >>> create a new side project. Maven is more mature for sure, but we will
>> >>> likely need multi language support so currently Bazel seems to be the
>> >>> winner (I personally vote for Bazel 1.0 because the backward
>> compatibility
>> >>> has been bad so far).
>> >>>
>> >>> Any ideas or suggestions, please feel free to reply.
>> >>>
>> >>> Regards,
>> >>> --ning
>> >>>
>> >> --
>> > Sent from Gmail Mobile
>> >
>>
> --
> Sent from A Mobile Device
>
-- 
Sent from Gmail Mobile

Re: Heron Spouts Code

[Simon Weng]

As long as it does not have to be a single release train for all of the
spout, I’m good with a single separate repo hosting all of spout. We just
need a bit upfront effort to setup Bazel for the project.

However, I expect once one Spout project folder is setup, it can serve as
sample for other spouts of same language, for example, once Kafka Spout is
migrated from Maven to Bazel, other Java-based spout should be easy to
setup.

Seems time to pick up some Bazel knowledge.

On Thu, Jan 17, 2019 at 2:25 AM Ning Wang  wrote:

> This is an option. I have a few concerns about it:
> - There will be a lot of repos and it will be messy to manage and it might
> be harder for users to find it. I am expecting at least more than ten
> (different services times different languages).
> - There will be some duplicated code such as build/release configs,
> scripts. etc.
>
> I think we should be able to achieve the first reason with a single repo.
> Different spouts should likely be in different folders and they can evolve
> separately.
> The second reason is valid, but duplicated code is a side effect.
> The third reason depends on building tool I feel. Bazel is powerful, but
> it is just changing time by time. :(
>
> Just my two cents.
>
>
>
>
>
> On Wed, Jan 16, 2019 at 8:09 PM Simon Weng  wrote:
>
>> Hi, all:
>>
>> Can it also be one of the options to even have separate repo for each
>> type of spouts? The reasons it is worth considering are:
>>
>> 1. Allow each spout to evolve and release in different pace because each
>> is technically driven by external source software. For example, the
>> community may need different versions of the Kafka Spout to be compatible
>> with their deployed Kafka cluster in production
>> 2. Allow each spout project to use the de facto build tool that suits the
>> external SDK best. This will help to minimize the learning curve for
>> constributors who specialize in different source software stack
>> 3. Simply the maintainence of the build and CI
>>
>> I’m not familiar with the capability of Bazel, so certainly I’m not
>> against it. If it can help to achieve some of the above, I guess one single
>> repo will also work then.
>>
>> SiMing
>>
>> On Wed, Jan 16, 2019 at 5:34 PM Ning Wang  wrote:
>>
>>> +Siming
>>>
>>> On Tue, Jan 15, 2019 at 11:35 PM Ning Wang  wrote:
>>>
>>>> Hi, all,
>>>>
>>>> A few of us (Spencer, Saikat, Siming, Karthik, Josh, Sree) discussed
>>>> today in our general slack channel that we should have spouts code
>>>> somewhere so that people can reuse them (spouts are highly reusable in
>>>> general) and contribute improvements. This is just a recap of the idea and
>>>> some updates.
>>>>
>>>> We have two options:
>>>> 1. add a spouts/ dir in heron project.
>>>> 2. create a new project in github.
>>>>
>>>> For option 1, it is easy to start. But the iteration and release will
>>>> be coupled with Heron project itself. It is likely there will be quite some
>>>> activities around spouts time by time when new spouts are added. Also,
>>>> Heron itself is basically the engine itself plus APIs and tooling, while
>>>> there could be quite some spouts in future with many new dependencies like
>>>> Kafka, pubsub, neo4j and neptune, etc. It is debatable to have spout
>>>> implementations in Heron project, and these extra dependencies could add
>>>> some unnecessary complexity.
>>>>
>>>> For option 2, there will be some work up front. but it will be much
>>>> easier to manage and evolve. And here will be less concerns about new
>>>> spouts (in different languages) and dependencies because spouts are
>>>> relatively independent to each other and we may generate artifacts per
>>>> spout.
>>>>
>>>> Overall most people prefer option 2 for its cleanness.
>>>>
>>>> I talked with Twitter OSS team. They are happy to support the
>>>> initiative and suggest us to check with Apache team and see what is the
>>>> best process. First question is that should this new side project be under
>>>> Apache or not? This might be a question to mentors. What do you
>>>> think/suggest?
>>>>
>>>> Another topic being discussed is the build tool in case we decide to
>>>> create a new side project. Maven is more mature for sure, but we will
>>>> likely need multi language support so currently Bazel seems to be the
>>>> winner (I personally vote for Bazel 1.0 because the backward compatibility
>>>> has been bad so far).
>>>>
>>>> Any ideas or suggestions, please feel free to reply.
>>>>
>>>> Regards,
>>>> --ning
>>>>
>>> --
>> Sent from Gmail Mobile
>>
> --
Sent from Gmail Mobile

Re: Heron Spouts Code

[Simon Weng]

Hi, all:

Can it also be one of the options to even have separate repo for each type
of spouts? The reasons it is worth considering are:

1. Allow each spout to evolve and release in different pace because each is
technically driven by external source software. For example, the community
may need different versions of the Kafka Spout to be compatible with their
deployed Kafka cluster in production
2. Allow each spout project to use the de facto build tool that suits the
external SDK best. This will help to minimize the learning curve for
constributors who specialize in different source software stack
3. Simply the maintainence of the build and CI

I’m not familiar with the capability of Bazel, so certainly I’m not against
it. If it can help to achieve some of the above, I guess one single repo
will also work then.

SiMing

On Wed, Jan 16, 2019 at 5:34 PM Ning Wang  wrote:

> +Siming
>
> On Tue, Jan 15, 2019 at 11:35 PM Ning Wang  wrote:
>
>> Hi, all,
>>
>> A few of us (Spencer, Saikat, Siming, Karthik, Josh, Sree) discussed
>> today in our general slack channel that we should have spouts code
>> somewhere so that people can reuse them (spouts are highly reusable in
>> general) and contribute improvements. This is just a recap of the idea and
>> some updates.
>>
>> We have two options:
>> 1. add a spouts/ dir in heron project.
>> 2. create a new project in github.
>>
>> For option 1, it is easy to start. But the iteration and release will be
>> coupled with Heron project itself. It is likely there will be quite some
>> activities around spouts time by time when new spouts are added. Also,
>> Heron itself is basically the engine itself plus APIs and tooling, while
>> there could be quite some spouts in future with many new dependencies like
>> Kafka, pubsub, neo4j and neptune, etc. It is debatable to have spout
>> implementations in Heron project, and these extra dependencies could add
>> some unnecessary complexity.
>>
>> For option 2, there will be some work up front. but it will be much
>> easier to manage and evolve. And here will be less concerns about new
>> spouts (in different languages) and dependencies because spouts are
>> relatively independent to each other and we may generate artifacts per
>> spout.
>>
>> Overall most people prefer option 2 for its cleanness.
>>
>> I talked with Twitter OSS team. They are happy to support the initiative
>> and suggest us to check with Apache team and see what is the best process.
>> First question is that should this new side project be under Apache or not?
>> This might be a question to mentors. What do you think/suggest?
>>
>> Another topic being discussed is the build tool in case we decide to
>> create a new side project. Maven is more mature for sure, but we will
>> likely need multi language support so currently Bazel seems to be the
>> winner (I personally vote for Bazel 1.0 because the backward compatibility
>> has been bad so far).
>>
>> Any ideas or suggestions, please feel free to reply.
>>
>> Regards,
>> --ning
>>
> --
Sent from Gmail Mobile