Re: Moving Developer Section of cwiki to the git repository

[Balazs Donat Bessenyei]

Thank you.

Wouldn't committing (and PRing) them one-by-one pollute the git log and
mailing lists too much?

On Oct 24, 2016 5:26 PM, "Mike Percy" <mpe...@apache.org> wrote:

> Feel free to do it per-file if you want. Looks good to me
>
> On Mon, Oct 24, 2016 at 4:14 PM, Balazs Donat Bessenyei <
> bes...@cloudera.com
> > wrote:
>
> > I have commenced moving the Developer Section to the git repo.
> >
> > There is a WIP pull request available at https://github.com/apache/
> > flume/pull/77
> >
> > I'm planning to add more e-mail templates to the HowToRelease.md and
> > commit all docs in a single commit, but reviews and comments are
> > welcome in the meanwhile.
> >
> >
> > Thank you,
> >
> > Donat
> >
> > On Fri, Oct 21, 2016 at 6:07 PM, Mike Percy <mpe...@apache.org> wrote:
> > > Thanks, Donat. +1 on moving the docs.
> > >
> > > Regarding the web site source code, it can reside in svn or git however
> > it
> > > should continue using the standard ASF site infrastructure [1] for
> > hosting
> > > content which consists only of static rendered HTML content. Our
> options
> > > for pushing that rendered HTML are either svnpubsub[2] or gitpubsub[3]
> > > which are basically the same thing - commit HTML and it gets pushed
> live
> > to
> > > the web site.
> > >
> > > If someone wants to redesign the site or improve the site template, I
> > would
> > > certainly welcome that.
> > >
> > > As a comparison point with another ASF project I am involved in, on
> > Apache
> > > Kudu we use the software used for GitHub pages (Jekyll) to render the
> > site.
> > > The web site source code is in the gh-pages branch in Git [4]. The site
> > is
> > > mostly written in Markdown and we wrote a script [5] that locally
> renders
> > > the site to static HTML content and checks in the rendered content to
> the
> > > Kudu gitpubsub repository [6].
> > >
> > > In Flume, we basically do the same thing but but instead of Jekyll we
> use
> > > Sphinx and the site source code is primarily ReStructuredText. The site
> > > lives in its own svn repo [7]. We push rendered HTML tent to an
> svnpubsub
> > > repository for Flume [8] which is controlled by the ASF CMS [9]...
> IIRC,
> > > the CMS takes care of invoking the rendering part instead of having a
> > local
> > > script to do it. This is kind of documented in the How to Release guide
> > for
> > > Flume [10].
> > >
> > > If someone just wanted to change the look & feel of the web site they
> > could
> > > easily do it with a little HTML/CSS knowledge and some reading up on
> > > Sphinx. The web site template could be changed to something nicer (it's
> > > using some kind of stock template right now). The existing .rst content
> > > would not need to be modified at all, probably.
> > >
> > > If someone wanted to switch the whole web site to some other system,
> like
> > > Jekyll, obviously it's a bigger change but if they were determined
> then I
> > > don't think anyone would try to stop them, assuming it's an
> improvement!
> > >
> > > Hope this helps,
> > > Mike
> > >
> > > [1] https://www.apache.org/dev/project-site.html
> > > [2] http://svn.apache.org/viewvc/subversion/trunk/tools/
> > > server-side/svnpubsub/
> > > [3] https://www.apache.org/dev/gitpubsub.html
> > > [4] https://github.com/apache/kudu/tree/gh-pages
> > > [5] https://github.com/apache/kudu/blob/gh-pages/site_tool
> > > [6] https://git-wip-us.apache.org/repos/asf?p=kudu-site.git;a=
> > > shortlog;h=refs/heads/asf-site
> > > [7] https://svn.apache.org/repos/asf/flume/site/trunk/
> > > [8] https://svn.apache.org/repos/infra/websites/production/flume/
> > > [9] https://www.apache.org/dev/cms.html
> > > [10]
> > > https://cwiki.apache.org/confluence/display/FLUME/How+
> > to+Release#HowtoRelease-Updatethewebsite
> > >
> > > On Thu, Oct 20, 2016 at 2:59 PM, Balazs Donat Bessenyei <
> > bes...@cloudera.com
> > >> wrote:
> > >
> > >> As nobody has objected in ~48 hours, I'll open a pull request about
> this
> > >> soon.
> > >>
> > >> Regarding the whole site thing: AFAIK, the site contents have to
> > >> reside in SVN, because that's how ASF infrastructure works now.
> > >>
> > &g

Re: Moving Developer Section of cwiki to the git repository

[Balazs Donat Bessenyei]

I have commenced moving the Developer Section to the git repo.

There is a WIP pull request available at https://github.com/apache/flume/pull/77

I'm planning to add more e-mail templates to the HowToRelease.md and
commit all docs in a single commit, but reviews and comments are
welcome in the meanwhile.


Thank you,

Donat

On Fri, Oct 21, 2016 at 6:07 PM, Mike Percy <mpe...@apache.org> wrote:
> Thanks, Donat. +1 on moving the docs.
>
> Regarding the web site source code, it can reside in svn or git however it
> should continue using the standard ASF site infrastructure [1] for hosting
> content which consists only of static rendered HTML content. Our options
> for pushing that rendered HTML are either svnpubsub[2] or gitpubsub[3]
> which are basically the same thing - commit HTML and it gets pushed live to
> the web site.
>
> If someone wants to redesign the site or improve the site template, I would
> certainly welcome that.
>
> As a comparison point with another ASF project I am involved in, on Apache
> Kudu we use the software used for GitHub pages (Jekyll) to render the site.
> The web site source code is in the gh-pages branch in Git [4]. The site is
> mostly written in Markdown and we wrote a script [5] that locally renders
> the site to static HTML content and checks in the rendered content to the
> Kudu gitpubsub repository [6].
>
> In Flume, we basically do the same thing but but instead of Jekyll we use
> Sphinx and the site source code is primarily ReStructuredText. The site
> lives in its own svn repo [7]. We push rendered HTML tent to an svnpubsub
> repository for Flume [8] which is controlled by the ASF CMS [9]... IIRC,
> the CMS takes care of invoking the rendering part instead of having a local
> script to do it. This is kind of documented in the How to Release guide for
> Flume [10].
>
> If someone just wanted to change the look & feel of the web site they could
> easily do it with a little HTML/CSS knowledge and some reading up on
> Sphinx. The web site template could be changed to something nicer (it's
> using some kind of stock template right now). The existing .rst content
> would not need to be modified at all, probably.
>
> If someone wanted to switch the whole web site to some other system, like
> Jekyll, obviously it's a bigger change but if they were determined then I
> don't think anyone would try to stop them, assuming it's an improvement!
>
> Hope this helps,
> Mike
>
> [1] https://www.apache.org/dev/project-site.html
> [2] http://svn.apache.org/viewvc/subversion/trunk/tools/
> server-side/svnpubsub/
> [3] https://www.apache.org/dev/gitpubsub.html
> [4] https://github.com/apache/kudu/tree/gh-pages
> [5] https://github.com/apache/kudu/blob/gh-pages/site_tool
> [6] https://git-wip-us.apache.org/repos/asf?p=kudu-site.git;a=
> shortlog;h=refs/heads/asf-site
> [7] https://svn.apache.org/repos/asf/flume/site/trunk/
> [8] https://svn.apache.org/repos/infra/websites/production/flume/
> [9] https://www.apache.org/dev/cms.html
> [10]
> https://cwiki.apache.org/confluence/display/FLUME/How+to+Release#HowtoRelease-Updatethewebsite
>
> On Thu, Oct 20, 2016 at 2:59 PM, Balazs Donat Bessenyei <bes...@cloudera.com
>> wrote:
>
>> As nobody has objected in ~48 hours, I'll open a pull request about this
>> soon.
>>
>> Regarding the whole site thing: AFAIK, the site contents have to
>> reside in SVN, because that's how ASF infrastructure works now.
>>
>> First, I'll work on moving the Developer Section to
>> https://github.com/apache/flume , then we'll see what else we can do.
>>
>>
>> Thank you,
>>
>> Donat
>>
>> On Thu, Oct 20, 2016 at 12:41 PM, UMESH CHAUDHARY <umesh9...@gmail.com>
>> wrote:
>> > +1 for moving wiki and web contents into GitHub. It would be easier to
>> > manage. Also, +1 for improving our website.
>> >
>> > On Thu, 20 Oct 2016 at 15:42 Lior Zeno <liorz...@gmail.com> wrote:
>> >
>> > +1 for moving the whole site as well. I wish we could improve our
>> frontend
>> > to become a bit more appealing.
>> >
>> > On Thu, Oct 20, 2016 at 11:53 AM, Attila Simon <s...@cloudera.com>
>> wrote:
>> >
>> >> +1
>> >> Essentially moving whole site (all web content) to scm would help
>> >> contribution.
>> >>
>> >> Cheers,
>> >> Attila
>> >>
>> >> On Tuesday, 18 October 2016, Balazs Donat Bessenyei <
>> bes...@cloudera.com>
>> >> wrote:
>> >>
>> >> > Hi All,
>> >> >
>> >> > As it's kind of difficult to get permissions in the wiki to edit pages
>> >> > like https://cwiki.apache.org/confluence/display/FLUME/How+to+
>> Contribute
>> >> > , I suggest moving contents to the git repository into files like
>> >> > CONTRIBUTING.md, etc.
>> >> >
>> >> > I'd be happy to create the files and move the current texts.
>> >> >
>> >> > Please, let me know your thoughts.
>> >> >
>> >> >
>> >> > Thank you,
>> >> >
>> >> > Donat
>> >> >
>> >>
>> >>
>> >> --
>> >>
>> >> *Attila Simon*
>> >> Software Engineer
>> >> Email:   s...@cloudera.com
>> >>
>> >> [image: Cloudera Inc.]
>> >>
>>

Re: Enabling Travis-CI on Flume

[Balazs Donat Bessenyei]

Infra has enabled Travis-CI for us.

mvn clean install -DskipTests is executed on all PR-s and commits from now on.

However, builds occasionally fail with heap space error.
To fix that issue, I have opened a pull request:
https://github.com/apache/flume/pull/75 . Any reviews and comments are
welcome.


Thank you,

Donat


On Fri, Oct 21, 2016 at 6:14 PM, Mike Percy <mpe...@apache.org> wrote:
> Hi Lior!
>
> No, my message was not directed at you, or any person in particular. I
> intended this message for those paying attention to this topic to try and
> set expectations for how decision making for things like this usually (and
> hopefully) works in Apache: If you are willing to do the work to get
> something done, then it will probably get done the way you want!
>
> (Assuming the end result is something that others in the community want --
> in this case, it's very basic pre-commit checks. I think we have consensus
> that pre-commit checks are something that would be a net benefit to
> everyone.)
>
> Sorry for any confusion! And don't let me stop people from voicing their
> views and concerns.
>
> Best,
> Mike
>
> On Fri, Oct 21, 2016 at 4:44 PM, Lior Zeno <liorz...@gmail.com> wrote:
>
>> Mike, I was not holding Donat back. I was just suggesting ways to configure
>> Jenins, per Donat's request. I'm sorry if my former post delivered the
>> wrong message.
>>
>> On Fri, Oct 21, 2016 at 6:29 PM, Mike Percy <mpe...@apache.org> wrote:
>>
>> > Personally I prefer Jenkins over TravisCI for various reasons however if
>> > Donat is willing to do the work of adding pre-commit checks on PRs via
>> > Travis then I say let him do it, in the Apache spirit of "let they that
>> do
>> > the work make the decisions".
>> >
>> > If someone actually spends the time to set up Jenkins and configure it to
>> > do the same thing, then great, let's switch when it's ready.
>> >
>> > Note that only ASF committers have access to Jenkins so non-committers
>> will
>> > need to work with a committer to get it done if they want to help.
>> >
>> > Mike
>> >
>> > On Fri, Oct 21, 2016 at 3:46 PM, Lior Zeno <liorz...@gmail.com> wrote:
>> >
>> > > There are many ways to do it, for example:
>> > > https://www.theguild.nl/building-github-pull-requests-using-
>> > > jenkins-pipelines/
>> > > or https://www.theguild.nl/building-github-pull-requests-with-jenkins/
>> > for
>> > > earlier versions of Jenkins.
>> > > I do not really care if it would be Jenkins or Travis, but I do think
>> > that
>> > > we can get Jenkins configured faster since we already have it. I can
>> help
>> > > with the configuration.
>> > >
>> > > On Fri, Oct 21, 2016 at 5:17 PM, Balazs Donat Bessenyei <
>> > > bes...@cloudera.com
>> > > > wrote:
>> > >
>> > > > As I haven't received any objections to enabling Travis, I'm going to
>> > > > ask INFRA to enable it for Flume soon.
>> > > >
>> > > > This change would help submitting and reviewing pull requests.
>> > > >
>> > > > If someone figures out how we could use Jenkins for this purpose, we
>> > > > can always disable Travis.
>> > > >
>> > > > PS. there are more projects using Travis:
>> > > > https://issues.apache.org/jira/browse/INFRA-12757?jql=
>> > > > project%20%3D%20INFRA%20AND%20text%20~%20travis%20ORDER%
>> > > > 20BY%20updated%20DESC
>> > > >
>> > > > On Fri, Oct 14, 2016 at 5:41 PM, Attila Simon <s...@cloudera.com>
>> > wrote:
>> > > > > Denes I'm happy to help you in this endeavor of setting up jenkins
>> > job
>> > > > for
>> > > > > verifying pull requests.
>> > > > >
>> > > > >
>> > > > > *Attila Simon*
>> > > > > Software Engineer
>> > > > > Email:   s...@cloudera.com
>> > > > >
>> > > > > [image: Cloudera Inc.]
>> > > > >
>> > > > > On Fri, Oct 14, 2016 at 2:47 PM, Denes Arvay <de...@cloudera.com>
>> > > wrote:
>> > > > >
>> > > > >> I'd also vote for Jenkins with github PRs.
>> > > > >> I just checked Mesos and the PRs are checked by Travis, or at
>> least
>> > > they
>> > > > &

Re: Enabling Travis-CI on Flume

[Balazs Donat Bessenyei]

As I haven't received any objections to enabling Travis, I'm going to
ask INFRA to enable it for Flume soon.

This change would help submitting and reviewing pull requests.

If someone figures out how we could use Jenkins for this purpose, we
can always disable Travis.

PS. there are more projects using Travis:
https://issues.apache.org/jira/browse/INFRA-12757?jql=project%20%3D%20INFRA%20AND%20text%20~%20travis%20ORDER%20BY%20updated%20DESC

On Fri, Oct 14, 2016 at 5:41 PM, Attila Simon <s...@cloudera.com> wrote:
> Denes I'm happy to help you in this endeavor of setting up jenkins job for
> verifying pull requests.
>
>
> *Attila Simon*
> Software Engineer
> Email:   s...@cloudera.com
>
> [image: Cloudera Inc.]
>
> On Fri, Oct 14, 2016 at 2:47 PM, Denes Arvay <de...@cloudera.com> wrote:
>
>> I'd also vote for Jenkins with github PRs.
>> I just checked Mesos and the PRs are checked by Travis, or at least they
>> experienced with it, there's a short discussion regarding to Travis at
>> https://github.com/apache/mesos/pull/165
>>
>> As for the jenkins pull request job I'd be happy to set it up or help
>> setting it up.
>>
>> Denes
>>
>> On Fri, Oct 14, 2016 at 2:15 PM Lior Zeno <liorz...@gmail.com> wrote:
>>
>> Are we switching to PRs from patches + RB? In Apache Mesos, they have a
>>
>> review bot that can leave a comment on the patch, we could try and port it
>>
>> to Flume. I think they use Jenkins too.
>>
>>
>>
>> On Fri, Oct 14, 2016 at 3:11 PM, Balazs Donat Bessenyei <
>> bes...@cloudera.com
>>
>> > wrote:
>>
>>
>>
>> > If the same function can be achieved with Jenkins and it's easy
>>
>> > (+quick) to set up, I'm totally happy with that.
>>
>> >
>>
>> > What do we have to do to enable Jenkins builds on PR-s?
>>
>> >
>>
>> > On Fri, Oct 14, 2016 at 2:05 PM, Lior Zeno <liorz...@gmail.com> wrote:
>>
>> > > There are ways to do the same with Jenkins, for instance, see this SO
>>
>> > > thread
>>
>> > > http://stackoverflow.com/questions/37661602/how-to-set-
>>
>> > up-a-github-pull-request-build-in-a-jenkinsfile
>>
>> > >
>>
>> > > On Fri, Oct 14, 2016 at 11:09 AM, Balazs Donat Bessenyei <
>>
>> > > bes...@cloudera.com> wrote:
>>
>> > >
>>
>> > >> My primary reason for Travis (vs. Jenkins) was that I have experience
>>
>> > with
>>
>> > >> it.
>>
>> > >>
>>
>> > >> And it leaves these happy little checkmarks:
>>
>> > >> https://github.com/sebastianbergmann/phpunit/pull/1051/commits on the
>>
>> > >> commits and messages as seen on
>>
>> > >> https://github.com/apache/hive/pull/107 .
>>
>> > >>
>>
>> > >> Jenkins is probably configurable to achieve similar function. However,
>>
>> > >> I have no idea how to do such. (And could not find an example when I
>>
>> > >> did a quick search.)
>>
>> > >>
>>
>> > >> Are there any disadvantages of enabling Travis on Flume?
>>
>> > >>
>>
>> > >>
>>
>> > >> Thank you,
>>
>> > >>
>>
>> > >> Donat
>>
>> > >>
>>
>> > >> On Thu, Oct 13, 2016 at 6:06 PM, Lior Zeno <liorz...@gmail.com>
>> wrote:
>>
>> > >> > Jenkins can do PRs as well. If we can upgrade Jenkins to 2.0, we
>> will
>>
>> > be
>>
>> > >> > able to define the build step via Jenkinsfile which becomes very
>>
>> > similar
>>
>> > >> to
>>
>> > >> > Travis.
>>
>> > >> > Is there any reason to prefer Travis over Jenkins in our case?
>>
>> > >> >
>>
>> > >> > On Thu, Oct 13, 2016 at 7:01 PM, Balazs Donat Bessenyei <
>>
>> > >> bes...@cloudera.com
>>
>> > >> >> wrote:
>>
>> > >> >
>>
>> > >> >> Hi All,
>>
>> > >> >>
>>
>> > >> >> Having something that checks proposed patches (PR-s especially)
>>
>> > >> >> automatically would help a lot with the development on Flume.
>>
>> > >> >>
>>
>> > >> >> I think, Travis-CI could be an easy solution and (afaik) we'd only
>>
>> > have
>>
>> > >> to
>>
>> > >> >> ask infra to enable it for us.
>>
>> > >> >>
>>
>> > >> >> Please, let me know your thoughts.
>>
>> > >> >>
>>
>> > >> >> Thank you,
>>
>> > >> >>
>>
>> > >> >> Donat
>>
>> > >> >>
>>
>> > >>
>>
>> >
>>

Re: Official support for http sink?

[Balazs Donat Bessenyei]

I have contacted Ben Wheeler, the main author of
https://github.com/hmrc/flume-http-sink .

He said that they are open to contributing the feature to Apache Flume.

I will do my best to assist with the migration.


Thank you,

Donat


On Tue, Oct 18, 2016 at 5:59 PM, Shang Wu (CS) <shang...@symantec.com> wrote:
> Hi Donat,
>
> Thanks a lot for the consideration.
>
> BR,
> Shang Wu, Senior Architect, BlueCoat, a Symantec Company
>
> On 10/18/16, 1:56 AM, "Balazs Donat Bessenyei" <bes...@cloudera.com> wrote:
>
> Hi Shang Wu,
>
> Thank you for your e-mail.
> Both implementations seem to be good. (Also, their licences fit the
> Apache Flume codebase.)
> https://github.com/hmrc/flume-http-sink is more recent.
>
> I'll ping hmrc soon to see if they are willing to create a pull
> request. I think it would be a great improvement to Flume.
>
> If anyone has different ideas or objections, please let me know.
>
>
> Thank you,
>
> Donat
>
> On Mon, Oct 17, 2016 at 10:14 PM, Shang Wu (CS) <shang...@symantec.com> 
> wrote:
> > Hi,
> >
> >
> >
> > We are Symantec Cloud Security group and Apache Flume looks very 
> promising
> > for our backend components, especially its wide supports for those 
> popular
> > data infrastructures via sources and sinks.
> >
> > Having said that, we are looking for a very generic approach for restful
> > API-based data storage systems, such as http sink, now.
> >
> >
> >
> > We’ve found two http sink implementations in github, as follows, and 
> would
> > like to know any plan to include either of it, into the next release of
> > Apache Flume?
> >
> >
> >
> > https://github.com/josealvarezmuguerza/flume-http-sink
> >
> > https://github.com/hmrc/flume-http-sink
> >
> >
> >
> > Thanks adv.
> >
> > BR,
> >
> > Shang Wu, Senior Architect, BlueCoat, a Symantec Company.
>
>

Moving Developer Section of cwiki to the git repository

[Balazs Donat Bessenyei]

Hi All,

As it's kind of difficult to get permissions in the wiki to edit pages
like https://cwiki.apache.org/confluence/display/FLUME/How+to+Contribute
, I suggest moving contents to the git repository into files like
CONTRIBUTING.md, etc.

I'd be happy to create the files and move the current texts.

Please, let me know your thoughts.


Thank you,

Donat

Re: [ANNOUNCE] Apache Flume 1.7.0 released

[Balazs Donat Bessenyei]

Thanks for the suggestions, Lior.

I have just tagged the release and marked the version as released in JIRA.


Donat

On Tue, Oct 18, 2016 at 2:40 PM, Lior Zeno <liorz...@gmail.com> wrote:
> Congratulations everyone! A big thank you for all of those who made this
> possible.
>
> Donat, if you could please tag the final release in the repo that would be
> great. Also, mark the version as released on Jira.
>
> On Tue, Oct 18, 2016 at 3:32 PM, Balazs Donat Bessenyei <bes...@cloudera.com
>> wrote:
>
>> And thank you, Mike Percy for the mentoring and tremendous amounts of
>> assistance with the release!
>>
>> On Tue, Oct 18, 2016 at 12:46 PM, Balazs Donat Bessenyei
>> <bes...@cloudera.com> wrote:
>> > Thank you all very much who participated and helped with the release!
>> >
>> >
>> > Donat
>> >
>> >
>> > On Tue, Oct 18, 2016 at 12:37 PM, Mike Percy <mpe...@apache.org> wrote:
>> >> Woot! Congrats everyone!
>> >>
>> >> Thanks Donat for working so hard to get this version of Flume out the
>> door!
>> >>
>> >> Best,
>> >> Mike
>> >>
>> >>
>> >> On Tue, Oct 18, 2016 at 10:09 AM, Bessenyei Balázs Donát <
>> bes...@apache.org>
>> >> wrote:
>> >>>
>> >>> The Apache Flume team is pleased to announce the release of Flume
>> >>> version 1.7.0.
>> >>>
>> >>> Flume is a distributed, reliable, and available service for efficiently
>> >>> collecting, aggregating, and moving large amounts of log data.
>> >>>
>> >>> This release can be downloaded from the Flume download page at:
>> >>> http://flume.apache.org/download.html
>> >>>
>> >>> The change log and documentation are available on the 1.7.0 release
>> page:
>> >>> http://flume.apache.org/releases/1.7.0.html
>> >>>
>> >>> Your help and feedback is more than welcome. For more information on
>> how
>> >>> to report problems and to get involved, visit the project website at
>> >>> http://flume.apache.org/
>> >>>
>> >>> The Apache Flume Team
>> >>
>> >>
>>

Re: [ANNOUNCE] Apache Flume 1.7.0 released

[Balazs Donat Bessenyei]

And thank you, Mike Percy for the mentoring and tremendous amounts of
assistance with the release!

On Tue, Oct 18, 2016 at 12:46 PM, Balazs Donat Bessenyei
<bes...@cloudera.com> wrote:
> Thank you all very much who participated and helped with the release!
>
>
> Donat
>
>
> On Tue, Oct 18, 2016 at 12:37 PM, Mike Percy <mpe...@apache.org> wrote:
>> Woot! Congrats everyone!
>>
>> Thanks Donat for working so hard to get this version of Flume out the door!
>>
>> Best,
>> Mike
>>
>>
>> On Tue, Oct 18, 2016 at 10:09 AM, Bessenyei Balázs Donát <bes...@apache.org>
>> wrote:
>>>
>>> The Apache Flume team is pleased to announce the release of Flume
>>> version 1.7.0.
>>>
>>> Flume is a distributed, reliable, and available service for efficiently
>>> collecting, aggregating, and moving large amounts of log data.
>>>
>>> This release can be downloaded from the Flume download page at:
>>> http://flume.apache.org/download.html
>>>
>>> The change log and documentation are available on the 1.7.0 release page:
>>> http://flume.apache.org/releases/1.7.0.html
>>>
>>> Your help and feedback is more than welcome. For more information on how
>>> to report problems and to get involved, visit the project website at
>>> http://flume.apache.org/
>>>
>>> The Apache Flume Team
>>
>>

Re: [ANNOUNCE] Apache Flume 1.7.0 released

[Balazs Donat Bessenyei]

Thank you all very much who participated and helped with the release!


Donat


On Tue, Oct 18, 2016 at 12:37 PM, Mike Percy  wrote:
> Woot! Congrats everyone!
>
> Thanks Donat for working so hard to get this version of Flume out the door!
>
> Best,
> Mike
>
>
> On Tue, Oct 18, 2016 at 10:09 AM, Bessenyei Balázs Donát 
> wrote:
>>
>> The Apache Flume team is pleased to announce the release of Flume
>> version 1.7.0.
>>
>> Flume is a distributed, reliable, and available service for efficiently
>> collecting, aggregating, and moving large amounts of log data.
>>
>> This release can be downloaded from the Flume download page at:
>> http://flume.apache.org/download.html
>>
>> The change log and documentation are available on the 1.7.0 release page:
>> http://flume.apache.org/releases/1.7.0.html
>>
>> Your help and feedback is more than welcome. For more information on how
>> to report problems and to get involved, visit the project website at
>> http://flume.apache.org/
>>
>> The Apache Flume Team
>
>

Re: Flume channel error : Unable to put batch on required channel: FileChannel file-channel-1

[Balazs Donat Bessenyei]

As far as I know, apache.org mailing lists "ignore" attachments.
You should either upload them to some service like
https://gist.github.com/ or inline the relevant parts.

Btw, when do you get these errors? When starting Flume or after some time?


Thank you,

Donat

On Mon, Oct 17, 2016 at 7:51 PM, Dheeraj Rokade
 wrote:
> Please find the configuration file attached
>
> Thanks
> Dheeraj
>
> Sent from Yahoo Mail on Android
>
> On Mon, 17 Oct, 2016 at 18:23, Attila Simon
>  wrote:
> Hi,
> It might've been my mobile client for gmail but couldn't find the flume
> conf attached. Could you please double check?
> Cheers,
> Attila
>
> On Monday, 17 October 2016, Dheeraj Rokade 
> wrote:
>
>> Hello Team,
>>
>> My development activity is stuck due to below error in the Apache agent.
>>
>> *org.apache.flume.ChannelException: Unable to put batch on required
>> channel: FileChannel file-channel-1 *
>>
>> I have the following configuration
>> One Source - *Kafka*
>> Three channels - *File channel*
>> Four Sinks - *HDFS Sinks*
>>
>> Attached is the flume agent config file and below is the error message
>>
>> Things tried
>> 1) Lowering or increasing the batch size for Kafka source and Hive HDFS
>> sink
>> 2) Lowering or increasing the TransactionCapacity and Capacity for File
>> channel
>> 3) With some help from the net, tried keeping File channel
>> TransactionCapacity to be 5x of Kafka and HDFS Batch size but in vain.
>>
>> Please help advise
>>
>> Error message in detail :
>> 16/10/17 18:06:08 ERROR kafka.KafkaSource: KafkaSource EXCEPTION, {}
>> org.apache.flume.ChannelException: Unable to put batch on required
>> channel: FileChannel file-channel-1 { dataDirs:
>> [/home/dheeraj.rokade/flume/.flume/file-channel1/data] }
>>at org.apache.flume.channel.ChannelProcessor.processEventBatch(
>> ChannelProcessor.java:200)
>>at org.apache.flume.source.kafka.KafkaSource.process(
>> KafkaSource.java:123)
>>at org.apache.flume.source.PollableSourceRunner$PollingRunner.run(
>> PollableSourceRunner.java:139)
>>at java.lang.Thread.run(Thread.java:745)
>> Caused by: java.lang.NullPointerException
>>at org.apache.flume.channel.file.proto.ProtosFactory$
>> FlumeEventHeader$Builder.setValue(ProtosFactory.java:7415)
>>at org.apache.flume.channel.file.Put.writeProtos(Put.java:85)
>>at org.apache.flume.channel.file.TransactionEventRecord.
>> toByteBuffer(TransactionEventRecord.java:174)
>>at org.apache.flume.channel.file.Log.put(Log.java:642)
>>at org.apache.flume.channel.file.FileChannel$
>> FileBackedTransaction.doPut(FileChannel.java:468)
>>at org.apache.flume.channel.BasicTransactionSemantics.put(
>> BasicTransactionSemantics.java:93)
>>at org.apache.flume.channel.BasicChannelSemantics.put(
>> BasicChannelSemantics.java:80)
>>at org.apache.flume.channel.ChannelProcessor.processEventBatch(
>> ChannelProcessor.java:189)
>>... 3 more
>> 16/10/17 18:06:13 ERROR kafka.KafkaSource: KafkaSource EXCEPTION, {}
>> org.apache.flume.ChannelException: Unable to put batch on required
>> channel: FileChannel file-channel-1 { dataDirs:
>> [/home/dheeraj.rokade/flume/.flume/file-channel1/data] }
>>at org.apache.flume.channel.ChannelProcessor.processEventBatch(
>> ChannelProcessor.java:200)
>>at org.apache.flume.source.kafka.KafkaSource.process(
>> KafkaSource.java:123)
>>at org.apache.flume.source.PollableSourceRunner$PollingRunner.run(
>> PollableSourceRunner.java:139)
>>at java.lang.Thread.run(Thread.java:745)
>> Caused by: java.lang.NullPointerException
>>at org.apache.flume.channel.file.proto.ProtosFactory$
>> FlumeEventHeader$Builder.setValue(ProtosFactory.java:7415)
>>at org.apache.flume.channel.file.Put.writeProtos(Put.java:85)
>>at org.apache.flume.channel.file.TransactionEventRecord.
>> toByteBuffer(TransactionEventRecord.java:174)
>>at org.apache.flume.channel.file.Log.put(Log.java:642)
>>at org.apache.flume.channel.file.FileChannel$
>> FileBackedTransaction.doPut(FileChannel.java:468)
>>at org.apache.flume.channel.BasicTransactionSemantics.put(
>> BasicTransactionSemantics.java:93)
>>at org.apache.flume.channel.BasicChannelSemantics.put(
>> BasicChannelSemantics.java:80)
>>at org.apache.flume.channel.ChannelProcessor.processEventBatch(
>> ChannelProcessor.java:189)
>>... 3 more
>
>>
>>
>
> --
>
> *Attila Simon*
> Software Engineer
> Email:  s...@cloudera.com
>
> [image: Cloudera Inc.]
>

Re: [RESULT] Flume 1.7.0 release vote

[Balazs Donat Bessenyei]

Thank you for the help, Mike!

On Mon, Oct 17, 2016 at 1:53 PM, Mike Percy <mpe...@apache.org> wrote:
> Thanks for running the vote, Donat!
>
> I will help to deploy the staged artifacts.
>
> Mike
>
> On Mon, Oct 17, 2016 at 10:57 AM, Balazs Donat Bessenyei <
> bes...@cloudera.com> wrote:
>
>> The release vote for Apache Flume 1.7.0 has been completed in this
>> thread: https://lists.apache.org/thread.html/
>> 3cf114125dbea6e662b69d1312c34184690af327900e8dcccea5edde@%
>> 3Cdev.flume.apache.org%3E
>>
>> The vote has received 3 binding +1 votes from the following PMC members:
>> Mike Percy
>> Hari Shreedharan
>> Brock Noland
>>
>> Four non-binding +1 votes were received from:
>> Denes Arvay
>> Bessenyei Balázs Donát
>> Lior Zeno
>> Attila Simon
>>
>> No +0 or -1 votes were received.
>>
>> Since three +1 votes were received from the PMC with no -1 votes, the
>> vote passes and Flume 1.7.0 RC2 will be promoted to the Flume 1.7.0
>> release.
>>
>> Thanks to all who voted!
>>
>>
>> Thank you,
>>
>> Donat
>>

[RESULT] Flume 1.7.0 release vote

[Balazs Donat Bessenyei]

The release vote for Apache Flume 1.7.0 has been completed in this
thread: 
https://lists.apache.org/thread.html/3cf114125dbea6e662b69d1312c34184690af327900e8dcccea5edde@%3Cdev.flume.apache.org%3E

The vote has received 3 binding +1 votes from the following PMC members:
Mike Percy
Hari Shreedharan
Brock Noland

Four non-binding +1 votes were received from:
Denes Arvay
Bessenyei Balázs Donát
Lior Zeno
Attila Simon

No +0 or -1 votes were received.

Since three +1 votes were received from the PMC with no -1 votes, the
vote passes and Flume 1.7.0 RC2 will be promoted to the Flume 1.7.0
release.

Thanks to all who voted!


Thank you,

Donat

Re: [VOTE] Release Apache Flume version 1.7.0 RC2

[Balazs Donat Bessenyei]

This vote is now closed.

I will send out the results in a separate email.

Thanks to all who voted!


Thank you,

Donat

On Mon, Oct 17, 2016 at 3:31 AM, Brock Noland <br...@apache.org> wrote:
> +1 (binding)
>
> Checked sigs and hashes, manual inspection of various other aspects.
>
> On Thu, Oct 13, 2016 at 12:54 PM, Hari Shreedharan <hshreedha...@apache.org>
> wrote:
>
>> +1 (binding)
>>
>> Signatures and checksums look good
>> Top level files are all good.
>> Build runs fine, simple agent with Seq source, memory channel and HDFS
>> sink run fine as well.
>>
>> On Thu, Oct 13, 2016 at 8:42 AM, Mike Percy <mpe...@apache.org> wrote:
>> > +1 (binding)
>> >
>> > There are some flaky tests which are listed below but I don't think they
>> > are release blockers.
>> >
>> > I performed the following checks:
>> >
>> > Binary convenience artifact:
>> > * Signature and checksums match
>> > * LICENSE, NOTICE, and README.md files in the binary convenience artifact
>> > look accurate and complete relative to the jars in lib/
>> > * Ran a very quick test with the binary artifact and it
>> > worked: ./bin/flume-ng agent -c conf -f
>> > conf/flume-conf.properties.template -n
>> > agent -Dflume.root.logger=DEBUG,console
>> > * Checked that the documentation in docs/ renders: Flume User Guide and
>> > Flume Dev Guide are OK. Also spot-checked that the new Kafka security
>> > documentation was included in the User Guide
>> >
>> > Source artifact:
>> > * Signature and checksums match
>> > * Built Flume from the source artifact using Oracle 1.7.0_80 on Ubuntu
>> > Linux 16.04, sanity tested the resulting binary using the above method
>> and
>> > it worked
>> > * RAT checks passed
>> > * Built a new source artifact out of the official source artifact and
>> > compiled it
>> > * I ran the unit tests. Most passed but the below two failed. These are
>> > flaky tests (we have a bunch of them in Flume) so I think it's fine not
>> to
>> > block the release on them.
>> >   * TestExecSource.testMonitoredCounterGroup - looks like a racy test
>> >   * TestSpillableMemoryChannel - didn't investigate
>> >
>> > RC2 looks good to me.
>> >
>> > Thanks for running this release, Donat!
>> >
>> > Mike
>> >
>> > On Wed, Oct 12, 2016 at 9:29 PM, Balazs Donat Bessenyei <
>> bes...@cloudera.com
>> >> wrote:
>> >
>> >> Hi All,
>> >>
>> >> This is the tenth release for Apache Flume as a top-level project,
>> >> version 1.7.0. We are voting on release candidate RC2.
>> >>
>> >> It fixes the following issues:
>> >>   https://raw.githubusercontent.com/apache/flume/flume-1.7/CHANGELOG
>> >>
>> >> *** Please cast your vote within the next 72 hours ***
>> >>
>> >> The tarball (*.tar.gz), signature (*.asc), and checksums (*.md5, *.sha1)
>> >> for the source and binary artifacts can be found here:
>> >>   http://people.apache.org/~bessbd/apache-flume-1.7.0-rc2/
>> >>
>> >> Maven staging repo:
>> >>   https://repository.apache.org/content/repositories/
>> orgapacheflume-1020/
>> >>
>> >> The tag to be voted on:
>> >>   https://git-wip-us.apache.org/repos/asf?p=flume.git;a=
>> commit;h=511d868
>> >>
>> >> Flume's KEYS file containing PGP keys we use to sign the release:
>> >>   https://www.apache.org/dist/flume/KEYS
>> >>
>> >>
>> >> Thank you,
>> >>
>> >> Donat
>> >>
>>

Re: Enabling Travis-CI on Flume

[Balazs Donat Bessenyei]

If the same function can be achieved with Jenkins and it's easy
(+quick) to set up, I'm totally happy with that.

What do we have to do to enable Jenkins builds on PR-s?

On Fri, Oct 14, 2016 at 2:05 PM, Lior Zeno <liorz...@gmail.com> wrote:
> There are ways to do the same with Jenkins, for instance, see this SO
> thread
> http://stackoverflow.com/questions/37661602/how-to-set-up-a-github-pull-request-build-in-a-jenkinsfile
>
> On Fri, Oct 14, 2016 at 11:09 AM, Balazs Donat Bessenyei <
> bes...@cloudera.com> wrote:
>
>> My primary reason for Travis (vs. Jenkins) was that I have experience with
>> it.
>>
>> And it leaves these happy little checkmarks:
>> https://github.com/sebastianbergmann/phpunit/pull/1051/commits on the
>> commits and messages as seen on
>> https://github.com/apache/hive/pull/107 .
>>
>> Jenkins is probably configurable to achieve similar function. However,
>> I have no idea how to do such. (And could not find an example when I
>> did a quick search.)
>>
>> Are there any disadvantages of enabling Travis on Flume?
>>
>>
>> Thank you,
>>
>> Donat
>>
>> On Thu, Oct 13, 2016 at 6:06 PM, Lior Zeno <liorz...@gmail.com> wrote:
>> > Jenkins can do PRs as well. If we can upgrade Jenkins to 2.0, we will be
>> > able to define the build step via Jenkinsfile which becomes very similar
>> to
>> > Travis.
>> > Is there any reason to prefer Travis over Jenkins in our case?
>> >
>> > On Thu, Oct 13, 2016 at 7:01 PM, Balazs Donat Bessenyei <
>> bes...@cloudera.com
>> >> wrote:
>> >
>> >> Hi All,
>> >>
>> >> Having something that checks proposed patches (PR-s especially)
>> >> automatically would help a lot with the development on Flume.
>> >>
>> >> I think, Travis-CI could be an easy solution and (afaik) we'd only have
>> to
>> >> ask infra to enable it for us.
>> >>
>> >> Please, let me know your thoughts.
>> >>
>> >> Thank you,
>> >>
>> >> Donat
>> >>
>>

Re: Enabling Travis-CI on Flume

[Balazs Donat Bessenyei]

I only wanted to have (at least) mvn clean install -DskipTests run, to
help the reviews in such a way that the patch
* doesn't break the build,
* passes checkstyle,
* passes RAT checks.

I think, that would save time and cause less frustration for everyone.

That would fit in the 50 min time limit, I've checked it on my fork.

On Fri, Oct 14, 2016 at 1:29 PM, Hari Shreedharan
<hshreedha...@apache.org> wrote:
> I think the free version of Travis has a 50 minute limit on the total build
> time and last I checked flume full build took longer than that. Is there a
> way to workaround that? I think I have a branch somewhere on GitHub that I
> experimented with.
>
> One interesting possibility with Travis is to integrate code coverage
> reports into PR builds, which is pretty nice.
> On Fri, Oct 14, 2016 at 3:09 AM Attila Simon <s...@cloudera.com> wrote:
>
>> +1 on jenkins, and keeping our build infra as simple and intuitive as
>> possible
>> If migrating everything - we currently do with Jenkins - to Travis and
>> abandon Jenkins than I would be fine with that as well.
>>
>>
>> *Attila Simon*
>> Software Engineer
>> Email:   s...@cloudera.com
>>
>> [image: Cloudera Inc.]
>>
>> On Fri, Oct 14, 2016 at 10:09 AM, Balazs Donat Bessenyei <
>> bes...@cloudera.com> wrote:
>>
>> > My primary reason for Travis (vs. Jenkins) was that I have experience
>> with
>> > it.
>> >
>> > And it leaves these happy little checkmarks:
>> > https://github.com/sebastianbergmann/phpunit/pull/1051/commits on the
>> > commits and messages as seen on
>> > https://github.com/apache/hive/pull/107 .
>> >
>> > Jenkins is probably configurable to achieve similar function. However,
>> > I have no idea how to do such. (And could not find an example when I
>> > did a quick search.)
>> >
>> > Are there any disadvantages of enabling Travis on Flume?
>> >
>> >
>> > Thank you,
>> >
>> > Donat
>> >
>> > On Thu, Oct 13, 2016 at 6:06 PM, Lior Zeno <liorz...@gmail.com> wrote:
>> > > Jenkins can do PRs as well. If we can upgrade Jenkins to 2.0, we will
>> be
>> > > able to define the build step via Jenkinsfile which becomes very
>> similar
>> > to
>> > > Travis.
>> > > Is there any reason to prefer Travis over Jenkins in our case?
>> > >
>> > > On Thu, Oct 13, 2016 at 7:01 PM, Balazs Donat Bessenyei <
>> > bes...@cloudera.com
>> > >> wrote:
>> > >
>> > >> Hi All,
>> > >>
>> > >> Having something that checks proposed patches (PR-s especially)
>> > >> automatically would help a lot with the development on Flume.
>> > >>
>> > >> I think, Travis-CI could be an easy solution and (afaik) we'd only
>> have
>> > to
>> > >> ask infra to enable it for us.
>> > >>
>> > >> Please, let me know your thoughts.
>> > >>
>> > >> Thank you,
>> > >>
>> > >> Donat
>> > >>
>> >
>>

Re: Enabling Travis-CI on Flume

[Balazs Donat Bessenyei]

My primary reason for Travis (vs. Jenkins) was that I have experience with it.

And it leaves these happy little checkmarks:
https://github.com/sebastianbergmann/phpunit/pull/1051/commits on the
commits and messages as seen on
https://github.com/apache/hive/pull/107 .

Jenkins is probably configurable to achieve similar function. However,
I have no idea how to do such. (And could not find an example when I
did a quick search.)

Are there any disadvantages of enabling Travis on Flume?


Thank you,

Donat

On Thu, Oct 13, 2016 at 6:06 PM, Lior Zeno <liorz...@gmail.com> wrote:
> Jenkins can do PRs as well. If we can upgrade Jenkins to 2.0, we will be
> able to define the build step via Jenkinsfile which becomes very similar to
> Travis.
> Is there any reason to prefer Travis over Jenkins in our case?
>
> On Thu, Oct 13, 2016 at 7:01 PM, Balazs Donat Bessenyei <bes...@cloudera.com
>> wrote:
>
>> Hi All,
>>
>> Having something that checks proposed patches (PR-s especially)
>> automatically would help a lot with the development on Flume.
>>
>> I think, Travis-CI could be an easy solution and (afaik) we'd only have to
>> ask infra to enable it for us.
>>
>> Please, let me know your thoughts.
>>
>> Thank you,
>>
>> Donat
>>

Re: Flume bechmarks

[Balazs Donat Bessenyei]

I have just proposed enabling Travis on a different thread. That should
help with this. (Having a separate machine would be best, but I don't know
how we could get one. I'll do the homework for this.)

On Oct 13, 2016 5:57 PM, "Lior Zeno" <liorz...@gmail.com> wrote:

> Maybe getting an isolated environment? The CI environment might be shared
> among multiple users, adding too much noise to the performance test.
>
> On Thu, Oct 13, 2016 at 6:53 PM, Balazs Donat Bessenyei <
> bes...@cloudera.com
> > wrote:
>
> > +1
> >
> > I think this is a good idea!
> >
> > How can I help with setting it up?
> >
> > On Oct 13, 2016 5:20 PM, "Lior Zeno" <liorz...@gmail.com> wrote:
> >
> > > Hi All,
> > >
> > > Monitoring Flume's performance over time is an important step in every
> > > production-level application.  Benchmarking Flume on a nightly basis
> has
> > > the following advantages:
> > >
> > > * Better understanding of Flume's bottlenecks.
> > > * Allow users to compare the performance of different solutions, such
> as
> > > Logstash and Fluentd.
> > > * Better understanding of the influence of recent commits on
> performance.
> > >
> > > Logstash already conducts various performance tests, more details in
> this
> > > link:
> > > http://logstash-benchmarks.elastic.co/
> > >
> > > I propose adding a few micro-benchmarks showing Flume's TPS vs date (of
> > > course, in the ideal case where the input and/or output do not
> bottleneck
> > > the system), e.g. using the SeqGen source.
> > >
> > > Thoughts?
> > >
> > > Thanks
> > >
> >
>

Enabling Travis-CI on Flume

[Balazs Donat Bessenyei]

Hi All,

Having something that checks proposed patches (PR-s especially)
automatically would help a lot with the development on Flume.

I think, Travis-CI could be an easy solution and (afaik) we'd only have to
ask infra to enable it for us.

Please, let me know your thoughts.

Thank you,

Donat

Re: Flume bechmarks

[Balazs Donat Bessenyei]

+1

I think this is a good idea!

How can I help with setting it up?

On Oct 13, 2016 5:20 PM, "Lior Zeno"  wrote:

> Hi All,
>
> Monitoring Flume's performance over time is an important step in every
> production-level application.  Benchmarking Flume on a nightly basis has
> the following advantages:
>
> * Better understanding of Flume's bottlenecks.
> * Allow users to compare the performance of different solutions, such as
> Logstash and Fluentd.
> * Better understanding of the influence of recent commits on performance.
>
> Logstash already conducts various performance tests, more details in this
> link:
> http://logstash-benchmarks.elastic.co/
>
> I propose adding a few micro-benchmarks showing Flume's TPS vs date (of
> course, in the ideal case where the input and/or output do not bottleneck
> the system), e.g. using the SeqGen source.
>
> Thoughts?
>
> Thanks
>

Re: [VOTE] Release Apache Flume version 1.7.0 RC2

[Balazs Donat Bessenyei]

Hi All,

+1 for the RC2

* md5s, sha1s  match
* signatures are correct
* mvn clean install -DskipTests runs successfully in the src ; RAT passes
* binary works
* LICENSE, NOTICE and README.md files are in the tar.gz-s


Donat

On Thu, Oct 13, 2016 at 3:10 PM, Denes Arvay <de...@cloudera.com> wrote:
> Hi,
>
> +1 for the RC2
> - checksums and signatures match
> - source package successfully built (but skipped the tests), rat and
> checkstyle pass
> - I was able to start up flume extracted from the binary package with the
> sample configuration
>
> Denes
>
> On Wed, Oct 12, 2016 at 9:29 PM Balazs Donat Bessenyei <bes...@cloudera.com>
> wrote:
>
> Hi All,
>
>
>
> This is the tenth release for Apache Flume as a top-level project,
>
> version 1.7.0. We are voting on release candidate RC2.
>
>
>
> It fixes the following issues:
>
>   https://raw.githubusercontent.com/apache/flume/flume-1.7/CHANGELOG
>
>
>
> *** Please cast your vote within the next 72 hours ***
>
>
>
> The tarball (*.tar.gz), signature (*.asc), and checksums (*.md5, *.sha1)
>
> for the source and binary artifacts can be found here:
>
>   http://people.apache.org/~bessbd/apache-flume-1.7.0-rc2/
>
>
>
> Maven staging repo:
>
>   https://repository.apache.org/content/repositories/orgapacheflume-1020/
>
>
>
> The tag to be voted on:
>
>   https://git-wip-us.apache.org/repos/asf?p=flume.git;a=commit;h=511d868
>
>
>
> Flume's KEYS file containing PGP keys we use to sign the release:
>
>   https://www.apache.org/dist/flume/KEYS
>
>
>
>
>
> Thank you,
>
>
>
> Donat

[VOTE] Release Apache Flume version 1.7.0 RC2

[Balazs Donat Bessenyei]

Hi All,

This is the tenth release for Apache Flume as a top-level project,
version 1.7.0. We are voting on release candidate RC2.

It fixes the following issues:
  https://raw.githubusercontent.com/apache/flume/flume-1.7/CHANGELOG

*** Please cast your vote within the next 72 hours ***

The tarball (*.tar.gz), signature (*.asc), and checksums (*.md5, *.sha1)
for the source and binary artifacts can be found here:
  http://people.apache.org/~bessbd/apache-flume-1.7.0-rc2/

Maven staging repo:
  https://repository.apache.org/content/repositories/orgapacheflume-1020/

The tag to be voted on:
  https://git-wip-us.apache.org/repos/asf?p=flume.git;a=commit;h=511d868

Flume's KEYS file containing PGP keys we use to sign the release:
  https://www.apache.org/dist/flume/KEYS


Thank you,

Donat

[CANCEL] [VOTE] Release Apache Flume version 1.7.0 RC1

[Balazs Donat Bessenyei]

Hi All,

This vote is cancelled.

I'm about to send an e-mail about RC2.


Thank you,

Donat


On Tue, Oct 11, 2016 at 7:23 PM, Balazs Donat Bessenyei
<bes...@cloudera.com> wrote:
> Hi All,
>
> This is the tenth release for Apache Flume as a top-level project,
> version 1.7.0. We are voting on release candidate RC1.
>
> It fixes the following issues:
>   https://raw.githubusercontent.com/apache/flume/flume-1.7/CHANGELOG
>
> *** Please cast your vote within the next 72 hours ***
>
> The tarball (*.tar.gz), signature (*.asc), and checksums (*.md5, *.sha1)
> for the source and binary artifacts can be found here:
>   http://people.apache.org/~bessbd/apache-flume-1.7.0-rc1/
>
> Maven staging repo:
>   https://repository.apache.org/content/repositories/orgapacheflume-1018/
>
> The tag to be voted on:
>   https://git-wip-us.apache.org/repos/asf?p=flume.git;a=commit;h=1a62453
>
> Flume's KEYS file containing PGP keys we use to sign the release:
>   https://www.apache.org/dist/flume/KEYS
>
>
> Thank you,
>
> Donat

Re: [VOTE] Release Apache Flume version 1.7.0 RC1

[Balazs Donat Bessenyei]

Hi All,

It has come to my attention that the source artifact is corrupted.

I'll be sending an e-mail with information about rc2 soon.

Please, let me know if you found any problems besides the corrupted artifact.


Thank you,

Donat

On Tue, Oct 11, 2016 at 7:23 PM, Balazs Donat Bessenyei
<bes...@cloudera.com> wrote:
> Hi All,
>
> This is the tenth release for Apache Flume as a top-level project,
> version 1.7.0. We are voting on release candidate RC1.
>
> It fixes the following issues:
>   https://raw.githubusercontent.com/apache/flume/flume-1.7/CHANGELOG
>
> *** Please cast your vote within the next 72 hours ***
>
> The tarball (*.tar.gz), signature (*.asc), and checksums (*.md5, *.sha1)
> for the source and binary artifacts can be found here:
>   http://people.apache.org/~bessbd/apache-flume-1.7.0-rc1/
>
> Maven staging repo:
>   https://repository.apache.org/content/repositories/orgapacheflume-1018/
>
> The tag to be voted on:
>   https://git-wip-us.apache.org/repos/asf?p=flume.git;a=commit;h=1a62453
>
> Flume's KEYS file containing PGP keys we use to sign the release:
>   https://www.apache.org/dist/flume/KEYS
>
>
> Thank you,
>
> Donat

[VOTE] Release Apache Flume version 1.7.0 RC1

[Balazs Donat Bessenyei]

Hi All,

This is the tenth release for Apache Flume as a top-level project,
version 1.7.0. We are voting on release candidate RC1.

It fixes the following issues:
  https://raw.githubusercontent.com/apache/flume/flume-1.7/CHANGELOG

*** Please cast your vote within the next 72 hours ***

The tarball (*.tar.gz), signature (*.asc), and checksums (*.md5, *.sha1)
for the source and binary artifacts can be found here:
  http://people.apache.org/~bessbd/apache-flume-1.7.0-rc1/

Maven staging repo:
  https://repository.apache.org/content/repositories/orgapacheflume-1018/

The tag to be voted on:
  https://git-wip-us.apache.org/repos/asf?p=flume.git;a=commit;h=1a62453

Flume's KEYS file containing PGP keys we use to sign the release:
  https://www.apache.org/dist/flume/KEYS


Thank you,

Donat

Re: [DISCUSS] Flume 1.7 release plan

[Balazs Donat Bessenyei]

Hi All,

I moved all the open 1.7 tickets to 1.8, branched 1.7 and pushed the
tag release-1.7.0-rc1

I am going to create the rc1 release artifacts in the next 12 hours
and I'll start a vote thread at that time.


Donat

On Fri, Oct 7, 2016 at 2:05 PM, Mike Percy <mpe...@apache.org> wrote:
> +1 from me, keep up the good work guys.
>
> Mike
>
> On Fri, Oct 7, 2016 at 12:59 PM, Balazs Donat Bessenyei <bes...@cloudera.com
>> wrote:
>
>> Considering Attila's request, I'll do the branching on Monday and the
>> first RC soon after that.
>>
>> Is that okay for everyone?
>>
>>
>> Thank you,
>>
>> Donat
>>
>> On Fri, Oct 7, 2016 at 12:53 PM, Attila Simon <s...@cloudera.com> wrote:
>> > Hi,
>> >
>> > Since we have a few patches in flight from me,Tristan,etc let me suggest
>> > giving the weekend for code review rounds and branch on Monday.
>> >
>> > Cheers,
>> > Attila
>> >
>> >
>> > *Attila Simon*
>> > Software Engineer
>> > Email:   s...@cloudera.com
>> >
>> > [image: Cloudera Inc.]
>> >
>> > On Thu, Oct 6, 2016 at 12:05 PM, Balazs Donat Bessenyei <
>> bes...@cloudera.com
>> >> wrote:
>> >
>> >> Thank you all for the responses.
>> >>
>> >> @Lior: I have reviewed the change you have mentioned (FLUME-2994). If
>> >> we manage to have the code done and +1/2-d soon, I'll commit the
>> >> patch.
>> >>
>> >> @Denes: the change looks good to me. I'll leave some time for others
>> >> to review, though.
>> >>
>> >> @Saikat: I've wanted to put integrations in the User Guide for Flume
>> >> for so long. For example Flume with Spark (
>> >> http://spark.apache.org/docs/latest/streaming-flume-integration.html
>> >> ).
>> >> I know it is available on the internet and I don't think we should
>> >> increase redundancy this way, but I'd really like to have such
>> >> examples in the docs.
>> >> What do you all think of this?
>> >>
>> >> Misc: any help with the tickets I've mentioned in my last e-mail would
>> >> be super welcome. (Even just triaging them and maybe deciding if they
>> >> are not that important as I first thought.)
>> >>
>> >>
>> >> Thank you,
>> >>
>> >> Donat
>> >>
>> >> On Wed, Oct 5, 2016 at 2:03 AM, Hari Shreedharan
>> >> <hshreedha...@apache.org> wrote:
>> >> > +1 for the release, and the branching and RC dates
>> >> >
>> >> > On Tue, Oct 4, 2016 at 10:54 AM Saikat Kanjilal <sxk1...@hotmail.com>
>> >> wrote:
>> >> >
>> >> >> Hi Donat,
>> >> >>
>> >> >> I can pickup any documentation related issues, are there anymore at
>> this
>> >> >> point outside of the FLUME-2971, I would be willing to pickup one of
>> the
>> >> >> ones below but cant guarantee that it'll be fixed in the timeframe
>> for
>> >> 1.7
>> >> >> , let me know the best path for helping.
>> >> >>
>> >> >> Thanks
>> >> >>
>> >> >>
>> >> >> 
>> >> >> From: Balazs Donat Bessenyei <bes...@cloudera.com>
>> >> >> Sent: Tuesday, October 4, 2016 8:01 AM
>> >> >> To: dev@flume.apache.org
>> >> >> Subject: Re: [DISCUSS] Flume 1.7 release plan
>> >> >>
>> >> >> As there have been no objections, I am going to proceed with the plan
>> >> >> I have outlined in my original mail.
>> >> >>
>> >> >> I will try to work a little more on some flaky tests to help with the
>> >> >> release process. (Such as FLUME-3002: Some tests in TestBucketWriter
>> >> >> are flaky.)
>> >> >>
>> >> >> It would be awesome if someone could fix FLUME-2971 (Document
>> Kerberos
>> >> >> set-up for Kafka Sink and Kafka Source).
>> >> >>
>> >> >> Also, I have found a few more tickets that could use a little care:
>> >> >> FLUME-2689 (reloading conf file leads syslogTcpSource not receives
>> any
>> >> >> event) - probably only needs some more reviews,
>> >> >> FLUME-2716 (F

Re: [DISCUSS] Flume 1.7 release plan

[Balazs Donat Bessenyei]

Considering Attila's request, I'll do the branching on Monday and the
first RC soon after that.

Is that okay for everyone?


Thank you,

Donat

On Fri, Oct 7, 2016 at 12:53 PM, Attila Simon <s...@cloudera.com> wrote:
> Hi,
>
> Since we have a few patches in flight from me,Tristan,etc let me suggest
> giving the weekend for code review rounds and branch on Monday.
>
> Cheers,
> Attila
>
>
> *Attila Simon*
> Software Engineer
> Email:   s...@cloudera.com
>
> [image: Cloudera Inc.]
>
> On Thu, Oct 6, 2016 at 12:05 PM, Balazs Donat Bessenyei <bes...@cloudera.com
>> wrote:
>
>> Thank you all for the responses.
>>
>> @Lior: I have reviewed the change you have mentioned (FLUME-2994). If
>> we manage to have the code done and +1/2-d soon, I'll commit the
>> patch.
>>
>> @Denes: the change looks good to me. I'll leave some time for others
>> to review, though.
>>
>> @Saikat: I've wanted to put integrations in the User Guide for Flume
>> for so long. For example Flume with Spark (
>> http://spark.apache.org/docs/latest/streaming-flume-integration.html
>> ).
>> I know it is available on the internet and I don't think we should
>> increase redundancy this way, but I'd really like to have such
>> examples in the docs.
>> What do you all think of this?
>>
>> Misc: any help with the tickets I've mentioned in my last e-mail would
>> be super welcome. (Even just triaging them and maybe deciding if they
>> are not that important as I first thought.)
>>
>>
>> Thank you,
>>
>> Donat
>>
>> On Wed, Oct 5, 2016 at 2:03 AM, Hari Shreedharan
>> <hshreedha...@apache.org> wrote:
>> > +1 for the release, and the branching and RC dates
>> >
>> > On Tue, Oct 4, 2016 at 10:54 AM Saikat Kanjilal <sxk1...@hotmail.com>
>> wrote:
>> >
>> >> Hi Donat,
>> >>
>> >> I can pickup any documentation related issues, are there anymore at this
>> >> point outside of the FLUME-2971, I would be willing to pickup one of the
>> >> ones below but cant guarantee that it'll be fixed in the timeframe for
>> 1.7
>> >> , let me know the best path for helping.
>> >>
>> >> Thanks
>> >>
>> >>
>> >> 
>> >> From: Balazs Donat Bessenyei <bes...@cloudera.com>
>> >> Sent: Tuesday, October 4, 2016 8:01 AM
>> >> To: dev@flume.apache.org
>> >> Subject: Re: [DISCUSS] Flume 1.7 release plan
>> >>
>> >> As there have been no objections, I am going to proceed with the plan
>> >> I have outlined in my original mail.
>> >>
>> >> I will try to work a little more on some flaky tests to help with the
>> >> release process. (Such as FLUME-3002: Some tests in TestBucketWriter
>> >> are flaky.)
>> >>
>> >> It would be awesome if someone could fix FLUME-2971 (Document Kerberos
>> >> set-up for Kafka Sink and Kafka Source).
>> >>
>> >> Also, I have found a few more tickets that could use a little care:
>> >> FLUME-2689 (reloading conf file leads syslogTcpSource not receives any
>> >> event) - probably only needs some more reviews,
>> >> FLUME-2716 (File Channel cannot handle capacity Integer.MAX_VALUE) -
>> >> missing a test,
>> >> FLUME-2461 (memoryChannel bytesRemaining counting error) - missing test
>> >> Patch-less (Needs some investigation and a fix. I am not sure about
>> >> their complexity):
>> >> FLUME-2912 (thrift Sources/Sinks can only authenticate with kerberos
>> >> principal in format with hostname)
>> >> FLUME-2871 (avro sink reset-connection-interval cause
>> >> EventDeliveryException)
>> >> FLUME-2811 (Taildir source doesn't call stop() on graceful shutdown)
>> >>
>> >> However, none of these seem to be a release blocker, so if they are
>> >> not done by Oct 7-10, it would still be fine.
>> >>
>> >>
>> >> Please, let me know your thoughts
>> >>
>> >> Donat
>> >>
>> >> On Mon, Oct 3, 2016 at 3:23 PM, Mike Percy <mpe...@apache.org> wrote:
>> >> > On Fri, Sep 30, 2016 at 7:01 PM, Saikat Kanjilal <sxk1...@hotmail.com
>> >
>> >> > wrote:
>> >> >
>> >> >> I'd be willing to help, I've been heads down on other stuff and have
>> had
>> >> >> to postpone the graph sink implementation (
>> https://issues.apache.org/
>> >> >> jira/browse/FLUME-2035) but am looking to resume work and targeting
>> an
>> >> >> initial implementation by mid November sometime.  In the meantime
>> let me
>> >> >> know how I can get more deeply involved in the next release.
>> >> >>
>> >> >
>> >> > I'd recommend taking a look at things like documentation and filling
>> in
>> >> > gaps so that we don't release with undocumented stuff. Other types of
>> >> > polish seem like a helpful thing to do right before a release as well.
>> >> >
>> >> > Mike
>> >>
>>

Re: [DISCUSS] Flume 1.7 release plan

[Balazs Donat Bessenyei]

Thank you all for the responses.

@Lior: I have reviewed the change you have mentioned (FLUME-2994). If
we manage to have the code done and +1/2-d soon, I'll commit the
patch.

@Denes: the change looks good to me. I'll leave some time for others
to review, though.

@Saikat: I've wanted to put integrations in the User Guide for Flume
for so long. For example Flume with Spark (
http://spark.apache.org/docs/latest/streaming-flume-integration.html
).
I know it is available on the internet and I don't think we should
increase redundancy this way, but I'd really like to have such
examples in the docs.
What do you all think of this?

Misc: any help with the tickets I've mentioned in my last e-mail would
be super welcome. (Even just triaging them and maybe deciding if they
are not that important as I first thought.)


Thank you,

Donat

On Wed, Oct 5, 2016 at 2:03 AM, Hari Shreedharan
<hshreedha...@apache.org> wrote:
> +1 for the release, and the branching and RC dates
>
> On Tue, Oct 4, 2016 at 10:54 AM Saikat Kanjilal <sxk1...@hotmail.com> wrote:
>
>> Hi Donat,
>>
>> I can pickup any documentation related issues, are there anymore at this
>> point outside of the FLUME-2971, I would be willing to pickup one of the
>> ones below but cant guarantee that it'll be fixed in the timeframe for 1.7
>> , let me know the best path for helping.
>>
>> Thanks
>>
>>
>> 
>> From: Balazs Donat Bessenyei <bes...@cloudera.com>
>> Sent: Tuesday, October 4, 2016 8:01 AM
>> To: dev@flume.apache.org
>> Subject: Re: [DISCUSS] Flume 1.7 release plan
>>
>> As there have been no objections, I am going to proceed with the plan
>> I have outlined in my original mail.
>>
>> I will try to work a little more on some flaky tests to help with the
>> release process. (Such as FLUME-3002: Some tests in TestBucketWriter
>> are flaky.)
>>
>> It would be awesome if someone could fix FLUME-2971 (Document Kerberos
>> set-up for Kafka Sink and Kafka Source).
>>
>> Also, I have found a few more tickets that could use a little care:
>> FLUME-2689 (reloading conf file leads syslogTcpSource not receives any
>> event) - probably only needs some more reviews,
>> FLUME-2716 (File Channel cannot handle capacity Integer.MAX_VALUE) -
>> missing a test,
>> FLUME-2461 (memoryChannel bytesRemaining counting error) - missing test
>> Patch-less (Needs some investigation and a fix. I am not sure about
>> their complexity):
>> FLUME-2912 (thrift Sources/Sinks can only authenticate with kerberos
>> principal in format with hostname)
>> FLUME-2871 (avro sink reset-connection-interval cause
>> EventDeliveryException)
>> FLUME-2811 (Taildir source doesn't call stop() on graceful shutdown)
>>
>> However, none of these seem to be a release blocker, so if they are
>> not done by Oct 7-10, it would still be fine.
>>
>>
>> Please, let me know your thoughts
>>
>> Donat
>>
>> On Mon, Oct 3, 2016 at 3:23 PM, Mike Percy <mpe...@apache.org> wrote:
>> > On Fri, Sep 30, 2016 at 7:01 PM, Saikat Kanjilal <sxk1...@hotmail.com>
>> > wrote:
>> >
>> >> I'd be willing to help, I've been heads down on other stuff and have had
>> >> to postpone the graph sink implementation (https://issues.apache.org/
>> >> jira/browse/FLUME-2035) but am looking to resume work and targeting an
>> >> initial implementation by mid November sometime.  In the meantime let me
>> >> know how I can get more deeply involved in the next release.
>> >>
>> >
>> > I'd recommend taking a look at things like documentation and filling in
>> > gaps so that we don't release with undocumented stuff. Other types of
>> > polish seem like a helpful thing to do right before a release as well.
>> >
>> > Mike
>>

Re: [DISCUSS] Flume 1.7 release plan

[Balazs Donat Bessenyei]

As there have been no objections, I am going to proceed with the plan
I have outlined in my original mail.

I will try to work a little more on some flaky tests to help with the
release process. (Such as FLUME-3002: Some tests in TestBucketWriter
are flaky.)

It would be awesome if someone could fix FLUME-2971 (Document Kerberos
set-up for Kafka Sink and Kafka Source).

Also, I have found a few more tickets that could use a little care:
FLUME-2689 (reloading conf file leads syslogTcpSource not receives any
event) - probably only needs some more reviews,
FLUME-2716 (File Channel cannot handle capacity Integer.MAX_VALUE) -
missing a test,
FLUME-2461 (memoryChannel bytesRemaining counting error) - missing test
Patch-less (Needs some investigation and a fix. I am not sure about
their complexity):
FLUME-2912 (thrift Sources/Sinks can only authenticate with kerberos
principal in format with hostname)
FLUME-2871 (avro sink reset-connection-interval cause EventDeliveryException)
FLUME-2811 (Taildir source doesn't call stop() on graceful shutdown)

However, none of these seem to be a release blocker, so if they are
not done by Oct 7-10, it would still be fine.


Please, let me know your thoughts

Donat

On Mon, Oct 3, 2016 at 3:23 PM, Mike Percy  wrote:
> On Fri, Sep 30, 2016 at 7:01 PM, Saikat Kanjilal 
> wrote:
>
>> I'd be willing to help, I've been heads down on other stuff and have had
>> to postpone the graph sink implementation (https://issues.apache.org/
>> jira/browse/FLUME-2035) but am looking to resume work and targeting an
>> initial implementation by mid November sometime.  In the meantime let me
>> know how I can get more deeply involved in the next release.
>>
>
> I'd recommend taking a look at things like documentation and filling in
> gaps so that we don't release with undocumented stuff. Other types of
> polish seem like a helpful thing to do right before a release as well.
>
> Mike

Re: [DISCUSS] Flume 1.7 release plan

[Balazs Donat Bessenyei]

Hi All,

Thank you for all the feedback and responses.

Regarding the open tickets, I had something similar to Mike's idea in
mind: we review the tickets and if there's nothing blocking a release,
we reschedule the unresolved ones for 1.8 or so.

I agree that we should have more frequent releases. Aiming for
something quarterly sounds like a good thing.


Thank you,

Donat


On Fri, Sep 30, 2016 at 8:01 PM, Saikat Kanjilal  wrote:
> I'd be willing to help, I've been heads down on other stuff and have had to 
> postpone the graph sink implementation 
> (https://issues.apache.org/jira/browse/FLUME-2035) but am looking to resume 
> work and targeting an initial implementation by mid November sometime.  In 
> the meantime let me know how I can get more deeply involved in the next 
> release.
>
>
> Regards
>
>
> 
> From: Ralph Goers 
> Sent: Friday, September 30, 2016 10:15 AM
> To: dev@flume.apache.org
> Subject: Re: [DISCUSS] Flume 1.7 release plan
>
> FWIW, missing the train wouldn't be that big a deal if you knew the next 
> release wouldn't be too far out.  A release a quarter would be a nice target.
>
> Ralph
>
>> On Sep 30, 2016, at 8:36 AM, Mike Percy  wrote:
>>
>> On Fri, Sep 30, 2016 at 2:13 PM, Denes Arvay  wrote:
>>
>>> What are your plans regarding to the mentioned 18 issues in patch available
>>> state? Should we decide on a one-by-one basis whether we'd like to include
>>> in the release or should we reschedule all of them to a next release?
>>>
>>
>> I'd personally recommend committing whatever is ready and reasonably
>> low-risk, and whatever doesn't make the deadline should "miss the train"
>> unless it's a particularly critical issue. This approach has worked well
>> for me in the past.
>>
>> Best,
>> Mike
>
>

Labels on issues blocking the Flume 1.7.0 release

[Balazs Donat Bessenyei]

Hello All,

Some fellow collaborators (@denes, @sati) and I would like to help out
with the 1.7.0 release.
In order to do that, we started triaging and reviewing patches. (
https://issues.apache.org/jira/browse/FLUME-2953?filter=12338032 )
As part of this process, we are labelling tickets
(reviewboard-missing, unit-test-missing, docs-missing and
rebase-needed) so that it would be easier to track the progress with
the issues (in a queryable way).

If you have any ideas how this could be improved, please do tell.


Thank you

Donat