Re: [DISCUSS] New Pulsar Documentation Website

[Jai Asher]

It looks really nice and has search functionality as well.

Nice work @Chris.

On Fri, Jul 27, 2018 at 2:18 PM, Sijie Guo  wrote:

> Hi all,
>
> In the past few weeks, Chris Kellogg has been working on a new Pulsar
> documentation website using Docusaurus . It is a
> Facebook open source project, designed for easy to maintain open source
> documentation websites.
>
> It is a very popular and documentation-focused framework and used by a lot
> of open source projects. It addressed a few problems that we had in current
> documentation website.
>
> - Sidebar: sidebar doesn't work well in current website. for example, it is
> impossible to navigate to any documentation page on mobile phones; if you
> have a smaller screen, you can not scroll down the sidebar to navigate to
> the pages at the bottom.
> - Hyperlinks on sections: there are some random characters generated in the
> hyperlinks for markdown sections. It is non-deterministic and will cause
> problems when linking documents.
>
> Docusaurus gives us more benefits than current website.
>
> - documentation focused: Docusaurus is designed for documentation websites.
> so it brings the focus on writing documents in markdown. The framework
> takes care of the rest of stuffs like versioning, sidebar and even
> translations.
> - much better sidebar: sidebar is working well across different browsers
> and mobile phones.
> - simpler hyperlinks: you can just link to other documents using document
> filenames. it is working well for both website and navigating at github
> directly. You don't have to compute any relative links.
> - versioning: it manages all the versioning stuff.
> - translations: it integrates Crowdin for translation contributions.
> - search-box: a search-box integrated with Algolia.
> - edit/translate button: for each document, it has edit button (for english
> language) and translate button (for non-english language). so people can
> quickly contribute documentation changes (such as fixing typos) by clicking
> those buttons. It lowers
> the contribution barrier.
>
>
> Currently the new site is alive under
> https://pulsar.incubator.apache.org/staging/. Please take a look at the
> new
> website, try it out and give us feedbacks.
>
> For contributors, I also wrote an instruction on how to contribute the
> documentation. So you can get a sense on how to contribute documents in the
> new site.
>
> https://github.com/sijie/incubator-pulsar/blob/
> 4a3a938d6ae1d9a5b1ae83d49293513c6e4f1711/site2/README.md
>
> Last, big shout out to Chris for his great work on this!
>
> - Sijie
>

Re: [VOTE] Pulsar Release 1.22.1-incubating Candidate 2

[Jai Asher]

The vote for releasing Apache Pulsar 1.22.1-incubating is now closed.

With a total of +3 binding votes and no -1 votes, the vote passes.

+1s (binding):
Sijie
Dave
Nozomi


On Fri, Jun 1, 2018 at 10:37 AM, Sijie Guo  wrote:

> On Fri, Jun 1, 2018 at 9:56 AM, Matteo Merli 
> wrote:
>
> > > Thanks Dave. I filed a github issue to take MD5 out of our website as
> > well.
> >
> > The md5 signatures were already taken out (since 2.0) though the website
> > still has a link to the non-existing md5 signatures
> >
>
> yes we need to update the website.
>
>
> >
> > On Fri, Jun 1, 2018 at 9:49 AM Sijie Guo  wrote:
> >
> > > +1 (binding)
> > >
> > > Environment: MacOS 10.13.1
> > >
> > > * check signatures, all are good.
> > > * Bin distribution
> > > - start standalone
> > > - test producer/consumer using pulsar-client and pulsar-perf
> > > * Src distribution
> > > - compile and unit tests
> > > - RAT check
> > > - start standalone
> > > - test producer/consumer using pulsar-client and pulsar-perf
> > >
> > > - Sijie
> > >
> > > On Fri, Jun 1, 2018 at 12:29 AM, Jai Asher 
> > wrote:
> > >
> > > > This is the second release candidate for Apache Pulsar, patch release
> > > > version 1.22.1-incubating.
> > > >
> > > > It fixes the following issues:
> > > > https://github.com/apache/incubator-pulsar/milestone/15?closed=1
> > > >
> > > > *** Please download, test and vote on this release. This vote will
> stay
> > > > open for at least 72 hours ***
> > > >
> > > > Note that we are voting upon the source (tag), binaries are provided
> > for
> > > > convenience.
> > > >
> > > > Source and binary files:
> > > > https://dist.apache.org/repos/dist/dev/incubator/pulsar/
> > > > pulsar-1.22.1-incubating-candidate-2/
> > > >
> > > > SHA-1 checksums:
> > > >
> > > > f2d29aa5e046c5bdefd8f466bce8e9ead80a2e09
> > > > apache-pulsar-1.22.1-incubating-src.tar.gz
> > > > f2704cae22b7fb3c1b72daab6ebe7d484fdfec6b
> > > > apache-pulsar-1.22.1-incubating-bin.tar.gz
> > > >
> > > > Maven staging repo:
> > > > https://repository.apache.org/content/repositories/
> > orgapachepulsar-1018/
> > > >
> > > > The tag to be voted upon:
> > > > v1.22.1-incubating-candidate-1 (c9a369936af3b3ecc663b86ae959a3
> > fbfa627aca)
> > > > https://github.com/apache/incubator-pulsar/releases/tag/
> > > > v1.22.1-incubating-candidate-2
> > > >
> > > > Pulsar's KEYS file containing PGP keys we use to sign the release:
> > > > https://dist.apache.org/repos/dist/release/incubator/pulsar/KEYS
> > > >
> > > > Please download the source package, and follow the README to build
> and
> > > run
> > > > the Pulsar standalone service.
> > > >
> > >
> > --
> > Matteo Merli
> > 
> >
>

[VOTE] Pulsar Release 1.22.1-incubating Candidate 2

[Jai Asher]

This is the second release candidate for Apache Pulsar, patch release
version 1.22.1-incubating.

It fixes the following issues:
https://github.com/apache/incubator-pulsar/milestone/15?closed=1

*** Please download, test and vote on this release. This vote will stay
open for at least 72 hours ***

Note that we are voting upon the source (tag), binaries are provided for
convenience.

Source and binary files:
https://dist.apache.org/repos/dist/dev/incubator/pulsar/pulsar-1.22.1-incubating-candidate-2/

SHA-1 checksums:

f2d29aa5e046c5bdefd8f466bce8e9ead80a2e09
apache-pulsar-1.22.1-incubating-src.tar.gz
f2704cae22b7fb3c1b72daab6ebe7d484fdfec6b
apache-pulsar-1.22.1-incubating-bin.tar.gz

Maven staging repo:
https://repository.apache.org/content/repositories/orgapachepulsar-1018/

The tag to be voted upon:
v1.22.1-incubating-candidate-1 (c9a369936af3b3ecc663b86ae959a3fbfa627aca)
https://github.com/apache/incubator-pulsar/releases/tag/v1.22.1-incubating-candidate-2

Pulsar's KEYS file containing PGP keys we use to sign the release:
https://dist.apache.org/repos/dist/release/incubator/pulsar/KEYS

Please download the source package, and follow the README to build and run
the Pulsar standalone service.

[VOTE] Pulsar Release 1.22.1-incubating Candidate 1

[Jai Asher]

This is the first release candidate for Apache Pulsar, patch release
version 1.22.1-incubating.

It fixes the following issues:
https://github.com/apache/incubator-pulsar/milestone/15?closed=1

*** Please download, test and vote on this release. This vote will stay
open for at least 72 hours ***

Note that we are voting upon the source (tag), binaries are provided for
convenience.

Source and binary files:
https://dist.apache.org/repos/dist/dev/incubator/pulsar/pulsar-1.22.1-incubating-candidate-1/

SHA-1 checksums:
242dd7772acae4bce0f30a6f9be51cb65864f560
apache-pulsar-1.22.1-incubating-bin.tar.gz
c5cf3ef97fde5f8ac613f2a886c3b1379d8bf27d
apache-pulsar-1.22.1-incubating-src.tar.gz

Maven staging repo:
https://repository.apache.org/content/repositories/orgapachepulsar-1016/

The tag to be voted upon:
v1.22.1-incubating-candidate-1 (0093a8024f2be1aea8d85406dca403c3ef01e17c)
https://github.com/apache/incubator-pulsar/releases/tag/v1.22.1-incubating-candidate-1

Pulsar's KEYS file containing PGP keys we use to sign the release:
https://dist.apache.org/repos/dist/release/incubator/pulsar/KEYS

Please download the source package, and follow the README to build
and run the Pulsar standalone service.

Re: Pulsar meetup in June/July

[Jai Asher]

Great idea! +1

On Mon, Apr 30, 2018 at 2:12 PM, Sijie Guo  wrote:

> +1
>
> On Mon, Apr 30, 2018 at 11:29 AM, Joe F  wrote:
>
> > In June, it going to be a year since Pulsar was accepted into the
> > incubator.
> >
> > I'm thinking of a meetup, with a few in-depth talks/presentations, what's
> > new in Pulsar 2.0 etc,   from Pulsar users and developers.  Are people
> > interested in a 1/2day event?   I can arrange for the venue and
> logistics.
> >
> >
> > Joe
> >
>

Re: minor/bugfix release

[Jai Asher]

For patch release 1.22.1, we are using Project: Pulsar 1.22.1 patch Release
<https://github.com/apache/incubator-pulsar/projects/8> to track the
proposed fixes to be backported to 1.22. Kindly go through them and let us
know if you want to add any more fixes to the patch release.

- Jai

On Fri, Apr 20, 2018 at 3:15 AM, Jia Zhai <zhaiji...@gmail.com> wrote:

> +1
>
> On Fri, Apr 20, 2018 at 2:19 PM, Sijie Guo <guosi...@gmail.com> wrote:
>
> > Thank you Jai!
> >
> > - Sijie
> >
> > On Thu, Apr 19, 2018 at 11:42 AM, Jai Asher <jai.ashe...@gmail.com>
> wrote:
> >
> > > +1 to 1.22.1 release - I think a bug fix (patch) release is a great
> idea
> > > especially if we have user-facing bugs. Since I was responsible for
> > > releasing 1.22, I don't mind taking up this task.
> > >
> > >
> > > On Thu, Apr 19, 2018 at 11:34 AM, Sijie Guo <guosi...@gmail.com>
> wrote:
> > >
> > > > Hi all,
> > > >
> > > > I think there are a couple of bugs reported on 1.22.0 when users are
> > > using
> > > > it. Shall we consider doing a 1.22.1 release?
> > > >
> > > > - Sijie
> > > >
> > > > On Wed, Apr 18, 2018 at 2:54 PM, Sijie Guo <guosi...@gmail.com>
> wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > Currently pulsar release schedule is mainly on feature releases.
> Any
> > > idea
> > > > > on bugfix releases?
> > > > >
> > > > > E.g. If there are bugs on 1.22.0, shall we consider releasing
> 1.22.1
> > > > > release with bug fixes?
> > > > >
> > > > > Thoughts?
> > > > >
> > > > > - Sijie
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: minor/bugfix release

[Jai Asher]

+1 to 1.22.1 release - I think a bug fix (patch) release is a great idea
especially if we have user-facing bugs. Since I was responsible for
releasing 1.22, I don't mind taking up this task.


On Thu, Apr 19, 2018 at 11:34 AM, Sijie Guo  wrote:

> Hi all,
>
> I think there are a couple of bugs reported on 1.22.0 when users are using
> it. Shall we consider doing a 1.22.1 release?
>
> - Sijie
>
> On Wed, Apr 18, 2018 at 2:54 PM, Sijie Guo  wrote:
>
> > Hi all,
> >
> > Currently pulsar release schedule is mainly on feature releases. Any idea
> > on bugfix releases?
> >
> > E.g. If there are bugs on 1.22.0, shall we consider releasing 1.22.1
> > release with bug fixes?
> >
> > Thoughts?
> >
> > - Sijie
> >
> >
>

Re: PIP review process - improvement

[Jai Asher]

> Sounds good. We can move the wiki pages content into same git repo and maybe
just keep index with links in wiki.

+1 for this. A good use of google drive would be for these kinds of votes
(Google forms) - hate spamming the inbox for every vote cast.

On Mon, Apr 9, 2018 at 11:31 AM, Sahaya Andrews  wrote:

> +1 from my side as well.
>
> Andrews.
>
> On Mon, Apr 9, 2018 at 9:55 AM, Matteo Merli 
> wrote:
> > Sounds good. We can move the wiki pages content into same git repo and
> > maybe just keep index with links in wiki.
> >
> > On Mon, Apr 9, 2018 at 1:35 AM Ivan Kelly  wrote:
> >
> >> > It will keep the discussion   public, convenient, archived, maintain
> >> history,
> >> > and will be mailed out (with github notifications).
> >> >
> >> > What do you all say?
> >>
> >> +1 for this. This is how BK does it, and I've found it much easier to
> use.
> >>
> >> -Ivan
> >>
> > --
> > Matteo Merli
> > 
>

[ANNOUNCE] Apache Pulsar 1.22.0-incubating released

[Jai Asher]

The Apache Pulsar team is proud to announce Apache Pulsar version
1.22.0-incubating.

This is the fourth Pulsar release after entering the Apache Incubator.

Pulsar is a highly scalable, low latency messaging platform running on
commodity hardware. It provides simple pub-sub semantics over topics,
guaranteed at-least-once delivery of messages, automatic cursor management
for
subscribers, and cross-datacenter replication.

For Pulsar release details and downloads, visit:

https://pulsar.incubator.apache.org/download

Release Notes are at:
https://github.com/apache/incubator-pulsar/releases/tag/v1.22.0-incubating

We would like to thank the contributors that made the release possible.

Regards,

The Pulsar Team

DISCLAIMER:
Apache Pulsar is an effort undergoing incubation at The Apache Software
Foundation (ASF), sponsored by the Apache Incubator PMC. Incubation is
required of all newly accepted projects until a further review indicates
that
the infrastructure, communications, and decision making process have
stabilized in a manner consistent with other successful ASF projects. While
incubation status is not necessarily a reflection of the completeness or
stability

Re: [VOTE] Pulsar 1.22.0-incubating Release Candidate 3

[Jai Asher]

Hi,

The vote for releasing Apache Pulsar 1.22.0-incubating is now closed.

With a total of +6 votes (5 binding) and no -1 votes, the vote passes.

+1s (binding):
Matteo
Sergio
Masahiro
Rajan
Nozomi

+1 (non binding):
Jia

Thanks a lot to all the reviewers for taking the time to validate this
release!

Regards,
Jai

On Fri, Feb 23, 2018 at 9:22 AM, Matteo Merli <mme...@apache.org> wrote:

> +1 (binding)
>
> Environment: MacOS 10.12 & Centos-7.4
>
> Checked:
>  * Signatures, crc
>  * Bin distribution:
>  - NOTICE, README, LICENSE
>  - Start standalone service and producer/consumer test
>  * Src distribution:
>  - NOTICE,  README, LICENSE
>  - Compile and unit tests
>  - Start standalone service
>  * Checked staging maven repository artifacts
>
> Matteo
>
> On Fri, Feb 23, 2018 at 2:44 AM Jia Zhai <zhaiji...@gmail.com> wrote:
>
> > +1 (non-binding)
> >
> > Environment: MacOS 10.13
> >
> > * Bin distribution
> > -checked md5, sha, asc,
> > -start standalone,
> > -producer/consumer worked,
> > * Src distribution
> > -checked md5, sha, asc,
> > -compile and unit tests,
> > -start standalone,
> > -producer/consumer worked
> >
> >
> >
> >
> > On Fri, Feb 23, 2018 at 4:02 PM, Masahiro Sakamoto <
> massa...@yahoo-corp.jp
> > >
> > wrote:
> >
> > > +1 (binding)
> > >
> > > Environment: CentOS 6.7
> > >
> > > * checked md5, sha, asc
> > > * ran standalone
> > > * producer/consumer worked normally
> > > * compiling and unit tests succeeded
> > > * rat check
> > >
> > > - Masahiro
> > >
> > > --
> > > Masahiro Sakamoto
> > > Yahoo Japan Corp.
> > > E-mail: massa...@yahoo-corp.jp
> > > --
> > >
> > > > -Original Message-
> > > > From: Jai Asher [mailto:jai.ashe...@gmail.com]
> > > > Sent: Wednesday, February 21, 2018 1:39 PM
> > > > To: dev@pulsar.incubator.apache.org
> > > > Subject: [VOTE] Pulsar 1.22.0-incubating Release Candidate 3
> > > >
> > > > This is the fourth release candidate for Apache Pulsar, version
> > > > 1.22.0-incubating.
> > > >
> > > > It fixes the following issues:
> > > > https://github.com/apache/incubator-pulsar/milestone/11?closed=1
> > > >
> > > > *** Please download, test and vote by Friday, Feb 23, 2018, 10:00
> GMT.
> > > >
> > > > Note that we are voting upon the source (tag), binaries are provided
> > for
> > > > convenience.
> > > >
> > > > Source and binary files:
> > > > https://dist.apache.org/repos/dist/dev/incubator/pulsar/
> pulsar-1.22.0-
> > > > incubating-candidate-3/
> > > >
> > > > Maven staging repo:
> > > > https://repository.apache.org/content/repositories/
> orgapachepulsar-101
> > > > 0/
> > > >
> > > > The tag to be voted upon:
> > > > v1.22.0-incubating-candidate-3
> > > > (5d14788e510faec23fd8ed189ed343e93b489dda)
> > > > https://github.com/apache/incubator-pulsar/releases/tag/
> v1.22.0-incuba
> > > > ting-candidate-3
> > > >
> > > > Pulsar's KEYS file containing PGP keys we use to sign the release:
> > > > https://dist.apache.org/repos/dist/release/incubator/pulsar/KEYS
> > > >
> > > > Please download the source package, and follow the README to build
> and
> > > run
> > > > the Pulsar standalone service.
> > >
> >
> --
> Matteo Merli
> <mme...@apache.org>
>

Re: [VOTE] Pulsar 1.22.0-incubating Release Candidate 2

[Jai Asher]

Closing this thread since we have a new release candidate.
Subject of the new voting mail chain "Pulsar 1.22.0-incubating Release
Candidate 3"

On Tue, Feb 20, 2018 at 1:38 PM, Jai Asher <jai.ashe...@gmail.com> wrote:

> This is the third release candidate for Apache Pulsar, version
> 1.22.0-incubating.
>
> It fixes the following issues:
> https://github.com/apache/incubator-pulsar/milestone/11?closed=1
>
> *** Please download, test and vote by Friday, Feb 23, 2018, 10:00 GMT.
>
> Note that we are voting upon the source (tag), binaries are provided for
> convenience.
>
> Source and binary files:
> https://dist.apache.org/repos/dist/dev/incubator/pulsar/
> pulsar-1.22.0-incubating-candidate-2/
>
> Maven staging repo:
> https://repository.apache.org/content/repositories/orgapachepulsar-1009/
>
> The tag to be voted upon:
> v1.22.0-incubating-candidate-2 (44fd82654fbf19f31a708b4c9d6ce1681e32a5fb)
> https://github.com/apache/incubator-pulsar/releases/tag/
> v1.22.0-incubating-candidate-2
>
> Pulsar's KEYS file containing PGP keys we use to sign the release:
> https://dist.apache.org/repos/dist/release/incubator/pulsar/KEYS
>
> Please download the source package, and follow the README to build
> and run the Pulsar standalone service.
>

[VOTE] Pulsar 1.22.0-incubating Release Candidate 3

[Jai Asher]

This is the fourth release candidate for Apache Pulsar, version
1.22.0-incubating.

It fixes the following issues:
https://github.com/apache/incubator-pulsar/milestone/11?closed=1

*** Please download, test and vote by Friday, Feb 23, 2018, 10:00 GMT.

Note that we are voting upon the source (tag), binaries are provided for
convenience.

Source and binary files:
https://dist.apache.org/repos/dist/dev/incubator/pulsar/pulsar-1.22.0-incubating-candidate-3/

Maven staging repo:
https://repository.apache.org/content/repositories/orgapachepulsar-1010/

The tag to be voted upon:
v1.22.0-incubating-candidate-3 (5d14788e510faec23fd8ed189ed343e93b489dda)
https://github.com/apache/incubator-pulsar/releases/tag/v1.22.0-incubating-candidate-3

Pulsar's KEYS file containing PGP keys we use to sign the release:
https://dist.apache.org/repos/dist/release/incubator/pulsar/KEYS

Please download the source package, and follow the README to build
and run the Pulsar standalone service.

[VOTE] Pulsar 1.22.0-incubating Release Candidate 2

[Jai Asher]

This is the third release candidate for Apache Pulsar, version
1.22.0-incubating.

It fixes the following issues:
https://github.com/apache/incubator-pulsar/milestone/11?closed=1

*** Please download, test and vote by Friday, Feb 23, 2018, 10:00 GMT.

Note that we are voting upon the source (tag), binaries are provided for
convenience.

Source and binary files:
https://dist.apache.org/repos/dist/dev/incubator/pulsar/pulsar-1.22.0-incubating-candidate-2/

Maven staging repo:
https://repository.apache.org/content/repositories/orgapachepulsar-1009/

The tag to be voted upon:
v1.22.0-incubating-candidate-2 (44fd82654fbf19f31a708b4c9d6ce1681e32a5fb)
https://github.com/apache/incubator-pulsar/releases/tag/v1.22.0-incubating-candidate-2

Pulsar's KEYS file containing PGP keys we use to sign the release:
https://dist.apache.org/repos/dist/release/incubator/pulsar/KEYS

Please download the source package, and follow the README to build
and run the Pulsar standalone service.

Re: [VOTE] Pulsar 1.22.0-incubating Release Candidate 1

[Jai Asher]

Thanks, Matteo :-)
I will have another release candidate ready by the End of Day.

On Mon, Feb 19, 2018 at 10:49 AM, Matteo Merli <mme...@apache.org> wrote:

> Jai,
>
> I have merged the mentioned fix in master & 1.22 branch.
>
> On Mon, Feb 19, 2018 at 9:53 AM Dave Fisher <dave2w...@comcast.net> wrote:
>
> > Hi -
> >
> > Given this comment I’ll wait until tomorrow to review the release.
> >
> > Regards,
> > Dave
> >
> > > On Feb 19, 2018, at 9:27 AM, Matteo Merli <mme...@apache.org> wrote:
> > >
> > > Jai, the artifacts look good.
> > >
> > > Though I think we should squeeze in the fix for
> > > https://github.com/apache/incubator-pulsar/pull/1251
> > > That makes partitions unusable on non-persistent topics.
> > >
> > > Matteo
> > >
> > > On Sat, Feb 17, 2018 at 1:13 AM Jai Asher <jai.ashe...@gmail.com>
> wrote:
> > >
> > >> This is the second release candidate for Apache Pulsar, version
> > >> 1.22.0-incubating.
> > >>
> > >> It fixes the following issues:
> > >> https://github.com/apache/incubator-pulsar/milestone/11?closed=1
> > >>
> > >> *** Please download, test and vote by Tuesday, Feb 20th, 2018, 10:00
> > GMT.
> > >>
> > >> Note that we are voting upon the source (tag), binaries are provided
> for
> > >> convenience.
> > >>
> > >> Source and binary files:
> > >>
> > >>
> > https://dist.apache.org/repos/dist/dev/incubator/pulsar/
> pulsar-1.22.0-incubating-candidate-1/
> > >>
> > >> Maven staging repo:
> > >>
> > https://repository.apache.org/content/repositories/orgapachepulsar-1008/
> > >>
> > >> The tag to be voted upon:
> > >> v1.22.0-incubating-candidate-0
> > (c7c8a408e377e979350453e06c68340bc66c512c)
> > >>
> > >>
> > https://github.com/apache/incubator-pulsar/releases/tag/
> v1.22.0-incubating-candidate-1
> > >>
> > >> Pulsar's KEYS file containing PGP keys we use to sign the release:
> > >> https://dist.apache.org/repos/dist/release/incubator/pulsar/KEYS
> > >>
> > >> Please download the source package, and follow the README to build
> > >> and run the Pulsar standalone service.
> > >>
> > > --
> > > Matteo Merli
> > > <mme...@apache.org>
> >
> > --
> Matteo Merli
> <mme...@apache.org>
>

[VOTE] Pulsar 1.22.0-incubating Release Candidate 1

[Jai Asher]

This is the second release candidate for Apache Pulsar, version
1.22.0-incubating.

It fixes the following issues:
https://github.com/apache/incubator-pulsar/milestone/11?closed=1

*** Please download, test and vote by Tuesday, Feb 20th, 2018, 10:00 GMT.

Note that we are voting upon the source (tag), binaries are provided for
convenience.

Source and binary files:
https://dist.apache.org/repos/dist/dev/incubator/pulsar/pulsar-1.22.0-incubating-candidate-1/

Maven staging repo:
https://repository.apache.org/content/repositories/orgapachepulsar-1008/

The tag to be voted upon:
v1.22.0-incubating-candidate-0 (c7c8a408e377e979350453e06c68340bc66c512c)
https://github.com/apache/incubator-pulsar/releases/tag/v1.22.0-incubating-candidate-1

Pulsar's KEYS file containing PGP keys we use to sign the release:
https://dist.apache.org/repos/dist/release/incubator/pulsar/KEYS

Please download the source package, and follow the README to build
and run the Pulsar standalone service.

Re: [VOTE] Pulsar 1.22.0-incubating Release Candidate 0

[Jai Asher]

Hi,
"1.22.0-incubating Release Candidate 0" can't be verified at the moment
since the TLS certs we used in our test suites have expired today.
We will get it fixed and release another candidate soon.

Regards,
Jai

On Fri, Feb 16, 2018 at 12:13 PM, Matteo Merli <mme...@apache.org> wrote:

> +1 (binding)
>
> Environment: MacOS 10.12 & Centos-7.4
>
> Checked:
>  * Signatures, crc
>  * Bin distribution:
>  - NOTICE, README, LICENSE
>  - Start standalone service and producer/consumer test
>  * Src distribution:
>  - NOTICE,  README, LICENSE
>  - Compile and unit tests
>  - Start standalone service
>  * Checked staging maven repository artifacts
>
> Matteo
>
> On Thu, Feb 15, 2018 at 10:19 PM Jai Asher <jai.ashe...@gmail.com> wrote:
>
> > This is the first release candidate for Apache Pulsar, version
> > 1.22.0-incubating.
> >
> > It fixes the following issues:
> > https://github.com/apache/incubator-pulsar/milestone/11?closed=1
> >
> > *** Please download, test and vote by Tuesday, Feb 20th, 2018, 10:00 GMT.
> >
> > Note that we are voting upon the source (tag), binaries are provided for
> > convenience.
> >
> > Source and binary files:
> >
> > https://dist.apache.org/repos/dist/dev/incubator/pulsar/
> pulsar-1.22.0-incubating-candidate-0/
> >
> > Maven staging repo:
> > https://repository.apache.org/content/repositories/orgapachepulsar-1007/
> >
> > The tag to be voted upon:
> > v1.22.0-incubating-candidate-0 (96d720c928b21485df8e055dde379a
> eeb7f81624)
> >
> > https://github.com/apache/incubator-pulsar/releases/tag/
> v1.22.0-incubating-candidate-0
> >
> > Pulsar's KEYS file containing PGP keys we use to sign the release:
> > https://dist.apache.org/repos/dist/release/incubator/pulsar/KEYS
> >
> > Please download the source package, and follow the README to build
> > and run the Pulsar standalone service.
> >
> --
> Matteo Merli
> <mme...@apache.org>
>

[VOTE] Pulsar 1.22.0-incubating Release Candidate 0

[Jai Asher]

This is the first release candidate for Apache Pulsar, version
1.22.0-incubating.

It fixes the following issues:
https://github.com/apache/incubator-pulsar/milestone/11?closed=1

*** Please download, test and vote by Tuesday, Feb 20th, 2018, 10:00 GMT.

Note that we are voting upon the source (tag), binaries are provided for
convenience.

Source and binary files:
https://dist.apache.org/repos/dist/dev/incubator/pulsar/pulsar-1.22.0-incubating-candidate-0/

Maven staging repo:
https://repository.apache.org/content/repositories/orgapachepulsar-1007/

The tag to be voted upon:
v1.22.0-incubating-candidate-0 (96d720c928b21485df8e055dde379aeeb7f81624)
https://github.com/apache/incubator-pulsar/releases/tag/v1.22.0-incubating-candidate-0

Pulsar's KEYS file containing PGP keys we use to sign the release:
https://dist.apache.org/repos/dist/release/incubator/pulsar/KEYS

Please download the source package, and follow the README to build
and run the Pulsar standalone service.

Re: [PROPOSAL] Planning for next releases

[Jai Asher]

Hi,
Rajan and I are planning to start the release process on Monday (12
Feb).
Is everyone fine with this? If yes then kindly get your PR merged by
Monday.

Regards,
Jai

On Sun, Jan 21, 2018 at 11:28 PM, Rajan Dhabalia 
wrote:

> Hi,
>
> Yes, we can target 1.22.0-incubating release by reviewing and merging
> pending PRs by end of this month and then we can prepare for 2.0 release.
>
> >> Also, are there any volunteers for release managers for both releases?
> Sure, I can be release manager for 1.22.0-incubating.
>
> Thanks,
> Rajan
>
>
>
> On Fri, Jan 19, 2018 at 3:54 PM, Matteo Merli  wrote:
>
> > Since it has already been a while since last release (at least since when
> > we closed the branch for 1.21), I would like to start discussion and from
> > my side propose the following:
> >
> > 1. Release 1.22.0-incubating at the end of this month
> >
> >We have already a considerable number of fixes and improvement over
> 1.21
> > and we should release that, along with pending PRs. I would suggest
> anyone
> > to mark current changes for either this or next release and help with
> > reviewing them.
> >
> > 2. Prepare for 2.0 Pulsar release for end of February.
> > We have been discussing a while back to take the opportunity to make
> > some changes in API or tools that break binary compatibility, and do them
> > all at once.
> > I had created a project in Github to track the proposals:
> > https://github.com/apache/incubator-pulsar/projects/4
> > For this I would also propose to quickly create a branch for 1.22 so
> > that the work that is targeting 2.0 can be merged into master before the
> > release is done.
> >
> > Please anyone share your thoughts on this proposal.
> >
> > Also, are there any volunteers for release managers for both releases? I
> > can also pick up one of them otherwise.
> >
> > Thanks,
> > Matteo
> >
> >
> >
> > --
> > Matteo Merli
> > 
> >
>

Re: [DISCUSS] PIP 12: Introduce builder for creating Producer Consumer Reader

[Jai Asher]

> ```
> Producer createProducer(String topic);
> CompletableFuture createProducerAsync(String topic);
>
> Producer createProducer(String topic,
>  ProducerConfiguration conf);
> CompletableFuture createProducerAsync(
>   String topic, ProducerConfiguration conf);
> ```

I agree that in this example we have two unnecessary constructors, but
adding a setTopicName method to ProducerConfiguration could be an easier
solution instead of a different builder.

> ```java
> Producer producer = client.newProducer()
> .topic("my-topic")
> .sendTimeout(30, TimeUnit.SECONDS)
> .create();
> ```

> Similarly,

> ```java
> CompletableFuture future = client.newProducer()
>.topic("my-topic")
>.createAsync();
> ```

Also in the example, you posted you still require two methods since the
basic problem is that you can overload based on return type - which would
be the same if you put topicName in ProducerConfiguration.

As for "Type information", not sure if Producer, Consumer or Reader are
generic enough to be typed but then we can have typed ProducerConfiguration
too.

So, in short, I am not against this proposal but I am not able to
understand the necessity.

On Mon, Jan 22, 2018 at 2:18 PM, Matteo Merli  wrote:

> I have created PR with the proposed changes, to make it easier to see the
> resulting code.
>
> https://github.com/apache/incubator-pulsar/pull/1089
>
> Since this is a kind of change that can be done in several different ways,
> it would be good to have feedback on the new API changes.
>
>
> Matteo
>
> On Mon, Jan 22, 2018 at 1:29 PM Matteo Merli  wrote:
>
> >
> >
> > https://github.com/apache/incubator-pulsar/wiki/PIP-12:-Intr
> oduce-builder-for-creating-Producer-Consumer-Reader
> >
> > ---
> >
> >
> > * **Status**: Proposal
> > * **Author**: Matteo Merli
> > * **Pull Request**: [ ]
> > * **Mailing List discussion**:
> >
> > ## Motivation
> >
> > The current client API is using a mix of configuration object and
> "builder"
> > pattern.
> >
> > `Client`, `Producer`, `Consumer` and `Reader` objects are created by
> > passing
> > an optional `ClientConfiguration` (`ProducerConfiguration`, ..) object.
> If
> > that
> > is not passed, all the default options are being used (same as passing
> > `new ClientConfiguration()`).
> >
> > For example:
> > ```java
> > ClientConfiguration conf = new ClientConfiguration();
> > conf.setIoThreads(8);
> > PulsarClient client = PulsarClient.create("pulsar://localhost:6650",
> conf);
> > ```
> >
> > On the contrary, when building a message to be published, we are
> following
> > the
> > builder pattern:
> >
> > ```java
> > Message msg = MessageBuilder.create().setPayload(myPayload).build();
> > producer.send(msg);
> > ```
> >
> > The main problem with the configuration objects is that they need a lot
> of
> > methods to be used. For example, we have multiple methods for creating a
> > producer, depending on wether they accept a `conf` parameter or wether
> they
> > are sync or async:
> >
> > ```java
> > Producer createProducer(String topic);
> > CompletableFuture createProducerAsync(String topic);
> >
> > Producer createProducer(String topic,
> >   ProducerConfiguration conf);
> > CompletableFuture createProducerAsync(
> >   String topic, ProducerConfiguration conf);
> > ```
> >
> > With that, we have 4 variations for creating a `Producer`, 4 for a
> > `Consumer`
> > and 4 for a `Reader`.
> >
> > If we want to add different ways to subscribe, for example subscribing to
> > a set of topics at once, or by specifying a regular expression matching
> > multiple topics, the number of methods in the `PulsarClient` API will
> just
> > explode, making it very difficult for people to navigate.
> >
> > The same problem we would have for adding a type information at the API
> > level, since some of the information included in the configuration might
> > be typed as well (eg: the `MessageListener` on a `Consumer` is set on the
> > `ConsumerConfiguration` but that would be depending on the "type" of the
> > message).
> >
> > ## Proposal
> >
> > This proposal is to introduce builder style constructors for `Producer`,
> > `Consumer`, `Reader` and also `PulsarClient` to have consistency across
> the
> > board.
> >
> > ```java
> > interface PulsarClient {
> > public static ClientBuilder builder() {
> >return new ClientBuilderImpl();
> >}
> >
> > ProducerBuilder newProducer();
> > ConsumerBuilder newConsumer();
> > ReaderBuilder newReader();
> >
> > // All other methods will be deprecated in 2.0
> > // and possibly removed in following versions
> > }
> > ```
> >
> > Another design choice made here is not treat the required arguments in a
> > a special way.
> >
> > For example, when creating a `Producer`, the "topic" name is a required
> > information and that's why the previous API was differentiating it from
> > the optional configs:
> 

Re: [DISCUSS] Adding more Security checks to Pulsar Proxy

[Jai Asher]

Hi Matteo,

> Image didn't make it on the email

Sorry for that - here is the link to the image

https://drive.google.com/file/d/1j6ElXghUwwlr6YqUClp-Nx5sBFOW8sfi/view?usp=sharing

> In some cases the authentication cannot be easily done again in the
broker, for example if that needs to validate the IP of the client.

Agreed so I will make sending roleToken/actual cert optional on the client
side and also have a flag on broker side to enforce this ( that the proxy
has sent certificate and not just a roleToken).
Flag name authenticateProxyClient=true

> I'm not fully understanding how adding the "proxyRole" can prevent a 
> compromised
proxy to impersonate a client. Cannot a proxy just vouch for itself? Or do
you mean that along with change (a) (passing the original certificate) that
should prevent it?

I don't fully understand the question but the attack I was talking about is
that once a proxy host is compromised it can choose to send
originalPrincipal as null which will make the broker treat it as a normal
client and authenticate/authorize the proxy as a normal client rather than
demanding to authorize the originalPrincipal.

So for example:-
A client using topic T1 with roleToken R1 going through proxy with roleToken
R3 will require granting AuthAction produce/consume on both R1 and R3
(proxy).
Similarly, a client using topic T2 with roleToken R2 going through the same
proxy will require granting AuthAction produce/consume on both R1 and R3
(proxy).

Now with my current PR if proxy is compromised it can access topics T1 and
T2 without requiring client roleToken (R1 and R2) at all by passing
originalPrincipal as
null.

If I add a broker side setting "proxyRoles=R3, R4", then as soon as I see
that the roleToken (R3) extracted from the incoming cert is in proxyRoles (R3,
R4) then I can return an error if originalPrincipal is null.


Regards,
Jai

On Fri, Jan 19, 2018 at 4:20 PM, Matteo Merli <matteo.me...@gmail.com>
wrote:

> > So what I propose is that we authorize the proxy like a normal client
> > against AuthAction consume/produce/admin. In other words, if a client has
> a
> > roleToken which is Authorized to produce but the Proxy roleToken doesn't
> > have AuthAction produce on it - then the request is denied.
>
> I see, so instead of requiring "super-user" permission for the proxy, you
> propose to only require access to the specific topic being proxied.
>
> Sounds good and seems to be very flexible. One can still add the proxy as
> the "super-user" and whitelist it to access all topics or choose the
> fine-grained approach.
>
>
> >1. Current auth flow - as implemented in this PR
> >[image: Inline image 1]
>
> Image didn't make it on the email
>
> > Code Change:- The proxy should send the client certificate to the broker
> > ? and the broker should authorize and authenticate the client as well as
> > the proxy.
>
> In some cases the authentication cannot be easily done again in the broker,
> for example if that needs to validate the IP of the client.
>
>
> > b. The broker should be able to distinguish between a proxy and a client
> so that no compromised proxy can impersonate a client.
> > Code Change:- add proxyRole as a config param like we do for
> superUserRole and enforce that originalPrinciple is passed when proxy tries
> to connect.
>
> I'm not fully understanding how adding the "proxyRole" can prevent a
> compromised proxy to impersonate a client. Cannot a proxy just vouch for
> itself? Or do you mean that along with change (a) (passing the original
> certificate) that should prevent it?
>
>
>
> On Fri, Jan 19, 2018 at 4:05 PM Jai Asher <jai.ashe...@gmail.com> wrote:
>
> > Hi,
> > We had a small discussion about this solution (internally) -
> > publishing the minutes and action items so that remaining pulsar devs can
> > chime in.
> >
> > 1. Current auth flow - as implemented in this PR
> > [image: Inline image 1]
> >
> >
> > 2. Maurice had pointed out some further enhancements we can
> > incorporate:-
> > a. Proxy extracting the roleToken and broker just authorizing
> this
> > roletoken is not a very secure model (kind of what we discussed earlier
> > here <https://github.com/apache/incubator-pulsar/issues/858>). One point
> > that he added to our initial discussion was that a roleToken (string) is
> > modifiable but a certificated is not - since it is signed,
> > Code Change:- The proxy should send the client certificate to the
> > broker and the broker should authorize and authenticate the client as
> well
> > as the proxy.
> >
> > b. The broker should be able to distinguis

Re: [DISCUSS] Adding more Security checks to Pulsar Proxy

[Jai Asher]

Hi,
We had a small discussion about this solution (internally) - publishing
the minutes and action items so that remaining pulsar devs can chime in.

1. Current auth flow - as implemented in this PR
[image: Inline image 1]


2. Maurice had pointed out some further enhancements we can
incorporate:-
a. Proxy extracting the roleToken and broker just authorizing this
roletoken is not a very secure model (kind of what we discussed earlier here
<https://github.com/apache/incubator-pulsar/issues/858>). One point that he
added to our initial discussion was that a roleToken (string) is modifiable
but a certificated is not - since it is signed,
Code Change:- The proxy should send the client certificate to the
broker and the broker should authorize and authenticate the client as well
as the proxy.

b. The broker should be able to distinguish between a proxy and a
client so that no compromised proxy can impersonate a client.
Code Change:- add proxyRole as a config param like we do for
superUserRole and enforce that originalPrinciple is passed when proxy tries
to connect.

As Rajan had suggested I will create a *separate* PR to address
Maurice's enhancements on top of the current implementation so that we can
make incremental progress.

3. Since this is an open source project I will make the flow as
customizable and components as pluggable as possible.

Thanks to Rajan, Andrews, Maurice and Joe for the inputs.

Regards,
Jai

On Tue, Jan 2, 2018 at 12:04 PM, Jai Asher <jai.ashe...@gmail.com> wrote:

> Hi all,
>  I've created PIP for Adding more Security checks to Pulsar Proxy.
>  High-level description:
> * The machine hosting the Pulsar proxy will have a public IP and
> susceptible to all kinds of web attacks. The aim of this PIP is to minimize
> the damage caused by a compromised proxy on the entire service.*
>
>  PIP:- https://github.com/apache/incubator-pulsar/wiki/PIP-9:-Addin
> g-more-Security-checks-to-Pulsar-Proxy
>  PR:- https://github.com/apache/incubator-pulsar/pull/1002
>  Issue:- https://github.com/apache/incubator-pulsar/issues/858
>
>  Can you please review and provide your feedback/comments.
>
> Regards,
> Jai
>

Re: [VOTE] Pulsar 1.19.0-incubating Release Candidate 0

[Jai Asher]

+1 Release!

- Checked build success
- Ran the binaries on my local machine

Jai

On Thu, Aug 3, 2017 at 4:19 PM, Sahaya Andrews  wrote:

> +1
>
> Environment: RHEL 6.4
> - Verified checksum
> - Ran local build against the src distribution
> - Ran standalone server from the binary distribution
> - Ran perf producer and consumer test against locally built binary and
> binary distribution
>
> I did not see any errors in producer/consumer/pulsar logs.
>
> Thanks,
> Andrews.
>
>
> On Thu, Aug 3, 2017 at 5:53 AM, Masakazu Kitajo  wrote:
> > +0
> >
> > Ran standalone server on macOS 10.12 / Java 8, with Java client.
> >
> > I saw lots of warnings and a few errors during building from the source
> > package, I was able to run standalone server and successfully
> transferred a
> > message with pulsar-client though. It seems they are not critical. I'll
> > create issues for each warnings and errors later.
> >
> > Also, I followed Getting started on our documentation, however, the steps
> > to build are only on README file. It's not user friendly. We should
> improve
> > it before announcing the release.
> >
> > Committers, it would be great if you could write down about your
> > environment (e.g. OS, Java version, compiler, etc) when you cast a vote.
> > Since we support Linux and macOS (and a few languages for clients), we
> > should confirm that the new version works on the environments we support.
> >
> > Thanks,
> > Masakazu
> >
> >
> >
> > On Thu, Aug 3, 2017 at 4:47 AM, Matteo Merli  wrote:
> >
> >> Remainder for all committers:
> >>
> >> please check the release artifacts and vote on this thread, before
> tomorrow
> >> 3pm PDT.
> >>
> >> Vote as in :
> >>
> >> [ ] +1 Release this package
> >> [ ] 0 I don't feel strongly about it, but don't object
> >> [ ] -1 Do not release this package because...
> >>
> >>
> >> The documentation on the release process and what needs to be included
> can
> >> be found at:
> >>- https://incubator.apache.org/guides/releasemanagement.html
> >>- http://www.apache.org/legal/release-policy.html
> >>- http://www.apache.org/dev/release-distribution.html
> >>- http://www.apache.org/dev/release-publishing.html
> >>- http://www.apache.org/dev/publishing-maven-artifacts.html
> >>
> >> --
> >> Matteo Merli
> >> 
> >>
> >> On Mon, Jul 31, 2017 at 2:39 PM, Matteo Merli 
> wrote:
> >>
> >> > This is the first release candidate for Apache Pulsar, version
> >> > 1.19.0-incubating.
> >> >
> >> > Major changes included in this release are:
> >> >  * Added stateless Pulsar proxy
> >> >  * Support for non-persistent topics
> >> >  * Upgraded RocksDB to comply with ASF policy
> >> >  * Instrumentation of ZooKeeper client to expose metrics
> >> >  * Fixes for TLS auth in WebSocket proxy
> >> >
> >> >
> >> > Complete list of changes can be found at:
> >> >
> >> > https://github.com/apache/incubator-pulsar/milestone/8?closed=1
> >> >
> >> > *** Please download, test and vote by August 3th 2017, 15:00 PDT.
> >> >
> >> > Note that we are voting upon the source (tag), binaries are provided
> for
> >> > convenience.
> >> >
> >> > Source and binary files:
> >> > https://dist.apache.org/repos/dist/dev/incubator/pulsar/
> >> > pulsar-1.19.0-incubating-candidate-0/
> >> >
> >> > Maven staging repo:
> >> > https://repository.apache.org/content/repositories/
> orgapachepulsar-1000/
> >> >
> >> > The tag to be voted upon:
> >> > v1.19.0-incubating-candidate-0 (5125279b50f7a3da8c211b698580e1
> >> d2f4dd65e2)
> >> >
> >> > Pulsar's KEYS file containing PGP keys we use to sign the release:
> >> > https://dist.apache.org/repos/dist/release/incubator/pulsar/KEYS
> >> >
> >> > Please download the the source package, and follow the README to build
> >> > and run the Pulsar standalone service.
> >> >
> >> >
> >> >
> >> > --
> >> > Matteo Merli
> >> > 
> >> >
> >>
>