Re: Pulsar release 2.2

[Sijie Guo]

+1 sgtm

On Mon, Sep 17, 2018 at 11:30 AM Joe F  wrote:

> Now that 2.1.1 is completed, I intend to start the Pulsar 2.2 release
> process in a few days, and would like to freeze the code this Friday.
>
> Please ensure that all your committed PR's are merged and complete. I still
> see 11 outstanding items.
> https://github.com/apache/incubator-pulsar/milestone/16
>
>  Are we OK with pushing some of these to the next release, if not completed
> by Friday ?
>
> Joe
>

Re: pulsar_precommit jobs taking up too much space

[Sijie Guo]

H Daniel,

I reduced the number of days to keep for the build. That should clean up
the job builds more aggressively.

- Sijie

On Sun, Sep 16, 2018 at 6:11 PM Daniel Pono Takamori 
wrote:

> Hey folks,
> Doing disk cleanups on our Jenkins nodes [0], and found about 40
> jpulsar_precommit* jobs on a single node taking up 2-5gb each.  We've
> moved to purging 15 day old files [1] on Jenkins nodes.  I'm wondering
> if we can be more aggressive on purging these old precommit jobs, or
> if there is something on your end which would also serve to reduce the
> disk overhead on keeping them around.
>
> Thanks for any ideas!
> -Pono on behalf of Infra
>
> [0] - https://issues.apache.org/jira/browse/INFRA-17024
> [1] -
> https://github.com/apache/infrastructure-puppet/pull/1159/commits/8438937327a63c2c5999baa3fefe956d659dedec
>

Re: [VOTE] Graduate Apache Pulsar (incubating) as a TLP

[Sijie Guo]

+1!! (non-binding)

- Sijie

On Wed, Sep 12, 2018 at 8:40 AM Dave Fisher  wrote:

> Hi -
>
> The Apache Pulsar project is ready to graduate as a TLP. They entered
> Incubation on June 1, 2017, have had many releases and have grown the
> community.
>
> Vote:
> [ ] +1 - Recommend Graduation of Apache Pulsar as a TLP
> [ ] -1 - Do not recommend graduation of Apache Pulsar because ….
>
> At the mentors request they did a maturity model analysis [1] and wrote
> contribution guidelines. [2]
>
> The Graduation Proposal was written and discussed on the dev list. [3] At
> the mentor's recommendation the By-Laws Clause was removed.
>
> The new prospective PMC is set and the VOTE thread in the podling is here
> [4] with these results. [5]
>
> Regards,
> Dave
>
> [1]
> https://github.com/apache/incubator-pulsar/wiki/Apache-Maturity-Model-Assessment-for-Pulsar
> [2] http://pulsar.incubator.apache.org/en/contributing/
> [3]
> https://lists.apache.org/thread.html/b0914461f57253237e4a3c9151342f6d4fa37359dfc98a07adf9f36f@%3Cdev.pulsar.apache.org%3E
> [4]
> https://lists.apache.org/thread.html/93198abe36564a9e11a2a1bfe3ea8f35998444dbafea830f8b39df7b@%3Cdev.pulsar.apache.org%3E
> [5]
> https://lists.apache.org/thread.html/64841a07ba3dee2271f4098f9142afd41acffae1736275592aab4c83@%3Cdev.pulsar.apache.org%3E
>
>
> 
>
> Establish the Apache Pulsar Project
>
> WHEREAS, the Board of Directors deems it to be in the best interests of
> the Foundation and consistent with the Foundation's purpose to establish
> a Project Management Committee charged with the creation and maintenance
> of open-source software, for distribution at no charge to the public,
> related to a highly scalable, low latency messaging platform running on
> commodity hardware. It provides simple pub-sub and queue semantics over
> topics, lightweight compute framework, automatic cursor management for
> subscribers, and cross-datacenter replication.
>
> NOW, THEREFORE, BE IT RESOLVED, that a Project Management Committee
> (PMC), to be known as the "Apache Pulsar Project", be and hereby is
> established pursuant to Bylaws of the Foundation; and be it further
>
> RESOLVED, that the Apache Pulsar Project be and hereby is responsible
> for the creation and maintenance of software related to a highly
> scalable, low latency messaging platform running on commodity hardware.
> It provides simple pub-sub and queue semantics over topics, lightweight
> compute framework, automatic cursor management for subscribers, and
> cross-datacenter replication; and be it further
>
> RESOLVED, that the office of "Vice President, Apache Pulsar" be and
> hereby is created, the person holding such office to serve at the
> direction of the Board of Directors as the chair of the Apache Pulsar
> Project, and to have primary responsibility for management of the
> projects within the scope of responsibility of the Apache Pulsar
> Project; and be it further
>
> RESOLVED, that the persons listed immediately below be and hereby are
> appointed to serve as the initial members of the Apache Pulsar Project:
>
> * Boyang Jerry Peng 
> * Brad McMillen 
> * David Fisher 
> * Francis Christopher Liu 
> * Hiroyuki Sakai 
> * Ivan Brendan Kelly 
> * Jai Asher 
> * Jia Zhai 
> * Jim Jagielski 
> * Joe Francis 
> * Ludwig Pummer 
> * Masahiro Sakamoto 
> * Masakazu Kitajo 
> * Matteo Merli 
> * Nozomi Kurihara 
> * P. Taylor Goetz 
> * Rajan Dhabalia 
> * Sahaya Andrews 
> * Sanjeev Kulkarni 
> * Sebastián Schepens 
> * Siddharth Boobna 
> * Sijie Guo 
> * Yuki Shiga 
>
> NOW, THEREFORE, BE IT FURTHER RESOLVED, that Matteo Merli be appointed
> to the office of Vice President, Apache Pulsar, to serve in accordance
> with and subject to the direction of the Board of Directors and the
> Bylaws of the Foundation until death, resignation, retirement, removal
> or disqualification, or until a successor is appointed; and be it
> further
>
> RESOLVED, that the Apache Pulsar Project be and hereby is tasked with
> the migration and rationalization of the Apache Incubator Pulsar
> podling; and be it further
>
> RESOLVED, that all responsibilities pertaining to the Apache Incubator
> Pulsar podling encumbered upon the Apache Incubator PMC are hereafter
> discharged.
>

Re: Planning for Pulsar 2.2 release

[Sijie Guo]

Hi all,

I have gone through the issues for 2.2 and moved the tasks that are not
needed for 2.2 out.

There are 13 pull requests open for 2.2, most of them are ready to go.
Please take a look at the issues. If you have issues that are needed to be
included at 2.2 release, please mark them for 2.2 release.
Once all the issues are merged, we should start the process for releasing
2.2

https://github.com/apache/incubator-pulsar/milestone/16

Thanks,
Sijie

On Mon, Aug 20, 2018 at 2:58 PM Sijie Guo  wrote:

> Thank you Joe! :)
>
> - Sijie
>
> On Mon, Aug 20, 2018 at 2:51 PM Joe F  wrote:
>
>> Ok if no one else is there i will take this
>>
>> Joe
>>
>> On Mon, Aug 20, 2018, 2:45 PM Sijie Guo  wrote:
>>
>> > Anyone volunteer for releasing 2.2?
>> >
>> > - Sijie
>> >
>> > On Thu, Aug 16, 2018 at 12:02 AM Sijie Guo  wrote:
>> >
>> > > Any committer volunteer for releasing 2.2?
>> > >
>> > > - Sijie
>> > >
>> > > On Wed, Aug 8, 2018 at 11:43 AM Sanjeev Kulkarni > >
>> > > wrote:
>> > >
>> > >> The timeline sounds good!
>> > >> One more thing that users have asked about is feature parity between
>> > >> python
>> > >> and java clients so we should have some work items associated with
>> this
>> > >> task as well.
>> > >>
>> > >>
>> > >> On Wed, Aug 8, 2018 at 11:40 AM Sijie Guo 
>> wrote:
>> > >>
>> > >> > Hi all,
>> > >> >
>> > >> > Although we just announced 2.1, we already have a lot of new
>> features
>> > >> > merged since 2.1 candidate is cut. For example, Pulsar SQL, GCS
>> > support
>> > >> as
>> > >> > tiered storage and such. Shall we target at releasing 2.2 at the
>> end
>> > of
>> > >> > August? Since releasing usually takes 1~2 weeks, that would be
>> > released
>> > >> > around Mid September.
>> > >> >
>> > >> > What are the thoughts around this?
>> > >> >
>> > >> > Any committers volunteer to drive 2.2 release?
>> > >> >
>> > >> > - Sijie
>> > >> >
>> > >>
>> > >
>> >
>>
>

Re: [VOTE] Pulsar Release 2.1.1-incubating Candidate 1

[Sijie Guo]

+1

Environment:
MacOS 10.13.1

- All asc, sha512 files are good. (sha1 files can be dropped, but minor
issues)
- binary is good, verified pub/sub, functions and connectors
- src is good, can compile, rat check, and can run pub/sub, functions and
connectors after compilation.

- Sijie

On Thu, Sep 6, 2018 at 5:48 PM Jerry Peng 
wrote:

> Sorry there was a typo in my previous email:
>
> "v2.2.1-incubating-candidate-1" => "v2.1.1-incubating-candidate-1"
>
> On Thu, Sep 6, 2018 at 5:31 PM Jerry Peng 
> wrote:
>
> > This is the first release candidate for Apache Pulsar, version
> > 2.1.1-incubating.
> >
> > It fixes the following issues:
> > https://github.com/apache/incubator-pulsar/milestone/17?closed=1
> >
> > Mostly bug fixes for 2.1.0
> >
> > *** Please download, test and vote on this release. This vote will stay
> > open for at least 72 hours ***
> >
> > Note that we are voting upon the source (tag), binaries are provided for
> > convenience.
> >
> > Source and binary files:
> >
> >
> https://dist.apache.org/repos/dist/dev/incubator/pulsar/pulsar-2.1.1-incubating-candidate-1/
> >
> > SHA-1 checksums:
> >
> > 506c172b13c1581f9b52769a4c6594541628b928
> >  apache-pulsar-2.1.1-incubating-bin.tar.gz
> > de3a74449a78d0713998544237dde59765ee121f
> >  apache-pulsar-2.1.1-incubating-src.tar.gz
> > 1d38554e58cfd733fc2d54ec1442747daeb8fc1b
> > apache-pulsar-io-connectors-2.1.1-incubating-bin.tar.gz
> >
> > Maven staging repo:
> > https://repository.apache.org/content/repositories/orgapachepulsar-1026
> >
> > The tag to be voted upon:
> > v2.2.1-incubating-candidate-1 68faf85083025b29bb6ab39028a1d467bd3d5217
> >
> >
> https://github.com/apache/incubator-pulsar/releases/tag/v2.1.1-incubating-candidate-1
> >
> > Pulsar's KEYS file containing PGP keys we use to sign the release:
> > https://dist.apache.org/repos/dist/release/incubator/pulsar/KEYS
> >
> > Please download the the source package, and follow the README to build
> and
> > run the Pulsar standalone service.
> >
> > Here is a guide for validating a release candidate:
> >
> >
> https://github.com/apache/incubator-pulsar/wiki/Release-Candidate-Validation
> >
>

Re: [VOTE] Graduation resolution proposal

[Sijie Guo]

+1

On Fri, Sep 7, 2018 at 5:22 PM Matteo Merli  wrote:

> Please vote on the proposal for Pulsar graduation to TLP to submit to
> the Incubator PMC.
>
> This vote will stay open for at least 72 hours.
>
>
> 
>
> Establish the Apache Pulsar Project
>
> WHEREAS, the Board of Directors deems it to be in the best interests of
> the Foundation and consistent with the Foundation's purpose to establish
> a Project Management Committee charged with the creation and maintenance
> of open-source software, for distribution at no charge to the public,
> related to a highly scalable, low latency messaging platform running on
> commodity hardware. It provides simple pub-sub and queue semantics over
> topics, lightweight compute framework, automatic cursor management for
> subscribers, and cross-datacenter replication.
>
> NOW, THEREFORE, BE IT RESOLVED, that a Project Management Committee
> (PMC), to be known as the "Apache Pulsar Project", be and hereby is
> established pursuant to Bylaws of the Foundation; and be it further
>
> RESOLVED, that the Apache Pulsar Project be and hereby is responsible
> for the creation and maintenance of software related to a highly
> scalable, low latency messaging platform running on commodity hardware.
> It provides simple pub-sub and queue semantics over topics, lightweight
> compute framework, automatic cursor management for subscribers, and
> cross-datacenter replication; and be it further
>
> RESOLVED, that the office of "Vice President, Apache Pulsar" be and
> hereby is created, the person holding such office to serve at the
> direction of the Board of Directors as the chair of the Apache Pulsar
> Project, and to have primary responsibility for management of the
> projects within the scope of responsibility of the Apache Pulsar
> Project; and be it further
>
> RESOLVED, that the persons listed immediately below be and hereby are
> appointed to serve as the initial members of the Apache Pulsar Project:
>
>  * Boyang Jerry Peng   
>  * Brad McMillen   
>  * David Fisher
>  * Francis Christopher Liu 
>  * Hiroyuki Sakai  
>  * Ivan Brendan Kelly  
>  * Jai Asher   
>  * Jia Zhai
>  * Jim Jagielski   
>  * Joe Francis 
>  * Ludwig Pummer   
>  * Masahiro Sakamoto   
>  * Masakazu Kitajo 
>  * Matteo Merli
>  * Nozomi Kurihara 
>  * P. Taylor Goetz 
>  * Rajan Dhabalia  
>  * Sahaya Andrews  
>  * Sanjeev Kulkarni
>  * Sebastián Schepens  
>  * Siddharth Boobna
>  * Sijie Guo   
>  * Yuki Shiga  
>
> NOW, THEREFORE, BE IT FURTHER RESOLVED, that Matteo Merli be appointed
> to the office of Vice President, Apache Pulsar, to serve in accordance
> with and subject to the direction of the Board of Directors and the
> Bylaws of the Foundation until death, resignation, retirement, removal
> or disqualification, or until a successor is appointed; and be it
> further
>
> RESOLVED, that the Apache Pulsar Project be and hereby is tasked with
> the migration and rationalization of the Apache Incubator Pulsar
> podling; and be it further
>
> RESOLVED, that all responsibilities pertaining to the Apache Incubator
> Pulsar podling encumbered upon the Apache Incubator PMC are hereafter
> discharged.
>
> 
>
>
>
> --
> Matteo Merli
> 
>

Re: [DRAFT] [UPDATED] Graduation resolution proposal

[Sijie Guo]

 and
> hereby
> > > is
> > > > >> established pursuant to Bylaws of the Foundation; and be it
> > > further
> > > > >>
> > > > >> RESOLVED, that the Apache Pulsar Project be and hereby is
> > > responsible
> > > > >> for the creation and maintenance of software related to a
> highly
> > > > >> scalable, low latency messaging platform running on commodity
> > > hardware.
> > > > >> It provides simple pub-sub and queue semantics over topics,
> > > > >> lightweight compute framework, automatic cursor management for
> > > > >> subscribers, and cross-datacenter replication; and be it
> further
> > > > >>
> > > > >> RESOLVED, that the office of "Vice President, Apache Pulsar"
> be
> > > and
> > > > >> hereby is created, the person holding such office to serve at
> > the
> > > > >> direction of the Board of Directors as the chair of the Apache
> > > Pulsar
> > > > >> Project, and to have primary responsibility for management of
> > the
> > > > >> projects within the scope of responsibility of the Apache
> Pulsar
> > > > >> Project; and be it further
> > > > >>
> > > > >> RESOLVED, that the persons listed immediately below be and
> > hereby
> > > are
> > > > >> appointed to serve as the initial members of the Apache Pulsar
> > > Project:
> > > > >>
> > > > >> * Boyang Jerry Peng   
> > > > >> * Brad McMillen   
> > > > >> * David Fisher
> > > > >> * Francis Christopher Liu 
> > > > >> * Hiroyuki Sakai  
> > > > >> * Ivan Brendan Kelly  
> > > > >> * Jai Asher   
> > > > >> * Jia Zhai
> > > > >> * Jim Jagielski   
> > > > >> * Joe Francis 
> > > > >> * Ludwig Pummer   
> > > > >> * Masahiro Sakamoto   
> > > > >> * Masakazu Kitajo 
> > > > >> * Matteo Merli
> > > > >> * Nozomi Kurihara 
> > > > >> * P. Taylor Goetz 
> > > > >> * Rajan Dhabalia  
> > > > >> * Sahaya Andrews  
> > > > >> * Sanjeev Kulkarni
> > > > >> * Sebastián Schepens  
> > > > >> * Siddharth Boobna
> > > > >> * Sijie Guo   
> > > > >> * Yuki Shiga  
> > > > >>
> > > > >> NOW, THEREFORE, BE IT FURTHER RESOLVED, that Matteo Merli be
> > > > >> appointed to the office of Vice President, Apache Pulsar, to
> > > serve in
> > > > >> accordance with and subject to the direction of the Board of
> > > > >> Directors and the Bylaws of the Foundation until death,
> > > resignation,
> > > > >> retirement, removal or disqualification, or until a successor
> is
> > > > >> appointed; and be it further
> > > > >>
> > > > >> RESOLVED, that the Apache Pulsar Project be and hereby is
> tasked
> > > with
> > > > >> the migration and rationalization of the Apache Incubator
> Pulsar
> > > > >> podling; and be it further
> > > > >>
> > > > >> RESOLVED, that all responsibilities pertaining to the Apache
> > > > >> Incubator Pulsar podling encumbered upon the Apache Incubator
> > PMC
> > > are
> > > > >> hereafter discharged.
> > > > >>
> > > > >> 
> > > > >>
> > > > >>
> > > > >> --
> > > > >> Matteo Merli
> > > > >> 
> > >
> > >
> > >
> >
> --
> Matteo Merli
> 
>

Re: [DRAFT] Podling report

[Sijie Guo]

+1 looks good

- Sijie

On Wed, Sep 5, 2018 at 5:36 PM Matteo Merli  wrote:

> Sorry for sending out at very last minute (deadline is today). Please take
> a look.
>
> I have added a paragraph regarding the branding discussion that happen in
> early June (as discussed with Taylor then) since the previous draft was
> already out at that point.
>
>
>
> ---
>
> Pulsar
> Pulsar is a highly scalable, low latency messaging platform running on
> commodity hardware. It provides simple pub-sub semantics over topics,
> guaranteed at-least-once delivery of messages, automatic cursor management
> for
> subscribers, and cross-datacenter replication.
>
> Pulsar has been incubating since 2017-06-01.
>
> Most important issues to address in the move towards graduation:
>
>   None
>
> Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
> aware of?
>
>   Earlier in June there have been few discussions on the private list
>   regarding communications regarding Pulsar that were not coming from
>   PPMC or that were not respecting the ASF policies.  Clarifications
>   followed between PPMC members, mentors and interested parties to
>   ensure the mistakes were made in good faith and, in particular, to
>   make sure everyone was fully has full understanding of ASF
>   policies. There was no other branding related issue after the first
>   occourence.
>
> How has the community developed since the last report?
>
>   The community added 8 new contributors that submitted pull-requests
>   which were merged into master.
>
>   The number of users approaching the team on the Slack channel has
>   kept steadily increasing since the last report. Many users have
>   actively deployed. Pulsar for evaluation and production use cases.
>
>   Different meetups were organized by project members and hosted by
>   Yahoo in Sunnyvale and Yahoo Japan in Tokyo. We have presented
>   Pulsar's introductions, updates on the state of the projects,
>   deep-dives and hands-on tutorial, including recorded podcasts.
>
>   One talk on Pulsar was presented at one at OSCon in July and there
>   are several scheduled talks: 2 at ApacheCon in September, and 2
>   others at Strata New York in September.
>
>   Since the last report the number of weekly-active-users on the Slack
>   channel has increased from 53 to 88.
>
>
> How has the project developed since the last report?
>
>   28 authors have pushed 494 commits to master in the last 3 months.
>
>   The project has made the its seventh release since joining the
>   Apache Incubator (2.1.0-incubating on Aug 2nd).
>
>   This release introduced these new features:
>
>* Pulsar IO: A connector framework for moving data in and out of
>  Apache Pulsar leveraging Pulsar Functions runtime.
>* A number of builtin connectors: (Aerospike, Cassandra, Kafka,
>  Kinesis, RabbitMQ, Twitter)
>* Tiered Storage: An extension in Pulsar segment store to offload
>  older segments into long term storage (e.g. HDFS, S3). S3 support
>  is supported in 2.1 release.
>* Stateful function: Pulsar Functions is able to use State API for
>  storing state within Pulsar.
>* Pulsar Go Client
>* Avro and Protobuf Schema support
>
>   Community is actively working on a bug-fix release
>   (2.1.1-incubating) and on the next milestone, 2.2 release for which
>   the biggest feature will be support for SQL within Pulsar.
>
>   Since June, 5 new PIPs (Pulsar Improvement Proposals) for
>   major feature/changes, have been submitted to the wiki and
>   discussed in the mailing list.
>
> PIP 23: Message Tracing By Interceptors
> PIP 22: Pulsar Dead Letter Topic
> PIP 21: Pulsar Edge Component
> PIP 20: Mechanism to revoke TLS authentication
> PIP 19: Pulsar SQL
>
>
> How would you assess the podling's maturity?
> Please feel free to add your own commentary.
>
>   [ ] Initial setup
>   [ ] Working towards first release
>   [ ] Community building
>   [X] Nearing graduation
>   [ ] Other:
>
> Date of last release:
>   2018-08-02, 2.1.0-incubating
>
> When were the last committers or PPMC members elected?
>
>   2018-06-11 - Ivan Kelly
>   2018-06-11 - Jia Zhai
>
>
> --
> Matteo Merli
> 
>

Re: [discuss] how to label milestone

[Sijie Guo]

My motivation is simple - I would like to see a process for managing
merging pull requests, add labels and milestone and cherry-picks for
releases. Otherwise things just become very messy with different people
merging pull requests in different ways.

I know there was a discussion about the merge script. but people doesn't
like the way how bk merge script closes the pull request.

http://mail-archives.apache.org/mod_mbox/pulsar-dev/201708.mbox/%3CCAGjw%2BkO1ucxp%3DKn0aK077kANhiZiVRcL5exhPJPgHkdfa7XSCA%40mail.gmail.com%3E

So I changed the merge script to use github pull request merge api on
merging a PR. here is the script:

https://github.com/apache/incubator-pulsar/pull/2526

This is one example PR how it was merged by the script:
https://github.com/sijie/incubator-pulsar/pull/8

Let me know what do you think.

- Sijie


On Mon, Aug 27, 2018 at 2:45 AM Sijie Guo  wrote:

> Hi all,
>
> I think we need to discuss how to label the issues and how to cherry-pick
> the issues for bug fixes. Currently it is painful for doing a bugfix
> release.
> All the bug fixes are deferred to release manager cherry-picking to bug
> fix branches. Since master and branches are quickly diverged with changes,
> it is very painful for cherry-picks later and there is no clear process
> for cherry-pick and labelling.
>
> Shall we consider following the process what BookKeeper is using, having a
> merge script to handle labelling issues/milestones and also cherry-picks?
>
> Or any suggestions on how to improve this?
>
> - Sijie
>
> On Mon, Aug 6, 2018 at 10:50 AM Sijie Guo  wrote:
>
>> Hi all,
>>
>> It seems we don't have a standard for labelling milestone. sometimes I am
>> not lost and confused when I see a change was merged to master but marked
>> in a bug fix milestone. so I am raising a discussion about this.
>>
>> I think every pull request that was created based on master, which would
>> be merged as a commit of master, those pull requests should be first marked
>> as major release milestone, for example: `2.2.0` release. If a pull request
>> that was created based on a branch, which would be merged as a commit of
>> that branch, those pull requests should be marked as minor release
>> milestones, for example: `2.1.1` release. so that we have a clearer way on
>> tracking what changes have been made to what releases. otherwise it is a
>> bit hard to track when we have multiple releases ongoing.
>>
>> Thoughts?
>>
>> Thanks,
>> Sijie
>>
>>

Re: [discuss] how to label milestone

[Sijie Guo]

On Mon, Sep 3, 2018 at 12:14 AM Ivan Kelly  wrote:

> > All the bug fixes are deferred to release manager cherry-picking to bug
> fix
> > branches. Since master and branches are quickly diverged with changes,
> > it is very painful for cherry-picks later and there is no clear process
> for
> > cherry-pick and labelling.
>
> What is painful here?



> If it's the conflict resolution, this pain would
> just be spread out, with it being felt by whomever is merging, but
> then it would also be worse, because cherry-picks would happen out of
> order.
>

Sure. But if when merging a PR also cherry-pick to corresponding branches,
the conflicts will be reduced a lot.

The pain comes from issues marked for a bug release milestone but it is
merged directly to master and never cherry-picked to branch for bugfix
releases.
So the release manager has to compare branches and master and also have to
compare labels, milestones and even look into the details of pull requests
to understood whether a change is cherry-picked or not.


>
> In the past I've had more success with processes which keep day to day
> development on master 100% of the time. If and when you want to cut a
> release branch, at that point, you go through the commits in master
> since the last release, and select those that should make the bugfix
> release. Once you have the list, cherry-picking, in-order, can occur.
> This creates more work for the release manager, but it removes the
> need to think about the releases from the developer in their
> day-to-day.
>

That works for me. what I really want is a process here that everyone
follows. I have been seeing issues merged to master but marked for bugfixes
releases,
and issues merged without labels and milestones. That will just kill the
release manager.


>
> > Shall we consider following the process what BookKeeper is using, having
> a
> > merge script to handle labelling issues/milestones and also cherry-picks?
>
> That script turns a 5 second task into a 5 minute task.


Sure it requires more steps on merging issues, however it enforces people
label components, types and releases correctly. If we don't label things
correctly,
the project and release processes will quickly become out of control. No
one is able to quickly figure out how a change lands at what releases.

People sometimes are just lazy. If they can just click a button to merge
one PR, they do it and things quickly become messy because there is no
enforcement.



> It also
> creates a decision point for the developer where the answer is not
> always clear.


I think as a committer, he/she should try to be aware of those things (e.g.
whether this bug is critical, whether it should be included to a release or
not). It is sort of the committer's commitment to the project.

If he doesn't know whether he should cherry-pick or not, he can just merge
to master and leave it to release manager to decide. The point here is we
need to some enforcement to make sure a merged change
is labelled correctly for a release. so when the release manager going
through the issues and commits, he is able to quickly know what he should
do.


> If no bugfix releases are currently planned, should I
> still cherry-pick in branch-1.22?
>



>
> -Ivan
>

[ANNOUNCE] Apache BookKeeper 4.7.2 released

[Sijie Guo]

The Apache BookKeeper team is proud to announce Apache BookKeeper version
4.7.2.

Apache BookKeeper is a scalable, fault-tolerant, and low-latency storage
service optimized for
real-time workloads. It has been used for a fundamental service to build
reliable services.
It is also the log segment store for Apache DistributedLog and the message
store for Apache Pulsar.

This is the 12th release of Apache BookKeeper.

This is a bugfix release, which fixes a bunch of issues reported from users
of 4.7.1.
These fixes include bug fixes around DbLedgerStorage, failure handling
around ensemble changes, bookie shutdown and such.

For BookKeeper release details and downloads, visit:

https://bookkeeper.apache.org/releases/

BookKeeper 4.7.2 Release Notes are at:

https://bookkeeper.apache.org/docs/4.7.2/overview/releaseNotes/

We would like to thank the contributors that made the release possible.

Regards,

The BookKeeper Team

Re: Podling Report Reminder - September 2018

[Sijie Guo]

We have a podling report due this wednesday. Anyone volunteers in drafting
the report?

- Sijie

On Sun, Sep 2, 2018 at 5:55 PM  wrote:

> Dear podling,
>
> This email was sent by an automated system on behalf of the Apache
> Incubator PMC. It is an initial reminder to give you plenty of time to
> prepare your quarterly board report.
>
> The board meeting is scheduled for Wed, 19 September 2018, 10:30 am PDT.
> The report for your podling will form a part of the Incubator PMC
> report. The Incubator PMC requires your report to be submitted 2 weeks
> before the board meeting, to allow sufficient time for review and
> submission (Wed, September 05).
>
> Please submit your report with sufficient time to allow the Incubator
> PMC, and subsequently board members to review and digest. Again, the
> very latest you should submit your report is 2 weeks prior to the board
> meeting.
>
> Candidate names should not be made public before people are actually
> elected, so please do not include the names of potential committers or
> PPMC members in your report.
>
> Thanks,
>
> The Apache Incubator PMC
>
> Submitting your Report
>
> --
>
> Your report should contain the following:
>
> *   Your project name
> *   A brief description of your project, which assumes no knowledge of
> the project or necessarily of its field
> *   A list of the three most important issues to address in the move
> towards graduation.
> *   Any issues that the Incubator PMC or ASF Board might wish/need to be
> aware of
> *   How has the community developed since the last report
> *   How has the project developed since the last report.
> *   How does the podling rate their own maturity.
>
> This should be appended to the Incubator Wiki page at:
>
> https://wiki.apache.org/incubator/September2018
>
> Note: This is manually populated. You may need to wait a little before
> this page is created from a template.
>
> Mentors
> ---
>
> Mentors should review reports for their project(s) and sign them off on
> the Incubator wiki page. Signing off reports shows that you are
> following the project - projects that are not signed may raise alarms
> for the Incubator PMC.
>
> Incubator PMC
>

Re: [DRAFT] Graduation resolution proposal

[Sijie Guo]

+1 ; #gopulsar 🚢🚢🚢

On Fri, Aug 31, 2018 at 9:44 AM Matteo Merli  wrote:

> I have prepared the draft for the graduation resolution using the
> Whimsy tool (https://whimsy.apache.org/roster/ppmc/pulsar).
>
> Maturity model self-assessment is at
>
> https://github.com/apache/incubator-pulsar/wiki/Apache-Maturity-Model-Assessment-for-Pulsar
> Contributing to Pulsar page is now available at
> http://pulsar.incubator.apache.org/contributing/
>
> Notes:
>
>  * I'm proposing myself as initial PMC chair -- Please comment if
> community is onboard with this or propose other persons as well


+1


>
>  * This draft includes all existing PPMC members and mentors into the new
> PMC.
>
>  - For all: please indicate if you want to keep being part of the
> PMC or if you prefer to be removed.


Would like to be part of PMC


>
>
> 
>
> Establish the Apache Pulsar Project
>
> WHEREAS, the Board of Directors deems it to be in the best interests of
> the Foundation and consistent with the Foundation's purpose to establish
> a Project Management Committee charged with the creation and maintenance
> of open-source software, for distribution at no charge to the public,
> related to a highly scalable, low latency messaging platform running on
> commodity hardware. It provides simple pub-sub and queue semantics over
> topics, lightweight compute framework, automatic cursor management for
> subscribers, and cross-datacenter replication.
>
> NOW, THEREFORE, BE IT RESOLVED, that a Project Management Committee
> (PMC), to be known as the "Apache Pulsar Project", be and hereby is
> established pursuant to Bylaws of the Foundation; and be it further
>
> RESOLVED, that the Apache Pulsar Project be and hereby is responsible
> for the creation and maintenance of software related to a highly
> scalable, low latency messaging platform running on commodity hardware.
> It provides simple pub-sub and queue semantics over topics, lightweight
> compute framework, automatic cursor management for subscribers, and
> cross-datacenter replication; and be it further
>
> RESOLVED, that the office of "Vice President, Apache Pulsar" be and
> hereby is created, the person holding such office to serve at the
> direction of the Board of Directors as the chair of the Apache Pulsar
> Project, and to have primary responsibility for management of the
> projects within the scope of responsibility of the Apache Pulsar
> Project; and be it further
>
> RESOLVED, that the persons listed immediately below be and hereby are
> appointed to serve as the initial members of the Apache Pulsar Project:
>
>  * Boyang Jerry Peng   
>  * Brad McMillen   
>  * David Fisher
>  * Francis Christopher Liu 
>  * Hiroyuki Sakai  
>  * Ivan Brendan Kelly  
>  * Jai Asher   
>  * Jia Zhai
>  * Jim Jagielski   
>  * Joe Francis 
>  * Ludwig Pummer   
>  * Masahiro Sakamoto   
>  * Masakazu Kitajo 
>  * Matteo Merli
>  * Nozomi Kurihara 
>  * P. Taylor Goetz 
>  * Rajan Dhabalia  
>  * Sahaya Andrews  
>  * Sanjeev Kulkarni
>  * Sebastián Schepens  
>  * Siddharth Boobna
>  * Sijie Guo   
>  * Yuki Shiga  
>
> NOW, THEREFORE, BE IT FURTHER RESOLVED, that Matteo Merli be appointed
> to the office of Vice President, Apache Pulsar, to serve in accordance
> with and subject to the direction of the Board of Directors and the
> Bylaws of the Foundation until death, resignation, retirement, removal
> or disqualification, or until a successor is appointed; and be it
> further
>
> RESOLVED, that the initial Apache Pulsar PMC be and hereby is tasked
> with the creation of a set of bylaws intended to encourage open
> development and increased participation in the Apache Pulsar Project;
> and be it further
>
> RESOLVED, that the Apache Pulsar Project be and hereby is tasked with
> the migration and rationalization of the Apache Incubator Pulsar
> podling; and be it further
>
> RESOLVED, that all responsibilities pertaining to the Apache Incubator
> Pulsar podling encumbered upon the Apache Incubator PMC are hereafter
> discharged.
>
> 
>
>
> --
> Matteo Merli
> 
> --
> Matteo Merli
> 
>

Re: [DISCUSSION] Pulsar maturity model assessment

[Sijie Guo]

On Wed, Aug 29, 2018 at 12:17 AM Matteo Merli 
wrote:

> On Tue, Aug 28, 2018 at 5:50 PM Dave Fisher  wrote:
>
> > This is very detailed and looks good with one important omission.
> >
> > How does one go on from being a contributor to committer / PMC status?
> > There should be a description about that.
> >
> > Please discuss!
> >
>
> That is a very good point. I have added a couple of sections to the
> "Contributing" page in the PR:
>  * https://github.com/apache/incubator-pulsar/pull/2459
>  * Preview link:
>
> https://github.com/apache/incubator-pulsar/blob/0515a9a12a77ca6ffdc8587125e21a32cdb61291/site2/website/contributing.md
>
> Also reporting it here for convenience:
>
>
> ---
> ## Becoming a committer
>
> Committers are community members that have write access to the project’s
> repositories, i.e., they can modify the code, documentation, and website
> by themselves and also accept other contributions.
>
> There is no strict protocol for becoming a committer. Candidates for new
> committers are typically people that are active contributors and
> community members.
>
> Being an active community member means participating on mailing list
> discussions, helping to answer questions, verifying release candidates,
> being respectful towards others, and following the meritocratic
> principles of community management. Since the
> [Apache Way](https://www.apache.org/foundation/governance/)
> has a strong focus on the project community, this part is very important.
>
> Of course, contributing code and documentation to the project is
> important as well. A good way to start is contributing improvements, new
> features, or bug fixes. You need to show that you take responsibility
> for the code that you contribute, add tests and documentation, and help
> maintaining it.
>
> Candidates for new committers are suggested by current committers or PMC
> members, and voted upon by the PMC.
>
> If you would like to become a committer, you should engage with the
> community and start contributing to Apache Pulsar in any of the above
> ways. You might also want to talk to other committers and ask for their
> advice and guidance.
>
> ## Becoming member of PMC
>
> The PMC is the project governance body. Committers or contributors that
> have demonstrated continued involvement with the community can be
> nominated to become members of the PMC.
>
> PMC members nominate new contributors to the project as either
> committers or as new PMC members, and PMC members cast votes on electing
> new committers or PMC members to the project. PMC members also have
> binding votes on any project matters. Refer to
> [ASF PMCs governance](
> http://www.apache.org/foundation/governance/pmcs.html)
> for a more detailed explanation of the duties and roles of the PMC.
>
> ---
>


👍

>
>
> > This is better and ties in with my main question above.
> >
>
> Created a separate PR to update the "team" page:
> https://github.com/apache/incubator-pulsar/pull/2470
>
> Matteo
> --
> Matteo Merli
> 
>

Re: [DISCUSSION] Pulsar maturity model assessment

[Sijie Guo]

Great work, Matteo!

+1; #shipit

- Sijie


On Mon, Aug 27, 2018 at 11:43 PM Matteo Merli  wrote:

> I have started a draft of the maturity model assessment.
>
> You can find it on the wiki:
>
> https://github.com/apache/incubator-pulsar/wiki/Apache-Maturity-Model-Assessment-for-Pulsar
>
> Please review it and provide feedback. Once there is agreement on this
> document, I think we should then do a vote to formalize the community
> position regarding TLP graduation.
>
> Matteo
> --
> Matteo Merli
> 
>

Re: [discuss] how to label milestone

[Sijie Guo]

Hi all,

I think we need to discuss how to label the issues and how to cherry-pick
the issues for bug fixes. Currently it is painful for doing a bugfix
release.
All the bug fixes are deferred to release manager cherry-picking to bug fix
branches. Since master and branches are quickly diverged with changes,
it is very painful for cherry-picks later and there is no clear process for
cherry-pick and labelling.

Shall we consider following the process what BookKeeper is using, having a
merge script to handle labelling issues/milestones and also cherry-picks?

Or any suggestions on how to improve this?

- Sijie

On Mon, Aug 6, 2018 at 10:50 AM Sijie Guo  wrote:

> Hi all,
>
> It seems we don't have a standard for labelling milestone. sometimes I am
> not lost and confused when I see a change was merged to master but marked
> in a bug fix milestone. so I am raising a discussion about this.
>
> I think every pull request that was created based on master, which would
> be merged as a commit of master, those pull requests should be first marked
> as major release milestone, for example: `2.2.0` release. If a pull request
> that was created based on a branch, which would be merged as a commit of
> that branch, those pull requests should be marked as minor release
> milestones, for example: `2.1.1` release. so that we have a clearer way on
> tracking what changes have been made to what releases. otherwise it is a
> bit hard to track when we have multiple releases ongoing.
>
> Thoughts?
>
> Thanks,
> Sijie
>
>

Re: PIP 23 : Message Tracing By Interceptors

[Sijie Guo]

On Fri, Aug 24, 2018 at 6:04 PM Matteo Merli  wrote:

> > Original, i just want to implement tracing by interceptors. We don’t must
> > to binding “tracing” and “interceptors” together.
> > I thougt that to implement message tracing in pulsar, we should implement
> > interceptors before. And interceptors might be useful in multiple
> scenarios.
>
> Yes, that's exactly what I meant. Interceptors are a good feature to have
> in general and tracing can be implemented on top of that.
> I think we should probably separate the 2 aspects and have this proposal
> just focus on interceptors.
>

I think the PIP was meant to focus on interceptors (and optionally a dump
client tracing interceptor).

We can have a separate PIP for "Integrating with OpenTracing" and another
PIP for "Integrating with OpenCensus" in future.
So we are able to reduce the scope and move PIP forward much faster.

- Sijie


>
> One other thing to consider is that for tracing to be "meaningful" we would
> also need to have support in broker side (and, as you mentioned, ideally in
> bookkeeper as well). That means that either:
>
>  * We provide "interceptors" hooks also for broker
>  * or we have some "fixed" tracing implementation that we support.
>
> In any case, my previous example was to show how, as a user, I'd like the
> enabling tracing in Pulsar should be. Since tracing will be a very common
> feature, we should aim to have a tight integration with few common tracing
> providers, while leaving the freedom for user to integrate with custom
> tracing systems.
>
> Matteo
>
>
>
> On Fri, Aug 24, 2018 at 3:02 AM 李鹏辉  wrote:
>
> > Original, i just want to implement tracing by interceptors. We don’t must
> > to binding “tracing” and “interceptors” together.
> >
> > I thougt that to implement message tracing in pulsar, we should implement
> > interceptors before. And interceptors might be useful in multiple
> > scenarios.
> >
> --
> Matteo Merli
> 
>

Re: PIP 23 : Message Tracing By Interceptors

[Sijie Guo]

On Thu, Aug 23, 2018 at 11:25 PM Matteo Merli 
wrote:

> Hi Penghui,
>
> the proposal looks good to me. I like the approach of using interceptors
> since they might be useful in multiple scenarios, for applications to add
> custom logging or processing.
>
> For tracing, in particular, I think we should also aim to have "out of the
> box" ready made solution as well, especially targeting standard APIs like
> OpenTracing. We shouldn't be dictating any particular implementation, but
> we should try to avoid the burden of integrating with tracing APIs to
> application developers.
> Ideally, as a user, I should be able to do something like :
>
> client.newConsumer()
>  .topic("xyz")
>  .subscriptionName("yyz")
>  .withTracing(OpenTracing.of(tracingLoggerConfig))
>

Why can't we use the proposed #intercept(...)? I am not sure we need to
distinguish "tracing" from other interceptors.

client.newConsumer()
.topic("xyz")
.subscriptionName("yyz")
.intercept(OpenTracingInterceptor.of(tracingLoggerConfig))




>  .subscribe();
>
> (Or maybe even configure that at the PulsarClient level)
>
>
> On Tue, Aug 21, 2018 at 11:13 AM Sijie Guo  wrote:
>
> > Updated. Thanks, Penghui!
> >
> > - Sijie
> >
> > On Tue, Aug 21, 2018 at 3:56 AM 李鹏辉  wrote:
> >
> > > Please add zhaijia to Authors.
> > >
> > > Sijie Guo  于2018年8月21日周二 下午6:48写道：
> > >
> > > > Thank you penghui! This is a great feature to add to Pulsar.
> > > >
> > > > I've copied your PIP to pulsar wiki as well :
> > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-pulsar/wiki/PIP-23:-Message-Tracing-By-Interceptors
> > > >
> > > > - Sijie
> > > >
> > > > On Tue, Aug 21, 2018 at 3:14 AM 李鹏辉  wrote:
> > > >
> > > > > Dear all
> > > > >
> > > > > This is a PIP to add feature of message trace implement by
> > > interceptors.
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1aigUKZ-AhQvugOfzUzypyMMeXfum5XlNkwuYK4KvxJs/edit?usp=sharing
> > > > >
> > > > > —
> > > > > Regards,
> > > > > Penghui Li
> > > > >
> > > >
> > >
> >
> --
> Matteo Merli
> 
>

Re: Starting the Graduation Process

[Sijie Guo]

On Thu, Aug 23, 2018 at 5:17 PM Dave Fisher  wrote:

> Hi -
>
> I think that the Pulsar community is just about ready to graduate to a Top
> Level Project.
>
> The graduation guide is here. [1]
> The incubation status file is here. [2] I made some changes today which
> have yet to propagate. They should by tomorrow.
> The Community Development project has maturity model self assessment. [3]
> It is recommended that the project do this assessment recording answers in
> the Wiki.
>
> Let’s get this started!


🎉🎉 +1 to start the process


>
> Regards,
> Dave
>
> [1] https://incubator.apache.org/guides/graduation.html
> [2] https://incubator.apache.org/projects/pulsar.html
> [3]
> https://community.apache.org/apache-way/apache-project-maturity-model.html
>

Re: PIP 23 : Message Tracing By Interceptors

[Sijie Guo]

Updated. Thanks, Penghui!

- Sijie

On Tue, Aug 21, 2018 at 3:56 AM 李鹏辉  wrote:

> Please add zhaijia to Authors.
>
> Sijie Guo  于2018年8月21日周二 下午6:48写道：
>
> > Thank you penghui! This is a great feature to add to Pulsar.
> >
> > I've copied your PIP to pulsar wiki as well :
> >
> >
> https://github.com/apache/incubator-pulsar/wiki/PIP-23:-Message-Tracing-By-Interceptors
> >
> > - Sijie
> >
> > On Tue, Aug 21, 2018 at 3:14 AM 李鹏辉  wrote:
> >
> > > Dear all
> > >
> > > This is a PIP to add feature of message trace implement by
> interceptors.
> > >
> > >
> > >
> >
> https://docs.google.com/document/d/1aigUKZ-AhQvugOfzUzypyMMeXfum5XlNkwuYK4KvxJs/edit?usp=sharing
> > >
> > > —
> > > Regards,
> > > Penghui Li
> > >
> >
>

Re: PIP 23 : Message Tracing By Interceptors

[Sijie Guo]

Thank you penghui! This is a great feature to add to Pulsar.

I've copied your PIP to pulsar wiki as well :
https://github.com/apache/incubator-pulsar/wiki/PIP-23:-Message-Tracing-By-Interceptors

- Sijie

On Tue, Aug 21, 2018 at 3:14 AM 李鹏辉  wrote:

> Dear all
>
> This is a PIP to add feature of message trace implement by interceptors.
>
>
> https://docs.google.com/document/d/1aigUKZ-AhQvugOfzUzypyMMeXfum5XlNkwuYK4KvxJs/edit?usp=sharing
>
> —
> Regards,
> Penghui Li
>

Re: Planning for Pulsar 2.2 release

[Sijie Guo]

Thank you Joe! :)

- Sijie

On Mon, Aug 20, 2018 at 2:51 PM Joe F  wrote:

> Ok if no one else is there i will take this
>
> Joe
>
> On Mon, Aug 20, 2018, 2:45 PM Sijie Guo  wrote:
>
> > Anyone volunteer for releasing 2.2?
> >
> > - Sijie
> >
> > On Thu, Aug 16, 2018 at 12:02 AM Sijie Guo  wrote:
> >
> > > Any committer volunteer for releasing 2.2?
> > >
> > > - Sijie
> > >
> > > On Wed, Aug 8, 2018 at 11:43 AM Sanjeev Kulkarni 
> > > wrote:
> > >
> > >> The timeline sounds good!
> > >> One more thing that users have asked about is feature parity between
> > >> python
> > >> and java clients so we should have some work items associated with
> this
> > >> task as well.
> > >>
> > >>
> > >> On Wed, Aug 8, 2018 at 11:40 AM Sijie Guo  wrote:
> > >>
> > >> > Hi all,
> > >> >
> > >> > Although we just announced 2.1, we already have a lot of new
> features
> > >> > merged since 2.1 candidate is cut. For example, Pulsar SQL, GCS
> > support
> > >> as
> > >> > tiered storage and such. Shall we target at releasing 2.2 at the end
> > of
> > >> > August? Since releasing usually takes 1~2 weeks, that would be
> > released
> > >> > around Mid September.
> > >> >
> > >> > What are the thoughts around this?
> > >> >
> > >> > Any committers volunteer to drive 2.2 release?
> > >> >
> > >> > - Sijie
> > >> >
> > >>
> > >
> >
>

Re: Planning for Pulsar 2.2 release

[Sijie Guo]

Thank you Ali. However the release manager has be an Apache committer
first, since it requires permissions to access ASF infrastructure for the
whole release process.

- Sijie

On Mon, Aug 20, 2018 at 2:50 PM Ali Ahmed  wrote:

> I will volunteer.
>
> -Ali
>
> On Mon, Aug 20, 2018 at 2:45 PM Sijie Guo  wrote:
>
> > Anyone volunteer for releasing 2.2?
> >
> > - Sijie
> >
> > On Thu, Aug 16, 2018 at 12:02 AM Sijie Guo  wrote:
> >
> > > Any committer volunteer for releasing 2.2?
> > >
> > > - Sijie
> > >
> > > On Wed, Aug 8, 2018 at 11:43 AM Sanjeev Kulkarni 
> > > wrote:
> > >
> > >> The timeline sounds good!
> > >> One more thing that users have asked about is feature parity between
> > >> python
> > >> and java clients so we should have some work items associated with
> this
> > >> task as well.
> > >>
> > >>
> > >> On Wed, Aug 8, 2018 at 11:40 AM Sijie Guo  wrote:
> > >>
> > >> > Hi all,
> > >> >
> > >> > Although we just announced 2.1, we already have a lot of new
> features
> > >> > merged since 2.1 candidate is cut. For example, Pulsar SQL, GCS
> > support
> > >> as
> > >> > tiered storage and such. Shall we target at releasing 2.2 at the end
> > of
> > >> > August? Since releasing usually takes 1~2 weeks, that would be
> > released
> > >> > around Mid September.
> > >> >
> > >> > What are the thoughts around this?
> > >> >
> > >> > Any committers volunteer to drive 2.2 release?
> > >> >
> > >> > - Sijie
> > >> >
> > >>
> > >
> >
>
>
> --
> -Ali
>

Re: Planning for Pulsar 2.2 release

[Sijie Guo]

Anyone volunteer for releasing 2.2?

- Sijie

On Thu, Aug 16, 2018 at 12:02 AM Sijie Guo  wrote:

> Any committer volunteer for releasing 2.2?
>
> - Sijie
>
> On Wed, Aug 8, 2018 at 11:43 AM Sanjeev Kulkarni 
> wrote:
>
>> The timeline sounds good!
>> One more thing that users have asked about is feature parity between
>> python
>> and java clients so we should have some work items associated with this
>> task as well.
>>
>>
>> On Wed, Aug 8, 2018 at 11:40 AM Sijie Guo  wrote:
>>
>> > Hi all,
>> >
>> > Although we just announced 2.1, we already have a lot of new features
>> > merged since 2.1 candidate is cut. For example, Pulsar SQL, GCS support
>> as
>> > tiered storage and such. Shall we target at releasing 2.2 at the end of
>> > August? Since releasing usually takes 1~2 weeks, that would be released
>> > around Mid September.
>> >
>> > What are the thoughts around this?
>> >
>> > Any committers volunteer to drive 2.2 release?
>> >
>> > - Sijie
>> >
>>
>

Re: PIP 20: Mechanism to revoke TLS authentication

[Sijie Guo]

On Mon, Aug 20, 2018 at 5:59 AM Ivan Kelly  wrote:

> > Instead of using zookeeper, can we consider using a managed ledger or a
> system topic for keeping all these revoked keys?
>
> Do we already have an example of a system topic in use?


Functions is using system topics. And all its topics are under `functions`
namespace.


> Is there a
> custom namespace that's restricted to the admin role?
>

Currently I don't think we restrict the namespace to admin role. But we
should probably do so.

- Sijie



>
> -Ivan
>

Re: Planning for Pulsar 2.2 release

[Sijie Guo]

Any committer volunteer for releasing 2.2?

- Sijie

On Wed, Aug 8, 2018 at 11:43 AM Sanjeev Kulkarni 
wrote:

> The timeline sounds good!
> One more thing that users have asked about is feature parity between python
> and java clients so we should have some work items associated with this
> task as well.
>
>
> On Wed, Aug 8, 2018 at 11:40 AM Sijie Guo  wrote:
>
> > Hi all,
> >
> > Although we just announced 2.1, we already have a lot of new features
> > merged since 2.1 candidate is cut. For example, Pulsar SQL, GCS support
> as
> > tiered storage and such. Shall we target at releasing 2.2 at the end of
> > August? Since releasing usually takes 1~2 weeks, that would be released
> > around Mid September.
> >
> > What are the thoughts around this?
> >
> > Any committers volunteer to drive 2.2 release?
> >
> > - Sijie
> >
>

[DISCUSS] Pulsar 2.1.1 Release

[Sijie Guo]

Hi all,

It has been 2 weeks since 2.1.0 released. There were bunch of issues
reported by users and fixes have been merged. I am thinking of starting a
2.1.1 release next week. Any thoughts?

- Sijie

Re: PIP 20: Mechanism to revoke TLS authentication

[Sijie Guo]

Ivan,

Thank you for writing this up. This PIP looks great to me! +1

just one question:

> This will add the subject key identifier to zookeeper under
/tls/revoked/. All brokers and proxies cache the children
/tls/revoked.

Instead of using zookeeper, can we consider using a managed ledger or a
system topic for keeping all these revoked keys?

- Sijie

On Tue, Aug 7, 2018 at 1:12 AM Ivan Kelly  wrote:

> Hi folks,
>
> This is a PIP to add a mechanism to block TLS client certs from
> accessing Pulsar if they have been compromised.
>
> This is a relatively small change, but I thought it best to put it to
> the community before moving ahead with it, as people may have opinions
> on the approach.
>
> The PIP is here:
>
> https://github.com/apache/incubator-pulsar/wiki/PIP-20%3A-Mechanism-to-revoke-TLS-authentication
>
> Cheers,
> Ivan
>

Re: PIP 21: Pulsar Edge Component

[Sijie Guo]

Thank you David for proposing this PIP.

I have also added this to pulsar wiki:
https://github.com/apache/incubator-pulsar/wiki/PIP-21:-Pulsar-Edge-Component

- Sijie

On Tue, Aug 7, 2018 at 2:12 PM David Kjerrumgaard  wrote:

> Hi folks,
>
> This is a PIP to add a Pulsar Edge component to enable to deployment of
> Pulsar functions on resource-constrained environments, such as IoT gateways
> to allow the processing to occur closer to the event source.
>
>
> https://docs.google.com/document/d/1BMGOPkXIqe4ux7DioG6snJ7LKbswFbUFRhJ0LBMvhtY/edit?usp=sharing
>
>
> --
> Regards,
> David Kjerrumgaard
>

Re: PIP 22 : Pulsar Dead Letter Topic

[Sijie Guo]

Thank you Penghui.

I have also created the wiki page for this PIP.

https://github.com/apache/incubator-pulsar/wiki/PIP-22:-Pulsar-Dead-Letter-Topic

- Sijie

On Fri, Aug 10, 2018 at 5:37 AM 李鹏辉gmail  wrote:

> Dear all
>
> This is a PIP to add feature of Dead Letter Topic to control the message
> redelivery behavior by users. If the delivery times, which was tracked in
> Redelivery Tracker, reached user defined threshold, we re-deliver the
> message to Dead Letter Topic, then mark the message acked in original
> subscription.
>
>
> https://docs.google.com/document/d/1ClfrGg87nihlrHVyeSwfXgoM6Qx3Fnjcx5bJsnEcWO4/edit?usp=sharing
>
> —
> Regards,
> Penghui Li

Pulsar Blog

[Sijie Guo]

FYI.

Now we have a blog entry in apache website:
https://pulsar.incubator.apache.org/blog/ for latests updates of Apache
Pulsar.

This is the RSS feed : https://pulsar.incubator.apache.org/blog/feed.xml

- Sijie

Planning for Pulsar 2.2 release

[Sijie Guo]

Hi all,

Although we just announced 2.1, we already have a lot of new features
merged since 2.1 candidate is cut. For example, Pulsar SQL, GCS support as
tiered storage and such. Shall we target at releasing 2.2 at the end of
August? Since releasing usually takes 1~2 weeks, that would be released
around Mid September.

What are the thoughts around this?

Any committers volunteer to drive 2.2 release?

- Sijie

Re: Consult: PIP work progress

[Sijie Guo]

Zongyang,

Thank you for reaching out to Pulsar community.

In general, PIP is used for proposing new features, improvements and user
impact changes. However I don't think Pulsar community has defined any
strict rules for people submitting PIP.
The general rule for PIP is for people in the community to communicate what
are the "big" changes are coming into Pulsar release. We'd love to collect
feedback from the community to improve the PIP process.

Other comments inline.

On Tue, Aug 7, 2018 at 12:29 AM Xiao Zongyang 
wrote:

> Hi Pulsar Community!
>
>
> Zongyang, here!
>
>
> I'm now reading PIP(Pulsar Improvement Proposal) on github. And I am a
> little confused about how PIP works and what are the principles of PIP.
>
>
> Disscused in more details, the actual problems are:
>
>
> 1. Do we need a State of PIP named released, which means the proposal has
> been implemented but is coming in next release?
>

Ideally we should have a field in the PIP about what release this PIP has
been shipped.
I have tried to update those PIPs with correct release labelled.


> 2. What is the boundary between providing a proposal of changes and not?
> Do we have some principles to obey?
>

- New features
- Improvements
- Protocol changes (adding new proto commands)
- API changes

I think protocol and API changes have to be a PIP, since the change has
huge-enough impact to all the users of Pulsar.

However for others, I don't think there is a strict rule for it. Although
we recommend people to write PIP for new features and improvements.
Baseline is if you think your change requires some discussions from
community, it is a PIP.
Otherwise it can go with regular pull requests.


> 3. What is the lifecycle of a PIP? How does the process of PIP works?
> What will we do for transforming the PIP from one State to the next?
>

We don't have any official vote process for PIP. We are kind of treating a
feature is accepted if there is no objection in the community after PIP is
sent out. You know engineers are just lazy :)


> 4. The KIP(Kafka Improvement Proposal) has a section "Compatibility,
> Deprecation, and Migration Plan" when describes the effect of existing
> users. It seems that we don't include that section, what are the
> considerations behind this?
>

Yes. If a PIP impacts users, it has to document "Compatibility and
Migration Plan". We don't have a template for people to follow. But when a
PIP is sent out and if this feature requires considering compatibility and
migration plans,
people in the community will request it.


As I said, we are still improving our process. If you have any idea on
helping us improving the process, feel free to contribute at this area. for
example, maybe you can help us drafting a process for PIP or a template for
PIP.=


Hope this helps.

>
> Thank you guys to answering my questions!
>
>
> Best Regards

Zongyang

[discuss] how to label milestone

[Sijie Guo]

Hi all,

It seems we don't have a standard for labelling milestone. sometimes I am
not lost and confused when I see a change was merged to master but marked
in a bug fix milestone. so I am raising a discussion about this.

I think every pull request that was created based on master, which would be
merged as a commit of master, those pull requests should be first marked as
major release milestone, for example: `2.2.0` release. If a pull request
that was created based on a branch, which would be merged as a commit of
that branch, those pull requests should be marked as minor release
milestones, for example: `2.1.1` release. so that we have a clearer way on
tracking what changes have been made to what releases. otherwise it is a
bit hard to track when we have multiple releases ongoing.

Thoughts?

Thanks,
Sijie

Re: [Discuss] Blog at pulsar website

[Sijie Guo]

As discussed, I try to enable the blog in pulsar website and have one
example for 2.1.0 release. The idea is to have a channel (e.g. rss feeds)
in Pulsar website for users to catch up latest information in Pulsar.

https://github.com/apache/incubator-pulsar/pull/2308

Let me know how do you guys think.

- Sijie

On Tue, Jun 26, 2018 at 11:46 PM Nozomi Kurihara 
wrote:

> Hi,
>
>
> I think it is good idea to have blogs/news at pulsar website since users
> can easily catch up latest information.
>
>
> Regards,
>
> Nozomi
>
> ________
> 差出人: Sijie Guo 
> 送信日時: 2018年6月27日 15:33:55
> 宛先: dev@pulsar.incubator.apache.org
> 件名: [Discuss] Blog at pulsar website
>
> Hi,
>
> Do you think it is worth having a blogs (or news) category at pulsar
> website, for publishing news about releases or even blogs about new
> features?
>
> I see quite a lot of projects are doing so, for example:
>
> Arrow: http://arrow.apache.org/blog/
> Beam: https://beam.apache.org/blog/
> RocketMQ: https://rocketmq.apache.org/year-archive/
>
> Both Arrow and Beam started publishing blog posts since they were in
> incubator.
>
> https://beam.apache.org/beam/release/2016/06/15/first-release.html
> https://rocketmq.apache.org/rocketmq/filter-messages-by-sql92-in-rocketmq/
>
>
> Thoughts?
>
> - Sijie
>

[Call for Contribution] Translating Pulsar Documentation

[Sijie Guo]

Hi all,

Now the new Pulsar website has integrated Crowdin for localization. If
anyone is interested in contributing translations to Pulsar documentation,
feel free to join the Crowdin project
https://crowdin.com/project/apache-pulsar and contribute translations
there.

https://github.com/apache/incubator-pulsar/blob/master/site2/README.md#translation-and-localization


just be aware that all the contributed translations will be licensed under
Apache License v2.

Let me know if you have any questions.

- Sijie

Re: [DISCUSS] New Pulsar Documentation Website

[Sijie Guo]

Now the new website is live: https://pulsar.incubator.apache.org/

All documentations are versioned and localized. By default, it is pointing
to the latest released version. Documentation for different versions are
accessible from https://pulsar.incubator.apache.org/en/versions/

If you are updating documentation for latest development version, you
should modify the docs under
https://github.com/apache/incubator-pulsar/tree/master/site2/docs
If you want to fix documentation in a released version, you can modify the
docs under `versioned_docs`:
https://github.com/apache/incubator-pulsar/tree/master/site2/website/versioned_docs

I also put up a README for the instructions on how to contribute
documentation changes:
https://github.com/apache/incubator-pulsar/blob/master/site2/README.md

Let me know if you have any questions. More contributions are welcome!

- Sijie

On Thu, Aug 2, 2018 at 2:01 PM Sijie Guo  wrote:

> Hi all,
>
> Now the new website has all the content in-sync with existing website:
> https://pulsar.incubator.apache.org/staging/
>
> Since I am doing 2.1.0 release, I am planning to cut over the current
> website to new website at the last step of 2.1.0 release.
> All the documentation development will happen in the new website after
> that.
>
> Let me know if you have any questions or concerns. If I don't see any
> concerns in a few hours, I will move forward to cutover the new website.
>
> Thanks,
> Sijie
>
>
>
> On Fri, Jul 27, 2018 at 3:22 PM Jai Asher  wrote:
>
>> It looks really nice and has search functionality as well.
>>
>> Nice work @Chris.
>>
>> On Fri, Jul 27, 2018 at 2:18 PM, Sijie Guo  wrote:
>>
>> > Hi all,
>> >
>> > In the past few weeks, Chris Kellogg has been working on a new Pulsar
>> > documentation website using Docusaurus <https://docusaurus.io/>. It is
>> a
>> > Facebook open source project, designed for easy to maintain open source
>> > documentation websites.
>> >
>> > It is a very popular and documentation-focused framework and used by a
>> lot
>> > of open source projects. It addressed a few problems that we had in
>> current
>> > documentation website.
>> >
>> > - Sidebar: sidebar doesn't work well in current website. for example,
>> it is
>> > impossible to navigate to any documentation page on mobile phones; if
>> you
>> > have a smaller screen, you can not scroll down the sidebar to navigate
>> to
>> > the pages at the bottom.
>> > - Hyperlinks on sections: there are some random characters generated in
>> the
>> > hyperlinks for markdown sections. It is non-deterministic and will cause
>> > problems when linking documents.
>> >
>> > Docusaurus gives us more benefits than current website.
>> >
>> > - documentation focused: Docusaurus is designed for documentation
>> websites.
>> > so it brings the focus on writing documents in markdown. The framework
>> > takes care of the rest of stuffs like versioning, sidebar and even
>> > translations.
>> > - much better sidebar: sidebar is working well across different browsers
>> > and mobile phones.
>> > - simpler hyperlinks: you can just link to other documents using
>> document
>> > filenames. it is working well for both website and navigating at github
>> > directly. You don't have to compute any relative links.
>> > - versioning: it manages all the versioning stuff.
>> > - translations: it integrates Crowdin for translation contributions.
>> > - search-box: a search-box integrated with Algolia.
>> > - edit/translate button: for each document, it has edit button (for
>> english
>> > language) and translate button (for non-english language). so people can
>> > quickly contribute documentation changes (such as fixing typos) by
>> clicking
>> > those buttons. It lowers
>> > the contribution barrier.
>> >
>> >
>> > Currently the new site is alive under
>> > https://pulsar.incubator.apache.org/staging/. Please take a look at the
>> > new
>> > website, try it out and give us feedbacks.
>> >
>> > For contributors, I also wrote an instruction on how to contribute the
>> > documentation. So you can get a sense on how to contribute documents in
>> the
>> > new site.
>> >
>> > https://github.com/sijie/incubator-pulsar/blob/
>> > 4a3a938d6ae1d9a5b1ae83d49293513c6e4f1711/site2/README.md
>> >
>> > Last, big shout out to Chris for his great work on this!
>> >
>> > - Sijie
>> >
>>
>

[ANNOUNCE] Apache Pulsar 2.1.0-incubating released

[Sijie Guo]

The Apache Pulsar team is proud to announce Apache Pulsar version
2.1.0-incubating.

This is the seventh Pulsar release after entering the Apache Incubator.

Pulsar is a highly scalable, low latency messaging streaming platform
running on
commodity hardware. It provides simple pub/sub semantics over topics,
supports
at-least-once and effectively-once delivery of messages, automatic cursor
management
for subscribers, and multi-tenancy and cross-datacenter replication.
Additionally,
it also provides lightweight stream native processing through Pulsar
Functions.

2.1.0-incubating release introduces several major features and improvements
from many contributors. These major features include:

- Pulsar IO : A
serverless inspired connector framework for moving data in and out of
Apache Pulsar leveraging Pulsar Functions
. 6
builtin connectors are included: Kafka
, Kinesis
, RabbitMQ
, Aerospike
, Cassandra
 and Twitter
Firehose .

- Tiered Storage
: An
offloading mechanism in Pulsar segment store to offload old segments from
Apache BookKeeper into long term storage such as HDFS, S3, GCS. S3 is
currently supported in this release.

- Stateful Function
: Pulsar
Functions introduces a new State API
 to store
function state back to Apache BookKeeper. It is a deep integration with
BookKeeper's table service and it will simplify developing stateful
applications using Pulsar Functions. It is available as a developer-preview
feature in Java SDK.

- Go Client
: A cgo
wrapper over existing c++ client.

- More Schemas are introduced, including Avro

and Protobuf

.


For Pulsar release details and downloads, visit:

https://pulsar.incubator.apache.org/download

Release Notes are at:
http://pulsar.apache.org/release-notes/#2.1.0-incubating

We would like to thank the contributors that made the release possible.

Regards,

The Pulsar Team

DISCLAIMER:
Apache Pulsar is an effort undergoing incubation at The Apache Software
Foundation (ASF), sponsored by the Apache Incubator PMC. Incubation is
required of all newly accepted projects until a further review indicates
that
the infrastructure, communications, and decision making process have
stabilized in a manner consistent with other successful ASF projects. While
incubation status is not necessarily a reflection of the completeness or
stability
of the code, it does indicate that the project has yet to be fully endorsed
by the ASF.

Re: [DISCUSS] New Pulsar Documentation Website

[Sijie Guo]

Hi all,

Now the new website has all the content in-sync with existing website:
https://pulsar.incubator.apache.org/staging/

Since I am doing 2.1.0 release, I am planning to cut over the current
website to new website at the last step of 2.1.0 release.
All the documentation development will happen in the new website after
that.

Let me know if you have any questions or concerns. If I don't see any
concerns in a few hours, I will move forward to cutover the new website.

Thanks,
Sijie



On Fri, Jul 27, 2018 at 3:22 PM Jai Asher  wrote:

> It looks really nice and has search functionality as well.
>
> Nice work @Chris.
>
> On Fri, Jul 27, 2018 at 2:18 PM, Sijie Guo  wrote:
>
> > Hi all,
> >
> > In the past few weeks, Chris Kellogg has been working on a new Pulsar
> > documentation website using Docusaurus <https://docusaurus.io/>. It is a
> > Facebook open source project, designed for easy to maintain open source
> > documentation websites.
> >
> > It is a very popular and documentation-focused framework and used by a
> lot
> > of open source projects. It addressed a few problems that we had in
> current
> > documentation website.
> >
> > - Sidebar: sidebar doesn't work well in current website. for example, it
> is
> > impossible to navigate to any documentation page on mobile phones; if you
> > have a smaller screen, you can not scroll down the sidebar to navigate to
> > the pages at the bottom.
> > - Hyperlinks on sections: there are some random characters generated in
> the
> > hyperlinks for markdown sections. It is non-deterministic and will cause
> > problems when linking documents.
> >
> > Docusaurus gives us more benefits than current website.
> >
> > - documentation focused: Docusaurus is designed for documentation
> websites.
> > so it brings the focus on writing documents in markdown. The framework
> > takes care of the rest of stuffs like versioning, sidebar and even
> > translations.
> > - much better sidebar: sidebar is working well across different browsers
> > and mobile phones.
> > - simpler hyperlinks: you can just link to other documents using document
> > filenames. it is working well for both website and navigating at github
> > directly. You don't have to compute any relative links.
> > - versioning: it manages all the versioning stuff.
> > - translations: it integrates Crowdin for translation contributions.
> > - search-box: a search-box integrated with Algolia.
> > - edit/translate button: for each document, it has edit button (for
> english
> > language) and translate button (for non-english language). so people can
> > quickly contribute documentation changes (such as fixing typos) by
> clicking
> > those buttons. It lowers
> > the contribution barrier.
> >
> >
> > Currently the new site is alive under
> > https://pulsar.incubator.apache.org/staging/. Please take a look at the
> > new
> > website, try it out and give us feedbacks.
> >
> > For contributors, I also wrote an instruction on how to contribute the
> > documentation. So you can get a sense on how to contribute documents in
> the
> > new site.
> >
> > https://github.com/sijie/incubator-pulsar/blob/
> > 4a3a938d6ae1d9a5b1ae83d49293513c6e4f1711/site2/README.md
> >
> > Last, big shout out to Chris for his great work on this!
> >
> > - Sijie
> >
>

Re: [ANNOUNCE] github.com/Comcast/pulsar-client-go

[Sijie Guo]

On Tue, Jul 31, 2018 at 4:43 PM Adam Williams  wrote:

> > the starting point would be making the interface consistent across these
> two clients
>
> I agree this is a good goal and understand the desire for consistency,
> although I'm not sure that 100% parity would make sense given the
> current interface. Definitely naming can be changed to match the cgo
> client whenever possible, eg Client.NewProducer vs
> Client.CreateProducer (note that the C++ and Java clients differ
> here). Some things though may not translate nicely into the native Go
> client, such as ClientOptions.IOThreads, since Go does not directly
> expose OS threads. And then there are design decisions that would be
> worth discussing, such as having a global client timeout for certain
> operations (ClientOptions.OperationTimeoutSeconds) versus using
> context.Context for operation cancelation/timeouts at the method
> level.
>
> Maybe there can be a more minimal interface both clients satisfy while
> allowing each to provide custom functionalities?


I think a minimal interface probably is a good approach to start with. We
can also document the differences between cgo client and native go client,
as as what you pointed out there are things applied to cgo client don't
apply to native go client.



> Another goal worth
> considering is consistent behavior between this client and the others
> (Java, C++).
>
>
>
>
>
> On Mon, Jul 30, 2018 at 7:19 PM Sijie Guo  wrote:
> >
> > Awesome work, Adam! It is really great to see a native go client under
> > development. It will be interesting to discuss a plan how to consolidate
> > the native go client and cgo client. probably one of the starting point
> > would be making the interface consistent across these two clients.
> >
> > - Sijie
> >
> >
> > On Mon, Jul 30, 2018 at 2:44 PM Adam Williams 
> wrote:
> >
> > > Hello,
> > >
> > > I'd like to announce a newly released Go client library for Pulsar:
> > > https://github.com/Comcast/pulsar-client-go.
> > >
> > > The client follows semantic versioning and as of today is tagged at
> > > v0.1.0. This indicates that it is unstable and that breaking changes
> are
> > > allowed (up until the v1.x release) and should be expected.
> > >
> > > There is already a Go client included in the official Pulsar repository
> > > (under /pulsar-client-go directory). It is a wrapper around the C++
> > > library. As such, cgo must be used when compiling. In contrast, this
> newly
> > > released client is 100% Go. This is the main reason it was open
> sourced, as
> > > we hope a 100% Go client will be a valuable addition to the Pulsar
> > > ecosystem since it avoids the complications associated with wrapping a
> C++
> > > library.
> > >
> > > We'd like to invite collaborators to help with the development of this
> > > library. The README lists steps to start contributing, goals, and also
> some
> > > of the major known deficiencies of the library. In addition to this
> mailing
> > > list, we propose that Github issues be used to coordinate efforts and
> track
> > > bugs and feature development.
> > >
> > > We hope this contribution is an additive one and look forward to
> working
> > > with the community to bring it up to par with the other client
> libraries.
> > >
>

[Result] [VOTE] Pulsar Release 2.1.0-incubating Candidate 5

[Sijie Guo]

The vote is now closed with 3 +1s and no -1s.

Binding +1s

 * Dave Fisher
 * Justin Mclean
 * Jim Jagielski


Thank you,
Sijie

On Wed, Jul 25, 2018 at 11:17 AM Sijie Guo  wrote:

> This is the 5th release candidate for Apache Pulsar, version
> 2.1.0-incubating.
>
> Link to the vote on dev@pulsar list:
> https://lists.apache.org/thread.html/9f1ec1266c02dde859d19e11a61416b3747eb4920d8bd8868dbb2988@%3Cdev.pulsar.apache.org%3E
>
> Pulsar is a highly scalable, low latency messaging platform running on
> commodity hardware.
> It provides simple pub-sub semantics over topics, guaranteed at-least-once
> and effective-once delivery of messages, automatic cursor management for
> subscribers, and geo-replication.
>
> This is a feature release, introducing following new features:
>
> - Pulsar I/O: a serverless connector framework built on top of Pulsar
> Functions.
> - TieredStorage: offloads bookkeeper ledgers to a cold storage (like S3).
> - Stateful Function: Pulsar Functions supports storing state back to
> bookkeeper.
> - Go Client: a Go client wrapping over existing cpp client.
> - Avro and Protobuf schema are supported.
>
> Full list of changes and fixes are available:
>
> https://github.com/apache/incubator-pulsar/milestone/13?closed=1
>
> *** Please download, test and vote on this release. This vote will stay
> open
> for at least 72 hours ***
>
> Note that we are voting upon the source (tag), binaries are provided for
> convenience.
>
> Source and binary files:
>
> https://dist.apache.org/repos/dist/dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-5/
>
> SHA-1 checksums:
>
> f0f6b6d7c48d119da36fc4fa1f934fd1f9ba8498  
> ./apache-pulsar-2.1.0-incubating-bin.tar.gz
>
> b6a968249a7c5d21be89ece27895b862efda9a92  
> ./apache-pulsar-2.1.0-incubating-src.tar.gz
>
> ddcf929eb18082ff2b9979507054453e8e803ed3  
> ./apache-pulsar-io-connectors-2.1.0-incubating-bin.tar.gz
>
>
> Maven staging repo:
> https://repository.apache.org/content/repositories/orgapachepulsar-1025/
>
> The tag to be voted upon:
> v2.1.0-incubating-candidate-5 (75f79b77f55fbad351cb517e7f3b8ba4ab2dc177)
>
> https://github.com/apache/incubator-pulsar/releases/tag/v2.1.0-incubating-candidate-5
>
> Pulsar's KEYS file containing PGP keys we use to sign the release:
> https://dist.apache.org/repos/dist/release/incubator/pulsar/KEYS
>
> Please download the the source package, and follow the README to build
> and run the Pulsar standalone service.
>
> A few notes for voting 2.1.0 release:
>
> - For compiling source distribution, use "mvn clean install  
> -Dmaven.gitcommitid.skip=true
> [-DskipTests]".
> - Starting from this release, Pulsar is shipping RPM & DEB packages for
> pulsar non-java clients.
> - Starting from this release, Pulsar is shipping a binary distribution
> called `apache-pulsar-io-connectors`
>   including all built-in connectors. Each connector is packaged with all
> their dependencies and
>   a META-INF/DEPEDENCIES file with licensing information for
> all dependencies.
> - Since there are a lot of features introduced in 2.1, You can use the
> instructions documented in the link for
>   validating the binary release candidate:
> https://github.com/apache/incubator-pulsar/wiki/Release-Candidate-Validation
>
> - Sijie
>
>

Re: [VOTE] Pulsar Release 2.1.0-incubating Candidate 5

[Sijie Guo]

Can any people from IPMC help review and vote this release?

Sijie

On Wed, Jul 25, 2018 at 11:17 AM Sijie Guo  wrote:

> This is the 5th release candidate for Apache Pulsar, version
> 2.1.0-incubating.
>
> Link to the vote on dev@pulsar list:
> https://lists.apache.org/thread.html/9f1ec1266c02dde859d19e11a61416b3747eb4920d8bd8868dbb2988@%3Cdev.pulsar.apache.org%3E
>
> Pulsar is a highly scalable, low latency messaging platform running on
> commodity hardware.
> It provides simple pub-sub semantics over topics, guaranteed at-least-once
> and effective-once delivery of messages, automatic cursor management for
> subscribers, and geo-replication.
>
> This is a feature release, introducing following new features:
>
> - Pulsar I/O: a serverless connector framework built on top of Pulsar
> Functions.
> - TieredStorage: offloads bookkeeper ledgers to a cold storage (like S3).
> - Stateful Function: Pulsar Functions supports storing state back to
> bookkeeper.
> - Go Client: a Go client wrapping over existing cpp client.
> - Avro and Protobuf schema are supported.
>
> Full list of changes and fixes are available:
>
> https://github.com/apache/incubator-pulsar/milestone/13?closed=1
>
> *** Please download, test and vote on this release. This vote will stay
> open
> for at least 72 hours ***
>
> Note that we are voting upon the source (tag), binaries are provided for
> convenience.
>
> Source and binary files:
>
> https://dist.apache.org/repos/dist/dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-5/
>
> SHA-1 checksums:
>
> f0f6b6d7c48d119da36fc4fa1f934fd1f9ba8498  
> ./apache-pulsar-2.1.0-incubating-bin.tar.gz
>
> b6a968249a7c5d21be89ece27895b862efda9a92  
> ./apache-pulsar-2.1.0-incubating-src.tar.gz
>
> ddcf929eb18082ff2b9979507054453e8e803ed3  
> ./apache-pulsar-io-connectors-2.1.0-incubating-bin.tar.gz
>
>
> Maven staging repo:
> https://repository.apache.org/content/repositories/orgapachepulsar-1025/
>
> The tag to be voted upon:
> v2.1.0-incubating-candidate-5 (75f79b77f55fbad351cb517e7f3b8ba4ab2dc177)
>
> https://github.com/apache/incubator-pulsar/releases/tag/v2.1.0-incubating-candidate-5
>
> Pulsar's KEYS file containing PGP keys we use to sign the release:
> https://dist.apache.org/repos/dist/release/incubator/pulsar/KEYS
>
> Please download the the source package, and follow the README to build
> and run the Pulsar standalone service.
>
> A few notes for voting 2.1.0 release:
>
> - For compiling source distribution, use "mvn clean install  
> -Dmaven.gitcommitid.skip=true
> [-DskipTests]".
> - Starting from this release, Pulsar is shipping RPM & DEB packages for
> pulsar non-java clients.
> - Starting from this release, Pulsar is shipping a binary distribution
> called `apache-pulsar-io-connectors`
>   including all built-in connectors. Each connector is packaged with all
> their dependencies and
>   a META-INF/DEPEDENCIES file with licensing information for
> all dependencies.
> - Since there are a lot of features introduced in 2.1, You can use the
> instructions documented in the link for
>   validating the binary release candidate:
> https://github.com/apache/incubator-pulsar/wiki/Release-Candidate-Validation
>
> - Sijie
>

Re: [ANNOUNCE] github.com/Comcast/pulsar-client-go

[Sijie Guo]

Awesome work, Adam! It is really great to see a native go client under
development. It will be interesting to discuss a plan how to consolidate
the native go client and cgo client. probably one of the starting point
would be making the interface consistent across these two clients.

- Sijie


On Mon, Jul 30, 2018 at 2:44 PM Adam Williams  wrote:

> Hello,
>
> I'd like to announce a newly released Go client library for Pulsar:
> https://github.com/Comcast/pulsar-client-go.
>
> The client follows semantic versioning and as of today is tagged at
> v0.1.0. This indicates that it is unstable and that breaking changes are
> allowed (up until the v1.x release) and should be expected.
>
> There is already a Go client included in the official Pulsar repository
> (under /pulsar-client-go directory). It is a wrapper around the C++
> library. As such, cgo must be used when compiling. In contrast, this newly
> released client is 100% Go. This is the main reason it was open sourced, as
> we hope a 100% Go client will be a valuable addition to the Pulsar
> ecosystem since it avoids the complications associated with wrapping a C++
> library.
>
> We'd like to invite collaborators to help with the development of this
> library. The README lists steps to start contributing, goals, and also some
> of the major known deficiencies of the library. In addition to this mailing
> list, we propose that Github issues be used to coordinate efforts and track
> bugs and feature development.
>
> We hope this contribution is an additive one and look forward to working
> with the community to bring it up to par with the other client libraries.
>

[DISCUSS] New Pulsar Documentation Website

[Sijie Guo]

Hi all,

In the past few weeks, Chris Kellogg has been working on a new Pulsar
documentation website using Docusaurus . It is a
Facebook open source project, designed for easy to maintain open source
documentation websites.

It is a very popular and documentation-focused framework and used by a lot
of open source projects. It addressed a few problems that we had in current
documentation website.

- Sidebar: sidebar doesn't work well in current website. for example, it is
impossible to navigate to any documentation page on mobile phones; if you
have a smaller screen, you can not scroll down the sidebar to navigate to
the pages at the bottom.
- Hyperlinks on sections: there are some random characters generated in the
hyperlinks for markdown sections. It is non-deterministic and will cause
problems when linking documents.

Docusaurus gives us more benefits than current website.

- documentation focused: Docusaurus is designed for documentation websites.
so it brings the focus on writing documents in markdown. The framework
takes care of the rest of stuffs like versioning, sidebar and even
translations.
- much better sidebar: sidebar is working well across different browsers
and mobile phones.
- simpler hyperlinks: you can just link to other documents using document
filenames. it is working well for both website and navigating at github
directly. You don't have to compute any relative links.
- versioning: it manages all the versioning stuff.
- translations: it integrates Crowdin for translation contributions.
- search-box: a search-box integrated with Algolia.
- edit/translate button: for each document, it has edit button (for english
language) and translate button (for non-english language). so people can
quickly contribute documentation changes (such as fixing typos) by clicking
those buttons. It lowers
the contribution barrier.


Currently the new site is alive under
https://pulsar.incubator.apache.org/staging/. Please take a look at the new
website, try it out and give us feedbacks.

For contributors, I also wrote an instruction on how to contribute the
documentation. So you can get a sense on how to contribute documents in the
new site.

https://github.com/sijie/incubator-pulsar/blob/4a3a938d6ae1d9a5b1ae83d49293513c6e4f1711/site2/README.md

Last, big shout out to Chris for his great work on this!

- Sijie

[VOTE] Pulsar Release 2.1.0-incubating Candidate 5

[Sijie Guo]

This is the 5th release candidate for Apache Pulsar, version
2.1.0-incubating.

Link to the vote on dev@pulsar list:
https://lists.apache.org/thread.html/9f1ec1266c02dde859d19e11a61416b3747eb4920d8bd8868dbb2988@%3Cdev.pulsar.apache.org%3E

Pulsar is a highly scalable, low latency messaging platform running on
commodity hardware.
It provides simple pub-sub semantics over topics, guaranteed at-least-once
and effective-once delivery of messages, automatic cursor management for
subscribers, and geo-replication.

This is a feature release, introducing following new features:

- Pulsar I/O: a serverless connector framework built on top of Pulsar
Functions.
- TieredStorage: offloads bookkeeper ledgers to a cold storage (like S3).
- Stateful Function: Pulsar Functions supports storing state back to
bookkeeper.
- Go Client: a Go client wrapping over existing cpp client.
- Avro and Protobuf schema are supported.

Full list of changes and fixes are available:

https://github.com/apache/incubator-pulsar/milestone/13?closed=1

*** Please download, test and vote on this release. This vote will stay open
for at least 72 hours ***

Note that we are voting upon the source (tag), binaries are provided for
convenience.

Source and binary files:
https://dist.apache.org/repos/dist/dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-5/

SHA-1 checksums:

f0f6b6d7c48d119da36fc4fa1f934fd1f9ba8498
./apache-pulsar-2.1.0-incubating-bin.tar.gz

b6a968249a7c5d21be89ece27895b862efda9a92
./apache-pulsar-2.1.0-incubating-src.tar.gz

ddcf929eb18082ff2b9979507054453e8e803ed3
./apache-pulsar-io-connectors-2.1.0-incubating-bin.tar.gz


Maven staging repo:
https://repository.apache.org/content/repositories/orgapachepulsar-1025/

The tag to be voted upon:
v2.1.0-incubating-candidate-5 (75f79b77f55fbad351cb517e7f3b8ba4ab2dc177)
https://github.com/apache/incubator-pulsar/releases/tag/v2.1.0-incubating-candidate-5

Pulsar's KEYS file containing PGP keys we use to sign the release:
https://dist.apache.org/repos/dist/release/incubator/pulsar/KEYS

Please download the the source package, and follow the README to build
and run the Pulsar standalone service.

A few notes for voting 2.1.0 release:

- For compiling source distribution, use "mvn clean install
-Dmaven.gitcommitid.skip=true
[-DskipTests]".
- Starting from this release, Pulsar is shipping RPM & DEB packages for
pulsar non-java clients.
- Starting from this release, Pulsar is shipping a binary distribution
called `apache-pulsar-io-connectors`
  including all built-in connectors. Each connector is packaged with all
their dependencies and
  a META-INF/DEPEDENCIES file with licensing information for
all dependencies.
- Since there are a lot of features introduced in 2.1, You can use the
instructions documented in the link for
  validating the binary release candidate:
https://github.com/apache/incubator-pulsar/wiki/Release-Candidate-Validation

- Sijie

[RESULT] [VOTE] Pulsar Release 2.1.0-incubating Candidate 5

[Sijie Guo]

The vote for releasing Apache Pulsar 2.1.0-incubating is now closed.

With a total of +3 binding votes and no -1 votes, the
vote passes.

+1s (binding):
 * Matteo Merli
 * Jerry Peng
 * Dave Fisher

Thank you to all the reviewers for taking the time to validate this release.

- Sijie

On Thu, Jul 19, 2018 at 4:43 PM Sijie Guo  wrote:

> This is the 5th release candidate for Apache Pulsar, version
> 2.1.0-incubating.
>
> It introduces following main features:
>
> - Pulsar I/O: a serverless connector framework built on top of Pulsar
> Functions.
> - TieredStorage: offloads bookkeeper ledgers to a cold storage (like S3).
> - Stateful Function: Pulsar Functions supports storing state back to
> bookkeeper.
> - Go Client: a Go client wrapping over existing cpp client.
> - Avro and Protobuf schema are supported.
>
> Full list of changes and fixes are available:
>
> https://github.com/apache/incubator-pulsar/milestone/13?closed=1
>
> *** Please download, test and vote on this release. This vote will stay
> open
> for at least 72 hours ***
>
> Note that we are voting upon the source (tag), binaries are provided for
> convenience.
>
> Source and binary files:
>
> https://dist.apache.org/repos/dist/dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-5/
>
> SHA-1 checksums:
>
> f0f6b6d7c48d119da36fc4fa1f934fd1f9ba8498  
> ./apache-pulsar-2.1.0-incubating-bin.tar.gz
>
> b6a968249a7c5d21be89ece27895b862efda9a92  
> ./apache-pulsar-2.1.0-incubating-src.tar.gz
>
> ddcf929eb18082ff2b9979507054453e8e803ed3  
> ./apache-pulsar-io-connectors-2.1.0-incubating-bin.tar.gz
>
>
> Maven staging repo:
> https://repository.apache.org/content/repositories/orgapachepulsar-1025/
>
> The tag to be voted upon:
> v2.1.0-incubating-candidate-5 (75f79b77f55fbad351cb517e7f3b8ba4ab2dc177)
>
> https://github.com/apache/incubator-pulsar/releases/tag/v2.1.0-incubating-candidate-5
>
> Pulsar's KEYS file containing PGP keys we use to sign the release:
> https://dist.apache.org/repos/dist/release/incubator/pulsar/KEYS
>
> Please download the the source package, and follow the README to build
> and run the Pulsar standalone service.
>
> A few notes for reviewing/voting 2.1.0 release:
>
> - Starting from this release, we are shipping RPM & DEB packages for
> pulsar non-java clients.
> - Starting from this release, we are shipping a binary distribution called
> `apache-pulsar-io-connectors`
>   including all built-in connectors. Each connector is packaged with all
> their dependencies and
>   a META-INF/DEPEDENCIES file with licensing information for
> all dependencies.
> - Since there are a lot of features introduced in 2.1, I put up an
> instruction for people to validate release candidate
>on different features:
> https://github.com/apache/incubator-pulsar/wiki/Release-Candidate-Validation
>
>
>
>
>

Apache Jenkins Trigger Phrases

[Sijie Guo]

Hi all,

I have updated the jenkins CI. now you can trigger individual jobs without
rerun all tests.

- `(re)run java8 tests` => for java8 tests
- `(re)run cpp tests` => for cpp tests
- `(re)run integration tests` => for integration tests

"(re)test this please" will still trigger running all 3 tests.

- Sijie

Re: [VOTE] Pulsar Release 2.1.0-incubating Candidate 5

[Sijie Guo]

any more review comments?

- Sijie

On Fri, Jul 20, 2018 at 1:13 PM Sijie Guo  wrote:

> Okay will remove it.
>
> But just to clarify, the KEYS file I put in the voting email is the one
> used for releases, the one contains my KEY. Please use the one in voting
> email to verify.
>
> - Sijie
>
> On Fri, Jul 20, 2018 at 1:07 PM Dave Fisher  wrote:
>
>> Hi Sijie,
>>
>> I was looking at the KEYS here:
>> https://dist.apache.org/repos/dist/dev/incubator/pulsar/KEYS
>>
>> Probably that should be removed or updated from the release version.
>>
>> Regards,
>> Dave
>>
>> > On Jul 20, 2018, at 12:58 PM, Sijie Guo  wrote:
>> >
>> > HI Dave,
>> >
>> > My key is already there :
>> > https://dist.apache.org/repos/dist/release/incubator/pulsar/KEYS
>> >
>> > Search "Sijie Guo"
>> >
>> > - Sijie
>> >
>> >
>> >
>> > On Fri, Jul 20, 2018 at 11:47 AM Dave Fisher 
>> wrote:
>> >
>> >> Hi Sijie,
>> >>
>> >> Please add your key to the KEYS file.
>> >>
>> >> Regards,
>> >> Dave
>> >>
>> >>> On Jul 19, 2018, at 4:43 PM, Sijie Guo  wrote:
>> >>>
>> >>> This is the 5th release candidate for Apache Pulsar, version
>> >>> 2.1.0-incubating.
>> >>>
>> >>> It introduces following main features:
>> >>>
>> >>> - Pulsar I/O: a serverless connector framework built on top of Pulsar
>> >>> Functions.
>> >>> - TieredStorage: offloads bookkeeper ledgers to a cold storage (like
>> S3).
>> >>> - Stateful Function: Pulsar Functions supports storing state back to
>> >>> bookkeeper.
>> >>> - Go Client: a Go client wrapping over existing cpp client.
>> >>> - Avro and Protobuf schema are supported.
>> >>>
>> >>> Full list of changes and fixes are available:
>> >>>
>> >>> https://github.com/apache/incubator-pulsar/milestone/13?closed=1
>> >>>
>> >>> *** Please download, test and vote on this release. This vote will
>> stay
>> >> open
>> >>> for at least 72 hours ***
>> >>>
>> >>> Note that we are voting upon the source (tag), binaries are provided
>> for
>> >>> convenience.
>> >>>
>> >>> Source and binary files:
>> >>>
>> >>
>> https://dist.apache.org/repos/dist/dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-5/
>> >>>
>> >>> SHA-1 checksums:
>> >>>
>> >>> f0f6b6d7c48d119da36fc4fa1f934fd1f9ba8498
>> >>> ./apache-pulsar-2.1.0-incubating-bin.tar.gz
>> >>>
>> >>> b6a968249a7c5d21be89ece27895b862efda9a92
>> >>> ./apache-pulsar-2.1.0-incubating-src.tar.gz
>> >>>
>> >>> ddcf929eb18082ff2b9979507054453e8e803ed3
>> >>> ./apache-pulsar-io-connectors-2.1.0-incubating-bin.tar.gz
>> >>>
>> >>>
>> >>> Maven staging repo:
>> >>>
>> https://repository.apache.org/content/repositories/orgapachepulsar-1025/
>> >>>
>> >>> The tag to be voted upon:
>> >>> v2.1.0-incubating-candidate-5
>> (75f79b77f55fbad351cb517e7f3b8ba4ab2dc177)
>> >>>
>> >>
>> https://github.com/apache/incubator-pulsar/releases/tag/v2.1.0-incubating-candidate-5
>> >>>
>> >>> Pulsar's KEYS file containing PGP keys we use to sign the release:
>> >>> https://dist.apache.org/repos/dist/release/incubator/pulsar/KEYS
>> >>>
>> >>> Please download the the source package, and follow the README to build
>> >>> and run the Pulsar standalone service.
>> >>>
>> >>> A few notes for reviewing/voting 2.1.0 release:
>> >>>
>> >>> - Starting from this release, we are shipping RPM & DEB packages for
>> >> pulsar
>> >>> non-java clients.
>> >>> - Starting from this release, we are shipping a binary distribution
>> >> called
>> >>> `apache-pulsar-io-connectors`
>> >>> including all built-in connectors. Each connector is packaged with all
>> >>> their dependencies and
>> >>> a META-INF/DEPEDENCIES file with licensing information for
>> >>> all dependencies.
>> >>> - Since there are a lot of features introduced in 2.1, I put up an
>> >>> instruction for people to validate release candidate
>> >>>  on different features:
>> >>>
>> >>
>> https://github.com/apache/incubator-pulsar/wiki/Release-Candidate-Validation
>> >>
>> >>
>>
>>

Re: [VOTE] Pulsar Release 2.1.0-incubating Candidate 5

[Sijie Guo]

Okay will remove it.

But just to clarify, the KEYS file I put in the voting email is the one
used for releases, the one contains my KEY. Please use the one in voting
email to verify.

- Sijie

On Fri, Jul 20, 2018 at 1:07 PM Dave Fisher  wrote:

> Hi Sijie,
>
> I was looking at the KEYS here:
> https://dist.apache.org/repos/dist/dev/incubator/pulsar/KEYS
>
> Probably that should be removed or updated from the release version.
>
> Regards,
> Dave
>
> > On Jul 20, 2018, at 12:58 PM, Sijie Guo  wrote:
> >
> > HI Dave,
> >
> > My key is already there :
> > https://dist.apache.org/repos/dist/release/incubator/pulsar/KEYS
> >
> > Search "Sijie Guo"
> >
> > - Sijie
> >
> >
> >
> > On Fri, Jul 20, 2018 at 11:47 AM Dave Fisher 
> wrote:
> >
> >> Hi Sijie,
> >>
> >> Please add your key to the KEYS file.
> >>
> >> Regards,
> >> Dave
> >>
> >>> On Jul 19, 2018, at 4:43 PM, Sijie Guo  wrote:
> >>>
> >>> This is the 5th release candidate for Apache Pulsar, version
> >>> 2.1.0-incubating.
> >>>
> >>> It introduces following main features:
> >>>
> >>> - Pulsar I/O: a serverless connector framework built on top of Pulsar
> >>> Functions.
> >>> - TieredStorage: offloads bookkeeper ledgers to a cold storage (like
> S3).
> >>> - Stateful Function: Pulsar Functions supports storing state back to
> >>> bookkeeper.
> >>> - Go Client: a Go client wrapping over existing cpp client.
> >>> - Avro and Protobuf schema are supported.
> >>>
> >>> Full list of changes and fixes are available:
> >>>
> >>> https://github.com/apache/incubator-pulsar/milestone/13?closed=1
> >>>
> >>> *** Please download, test and vote on this release. This vote will stay
> >> open
> >>> for at least 72 hours ***
> >>>
> >>> Note that we are voting upon the source (tag), binaries are provided
> for
> >>> convenience.
> >>>
> >>> Source and binary files:
> >>>
> >>
> https://dist.apache.org/repos/dist/dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-5/
> >>>
> >>> SHA-1 checksums:
> >>>
> >>> f0f6b6d7c48d119da36fc4fa1f934fd1f9ba8498
> >>> ./apache-pulsar-2.1.0-incubating-bin.tar.gz
> >>>
> >>> b6a968249a7c5d21be89ece27895b862efda9a92
> >>> ./apache-pulsar-2.1.0-incubating-src.tar.gz
> >>>
> >>> ddcf929eb18082ff2b9979507054453e8e803ed3
> >>> ./apache-pulsar-io-connectors-2.1.0-incubating-bin.tar.gz
> >>>
> >>>
> >>> Maven staging repo:
> >>>
> https://repository.apache.org/content/repositories/orgapachepulsar-1025/
> >>>
> >>> The tag to be voted upon:
> >>> v2.1.0-incubating-candidate-5
> (75f79b77f55fbad351cb517e7f3b8ba4ab2dc177)
> >>>
> >>
> https://github.com/apache/incubator-pulsar/releases/tag/v2.1.0-incubating-candidate-5
> >>>
> >>> Pulsar's KEYS file containing PGP keys we use to sign the release:
> >>> https://dist.apache.org/repos/dist/release/incubator/pulsar/KEYS
> >>>
> >>> Please download the the source package, and follow the README to build
> >>> and run the Pulsar standalone service.
> >>>
> >>> A few notes for reviewing/voting 2.1.0 release:
> >>>
> >>> - Starting from this release, we are shipping RPM & DEB packages for
> >> pulsar
> >>> non-java clients.
> >>> - Starting from this release, we are shipping a binary distribution
> >> called
> >>> `apache-pulsar-io-connectors`
> >>> including all built-in connectors. Each connector is packaged with all
> >>> their dependencies and
> >>> a META-INF/DEPEDENCIES file with licensing information for
> >>> all dependencies.
> >>> - Since there are a lot of features introduced in 2.1, I put up an
> >>> instruction for people to validate release candidate
> >>>  on different features:
> >>>
> >>
> https://github.com/apache/incubator-pulsar/wiki/Release-Candidate-Validation
> >>
> >>
>
>

Re: [VOTE] Pulsar Release 2.1.0-incubating Candidate 5

[Sijie Guo]

Hi Dave,

This is because gitcommit plugin in the maven profile. It assumes the
working directory is a git repo. It can be skipped with " -Dmaven.
*gitcommitid*.*skip*=true "

If you run "mvn clean install -DskipTests -Dmaven.gitcommitid.skip=true",
it should be working. I am not sure if that is an issue for blocking a
release, I would like to see what other people's opinions as well.

Also is this the only issue are you seeing?

I would appreciate if all reviewers can review the whole release candidate
and point out the problems that he found. so I can fix them in the next
release candidate.

- Sijie


On Fri, Jul 20, 2018 at 12:12 PM Dave Fisher  wrote:

> Hi -
>
> I’m currently -1 as I get errors building from source in:
>
> [*INFO*] Apache Pulsar :: Protobuf shaded ... *FAILURE*
> [  0.049 s]
>
> [*ERROR*] Failed to execute goal
> pl.project13.maven:git-commit-id-plugin:2.2.4:revision *(git-info)* on
> project protobuf-shaded: *.git directory is not found! Please specify a
> valid [dotGitDirectory] in your pom.xml* -> *[Help 1]*
>
> Regards,
> Dave
>
> On Jul 19, 2018, at 4:43 PM, Sijie Guo  wrote:
>
> This is the 5th release candidate for Apache Pulsar, version
> 2.1.0-incubating.
>
> It introduces following main features:
>
> - Pulsar I/O: a serverless connector framework built on top of Pulsar
> Functions.
> - TieredStorage: offloads bookkeeper ledgers to a cold storage (like S3).
> - Stateful Function: Pulsar Functions supports storing state back to
> bookkeeper.
> - Go Client: a Go client wrapping over existing cpp client.
> - Avro and Protobuf schema are supported.
>
> Full list of changes and fixes are available:
>
> https://github.com/apache/incubator-pulsar/milestone/13?closed=1
>
> *** Please download, test and vote on this release. This vote will stay
> open
> for at least 72 hours ***
>
> Note that we are voting upon the source (tag), binaries are provided for
> convenience.
>
> Source and binary files:
>
> https://dist.apache.org/repos/dist/dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-5/
>
> SHA-1 checksums:
>
> f0f6b6d7c48d119da36fc4fa1f934fd1f9ba8498
> ./apache-pulsar-2.1.0-incubating-bin.tar.gz
>
> b6a968249a7c5d21be89ece27895b862efda9a92
> ./apache-pulsar-2.1.0-incubating-src.tar.gz
>
> ddcf929eb18082ff2b9979507054453e8e803ed3
> ./apache-pulsar-io-connectors-2.1.0-incubating-bin.tar.gz
>
>
> Maven staging repo:
> https://repository.apache.org/content/repositories/orgapachepulsar-1025/
>
> The tag to be voted upon:
> v2.1.0-incubating-candidate-5 (75f79b77f55fbad351cb517e7f3b8ba4ab2dc177)
>
> https://github.com/apache/incubator-pulsar/releases/tag/v2.1.0-incubating-candidate-5
>
> Pulsar's KEYS file containing PGP keys we use to sign the release:
> https://dist.apache.org/repos/dist/release/incubator/pulsar/KEYS
>
> Please download the the source package, and follow the README to build
> and run the Pulsar standalone service.
>
> A few notes for reviewing/voting 2.1.0 release:
>
> - Starting from this release, we are shipping RPM & DEB packages for pulsar
> non-java clients.
> - Starting from this release, we are shipping a binary distribution called
> `apache-pulsar-io-connectors`
>  including all built-in connectors. Each connector is packaged with all
> their dependencies and
>  a META-INF/DEPEDENCIES file with licensing information for
> all dependencies.
> - Since there are a lot of features introduced in 2.1, I put up an
> instruction for people to validate release candidate
>   on different features:
>
> https://github.com/apache/incubator-pulsar/wiki/Release-Candidate-Validation
>
>
>

Re: [VOTE] Pulsar Release 2.1.0-incubating Candidate 5

[Sijie Guo]

HI Dave,

My key is already there :
https://dist.apache.org/repos/dist/release/incubator/pulsar/KEYS

Search "Sijie Guo"

- Sijie



On Fri, Jul 20, 2018 at 11:47 AM Dave Fisher  wrote:

> Hi Sijie,
>
> Please add your key to the KEYS file.
>
> Regards,
> Dave
>
> > On Jul 19, 2018, at 4:43 PM, Sijie Guo  wrote:
> >
> > This is the 5th release candidate for Apache Pulsar, version
> > 2.1.0-incubating.
> >
> > It introduces following main features:
> >
> > - Pulsar I/O: a serverless connector framework built on top of Pulsar
> > Functions.
> > - TieredStorage: offloads bookkeeper ledgers to a cold storage (like S3).
> > - Stateful Function: Pulsar Functions supports storing state back to
> > bookkeeper.
> > - Go Client: a Go client wrapping over existing cpp client.
> > - Avro and Protobuf schema are supported.
> >
> > Full list of changes and fixes are available:
> >
> > https://github.com/apache/incubator-pulsar/milestone/13?closed=1
> >
> > *** Please download, test and vote on this release. This vote will stay
> open
> > for at least 72 hours ***
> >
> > Note that we are voting upon the source (tag), binaries are provided for
> > convenience.
> >
> > Source and binary files:
> >
> https://dist.apache.org/repos/dist/dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-5/
> >
> > SHA-1 checksums:
> >
> > f0f6b6d7c48d119da36fc4fa1f934fd1f9ba8498
> > ./apache-pulsar-2.1.0-incubating-bin.tar.gz
> >
> > b6a968249a7c5d21be89ece27895b862efda9a92
> > ./apache-pulsar-2.1.0-incubating-src.tar.gz
> >
> > ddcf929eb18082ff2b9979507054453e8e803ed3
> > ./apache-pulsar-io-connectors-2.1.0-incubating-bin.tar.gz
> >
> >
> > Maven staging repo:
> > https://repository.apache.org/content/repositories/orgapachepulsar-1025/
> >
> > The tag to be voted upon:
> > v2.1.0-incubating-candidate-5 (75f79b77f55fbad351cb517e7f3b8ba4ab2dc177)
> >
> https://github.com/apache/incubator-pulsar/releases/tag/v2.1.0-incubating-candidate-5
> >
> > Pulsar's KEYS file containing PGP keys we use to sign the release:
> > https://dist.apache.org/repos/dist/release/incubator/pulsar/KEYS
> >
> > Please download the the source package, and follow the README to build
> > and run the Pulsar standalone service.
> >
> > A few notes for reviewing/voting 2.1.0 release:
> >
> > - Starting from this release, we are shipping RPM & DEB packages for
> pulsar
> > non-java clients.
> > - Starting from this release, we are shipping a binary distribution
> called
> > `apache-pulsar-io-connectors`
> >  including all built-in connectors. Each connector is packaged with all
> > their dependencies and
> >  a META-INF/DEPEDENCIES file with licensing information for
> > all dependencies.
> > - Since there are a lot of features introduced in 2.1, I put up an
> > instruction for people to validate release candidate
> >   on different features:
> >
> https://github.com/apache/incubator-pulsar/wiki/Release-Candidate-Validation
>
>

Re: PIP 18: Pulsar SQL

[Sijie Guo]

Yeah we should. it might be good to add "Next PIP number" to  the main page?

On Thu, Jul 19, 2018 at 10:02 PM Jerry Peng 
wrote:

> Ok I will update the PIP number.  We should probably still list the reject
> PIP on the wiki with the other PIPs.  How else will we know what the latest
> number is.
>
> Best,
>
> Jerry
>
> On Thu, Jul 19, 2018 at 9:54 PM Sijie Guo  wrote:
>
> > This is a good catch!
> >
> > @Jerry Peng  We need to change this to PIP-19.
> >
> > PIP-18 Replicator is actually rejected in favor of using Pulsar I/O
> > connector. we should still list in the PIP list but mark it as Rejected.
> >
> > Thoughts?
> >
> > - Sijie
> >
> > On Thu, Jul 19, 2018 at 9:32 PM Dave Fisher 
> wrote:
> >
> > > Replicator -
> > >
> >
> https://github.com/apache/incubator-pulsar/wiki/PIP-18:-Pulsar-Replicator
> > >
> > > Sent from my iPhone
> > >
> > > > On Jul 19, 2018, at 4:20 PM, Jerry Peng  >
> > > wrote:
> > > >
> > > > Hi Dave,
> > > >
> > > > What do you mean by two PIP-18s? Is there a another PIP-18? I can't
> > seem
> > > to
> > > > find another PIP tagged as 18.
> > > >
> > > > Best,
> > > >
> > > > Jerry
> > > >
> > > >> On Thu, Jul 19, 2018 at 4:08 PM Dave Fisher 
> > > wrote:
> > > >>
> > > >> Hi -
> > > >>
> > > >> Interesting extension. It looks like there are two PIP-18s.
> > > >>
> > > >> Regards,
> > > >> Dave
> > > >>
> > > >>> On Jul 19, 2018, at 12:44 PM, Jerry Peng <
> > jerry.boyang.p...@gmail.com>
> > > >> wrote:
> > > >>>
> > > >>> Hi everyone,
> > > >>>
> > > >>> I have added a PIP for adding a SQL querying layer for Pulsar by
> > > >> leveraging
> > > >>> Presto to allow users to query data in topics.  The link to the
> PIP:
> > > >>>
> > > >>> https://github.com/apache/incubator-pulsar/wiki/PIP-18:-Pulsar-SQL
> > > >>>
> > > >>> Please read the document. Any feedback or suggestions is valuable!
> > > >>>
> > > >>> Best,
> > > >>>
> > > >>> Jerry
> > > >>
> > > >>
> > >
> >
>

Re: PIP 18: Pulsar SQL

[Sijie Guo]

This is a good catch!

@Jerry Peng  We need to change this to PIP-19.

PIP-18 Replicator is actually rejected in favor of using Pulsar I/O
connector. we should still list in the PIP list but mark it as Rejected.

Thoughts?

- Sijie

On Thu, Jul 19, 2018 at 9:32 PM Dave Fisher  wrote:

> Replicator -
> https://github.com/apache/incubator-pulsar/wiki/PIP-18:-Pulsar-Replicator
>
> Sent from my iPhone
>
> > On Jul 19, 2018, at 4:20 PM, Jerry Peng 
> wrote:
> >
> > Hi Dave,
> >
> > What do you mean by two PIP-18s? Is there a another PIP-18? I can't seem
> to
> > find another PIP tagged as 18.
> >
> > Best,
> >
> > Jerry
> >
> >> On Thu, Jul 19, 2018 at 4:08 PM Dave Fisher 
> wrote:
> >>
> >> Hi -
> >>
> >> Interesting extension. It looks like there are two PIP-18s.
> >>
> >> Regards,
> >> Dave
> >>
> >>> On Jul 19, 2018, at 12:44 PM, Jerry Peng 
> >> wrote:
> >>>
> >>> Hi everyone,
> >>>
> >>> I have added a PIP for adding a SQL querying layer for Pulsar by
> >> leveraging
> >>> Presto to allow users to query data in topics.  The link to the PIP:
> >>>
> >>> https://github.com/apache/incubator-pulsar/wiki/PIP-18:-Pulsar-SQL
> >>>
> >>> Please read the document. Any feedback or suggestions is valuable!
> >>>
> >>> Best,
> >>>
> >>> Jerry
> >>
> >>
>

[VOTE] Pulsar Release 2.1.0-incubating Candidate 5

[Sijie Guo]

This is the 5th release candidate for Apache Pulsar, version
2.1.0-incubating.

It introduces following main features:

- Pulsar I/O: a serverless connector framework built on top of Pulsar
Functions.
- TieredStorage: offloads bookkeeper ledgers to a cold storage (like S3).
- Stateful Function: Pulsar Functions supports storing state back to
bookkeeper.
- Go Client: a Go client wrapping over existing cpp client.
- Avro and Protobuf schema are supported.

Full list of changes and fixes are available:

https://github.com/apache/incubator-pulsar/milestone/13?closed=1

*** Please download, test and vote on this release. This vote will stay open
for at least 72 hours ***

Note that we are voting upon the source (tag), binaries are provided for
convenience.

Source and binary files:
https://dist.apache.org/repos/dist/dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-5/

SHA-1 checksums:

f0f6b6d7c48d119da36fc4fa1f934fd1f9ba8498
./apache-pulsar-2.1.0-incubating-bin.tar.gz

b6a968249a7c5d21be89ece27895b862efda9a92
./apache-pulsar-2.1.0-incubating-src.tar.gz

ddcf929eb18082ff2b9979507054453e8e803ed3
./apache-pulsar-io-connectors-2.1.0-incubating-bin.tar.gz


Maven staging repo:
https://repository.apache.org/content/repositories/orgapachepulsar-1025/

The tag to be voted upon:
v2.1.0-incubating-candidate-5 (75f79b77f55fbad351cb517e7f3b8ba4ab2dc177)
https://github.com/apache/incubator-pulsar/releases/tag/v2.1.0-incubating-candidate-5

Pulsar's KEYS file containing PGP keys we use to sign the release:
https://dist.apache.org/repos/dist/release/incubator/pulsar/KEYS

Please download the the source package, and follow the README to build
and run the Pulsar standalone service.

A few notes for reviewing/voting 2.1.0 release:

- Starting from this release, we are shipping RPM & DEB packages for pulsar
non-java clients.
- Starting from this release, we are shipping a binary distribution called
`apache-pulsar-io-connectors`
  including all built-in connectors. Each connector is packaged with all
their dependencies and
  a META-INF/DEPEDENCIES file with licensing information for
all dependencies.
- Since there are a lot of features introduced in 2.1, I put up an
instruction for people to validate release candidate
   on different features:
https://github.com/apache/incubator-pulsar/wiki/Release-Candidate-Validation

Re: PIP 18: Pulsar SQL

[Sijie Guo]

Also /cc d...@bookkeeper.apache.org

One of the goals in this PIP is about resource isolation or read priorities
at bookies to support different classes of read requests. That's aligned
with some general discussion about IO prioritization (between writes, reads
and scan/catchup reads) in bookkeeper community.
It would be good to get thoughts and inputs from people from bookkeeper
community.

Resource Isolation

   - Read priorities at BookKeeper level - Bookies should be able to
   support different classes of read requests and prioritize regular consumers
   vs read requests coming from SQL queries
   - Queries from users or groups should not overly impact other queries
   from other users or groups


- Sijie

On Thu, Jul 19, 2018 at 2:09 PM Sijie Guo  wrote:

> Great writeup, Jerry and Matteo! Can you share a google doc version of the
> PIP? That might be better for people to comment on it.
>
> I am ccing users@ mailing list as well. We can also get feedback from
> pulsar users as well.
>
> - Sijie
>
> On Thu, Jul 19, 2018 at 12:44 PM Jerry Peng 
> wrote:
>
>> Hi everyone,
>>
>> I have added a PIP for adding a SQL querying layer for Pulsar by
>> leveraging
>> Presto to allow users to query data in topics.  The link to the PIP:
>>
>> https://github.com/apache/incubator-pulsar/wiki/PIP-18:-Pulsar-SQL
>>
>> Please read the document. Any feedback or suggestions is valuable!
>>
>> Best,
>>
>> Jerry
>>
>

Re: PIP 18: Pulsar SQL

[Sijie Guo]

Great writeup, Jerry and Matteo! Can you share a google doc version of the
PIP? That might be better for people to comment on it.

I am ccing users@ mailing list as well. We can also get feedback from
pulsar users as well.

- Sijie

On Thu, Jul 19, 2018 at 12:44 PM Jerry Peng 
wrote:

> Hi everyone,
>
> I have added a PIP for adding a SQL querying layer for Pulsar by leveraging
> Presto to allow users to query data in topics.  The link to the PIP:
>
> https://github.com/apache/incubator-pulsar/wiki/PIP-18:-Pulsar-SQL
>
> Please read the document. Any feedback or suggestions is valuable!
>
> Best,
>
> Jerry
>

Re: Release 2.1

[Sijie Guo]

branch 2.1 is cut and a new release candidate tag is created.

so when you merge new PRs, use 2.2 now :)

thanks,
Sijie

On Thu, Jul 19, 2018 at 9:57 AM Sijie Guo  wrote:

> Hi all,
>
> I am cutting 2.1 release now. If you have new PRs coming up, please label
> them as 2.2.
>
> Thank you,
> Sijie
>
> On Tue, Jul 17, 2018 at 1:56 AM Ivan Kelly  wrote:
>
>> The second change is https://github.com/apache/incubator-pulsar/pull/2177
>>
>> -Ivan
>>
>> On Tue, Jul 17, 2018 at 12:45 AM, Sijie Guo  wrote:
>> > Cool. I will wait for these two issues to be merged.
>> >
>> > - Sijie
>> >
>> > On Mon, Jul 16, 2018 at 3:25 PM Ivan Kelly  wrote:
>> >
>> >> I'd like to get in
>> >> https://github.com/apache/incubator-pulsar/pull/2172 and a followup
>> >> (not pushed yet, but ready to go when #2172 is in), which fix auth in
>> >> python based functions.
>> >>
>> >> -Ivan
>> >>
>> >> On Mon, Jul 16, 2018 at 11:07 PM, Sijie Guo 
>> wrote:
>> >> > We have cleared up most of the issues related to auth and
>> connectors. So
>> >> we
>> >> > are able to release 2.1. There is one fix about regex subscription
>> >> through
>> >> > proxy, which I will send out for review today.
>> >> >
>> >> > If you have any changes that are needed to be included in 2.1, can
>> you
>> >> ping
>> >> > this thread? so I know what are they.
>> >> >
>> >> > If I don't hear any more changes by tomorrow, I would start the
>> release
>> >> > process.
>> >> > Since we have quite a lot of changes going into master for 2.1
>> release, I
>> >> > will reset the branch-2.1 to master, restart the release process
>> from a
>> >> > clear state but I will still stick to the release candidate number
>> that I
>> >> > stopped last time.
>> >> >
>> >> > If you have any questions regarding 2.1 release, feel free to ping me
>> >> here.
>> >> >
>> >> > - Sijie
>> >> >
>> >> >
>> >> >
>> >> > On Mon, Jul 9, 2018 at 2:44 AM Ivan Kelly  wrote:
>> >> >
>> >> >> It would be good to get tls proxy auth fixes into 2.1 since there's
>> >> >> time to do so now. Without these changes auth + tls do not work if
>> >> >> using a proxy.
>> >> >>
>> >> >> -Ivan
>> >> >>
>> >> >> On Thu, Jul 5, 2018 at 10:13 PM, Sijie Guo 
>> wrote:
>> >> >> > Hi all,
>> >> >> >
>> >> >> > Due the connector packaging issue, we delayed 2.1 release on
>> fixing
>> >> that.
>> >> >> > During this period, there are also other changes coming in. so I
>> will
>> >> >> > suggest us still labelling the issues for 2.1 release, and I will
>> >> >> recreate
>> >> >> > 2.1 branch from master once we resolve all the issues for 2.1
>> release.
>> >> >> >
>> >> >> > I will manually update those issues that already resolved for 2.2
>> >> back to
>> >> >> > 2.1.
>> >> >> >
>> >> >> > - Sijie
>> >> >>
>> >>
>>
>

Re: Release 2.1

[Sijie Guo]

Hi all,

I am cutting 2.1 release now. If you have new PRs coming up, please label
them as 2.2.

Thank you,
Sijie

On Tue, Jul 17, 2018 at 1:56 AM Ivan Kelly  wrote:

> The second change is https://github.com/apache/incubator-pulsar/pull/2177
>
> -Ivan
>
> On Tue, Jul 17, 2018 at 12:45 AM, Sijie Guo  wrote:
> > Cool. I will wait for these two issues to be merged.
> >
> > - Sijie
> >
> > On Mon, Jul 16, 2018 at 3:25 PM Ivan Kelly  wrote:
> >
> >> I'd like to get in
> >> https://github.com/apache/incubator-pulsar/pull/2172 and a followup
> >> (not pushed yet, but ready to go when #2172 is in), which fix auth in
> >> python based functions.
> >>
> >> -Ivan
> >>
> >> On Mon, Jul 16, 2018 at 11:07 PM, Sijie Guo  wrote:
> >> > We have cleared up most of the issues related to auth and connectors.
> So
> >> we
> >> > are able to release 2.1. There is one fix about regex subscription
> >> through
> >> > proxy, which I will send out for review today.
> >> >
> >> > If you have any changes that are needed to be included in 2.1, can you
> >> ping
> >> > this thread? so I know what are they.
> >> >
> >> > If I don't hear any more changes by tomorrow, I would start the
> release
> >> > process.
> >> > Since we have quite a lot of changes going into master for 2.1
> release, I
> >> > will reset the branch-2.1 to master, restart the release process from
> a
> >> > clear state but I will still stick to the release candidate number
> that I
> >> > stopped last time.
> >> >
> >> > If you have any questions regarding 2.1 release, feel free to ping me
> >> here.
> >> >
> >> > - Sijie
> >> >
> >> >
> >> >
> >> > On Mon, Jul 9, 2018 at 2:44 AM Ivan Kelly  wrote:
> >> >
> >> >> It would be good to get tls proxy auth fixes into 2.1 since there's
> >> >> time to do so now. Without these changes auth + tls do not work if
> >> >> using a proxy.
> >> >>
> >> >> -Ivan
> >> >>
> >> >> On Thu, Jul 5, 2018 at 10:13 PM, Sijie Guo 
> wrote:
> >> >> > Hi all,
> >> >> >
> >> >> > Due the connector packaging issue, we delayed 2.1 release on fixing
> >> that.
> >> >> > During this period, there are also other changes coming in. so I
> will
> >> >> > suggest us still labelling the issues for 2.1 release, and I will
> >> >> recreate
> >> >> > 2.1 branch from master once we resolve all the issues for 2.1
> release.
> >> >> >
> >> >> > I will manually update those issues that already resolved for 2.2
> >> back to
> >> >> > 2.1.
> >> >> >
> >> >> > - Sijie
> >> >>
> >>
>

Re: Release 2.1

[Sijie Guo]

Cool. I will wait for these two issues to be merged.

- Sijie

On Mon, Jul 16, 2018 at 3:25 PM Ivan Kelly  wrote:

> I'd like to get in
> https://github.com/apache/incubator-pulsar/pull/2172 and a followup
> (not pushed yet, but ready to go when #2172 is in), which fix auth in
> python based functions.
>
> -Ivan
>
> On Mon, Jul 16, 2018 at 11:07 PM, Sijie Guo  wrote:
> > We have cleared up most of the issues related to auth and connectors. So
> we
> > are able to release 2.1. There is one fix about regex subscription
> through
> > proxy, which I will send out for review today.
> >
> > If you have any changes that are needed to be included in 2.1, can you
> ping
> > this thread? so I know what are they.
> >
> > If I don't hear any more changes by tomorrow, I would start the release
> > process.
> > Since we have quite a lot of changes going into master for 2.1 release, I
> > will reset the branch-2.1 to master, restart the release process from a
> > clear state but I will still stick to the release candidate number that I
> > stopped last time.
> >
> > If you have any questions regarding 2.1 release, feel free to ping me
> here.
> >
> > - Sijie
> >
> >
> >
> > On Mon, Jul 9, 2018 at 2:44 AM Ivan Kelly  wrote:
> >
> >> It would be good to get tls proxy auth fixes into 2.1 since there's
> >> time to do so now. Without these changes auth + tls do not work if
> >> using a proxy.
> >>
> >> -Ivan
> >>
> >> On Thu, Jul 5, 2018 at 10:13 PM, Sijie Guo  wrote:
> >> > Hi all,
> >> >
> >> > Due the connector packaging issue, we delayed 2.1 release on fixing
> that.
> >> > During this period, there are also other changes coming in. so I will
> >> > suggest us still labelling the issues for 2.1 release, and I will
> >> recreate
> >> > 2.1 branch from master once we resolve all the issues for 2.1 release.
> >> >
> >> > I will manually update those issues that already resolved for 2.2
> back to
> >> > 2.1.
> >> >
> >> > - Sijie
> >>
>

Re: Release 2.1

[Sijie Guo]

We have cleared up most of the issues related to auth and connectors. So we
are able to release 2.1. There is one fix about regex subscription through
proxy, which I will send out for review today.

If you have any changes that are needed to be included in 2.1, can you ping
this thread? so I know what are they.

If I don't hear any more changes by tomorrow, I would start the release
process.
Since we have quite a lot of changes going into master for 2.1 release, I
will reset the branch-2.1 to master, restart the release process from a
clear state but I will still stick to the release candidate number that I
stopped last time.

If you have any questions regarding 2.1 release, feel free to ping me here.

- Sijie



On Mon, Jul 9, 2018 at 2:44 AM Ivan Kelly  wrote:

> It would be good to get tls proxy auth fixes into 2.1 since there's
> time to do so now. Without these changes auth + tls do not work if
> using a proxy.
>
> -Ivan
>
> On Thu, Jul 5, 2018 at 10:13 PM, Sijie Guo  wrote:
> > Hi all,
> >
> > Due the connector packaging issue, we delayed 2.1 release on fixing that.
> > During this period, there are also other changes coming in. so I will
> > suggest us still labelling the issues for 2.1 release, and I will
> recreate
> > 2.1 branch from master once we resolve all the issues for 2.1 release.
> >
> > I will manually update those issues that already resolved for 2.2 back to
> > 2.1.
> >
> > - Sijie
>

Re: SF Bay Area Apache Pulsar Meetup . July 10 2019

[Sijie Guo]

Thanks Yahoo folks for hosting the meetup! It was great to get together to
look back at those features and production usage in the past year.

How do people think about next meetup? e.g. location, time.

Or maybe we can start a survey in the community about this :)

- Sijie

On Tue, Jul 3, 2018 at 12:35 AM Joe F  wrote:

> https://www.meetup.com/SF-Bay-Area-Apache-Pulsar-Meetup/events/252312389/
>
>
> Pulsar Functions integrates streaming with compute, and the Pulsar
> connector framework enables effortless streaming between Pulsar and other
> systems as data sources and sinks . Come listen to the developers and users
> dive into the architecture, and share experience from production use cases
> of Pulsar.
>
>

Release 2.1

[Sijie Guo]

Hi all,

Due the connector packaging issue, we delayed 2.1 release on fixing that.
During this period, there are also other changes coming in. so I will
suggest us still labelling the issues for 2.1 release, and I will recreate
2.1 branch from master once we resolve all the issues for 2.1 release.

I will manually update those issues that already resolved for 2.2 back to
2.1.

- Sijie

Re: [Cancelled] [VOTE] Pulsar Release 2.1.0-incubating Candidate 4

[Sijie Guo]

Pulsar IO (connector) is a new feature introduced in 2.1.  The code is
okay, it is a problem about packaging which dependencies are not packages.
It got caught when some people try this feature out.


On Fri, Jun 29, 2018 at 10:13 PM Dave Fisher  wrote:

> As Release Manager you are free to pull the candidate. I want to
> understand that this would be a regression from the last release.
>
> Best Regards,
> Dave
>
> Sent from my iPhone
>
> > On Jun 29, 2018, at 3:13 PM, Sijie Guo  wrote:
> >
> > Cancelled RC4. Connectors don't really work.
> >
> > - Sijie
> >
> >> On Fri, Jun 29, 2018 at 2:37 AM Sijie Guo  wrote:
> >>
> >> This is the 4th release candidate for Apache Pulsar, version
> >> 2.1.0-incubating.
> >>
> >> It fixes the following issues:
> >> https://github.com/apache/incubator-pulsar/milestone/13?closed=1
> >>
> >> *** Please download, test and vote on this release. This vote will stay
> >> open
> >> for at least 72 hours ***
> >>
> >> Note that we are voting upon the source (tag), binaries are provided for
> >> convenience.
> >>
> >> Source and binary files:
> >>
> >>
> https://dist.apache.org/repos/dist/dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-4/
> >>
> >> SHA-1 checksums:
> >>
> >> 514d1c384c752d74a9e72fb60a06be1c2f632824
> apache-pulsar-2.1.0-incubating-bin.tar.gz
> >>
> >> e73fba881bc26d98ea9dd5eaaf0cd7ecd4d305d9
> apache-pulsar-2.1.0-incubating-src.tar.gz
> >>
> >>
> >> Maven staging repo:
> >>
> https://repository.apache.org/content/repositories/orgapachepulsar-1024/
> >>
> >> The tag to be voted upon:
> >> v2.1.0-incubating-candidate-4 (d3261ed4dc86f4cdf5ad6a246c532212f064967a)
> >>
> >>
> https://github.com/apache/incubator-pulsar/releases/tag/v2.1.0-incubating-candidate-4
> >>
> >> Pulsar's KEYS file containing PGP keys we use to sign the release:
> >> https://dist.apache.org/repos/dist/release/incubator/pulsar/KEYS
> >>
> >> Please download the the source package, and follow the README to build
> >> and run the Pulsar standalone service.
> >>
> >> NOTES: starting from this release, we are shipping RPM & DEB packages
> for
> >> pulsar cpp
> >> clients along with the source and binary tarballs.
> >>
> >> - Sijie
> >>
>
>

[Cancelled] [VOTE] Pulsar Release 2.1.0-incubating Candidate 4

[Sijie Guo]

Cancelled RC4. Connectors don't really work.

- Sijie

On Fri, Jun 29, 2018 at 2:37 AM Sijie Guo  wrote:

> This is the 4th release candidate for Apache Pulsar, version
> 2.1.0-incubating.
>
> It fixes the following issues:
> https://github.com/apache/incubator-pulsar/milestone/13?closed=1
>
> *** Please download, test and vote on this release. This vote will stay
> open
> for at least 72 hours ***
>
> Note that we are voting upon the source (tag), binaries are provided for
> convenience.
>
> Source and binary files:
>
> https://dist.apache.org/repos/dist/dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-4/
>
> SHA-1 checksums:
>
> 514d1c384c752d74a9e72fb60a06be1c2f632824  
> apache-pulsar-2.1.0-incubating-bin.tar.gz
>
> e73fba881bc26d98ea9dd5eaaf0cd7ecd4d305d9  
> apache-pulsar-2.1.0-incubating-src.tar.gz
>
>
> Maven staging repo:
> https://repository.apache.org/content/repositories/orgapachepulsar-1024/
>
> The tag to be voted upon:
> v2.1.0-incubating-candidate-4 (d3261ed4dc86f4cdf5ad6a246c532212f064967a)
>
> https://github.com/apache/incubator-pulsar/releases/tag/v2.1.0-incubating-candidate-4
>
> Pulsar's KEYS file containing PGP keys we use to sign the release:
> https://dist.apache.org/repos/dist/release/incubator/pulsar/KEYS
>
> Please download the the source package, and follow the README to build
> and run the Pulsar standalone service.
>
> NOTES: starting from this release, we are shipping RPM & DEB packages for
> pulsar cpp
> clients along with the source and binary tarballs.
>
> - Sijie
>

[VOTE] Pulsar Release 2.1.0-incubating Candidate 4

[Sijie Guo]

This is the 4th release candidate for Apache Pulsar, version
2.1.0-incubating.

It fixes the following issues:
https://github.com/apache/incubator-pulsar/milestone/13?closed=1

*** Please download, test and vote on this release. This vote will stay open
for at least 72 hours ***

Note that we are voting upon the source (tag), binaries are provided for
convenience.

Source and binary files:
https://dist.apache.org/repos/dist/dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-4/

SHA-1 checksums:

514d1c384c752d74a9e72fb60a06be1c2f632824
apache-pulsar-2.1.0-incubating-bin.tar.gz

e73fba881bc26d98ea9dd5eaaf0cd7ecd4d305d9
apache-pulsar-2.1.0-incubating-src.tar.gz


Maven staging repo:
https://repository.apache.org/content/repositories/orgapachepulsar-1024/

The tag to be voted upon:
v2.1.0-incubating-candidate-4 (d3261ed4dc86f4cdf5ad6a246c532212f064967a)
https://github.com/apache/incubator-pulsar/releases/tag/v2.1.0-incubating-candidate-4

Pulsar's KEYS file containing PGP keys we use to sign the release:
https://dist.apache.org/repos/dist/release/incubator/pulsar/KEYS

Please download the the source package, and follow the README to build
and run the Pulsar standalone service.

NOTES: starting from this release, we are shipping RPM & DEB packages for
pulsar cpp
clients along with the source and binary tarballs.

- Sijie

[Cancel] [VOTE] Pulsar Release 2.1.0-incubating Candidate 3

[Sijie Guo]

Sorry have to cancel this RC again. Missed one cherry-pick :
https://github.com/apache/incubator-pulsar/pull/2046#issuecomment-401286885

- Sijie

On Thu, Jun 28, 2018 at 5:33 PM Sijie Guo  wrote:

> This is the 3rd release candidate for Apache Pulsar, version
> 2.1.0-incubating.
>
> It fixes the following issues:
> https://github.com/apache/incubator-pulsar/milestone/13?closed=1
>
> *** Please download, test and vote on this release. This vote will stay
> open
> for at least 72 hours ***
>
> Note that we are voting upon the source (tag), binaries are provided for
> convenience.
>
> Source and binary files:
>
> https://dist.apache.org/repos/dist/dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-3/
>
> SHA-1 checksums:
>
> d5a9536b74532cc13390547b7f5a4a796ba4e400  
> apache-pulsar-2.1.0-incubating-bin.tar.gz
>
> 9d0184968816ca04cd14319fd8716dbc532fc808  
> apache-pulsar-2.1.0-incubating-src.tar.gz
>
>
> Maven staging repo:
> https://repository.apache.org/content/repositories/orgapachepulsar-1023/
>
> The tag to be voted upon:
> v2.1.0-incubating-candidate-2 (fde1da6875eba7fcd56a9f33e2ed100aba095aed)
>
> https://github.com/apache/incubator-pulsar/releases/tag/v2.1.0-incubating-candidate-3
>
> Pulsar's KEYS file containing PGP keys we use to sign the release:
> https://dist.apache.org/repos/dist/release/incubator/pulsar/KEYS
>
> Please download the the source package, and follow the README to build
> and run the Pulsar standalone service.
>
> NOTES: starting from this release, we are shipping RPM & DEB packages for
> pulsar cpp
> clients along with the source and binary tarballs.
>
> - Sijie
>

[VOTE] Pulsar Release 2.1.0-incubating Candidate 3

[Sijie Guo]

This is the 3rd release candidate for Apache Pulsar, version
2.1.0-incubating.

It fixes the following issues:
https://github.com/apache/incubator-pulsar/milestone/13?closed=1

*** Please download, test and vote on this release. This vote will stay open
for at least 72 hours ***

Note that we are voting upon the source (tag), binaries are provided for
convenience.

Source and binary files:
https://dist.apache.org/repos/dist/dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-3/

SHA-1 checksums:

d5a9536b74532cc13390547b7f5a4a796ba4e400
apache-pulsar-2.1.0-incubating-bin.tar.gz

9d0184968816ca04cd14319fd8716dbc532fc808
apache-pulsar-2.1.0-incubating-src.tar.gz


Maven staging repo:
https://repository.apache.org/content/repositories/orgapachepulsar-1023/

The tag to be voted upon:
v2.1.0-incubating-candidate-2 (fde1da6875eba7fcd56a9f33e2ed100aba095aed)
https://github.com/apache/incubator-pulsar/releases/tag/v2.1.0-incubating-candidate-3

Pulsar's KEYS file containing PGP keys we use to sign the release:
https://dist.apache.org/repos/dist/release/incubator/pulsar/KEYS

Please download the the source package, and follow the README to build
and run the Pulsar standalone service.

NOTES: starting from this release, we are shipping RPM & DEB packages for
pulsar cpp
clients along with the source and binary tarballs.

- Sijie

[CANCELLED] [VOTE] Pulsar Release 2.1.0-incubating Candidate 2

[Sijie Guo]

Cancel the RC due to license issues.

On Tue, Jun 26, 2018 at 6:53 PM Sijie Guo  wrote:

> This is the second release candidate for Apache Pulsar, version
> 2.1.0-incubating.
>
> It fixes the following issues:
> https://github.com/apache/incubator-pulsar/milestone/13?closed=1
>
> *** Please download, test and vote on this release. This vote will stay
> open
> for at least 72 hours ***
>
> Note that we are voting upon the source (tag), binaries are provided for
> convenience.
>
> Source and binary files:
>
> https://dist.apache.org/repos/dist/dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-2/
>
> SHA-1 checksums:
>
> 80fbd78c8b60c5207b8e5fed185d6943a42779dd  
> apache-pulsar-2.1.0-incubating-bin.tar.gz
>
> 20c15d3518fd2cd38e3fa1b81dbf8e3bd889c238  
> apache-pulsar-2.1.0-incubating-src.tar.gz
>
>
> Maven staging repo:
> https://repository.apache.org/content/repositories/orgapachepulsar-1022/
>
> The tag to be voted upon:
> v2.1.0-incubating-candidate-2 (30f61280677887ccbbf6f406c4ba12967caa3fa0)
>
> https://github.com/apache/incubator-pulsar/releases/tag/v2.1.0-incubating-candidate-2
>
> Pulsar's KEYS file containing PGP keys we use to sign the release:
> https://dist.apache.org/repos/dist/release/incubator/pulsar/KEYS
>
> Please download the the source package, and follow the README to build
> and run the Pulsar standalone service.
>
> NOTES: starting from this release, we are shipping RPM & DEB packages for
> pulsar cpp
> clients along with the source and binary tarballs.
>
>

Re: [VOTE] Pulsar Release 2.1.0-incubating Candidate 2

[Sijie Guo]

Cancel the vote. There are still license issues due to version upgrades on
2.1.

- Sijie

On Tue, Jun 26, 2018 at 6:53 PM Sijie Guo  wrote:

> This is the second release candidate for Apache Pulsar, version
> 2.1.0-incubating.
>
> It fixes the following issues:
> https://github.com/apache/incubator-pulsar/milestone/13?closed=1
>
> *** Please download, test and vote on this release. This vote will stay
> open
> for at least 72 hours ***
>
> Note that we are voting upon the source (tag), binaries are provided for
> convenience.
>
> Source and binary files:
>
> https://dist.apache.org/repos/dist/dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-2/
>
> SHA-1 checksums:
>
> 80fbd78c8b60c5207b8e5fed185d6943a42779dd  
> apache-pulsar-2.1.0-incubating-bin.tar.gz
>
> 20c15d3518fd2cd38e3fa1b81dbf8e3bd889c238  
> apache-pulsar-2.1.0-incubating-src.tar.gz
>
>
> Maven staging repo:
> https://repository.apache.org/content/repositories/orgapachepulsar-1022/
>
> The tag to be voted upon:
> v2.1.0-incubating-candidate-2 (30f61280677887ccbbf6f406c4ba12967caa3fa0)
>
> https://github.com/apache/incubator-pulsar/releases/tag/v2.1.0-incubating-candidate-2
>
> Pulsar's KEYS file containing PGP keys we use to sign the release:
> https://dist.apache.org/repos/dist/release/incubator/pulsar/KEYS
>
> Please download the the source package, and follow the README to build
> and run the Pulsar standalone service.
>
> NOTES: starting from this release, we are shipping RPM & DEB packages for
> pulsar cpp
> clients along with the source and binary tarballs.
>
>

[Discuss] Blog at pulsar website

[Sijie Guo]

Hi,

Do you think it is worth having a blogs (or news) category at pulsar
website, for publishing news about releases or even blogs about new
features?

I see quite a lot of projects are doing so, for example:

Arrow: http://arrow.apache.org/blog/
Beam: https://beam.apache.org/blog/
RocketMQ: https://rocketmq.apache.org/year-archive/

Both Arrow and Beam started publishing blog posts since they were in
incubator.

https://beam.apache.org/beam/release/2016/06/15/first-release.html
https://rocketmq.apache.org/rocketmq/filter-messages-by-sql92-in-rocketmq/


Thoughts?

- Sijie

[VOTE] Pulsar Release 2.1.0-incubating Candidate 2

[Sijie Guo]

This is the second release candidate for Apache Pulsar, version
2.1.0-incubating.

It fixes the following issues:
https://github.com/apache/incubator-pulsar/milestone/13?closed=1

*** Please download, test and vote on this release. This vote will stay open
for at least 72 hours ***

Note that we are voting upon the source (tag), binaries are provided for
convenience.

Source and binary files:
https://dist.apache.org/repos/dist/dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-2/

SHA-1 checksums:

80fbd78c8b60c5207b8e5fed185d6943a42779dd
apache-pulsar-2.1.0-incubating-bin.tar.gz

20c15d3518fd2cd38e3fa1b81dbf8e3bd889c238
apache-pulsar-2.1.0-incubating-src.tar.gz


Maven staging repo:
https://repository.apache.org/content/repositories/orgapachepulsar-1022/

The tag to be voted upon:
v2.1.0-incubating-candidate-2 (30f61280677887ccbbf6f406c4ba12967caa3fa0)
https://github.com/apache/incubator-pulsar/releases/tag/v2.1.0-incubating-candidate-2

Pulsar's KEYS file containing PGP keys we use to sign the release:
https://dist.apache.org/repos/dist/release/incubator/pulsar/KEYS

Please download the the source package, and follow the README to build
and run the Pulsar standalone service.

NOTES: starting from this release, we are shipping RPM & DEB packages for
pulsar cpp
clients along with the source and binary tarballs.

[CANCELLED] [VOTE] Pulsar Release 2.1.0-incubating Candidate 1

[Sijie Guo]

Cancel RC1 due to OpenSSL linking issues on the pre-built PRM/DEB packages.

On Mon, Jun 25, 2018 at 4:24 PM Sijie Guo  wrote:

> This is the first release candidate for Apache Pulsar, version
> 2.1.0-incubating.
>
> It fixes the following issues:
> https://github.com/apache/incubator-pulsar/milestone/13?closed=1
>
> *** Please download, test and vote on this release. This vote will stay
> open
> for at least 72 hours ***
>
> Note that we are voting upon the source (tag), binaries are provided for
> convenience.
>
> Source and binary files:
>
> https://dist.apache.org/repos/dist/dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-1/
>
> SHA-1 checksums:
>
> 944d65e9ef5862aa4b0d1f75bb73befad497ece9
> apache-pulsar-2.1.0-incubating-bin.tar.gz
> eb8ab08fa2feee1346f6b77bc7608bf155688763
> apache-pulsar-2.1.0-incubating-src.tar.gz
>
> Maven staging repo:
> https://repository.apache.org/content/repositories/orgapachepulsar-1021/
>
> The tag to be voted upon:
> v2.1.0-incubating-candidate-1 (840ad88eff88b22e21e567a889c14d04d6a4d91b)
>
> https://github.com/apache/incubator-pulsar/releases/tag/v2.1.0-incubating-candidate-1
>
> Pulsar's KEYS file containing PGP keys we use to sign the release:
> https://dist.apache.org/repos/dist/release/incubator/pulsar/KEYS
>
> Please download the the source package, and follow the README to build
> and run the Pulsar standalone service.
>
> NOTES: starting from this release, we are shipping RPM & DEB packages for
> pulsar cpp
> clients along with the source and binary tarballs.
>

Re: [VOTE] Pulsar Release 2.1.0-incubating Candidate 1

[Sijie Guo]

Thank you Matteo. Cancelling this candidate.

On Tue, Jun 26, 2018 at 9:46 AM Matteo Merli  wrote:

> There is an issue with the RPM/Deb packages included. Some symbols for
> OpenSSL are not found when linking from Go.
>
> I'm working on a fix for that.
>
> Matteo
>
> On Mon, Jun 25, 2018 at 4:25 PM Sijie Guo  wrote:
>
> > This is the first release candidate for Apache Pulsar, version
> > 2.1.0-incubating.
> >
> > It fixes the following issues:
> > https://github.com/apache/incubator-pulsar/milestone/13?closed=1
> >
> > *** Please download, test and vote on this release. This vote will stay
> > open
> > for at least 72 hours ***
> >
> > Note that we are voting upon the source (tag), binaries are provided for
> > convenience.
> >
> > Source and binary files:
> >
> >
> https://dist.apache.org/repos/dist/dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-1/
> >
> > SHA-1 checksums:
> >
> > 944d65e9ef5862aa4b0d1f75bb73befad497ece9
> > apache-pulsar-2.1.0-incubating-bin.tar.gz
> > eb8ab08fa2feee1346f6b77bc7608bf155688763
> > apache-pulsar-2.1.0-incubating-src.tar.gz
> >
> > Maven staging repo:
> > https://repository.apache.org/content/repositories/orgapachepulsar-1021/
> >
> > The tag to be voted upon:
> > v2.1.0-incubating-candidate-1 (840ad88eff88b22e21e567a889c14d04d6a4d91b)
> >
> >
> https://github.com/apache/incubator-pulsar/releases/tag/v2.1.0-incubating-candidate-1
> >
> > Pulsar's KEYS file containing PGP keys we use to sign the release:
> > https://dist.apache.org/repos/dist/release/incubator/pulsar/KEYS
> >
> > Please download the the source package, and follow the README to build
> > and run the Pulsar standalone service.
> >
> > NOTES: starting from this release, we are shipping RPM & DEB packages for
> > pulsar cpp
> > clients along with the source and binary tarballs.
> >
> --
> Matteo Merli
> 
>

[VOTE] Pulsar Release 2.1.0-incubating Candidate 1

[Sijie Guo]

This is the first release candidate for Apache Pulsar, version
2.1.0-incubating.

It fixes the following issues:
https://github.com/apache/incubator-pulsar/milestone/13?closed=1

*** Please download, test and vote on this release. This vote will stay open
for at least 72 hours ***

Note that we are voting upon the source (tag), binaries are provided for
convenience.

Source and binary files:
https://dist.apache.org/repos/dist/dev/incubator/pulsar/pulsar-2.1.0-incubating-candidate-1/

SHA-1 checksums:

944d65e9ef5862aa4b0d1f75bb73befad497ece9
apache-pulsar-2.1.0-incubating-bin.tar.gz
eb8ab08fa2feee1346f6b77bc7608bf155688763
apache-pulsar-2.1.0-incubating-src.tar.gz

Maven staging repo:
https://repository.apache.org/content/repositories/orgapachepulsar-1021/

The tag to be voted upon:
v2.1.0-incubating-candidate-1 (840ad88eff88b22e21e567a889c14d04d6a4d91b)
https://github.com/apache/incubator-pulsar/releases/tag/v2.1.0-incubating-candidate-1

Pulsar's KEYS file containing PGP keys we use to sign the release:
https://dist.apache.org/repos/dist/release/incubator/pulsar/KEYS

Please download the the source package, and follow the README to build
and run the Pulsar standalone service.

NOTES: starting from this release, we are shipping RPM & DEB packages for
pulsar cpp
clients along with the source and binary tarballs.

Re: Planning for release 2.1

[Sijie Guo]

FYI Cut branch-2.1. Starting the release process.

On Thu, Jun 21, 2018 at 12:30 PM Sijie Guo  wrote:

> I am planning to cut the branch and start the release process tomorrow.
>
> If you have any items outstanding for 2.1 or would like them to be
> included 2.1, please label them for 2.1.0 release.
> https://github.com/apache/incubator-pulsar/milestone/13
>
> Thank you,
> Sijie
>
> On Mon, Jun 18, 2018 at 12:49 PM Sijie Guo  wrote:
>
>> I am happy to contribute if no one else is working on that.
>>
>> - Sijie
>>
>> On Mon, Jun 18, 2018 at 12:01 PM Matteo Merli  wrote:
>>
>>> As we are about to release 2.0.1, we have already a batch of new items
>>> ready to be released. Biggest ticket items are:
>>>  * Connector framework
>>>  * Go client library
>>>  * Schema support for Avro and Protobuf
>>>
>>> I'd like to propose to cut the release later this week.
>>> Please point out any pending change you want to include in 2.1.
>>>
>>> Also, is anyone volunteering as release manager for 2.1?
>>>
>>> Matteo
>>>
>>> --
>>> Matteo Merli
>>> 
>>>
>>

Re: Planning for release 2.1

[Sijie Guo]

I am planning to cut the branch and start the release process tomorrow.

If you have any items outstanding for 2.1 or would like them to be included
2.1, please label them for 2.1.0 release.
https://github.com/apache/incubator-pulsar/milestone/13

Thank you,
Sijie

On Mon, Jun 18, 2018 at 12:49 PM Sijie Guo  wrote:

> I am happy to contribute if no one else is working on that.
>
> - Sijie
>
> On Mon, Jun 18, 2018 at 12:01 PM Matteo Merli  wrote:
>
>> As we are about to release 2.0.1, we have already a batch of new items
>> ready to be released. Biggest ticket items are:
>>  * Connector framework
>>  * Go client library
>>  * Schema support for Avro and Protobuf
>>
>> I'd like to propose to cut the release later this week.
>> Please point out any pending change you want to include in 2.1.
>>
>> Also, is anyone volunteering as release manager for 2.1?
>>
>> Matteo
>>
>> --
>> Matteo Merli
>> 
>>
>

Re: [VOTE] Maintaining "Apache Pulsar" name

[Sijie Guo]

+1 for keeping "Apache Pulsar"

On Wed, Jun 20, 2018 at 5:06 PM Matteo Merli  wrote:

> Following the previous discussion at
>
> https://lists.apache.org/thread.html/fe60c21b7cdca00918fe25e7ddf9772677342063b9da24c9b84b3329@%3Cdev.pulsar.apache.org%3E
> I am calling a formal vote for keep using "Apache Pulsar" name for the
> project.
>
> This vote will be open for at least 72 hours. After that we will
> communicate the decision to trademarks@apache
>
> +1 from me
>
> Matteo
> --
> Matteo Merli
> 
>

Re: Non-serializable builders

[Sijie Guo]

Hi Geoffroy,

I put up a PR https://github.com/apache/incubator-pulsar/pull/2004 for
addressing the issue. If you can take a look and see if that approach
addresses your problem.

- Sijie

On Fri, Jun 8, 2018 at 8:32 AM Sijie Guo  wrote:

> Hi Geoffroy,
>
> I created an issue for
> https://github.com/apache/incubator-pulsar/issues/1943.
>
> We will address it for the upcoming 2.1 release.
>
> - Sijie
>
> On Fri, Jun 8, 2018 at 6:39 AM Geoffroy Fouquier <
> geoffroy.fouqu...@exensa.com> wrote:
>
>> On 08/06/2018 12:01, Sijie Guo wrote:
>> > Geoffroy,
>> >
>> > Thank you for reporting this. The change was made to evolve pulsar into
>> a
>> > type-safe client with schema and more validations. Along with the idea,
>> > we changed the client to become a more fluent style builder and hide the
>> > configuration data as an implementation detail to the pulsar client
>> > implementation.
>> >
>> > However we missed the use cases in data computing world, people tends to
>> > use java Serializable on sharing configuration.
>> >
>> > We would incorporate your use cases and try to bring the configuration
>> > object back.
>> >
>> > - Sijie
>>
>> That would be great, thanks!
>>
>> >
>> >
>> > On Fri, Jun 8, 2018 at 1:09 AM Geoffroy Fouquier <
>> > geoffroy.fouqu...@exensa.com> wrote:
>> >
>> >> We are using pulsar 1.22 within a spark framework and I am currently
>> >> upgrading my cluster to pulsar 2.0. One of the main change concerns the
>> >> configuration classes, replace by builders. Although ProducerBuilder
>> and
>> >> ConsumerBuilder interfaces implement Serializable, corresponding
>> >> implementations aren't, because a pulsar client object is embedded. The
>> >> schema object is also a problem for serialization.
>> >>
>> >>
>> >> Basically, with 1.22 my producer app. instanciate a client and a
>> >> producer on each executor and only configurations classes need to be
>> >> serialized
>> >>
>> >> ---8<---
>> >>
>> >> WarcRecordExtractor
>> >>   .load(sc, input)
>> >>   .foreachPartition{ ite =>
>> >>
>> >>   val client = PulsarClient.create(ServiceUrl, clientConf)
>> >>   val producer = client.createProducer(topic, producerConf)
>> >>   ite.foreach{ x => producer.sendAsync(x._2.content) }
>> >>   producer.close()
>> >>
>> >>   client.close()
>> >>
>> >>   }
>> >>
>> >> ---8<---
>> >>
>> >>
>> >> With 2.0 I try to replace the configuration classes with the
>> >> corresponding builders, but this setting doesn't work since the
>> >> producerBuilder isn't serializable :
>> >>
>> >> ---8<---
>> >>
>> >>   [...]
>> >>
>> >>   val client = clientBuilder.serviceUrl(serviceUrl).build()
>> >>   val producer = producerBuilder.topic(topicName).create
>> >>
>> >>   [...]
>> >>
>> >> ---8<---
>> >>
>> >>
>> >> My actual workaround is to rewrite my own ProducerBuilderImpl without
>> >> pulsar client or schemas (the createAsync method still exists but throw
>> >> an exception). And then to instanciate a producer like this :
>> >>
>> >> ---8<---
>> >>
>> >>   [...]
>> >>
>> >>   val client : PulsarClientImpl =
>> >> clientBuilder.serviceUrl(broker).build().asInstanceOf[PulsarClientImpl]
>> >>   val producer =
>> >> client.createProducerAsync(producerBuilder.topic(topic).getConf(),
>> >> org.apache.pulsar.client.api.Schema.BYTES).get()
>> >>
>> >>   [...]
>> >>
>> >> ---8<---
>> >>
>> >> where producerBuilder is my own implementation which implements
>> >> ProducerBuilder interface.
>> >>
>> >>
>> >> I did the same thing for ConsumerBuilderImpl and also rewrite
>> >> SparkStreamingPulsarReceiver with the same kind of workaround. I don't
>> >> know why the configuration classes have been replaced by builders, but
>> >> it became more difficult to use it with spark (and probably with other
>> >> distributed framework), but maybe i did it wrong.
>> >>
>> >>
>> >>
>> >>
>>
>>

[ANNOUNCE] Apache BookKeeper 4.7.1 released

[Sijie Guo]

The Apache BookKeeper team is proud to announce Apache BookKeeper version
4.7.1.

Apache BookKeeper is a scalable, fault-tolerant, and low-latency storage
service optimized for
real-time workloads. It has been used for a fundamental service to build
reliable services.
It is also the log segment store for Apache DistributedLog and the message
store for Apache Pulsar.

This is the 11th release of Apache BookKeeper.

This is a bugfix release, which fixes a bunch of issues reported from users
of 4.7.0.
These changes include bug fixes around ledger cache and object pooling,
performance
enhancement avoid memory copies and various bug fixes and improvements
around
bookkeeper table service.

For BookKeeper release details and downloads, visit:

https://bookkeeper.apache.org/releases/

BookKeeper 4.7.1 Release Notes are at:

https://bookkeeper.apache.org/docs/4.7.1/overview/releaseNotes/

We would like to thank the contributors that made the release possible.

Regards,

The BookKeeper Team

Re: Non-persistent topics

[Sijie Guo]

2.1.0-SNAPSHOT is published to apache staging nightly.

I will try to include this fix for 2.1.0 release in this week. So it can go
out with 2.1 release.

On Tue, Jun 19, 2018 at 11:40 PM Geoffroy Fouquier <
geoffroy.fouqu...@exensa.com> wrote:

> On 19/06/2018 21:58, Sijie Guo wrote:
> > I put up a fix - https://github.com/apache/incubator-pulsar/pull/1994
> > verified that it fixes the problem. however I would like to get feedback
> > before I added unit tests / integration tests.
> >
> > - Sijie
>
> Great, thanks !
>
> Is there some nightly build which produces binary distribution ?
>
>

Re: Non-persistent topics

[Sijie Guo]

I put up a fix - https://github.com/apache/incubator-pulsar/pull/1994
verified that it fixes the problem. however I would like to get feedback
before I added unit tests / integration tests.

- Sijie

On Tue, Jun 19, 2018 at 12:22 PM Sijie Guo  wrote:

> Actually there is already an issue for that -
> https://github.com/apache/incubator-pulsar/issues/1967  will reuse that.
>
> The fix can be simple, but would like to get some clarifications from
> yahoo folks. as I understand, all the messages in non-persistent topic have
> same message ids. it makes the behavior different from persistent topic,
> which I think acknowledgements for non-persistent topic are ignored. if
> that is the expected semantic, one straightforward fix is disable
> acknowledgement grouping for non-persistent topic. Can someone from yahoo
> clarify it before I started the fix?
>
> - Sijie
>
> On Tue, Jun 19, 2018 at 12:15 PM Sijie Guo  wrote:
>
>> I found the issue. So in 2.0 there is an optimization change on grouping
>> acknowledgements. The optimization change here
>> <https://github.com/apache/incubator-pulsar/commit/19dd2c502725e3b45e56541576508626c4213091#diff-debb36270f152d2533b03902f214fa9aR674>
>>  avoided
>> delivered messages that are already "delivered".
>> However I think in non-persistent topic, since we don't store entries to
>> bookkeeper storage, we end up using 0 as ledger id and entry id. so when
>> the optimization change treats those
>> messages as already "delivered" so it never pops those messages to client
>> even client already receive them from the wire.
>>
>> Filing a github issue for it.
>>
>> - Sijie
>>
>> On Tue, Jun 19, 2018 at 11:27 AM Sijie Guo  wrote:
>>
>>> Hi Geoffroy,
>>>
>>> Sorry for late response. I tried your sequence and observed same
>>> behavior. so there might be something wrong there. I am looking into it now.
>>>
>>> - Sijie
>>>
>>> On Tue, Jun 19, 2018 at 11:13 AM  wrote:
>>>
>>>> Le 2018-06-19 19:08, Ali Ahmed a écrit :
>>>> > Is the server version is both cases 2.0 ?
>>>>
>>>> No, I always use the same version for pulsar-client and the server. For
>>>> a given pulsar version, my test script download an archive and make all
>>>> tests using this archive.
>>>>
>>>> > -Ali
>>>> >
>>>> >
>>>> > On Tue, Jun 19, 2018 at 7:47 AM, Geoffroy Fouquier <
>>>> > geoffroy.fouqu...@exensa.com> wrote:
>>>> >
>>>> >>
>>>> >> I recently describe on pulsar-users my issue concerning
>>>> non-persistent
>>>> >> topics. This time, I reproduce the faultly behaviour using only
>>>> >> pulsar-client and test the new pulsar 2.0.1 release, but my problem
>>>> >> remains.
>>>> >>
>>>> >> It's quite easy to reproduce this behaviour:
>>>> >>
>>>> >>  - I made tests with fresh installations only and without any
>>>> >> configurations.
>>>> >>
>>>> >>  - I start a standalone server, then after a few seconds a
>>>> >> pulsar-client
>>>> >> which consume a topic.
>>>> >>
>>>> >>  - Then i use pulsar-client to produce messages on the same topic
>>>> (and
>>>> >> repeat with a few seconds of delay between each batch).
>>>> >>
>>>> >>
>>>> >> if I send 10 times 1000 short messages ("foo bar baz")
>>>> >>
>>>> >> with pulsar 1.22, I receive:
>>>> >>
>>>> >> 100% of messages on a persistent topic
>>>> >>
>>>> >> 100% of messages on a non-persistent topic
>>>> >>
>>>> >> with pulsar 2.0.1, I receive:
>>>> >>
>>>> >> 100% of messages on a persistent topic.
>>>> >>
>>>> >>0,09% of messages on a non-persistent topic (9/1)
>>>> >>
>>>> >>
>>>> >> In fact, the nine messages are received with the first batch, and
>>>> >> nothing
>>>> >> is receive after the first one. I understand that if i send too many
>>>> >> messages the broker might start to drop messages. But 1000 messages
>>>> >> aren't
>>>> >> such a burden and pulsar 1.

Re: Non-persistent topics

[Sijie Guo]

Actually there is already an issue for that -
https://github.com/apache/incubator-pulsar/issues/1967  will reuse that.

The fix can be simple, but would like to get some clarifications from yahoo
folks. as I understand, all the messages in non-persistent topic have same
message ids. it makes the behavior different from persistent topic,
which I think acknowledgements for non-persistent topic are ignored. if
that is the expected semantic, one straightforward fix is disable
acknowledgement grouping for non-persistent topic. Can someone from yahoo
clarify it before I started the fix?

- Sijie

On Tue, Jun 19, 2018 at 12:15 PM Sijie Guo  wrote:

> I found the issue. So in 2.0 there is an optimization change on grouping
> acknowledgements. The optimization change here
> <https://github.com/apache/incubator-pulsar/commit/19dd2c502725e3b45e56541576508626c4213091#diff-debb36270f152d2533b03902f214fa9aR674>
>  avoided
> delivered messages that are already "delivered".
> However I think in non-persistent topic, since we don't store entries to
> bookkeeper storage, we end up using 0 as ledger id and entry id. so when
> the optimization change treats those
> messages as already "delivered" so it never pops those messages to client
> even client already receive them from the wire.
>
> Filing a github issue for it.
>
> - Sijie
>
> On Tue, Jun 19, 2018 at 11:27 AM Sijie Guo  wrote:
>
>> Hi Geoffroy,
>>
>> Sorry for late response. I tried your sequence and observed same
>> behavior. so there might be something wrong there. I am looking into it now.
>>
>> - Sijie
>>
>> On Tue, Jun 19, 2018 at 11:13 AM  wrote:
>>
>>> Le 2018-06-19 19:08, Ali Ahmed a écrit :
>>> > Is the server version is both cases 2.0 ?
>>>
>>> No, I always use the same version for pulsar-client and the server. For
>>> a given pulsar version, my test script download an archive and make all
>>> tests using this archive.
>>>
>>> > -Ali
>>> >
>>> >
>>> > On Tue, Jun 19, 2018 at 7:47 AM, Geoffroy Fouquier <
>>> > geoffroy.fouqu...@exensa.com> wrote:
>>> >
>>> >>
>>> >> I recently describe on pulsar-users my issue concerning non-persistent
>>> >> topics. This time, I reproduce the faultly behaviour using only
>>> >> pulsar-client and test the new pulsar 2.0.1 release, but my problem
>>> >> remains.
>>> >>
>>> >> It's quite easy to reproduce this behaviour:
>>> >>
>>> >>  - I made tests with fresh installations only and without any
>>> >> configurations.
>>> >>
>>> >>  - I start a standalone server, then after a few seconds a
>>> >> pulsar-client
>>> >> which consume a topic.
>>> >>
>>> >>  - Then i use pulsar-client to produce messages on the same topic (and
>>> >> repeat with a few seconds of delay between each batch).
>>> >>
>>> >>
>>> >> if I send 10 times 1000 short messages ("foo bar baz")
>>> >>
>>> >> with pulsar 1.22, I receive:
>>> >>
>>> >> 100% of messages on a persistent topic
>>> >>
>>> >> 100% of messages on a non-persistent topic
>>> >>
>>> >> with pulsar 2.0.1, I receive:
>>> >>
>>> >> 100% of messages on a persistent topic.
>>> >>
>>> >>0,09% of messages on a non-persistent topic (9/1)
>>> >>
>>> >>
>>> >> In fact, the nine messages are received with the first batch, and
>>> >> nothing
>>> >> is receive after the first one. I understand that if i send too many
>>> >> messages the broker might start to drop messages. But 1000 messages
>>> >> aren't
>>> >> such a burden and pulsar 1.22 doesn't have any problem to handle it.
>>> >> But
>>> >> ok, maybe I send too many messages at the same times, so I try to send
>>> >> smaller batch (resp. 10 and 2) :
>>> >>
>>> >>  - If I send 10 times 10 messages, I receive 12 / 100 messages.
>>> >>
>>> >>  - If a send 10 times 2 messages, I receive 11 / 20 messages.
>>> >>
>>> >>
>>> >> So I think there is a bug with pulsar 2 and non-persistent topic, but
>>> >> maybe I am wrong.
>>> >>
>>> >>
>

Re: Non-persistent topics

[Sijie Guo]

I found the issue. So in 2.0 there is an optimization change on grouping
acknowledgements. The optimization change here
<https://github.com/apache/incubator-pulsar/commit/19dd2c502725e3b45e56541576508626c4213091#diff-debb36270f152d2533b03902f214fa9aR674>
avoided
delivered messages that are already "delivered".
However I think in non-persistent topic, since we don't store entries to
bookkeeper storage, we end up using 0 as ledger id and entry id. so when
the optimization change treats those
messages as already "delivered" so it never pops those messages to client
even client already receive them from the wire.

Filing a github issue for it.

- Sijie

On Tue, Jun 19, 2018 at 11:27 AM Sijie Guo  wrote:

> Hi Geoffroy,
>
> Sorry for late response. I tried your sequence and observed same behavior.
> so there might be something wrong there. I am looking into it now.
>
> - Sijie
>
> On Tue, Jun 19, 2018 at 11:13 AM  wrote:
>
>> Le 2018-06-19 19:08, Ali Ahmed a écrit :
>> > Is the server version is both cases 2.0 ?
>>
>> No, I always use the same version for pulsar-client and the server. For
>> a given pulsar version, my test script download an archive and make all
>> tests using this archive.
>>
>> > -Ali
>> >
>> >
>> > On Tue, Jun 19, 2018 at 7:47 AM, Geoffroy Fouquier <
>> > geoffroy.fouqu...@exensa.com> wrote:
>> >
>> >>
>> >> I recently describe on pulsar-users my issue concerning non-persistent
>> >> topics. This time, I reproduce the faultly behaviour using only
>> >> pulsar-client and test the new pulsar 2.0.1 release, but my problem
>> >> remains.
>> >>
>> >> It's quite easy to reproduce this behaviour:
>> >>
>> >>  - I made tests with fresh installations only and without any
>> >> configurations.
>> >>
>> >>  - I start a standalone server, then after a few seconds a
>> >> pulsar-client
>> >> which consume a topic.
>> >>
>> >>  - Then i use pulsar-client to produce messages on the same topic (and
>> >> repeat with a few seconds of delay between each batch).
>> >>
>> >>
>> >> if I send 10 times 1000 short messages ("foo bar baz")
>> >>
>> >> with pulsar 1.22, I receive:
>> >>
>> >> 100% of messages on a persistent topic
>> >>
>> >> 100% of messages on a non-persistent topic
>> >>
>> >> with pulsar 2.0.1, I receive:
>> >>
>> >> 100% of messages on a persistent topic.
>> >>
>> >>0,09% of messages on a non-persistent topic (9/1)
>> >>
>> >>
>> >> In fact, the nine messages are received with the first batch, and
>> >> nothing
>> >> is receive after the first one. I understand that if i send too many
>> >> messages the broker might start to drop messages. But 1000 messages
>> >> aren't
>> >> such a burden and pulsar 1.22 doesn't have any problem to handle it.
>> >> But
>> >> ok, maybe I send too many messages at the same times, so I try to send
>> >> smaller batch (resp. 10 and 2) :
>> >>
>> >>  - If I send 10 times 10 messages, I receive 12 / 100 messages.
>> >>
>> >>  - If a send 10 times 2 messages, I receive 11 / 20 messages.
>> >>
>> >>
>> >> So I think there is a bug with pulsar 2 and non-persistent topic, but
>> >> maybe I am wrong.
>> >>
>> >>
>> >> Some logs about my experiments:
>> >>
>> >> % NBITE=10 NB_MESSAGE=1000 PERS=non-persistent
>> >> PULSAR_VERSION=1.22.0-incubating ./test-pulsar.sh
>> >> Pulsar version: 1.22.0-incubating
>> >> Starting standalone pulsar with pid 32128
>> >> Starting pulsar consumer (pid 32541) on non-persistent://tenant/standa
>> >> lone/ns/topic
>> >>
>> >> Writing 1000 messages (foo bar baz) on non-persistent://tenant/standa
>> >> lone/ns/topic
>> >> (1/10) Nb received messages: 1000 (should be: 1000)
>> >> (2/10) Nb received messages: 2000 (should be: 2000)
>> >> (3/10) Nb received messages: 3000 (should be: 3000)
>> >> (4/10) Nb received messages: 4000 (should be: 4000)
>> >> (5/10) Nb received messages: 5000 (should be: 5000)
>> >> (6/10) Nb received messages: 6000 (should be: 6000)
>> >> (7/10) Nb received messages: 

Re: Non-persistent topics

[Sijie Guo]

Hi Geoffroy,

Sorry for late response. I tried your sequence and observed same behavior.
so there might be something wrong there. I am looking into it now.

- Sijie

On Tue, Jun 19, 2018 at 11:13 AM  wrote:

> Le 2018-06-19 19:08, Ali Ahmed a écrit :
> > Is the server version is both cases 2.0 ?
>
> No, I always use the same version for pulsar-client and the server. For
> a given pulsar version, my test script download an archive and make all
> tests using this archive.
>
> > -Ali
> >
> >
> > On Tue, Jun 19, 2018 at 7:47 AM, Geoffroy Fouquier <
> > geoffroy.fouqu...@exensa.com> wrote:
> >
> >>
> >> I recently describe on pulsar-users my issue concerning non-persistent
> >> topics. This time, I reproduce the faultly behaviour using only
> >> pulsar-client and test the new pulsar 2.0.1 release, but my problem
> >> remains.
> >>
> >> It's quite easy to reproduce this behaviour:
> >>
> >>  - I made tests with fresh installations only and without any
> >> configurations.
> >>
> >>  - I start a standalone server, then after a few seconds a
> >> pulsar-client
> >> which consume a topic.
> >>
> >>  - Then i use pulsar-client to produce messages on the same topic (and
> >> repeat with a few seconds of delay between each batch).
> >>
> >>
> >> if I send 10 times 1000 short messages ("foo bar baz")
> >>
> >> with pulsar 1.22, I receive:
> >>
> >> 100% of messages on a persistent topic
> >>
> >> 100% of messages on a non-persistent topic
> >>
> >> with pulsar 2.0.1, I receive:
> >>
> >> 100% of messages on a persistent topic.
> >>
> >>0,09% of messages on a non-persistent topic (9/1)
> >>
> >>
> >> In fact, the nine messages are received with the first batch, and
> >> nothing
> >> is receive after the first one. I understand that if i send too many
> >> messages the broker might start to drop messages. But 1000 messages
> >> aren't
> >> such a burden and pulsar 1.22 doesn't have any problem to handle it.
> >> But
> >> ok, maybe I send too many messages at the same times, so I try to send
> >> smaller batch (resp. 10 and 2) :
> >>
> >>  - If I send 10 times 10 messages, I receive 12 / 100 messages.
> >>
> >>  - If a send 10 times 2 messages, I receive 11 / 20 messages.
> >>
> >>
> >> So I think there is a bug with pulsar 2 and non-persistent topic, but
> >> maybe I am wrong.
> >>
> >>
> >> Some logs about my experiments:
> >>
> >> % NBITE=10 NB_MESSAGE=1000 PERS=non-persistent
> >> PULSAR_VERSION=1.22.0-incubating ./test-pulsar.sh
> >> Pulsar version: 1.22.0-incubating
> >> Starting standalone pulsar with pid 32128
> >> Starting pulsar consumer (pid 32541) on non-persistent://tenant/standa
> >> lone/ns/topic
> >>
> >> Writing 1000 messages (foo bar baz) on non-persistent://tenant/standa
> >> lone/ns/topic
> >> (1/10) Nb received messages: 1000 (should be: 1000)
> >> (2/10) Nb received messages: 2000 (should be: 2000)
> >> (3/10) Nb received messages: 3000 (should be: 3000)
> >> (4/10) Nb received messages: 4000 (should be: 4000)
> >> (5/10) Nb received messages: 5000 (should be: 5000)
> >> (6/10) Nb received messages: 6000 (should be: 6000)
> >> (7/10) Nb received messages: 7000 (should be: 7000)
> >> (8/10) Nb received messages: 8000 (should be: 8000)
> >> (9/10) Nb received messages: 9000 (should be: 9000)
> >> (10/10) Nb received messages: 1 (should be: 1)
> >>
> >> % NBITE=10 NB_MESSAGE=1000 PERS=non-persistent
> >> PULSAR_VERSION=2.0.1-incubating ./test-pulsar.sh
> >> Pulsar version: 2.0.1-incubating
> >> Starting standalone pulsar with pid 2382
> >> Starting pulsar consumer (pid 3080) on
> >> non-persistent://tenant/ns/topic
> >>
> >> Writing 1000 messages (foo bar baz) on
> >> non-persistent://tenant/ns/topic
> >>   [...16:07:36.727 [main] INFO
> >> org.apache.pulsar.client.cli.PulsarClientTool
> >> - 1000 messages successfully produced]
> >> (1/10) Nb received messages: 9 (should be: 1000)
> >>   [...16:07:43.792 [main] INFO
> >> org.apache.pulsar.client.cli.PulsarClientTool
> >> - 1000 messages successfully produced]
> >> (2/10) Nb received messages: 9 (should be: 2000)
> >>   [...16:07:50.963 [main] INFO
> >> org.apache.pulsar.client.cli.PulsarClientTool
> >> - 1000 messages successfully produced]
> >> (3/10) Nb received messages: 9 (should be: 3000)
> >>   [...16:07:58.158 [main] INFO
> >> org.apache.pulsar.client.cli.PulsarClientTool
> >> - 1000 messages successfully produced]
> >> (4/10) Nb received messages: 9 (should be: 4000)
> >>   [...16:08:05.419 [main] INFO
> >> org.apache.pulsar.client.cli.PulsarClientTool
> >> - 1000 messages successfully produced]
> >> (5/10) Nb received messages: 9 (should be: 5000)
> >>   [...16:08:12.650 [main] INFO
> >> org.apache.pulsar.client.cli.PulsarClientTool
> >> - 1000 messages successfully produced]
> >> (6/10) Nb received messages: 9 (should be: 6000)
> >>   [...16:08:19.780 [main] INFO
> >> org.apache.pulsar.client.cli.PulsarClientTool
> >> - 1000 messages successfully produced]
> >> (7/10) Nb received messages: 9 (should be: 7000)
> >> 

Re: [ANNOUNCE] Apache Pulsar 2.0.1-incubating released

[Sijie Guo]

👍

On Tue, Jun 19, 2018 at 12:43 AM Matteo Merli  wrote:

> The Apache Pulsar team is proud to announce Apache Pulsar version
> 2.0.1-incubating.
>
> This release fixes issues reported for 2.0.0-rc1-incubating.
> Pulsar is a highly scalable, low latency messaging platform running on
> commodity hardware. It provides simple pub-sub semantics over topics,
> guaranteed at-least-once delivery of messages, automatic cursor management
> for
> subscribers, and cross-datacenter replication.
>
> For Pulsar release details and downloads, visit:
>
> https://pulsar.incubator.apache.org/download
>
> Release Notes are at:
> http://pulsar.apache.org/release-notes/#2.0.1-incubating
>
> We would like to thank the contributors that made the release possible.
>
> Regards,
>
> The Pulsar Team
>
> DISCLAIMER:
> Apache Pulsar is an effort undergoing incubation at The Apache Software
> Foundation (ASF), sponsored by the Apache Incubator PMC. Incubation is
> required of all newly accepted projects until a further review indicates
> that
> the infrastructure, communications, and decision making process have
> stabilized in a manner consistent with other successful ASF projects. While
> incubation status is not necessarily a reflection of the completeness or
> stability
> of the code, it does indicate that the project has yet to be fully endorsed
> by the ASF.
>
>
>
> --
> Matteo Merli
> 
>

Re: CloudEvents

[Sijie Guo]

FYI I started a PR for it. https://github.com/cloudevents/spec/pull/237

On Mon, Jun 18, 2018 at 12:31 AM Sijie Guo  wrote:

> Hi all,
>
> cloudevents <https://cloudevents.io/> is a specification for describing
> event data in a common way. I wrote a transport binding proposal for pulsar
> - how to map pulsar messages into cloudevents specification. Here is a
> draft: https://gist.github.com/sijie/082c59e66b9ed175c1bb48466f3629f0
>
> I would like to get some feedback from the community. If people don't see
> a major problem, I will submit a PR to cloudevents to contribute this
> pulsar transport binding proposal.
>
> Thank,
> Sijie
>

Re: Planning for release 2.1

[Sijie Guo]

I am happy to contribute if no one else is working on that.

- Sijie

On Mon, Jun 18, 2018 at 12:01 PM Matteo Merli  wrote:

> As we are about to release 2.0.1, we have already a batch of new items
> ready to be released. Biggest ticket items are:
>  * Connector framework
>  * Go client library
>  * Schema support for Avro and Protobuf
>
> I'd like to propose to cut the release later this week.
> Please point out any pending change you want to include in 2.1.
>
> Also, is anyone volunteering as release manager for 2.1?
>
> Matteo
>
> --
> Matteo Merli
> 
>

CloudEvents

[Sijie Guo]

Hi all,

cloudevents  is a specification for describing
event data in a common way. I wrote a transport binding proposal for pulsar
- how to map pulsar messages into cloudevents specification. Here is a
draft: https://gist.github.com/sijie/082c59e66b9ed175c1bb48466f3629f0

I would like to get some feedback from the community. If people don't see a
major problem, I will submit a PR to cloudevents to contribute this pulsar
transport binding proposal.

Thank,
Sijie

triage labels

[Sijie Guo]

Hi all,

Ivan and me started the idea of doing bug triages on pulsar github issues,
to make sure we don't miss any issues reported from users at mailing list
or slack channel and make sure every issue is helped. We are using
`triage/week-` to label the issues. so if you are seeing such labels,
don't be freaked out about that :)

if you volunteer to help and contribute to issue triages, feel free to ping
Ivan and me as well :)

- Sijie

Re: Pulsar meetup in June/July

[Sijie Guo]

That's great! Thank you, Joe!

Anyone wanna contribute the talks?

So we can start finalizing an agenda, and send out meetup invitations?

On Fri, Jun 8, 2018 at 10:39 AM Joe F  wrote:

> I have reserved the hall at Bldg C,  701, First Ave, Sunnyvale . This is in
> Oath Campus.
>
> Date is July 10th
>
> Joe
>
>
>
> On Thu, Jun 7, 2018 at 4:55 PM, Sijie Guo  wrote:
>
> > +1 go pulsar!
> >
> > - Sijie
> >
> > On Thu, Jun 7, 2018 at 4:49 PM Sanjeev Kulkarni 
> > wrote:
> >
> > > July 10th works for me as well!
> > >
> > > On Tue, Jun 5, 2018 at 11:21 AM Joe F  wrote:
> > >
> > > > All,
> > > >
> > > > I got the availability for the venue in the 2nd week of July  (50
> > > people),
> > > > and its available on  July 9th, 10th, or 13th .  How does that sound?
> > I
> > > > need to reserve a firm date...
> > > >
> > > > My preference is for 10th  [since its not a Monday or Friday :-) ]
> but
> > > I'm
> > > > fine with a whatever is convenient for all ..
> > > >
> > > >
> > > > Joe
> > > >
> > > > On Sun, Jun 3, 2018 at 7:21 PM, Sijie Guo 
> wrote:
> > > >
> > > > > Any updates?
> > > > >
> > > > > - Sijie
> > > > >
> > > > > On Mon, Apr 30, 2018 at 7:29 PM, Joe F  wrote:
> > > > >
> > > > > > I would expect people to volunteer and sign up... that's why I
> said
> > > > > "from
> > > > > > Pulsar users and developers".
> > > > > >
> > > > > >
> > > > > > Joe
> > > > > >
> > > > > > On Mon, Apr 30, 2018 at 6:44 PM, Dave Fisher <
> > dave2w...@comcast.net>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi -
> > > > > > >
> > > > > > > What would be the agenda?
> > > > > > >
> > > > > > > How can this event be promoted to help grow the community?
> > > > > > >
> > > > > > > Regards,
> > > > > > > Dave
> > > > > > >
> > > > > > > Sent from my iPhone
> > > > > > >
> > > > > > > > On Apr 30, 2018, at 11:29 AM, Joe F 
> > > wrote:
> > > > > > > >
> > > > > > > > In June, it going to be a year since Pulsar was accepted into
> > the
> > > > > > > incubator.
> > > > > > > >
> > > > > > > > I'm thinking of a meetup, with a few in-depth
> > > talks/presentations,
> > > > > > what's
> > > > > > > > new in Pulsar 2.0 etc,   from Pulsar users and developers.
> Are
> > > > > people
> > > > > > > > interested in a 1/2day event?   I can arrange for the venue
> and
> > > > > > > logistics.
> > > > > > > >
> > > > > > > >
> > > > > > > > Joe
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Non-serializable builders

[Sijie Guo]

Hi Geoffroy,

I created an issue for
https://github.com/apache/incubator-pulsar/issues/1943.

We will address it for the upcoming 2.1 release.

- Sijie

On Fri, Jun 8, 2018 at 6:39 AM Geoffroy Fouquier <
geoffroy.fouqu...@exensa.com> wrote:

> On 08/06/2018 12:01, Sijie Guo wrote:
> > Geoffroy,
> >
> > Thank you for reporting this. The change was made to evolve pulsar into a
> > type-safe client with schema and more validations. Along with the idea,
> > we changed the client to become a more fluent style builder and hide the
> > configuration data as an implementation detail to the pulsar client
> > implementation.
> >
> > However we missed the use cases in data computing world, people tends to
> > use java Serializable on sharing configuration.
> >
> > We would incorporate your use cases and try to bring the configuration
> > object back.
> >
> > - Sijie
>
> That would be great, thanks!
>
> >
> >
> > On Fri, Jun 8, 2018 at 1:09 AM Geoffroy Fouquier <
> > geoffroy.fouqu...@exensa.com> wrote:
> >
> >> We are using pulsar 1.22 within a spark framework and I am currently
> >> upgrading my cluster to pulsar 2.0. One of the main change concerns the
> >> configuration classes, replace by builders. Although ProducerBuilder and
> >> ConsumerBuilder interfaces implement Serializable, corresponding
> >> implementations aren't, because a pulsar client object is embedded. The
> >> schema object is also a problem for serialization.
> >>
> >>
> >> Basically, with 1.22 my producer app. instanciate a client and a
> >> producer on each executor and only configurations classes need to be
> >> serialized
> >>
> >> ---8<---
> >>
> >> WarcRecordExtractor
> >>   .load(sc, input)
> >>   .foreachPartition{ ite =>
> >>
> >>   val client = PulsarClient.create(ServiceUrl, clientConf)
> >>   val producer = client.createProducer(topic, producerConf)
> >>   ite.foreach{ x => producer.sendAsync(x._2.content) }
> >>   producer.close()
> >>
> >>   client.close()
> >>
> >>   }
> >>
> >> ---8<---
> >>
> >>
> >> With 2.0 I try to replace the configuration classes with the
> >> corresponding builders, but this setting doesn't work since the
> >> producerBuilder isn't serializable :
> >>
> >> ---8<---
> >>
> >>   [...]
> >>
> >>   val client = clientBuilder.serviceUrl(serviceUrl).build()
> >>   val producer = producerBuilder.topic(topicName).create
> >>
> >>   [...]
> >>
> >> ---8<---
> >>
> >>
> >> My actual workaround is to rewrite my own ProducerBuilderImpl without
> >> pulsar client or schemas (the createAsync method still exists but throw
> >> an exception). And then to instanciate a producer like this :
> >>
> >> ---8<---
> >>
> >>   [...]
> >>
> >>   val client : PulsarClientImpl =
> >> clientBuilder.serviceUrl(broker).build().asInstanceOf[PulsarClientImpl]
> >>   val producer =
> >> client.createProducerAsync(producerBuilder.topic(topic).getConf(),
> >> org.apache.pulsar.client.api.Schema.BYTES).get()
> >>
> >>   [...]
> >>
> >> ---8<---
> >>
> >> where producerBuilder is my own implementation which implements
> >> ProducerBuilder interface.
> >>
> >>
> >> I did the same thing for ConsumerBuilderImpl and also rewrite
> >> SparkStreamingPulsarReceiver with the same kind of workaround. I don't
> >> know why the configuration classes have been replaced by builders, but
> >> it became more difficult to use it with spark (and probably with other
> >> distributed framework), but maybe i did it wrong.
> >>
> >>
> >>
> >>
>
>

Re: Non-serializable builders

[Sijie Guo]

Geoffroy,

Thank you for reporting this. The change was made to evolve pulsar into a
type-safe client with schema and more validations. Along with the idea,
we changed the client to become a more fluent style builder and hide the
configuration data as an implementation detail to the pulsar client
implementation.

However we missed the use cases in data computing world, people tends to
use java Serializable on sharing configuration.

We would incorporate your use cases and try to bring the configuration
object back.

- Sijie



On Fri, Jun 8, 2018 at 1:09 AM Geoffroy Fouquier <
geoffroy.fouqu...@exensa.com> wrote:

>
> We are using pulsar 1.22 within a spark framework and I am currently
> upgrading my cluster to pulsar 2.0. One of the main change concerns the
> configuration classes, replace by builders. Although ProducerBuilder and
> ConsumerBuilder interfaces implement Serializable, corresponding
> implementations aren't, because a pulsar client object is embedded. The
> schema object is also a problem for serialization.
>
>
> Basically, with 1.22 my producer app. instanciate a client and a
> producer on each executor and only configurations classes need to be
> serialized
>
> ---8<---
>
> WarcRecordExtractor
>  .load(sc, input)
>  .foreachPartition{ ite =>
>
>  val client = PulsarClient.create(ServiceUrl, clientConf)
>  val producer = client.createProducer(topic, producerConf)
>  ite.foreach{ x => producer.sendAsync(x._2.content) }
>  producer.close()
>
>  client.close()
>
>  }
>
> ---8<---
>
>
> With 2.0 I try to replace the configuration classes with the
> corresponding builders, but this setting doesn't work since the
> producerBuilder isn't serializable :
>
> ---8<---
>
>  [...]
>
>  val client = clientBuilder.serviceUrl(serviceUrl).build()
>  val producer = producerBuilder.topic(topicName).create
>
>  [...]
>
> ---8<---
>
>
> My actual workaround is to rewrite my own ProducerBuilderImpl without
> pulsar client or schemas (the createAsync method still exists but throw
> an exception). And then to instanciate a producer like this :
>
> ---8<---
>
>  [...]
>
>  val client : PulsarClientImpl =
> clientBuilder.serviceUrl(broker).build().asInstanceOf[PulsarClientImpl]
>  val producer =
> client.createProducerAsync(producerBuilder.topic(topic).getConf(),
> org.apache.pulsar.client.api.Schema.BYTES).get()
>
>  [...]
>
> ---8<---
>
> where producerBuilder is my own implementation which implements
> ProducerBuilder interface.
>
>
> I did the same thing for ConsumerBuilderImpl and also rewrite
> SparkStreamingPulsarReceiver with the same kind of workaround. I don't
> know why the configuration classes have been replaced by builders, but
> it became more difficult to use it with spark (and probably with other
> distributed framework), but maybe i did it wrong.
>
>
>
>

Re: [DISCUSSION] Podling name search

[Sijie Guo]

On Thu, Jun 7, 2018 at 6:48 PM Jerry Peng 
wrote:

> I completely agree with want Matteo said.  Changing the name will
> effectively kill the project.   Changing the project will probably also
> implying changing the package names in the code itself which will be a
> breaking and non-backwards compatible change.  Any reasonable developer
> will be turned off by this.
>




>
> To be honest, considering something that might happen in 4 years is
> analogous to preventing a plane from taking off because someone might chock
> on peanuts during the flight.  Lets first get the plane to take off.
>

I like this, strongly +1. #shipit




>
> On Thu, Jun 7, 2018 at 6:39 PM Sijie Guo  wrote:
>
> > +1 for using the name "Apache Pulsar".
> >
> > - Sijie
> >
> > On Thu, Jun 7, 2018 at 5:01 PM Matteo Merli  wrote:
> >
> > > As some in Pulsar community are already aware, one of the pending
> > > tasks for Pulsar project to complete the "name search task".
> > >
> > > A JIRA task was opened to collect facts around usages of Pulsar
> > > name: https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-141
> > >
> > > I had forwarded that to trademarks@ and this was the response
> > > from Mark Thomas.
> > >
> > > > Pulsar is a very popular name for software.
> > > >
> > > > I don't see any obvious conflicts but given the popularity of the
> name
> > > > it is likely, in a global marketplace, that there will be some.
> > > >
> > > > Should the project wish to register the "PULSAR" mark in the future,
> it
> > > > is uncertain whether the ASF would be able to. Registering "APACHE
> > > > PULSAR" is unlikely to hit any difficulties.
> > > >
> > > > It is more likely than usual that an infringement will emerge in the
> > > > future that would require the project to rename.
> > > >
> > > > The podling needs to make a choice. Either:
> > > >
> > > > a) continue using PULSAR and accept that:
> > > >- there are likely to be some restrictions on how the name is used
> > > >  (primarily that it always has to be APACHE PULSAR)
> > > >- it is more likely than for most ASF projects that the project
> > > >  will be required to rename due to an infringement
> > > >
> > > > or
> > > >
> > > > b) pick a new name
> > > >
> > > > No rush on this. The podling should take time to discuss this.
> > >
> > >
> > > Therefore we need to take a decision on this matter.
> > >
> > > My personal inclination is to continue to use "Apache Pulsar" and
> > > have that to be registered as a trademark of ASF.
> > >
> > > My reasoning:
> > >
> > >  * We are anyway already always referring to "Apache Pulsar"
> > >rather than just "Pulsar"
> > >
> > >  * Changing name at this point in life of the project would be
> > >very disruptive. We just spent the past year into building
> > >some naming awareness and it would all vanish, not to mention
> > >the huge amount of work in updating documentation, package
> > >names, etc.
> > >
> > >  * If a conflict arise later on, it might be easier to change
> > >name at that point. Once the project has a larger community
> > >and bigger recognition, changing name would not mean to
> > >restart from scratch.
> > >
> > >
> > > I propose to have anyone share they thoughs on this issue. Once
> > > there is a prevalent inclination in the community, we can vote a
> > > final resolution on the subject.
> > >
> > > Thanks,
> > > Matteo
> > >
> > >
> > >
> > > --
> > > Matteo Merli
> > > 
> > >
> >
>

Re: [DISCUSSION] Podling name search

[Sijie Guo]

+1 for using the name "Apache Pulsar".

- Sijie

On Thu, Jun 7, 2018 at 5:01 PM Matteo Merli  wrote:

> As some in Pulsar community are already aware, one of the pending
> tasks for Pulsar project to complete the "name search task".
>
> A JIRA task was opened to collect facts around usages of Pulsar
> name: https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-141
>
> I had forwarded that to trademarks@ and this was the response
> from Mark Thomas.
>
> > Pulsar is a very popular name for software.
> >
> > I don't see any obvious conflicts but given the popularity of the name
> > it is likely, in a global marketplace, that there will be some.
> >
> > Should the project wish to register the "PULSAR" mark in the future, it
> > is uncertain whether the ASF would be able to. Registering "APACHE
> > PULSAR" is unlikely to hit any difficulties.
> >
> > It is more likely than usual that an infringement will emerge in the
> > future that would require the project to rename.
> >
> > The podling needs to make a choice. Either:
> >
> > a) continue using PULSAR and accept that:
> >- there are likely to be some restrictions on how the name is used
> >  (primarily that it always has to be APACHE PULSAR)
> >- it is more likely than for most ASF projects that the project
> >  will be required to rename due to an infringement
> >
> > or
> >
> > b) pick a new name
> >
> > No rush on this. The podling should take time to discuss this.
>
>
> Therefore we need to take a decision on this matter.
>
> My personal inclination is to continue to use "Apache Pulsar" and
> have that to be registered as a trademark of ASF.
>
> My reasoning:
>
>  * We are anyway already always referring to "Apache Pulsar"
>rather than just "Pulsar"
>
>  * Changing name at this point in life of the project would be
>very disruptive. We just spent the past year into building
>some naming awareness and it would all vanish, not to mention
>the huge amount of work in updating documentation, package
>names, etc.
>
>  * If a conflict arise later on, it might be easier to change
>name at that point. Once the project has a larger community
>and bigger recognition, changing name would not mean to
>restart from scratch.
>
>
> I propose to have anyone share they thoughs on this issue. Once
> there is a prevalent inclination in the community, we can vote a
> final resolution on the subject.
>
> Thanks,
> Matteo
>
>
>
> --
> Matteo Merli
> 
>

Re: Pulsar meetup in June/July

[Sijie Guo]

+1 go pulsar!

- Sijie

On Thu, Jun 7, 2018 at 4:49 PM Sanjeev Kulkarni  wrote:

> July 10th works for me as well!
>
> On Tue, Jun 5, 2018 at 11:21 AM Joe F  wrote:
>
> > All,
> >
> > I got the availability for the venue in the 2nd week of July  (50
> people),
> > and its available on  July 9th, 10th, or 13th .  How does that sound?  I
> > need to reserve a firm date...
> >
> > My preference is for 10th  [since its not a Monday or Friday :-) ] but
> I'm
> > fine with a whatever is convenient for all ..
> >
> >
> > Joe
> >
> > On Sun, Jun 3, 2018 at 7:21 PM, Sijie Guo  wrote:
> >
> > > Any updates?
> > >
> > > - Sijie
> > >
> > > On Mon, Apr 30, 2018 at 7:29 PM, Joe F  wrote:
> > >
> > > > I would expect people to volunteer and sign up... that's why I said
> > > "from
> > > > Pulsar users and developers".
> > > >
> > > >
> > > > Joe
> > > >
> > > > On Mon, Apr 30, 2018 at 6:44 PM, Dave Fisher 
> > > > wrote:
> > > >
> > > > > Hi -
> > > > >
> > > > > What would be the agenda?
> > > > >
> > > > > How can this event be promoted to help grow the community?
> > > > >
> > > > > Regards,
> > > > > Dave
> > > > >
> > > > > Sent from my iPhone
> > > > >
> > > > > > On Apr 30, 2018, at 11:29 AM, Joe F 
> wrote:
> > > > > >
> > > > > > In June, it going to be a year since Pulsar was accepted into the
> > > > > incubator.
> > > > > >
> > > > > > I'm thinking of a meetup, with a few in-depth
> talks/presentations,
> > > > what's
> > > > > > new in Pulsar 2.0 etc,   from Pulsar users and developers.  Are
> > > people
> > > > > > interested in a 1/2day event?   I can arrange for the venue and
> > > > > logistics.
> > > > > >
> > > > > >
> > > > > > Joe
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DRAFT] Pulsar Podling report June 2018

[Sijie Guo]

+1 this is a great milestone. exciting to see we are nearing graduation!

- Sijie

On Wed, Jun 6, 2018 at 8:57 AM, Matteo Merli  wrote:

> Here is the draft for the podling report. Please submit feedback soon, the
> deadline is today (sorry for sending draft at last moment).
>
>
> 
>
> Pulsar is a highly scalable, low latency messaging platform running on
> commodity hardware. It provides simple pub-sub semantics over topics,
> guaranteed at-least-once delivery of messages, automatic cursor management
> for
> subscribers, and cross-datacenter replication.
>
> Pulsar has been incubating since 2017-06-01.
>
> Most important issues to address in the move towards graduation:
>
>   1. Complete the Podling name search tasks. The task is in progress right
> now.
>
> Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be aware
> of?
>
>   None
>
> How has the community developed since the last report?
>
>   The community added 7 new contributors that submitted pull-requests which
>   were merged into master.
>
>   The number of users approaching the team on the Slack channel has kept
>   steadily increasing since the last report. Many users have actively
> deployed
>   Pulsar for evaluation and production use cases.
>
>   Project members from several companies have organized or participated in
>   several meetups, presenting Pulsar's introductions, deep-dives and
> hands-on
>   tutorial, including recorded podcasts. We have several scheduled talks on
>   Pulsar at various conferences, 2 at ApacheCon in September, one at OSCon
> in
>   July and 2 others at Strata New York in September. A Pulsar dedicated
> meetup
>   is being organized for next July.
>
>   Since the last report the number of weekly-active-users on the Slack
> channel
>   has increased from 53 to 88.
>
>   We have reached the 1 year mark since Pulsar entering the Apache
> Incubator.
>   Here is a summary of the community developments over the past year:
>
>
>   1. Pulsar community has done 5 Apache releases since entering
>  incubator. The release process is well documented and we have
>  had 4 different release managers from 3 different companies.
>
>   2. We have added 3 committers and PPMC members since incubation and
>  there are also other candidates who have already made significant
>  contributions to the project.
>
>   3. Community of users and people interested in Pulsar has expanded
>  considerably. Thanks to the months long work in improving ease of
>  use, documentation and blogs, many people became aware of Pulsar
>  and started playing with it, then evaluating it and finally
>  putting it in production for critical use cases.
>
>   4. We have tried to help users getting started through any
>  communication channel. Even though we keep trying to encourage
>  people to use the mailing list, most of the first interactions
>  have been happening through the Slack channel. We also did make
>  sure that:
>
>  a) No decisions are taken in Slack channel
>
>  b) Developers technical discussion happen mostly in Github
> issue/Pull-Request or in developers mailing list
>
>  c) Conversations in Slack are sent to dev/user mailing list in a
> daily digest form for archival and to be searchable
>
>  In any case Slack has been working fairly well in engaging with
>  users, by providing a tool to have very quick informal
>  question/answer interactions that were very appreciated by users.
>
>5. Overall, there were a lot of healthy discussions, with feedback
>   and collaborations from people from different companies and
>   different perspectives that resulted in much stronger design
>   decisions and ultimately a better system.
>
>6. We have taken several steps to increase awareness, like blog
>   posts, meetups (both dedicated to Pulsar or dedicated to similar
>   topics) and presentations to conferences, like Strata or
>   ApacheCon (where we have 2 talks scheduled for next September).
>
> How has the project developed since the last report?
>
>   23 authors have pushed 469 commits to master in the last 3 months.
>
>   The project has made the its fifth release since joining the
>   Apache Incubator (2.0.0-rc1-incubating on May 29th). This was a
>   major release that culminated several months of works and lays the
>   foundation for the next stage in Pulsar development. New major
>   features include:
>* Pulsar Functions (Lightweight compute framework)
>* Schema registry
>* Topic compaction
>
>   Community is actively working on next milestone, 2.1 release that
>   will include several new features including:
>* Pulsar IO connector framework
>* Tiered storage
>* Go client library
>
>   Since March, 3 new PIPs (Pulsar Improvement Proposals) for major
>   feature/changes, have been submitted to the wiki and discussed in the
>   mailing list.
>
>   To recap the project developments s

Re: [VOTE] Pulsar Release 2.0.1-incubating Candidate 1

[Sijie Guo]

+1 (binding)

Environment: MacOS 10.13.1

* check signatures, all are good.
* Bin distribution
- start standalone
- test producer/consumer using pulsar-client and pulsar-perf
* Src distribution
- compile and unit tests
- RAT check
- start standalone
- test producer/consumer using pulsar-client and pulsar-perf

- Sijie

On Tue, Jun 5, 2018 at 2:36 PM, Dave Fisher  wrote:

> +1 (binding)
>
> Signatures checked - asc
> Checksums checked - sha1 and sha512
> DISCLAIMER, NOTICE, and LICENSE pass
> README includes encryption notice.
> Source mvn apache-rat:check passes
> Source mvn install -DskipTests succeeds
> bin/pulsar standalone succeeds
>
> Great job!
>
> Regards,
> Dave
>
> > On Jun 4, 2018, at 10:43 PM, Matteo Merli  wrote:
> >
> > This is the first release candidate for Apache Pulsar, version
> > 2.0.1-incubating.
> >
> > This is a patch release that fixes few issues identified in previous
> > release 2.0.0-rc1-incubating.
> >
> > It fixes the following issues:
> > https://github.com/apache/incubator-pulsar/milestone/14?closed=1
> >
> > *** Please download, test and vote on this release. This vote will stay
> > open
> > for at least 72 hours ***
> >
> > Note that we are voting upon the source (tag), binaries are provided for
> > convenience.
> >
> > Source and binary files:
> > https://dist.apache.org/repos/dist/dev/incubator/pulsar/
> pulsar-2.0.1-incubating-candidate-1/
> >
> > SHA-1 checksums:
> >
> > ae1bd6f658ebff0745d5eb82961f400f9872f1da
> > apache-pulsar-2.0.1-incubating-bin.tar.gz
> > 2a22e40f290329ae5fac1d33e9754a67e01e882f
> > apache-pulsar-2.0.1-incubating-src.tar.gz
> >
> > Maven staging repo:
> > https://repository.apache.org/content/repositories/orgapachepulsar-1019
> >
> > The tag to be voted upon:
> > v2.0.1-incubating-candidate-1 (b18a2689d592ac676ab70a5447cc023e378eda3b)
> > https://github.com/apache/incubator-pulsar/releases/tag/
> v2.0.1-incubating-candidate-1
> >
> > Pulsar's KEYS file containing PGP keys we use to sign the release:
> > https://dist.apache.org/repos/dist/release/incubator/pulsar/KEYS
> >
> > Please download the the source package, and follow the README to build
> > and run the Pulsar standalone service.
> > --
> > Matteo Merli
> > 
>
>

[ANNOUNCE] Jerry Peng and Sanjeev Kulkarni as Apache Pulsar committers and PPMC members

[Sijie Guo]

Greetings,

The Apache Pulsar PPMC has voted to accept Jerry Peng (@jerrypeng) and
Sanjeev Kulkarni (@srkukarni) as a Pulsar committer and PPMC member.

Jerry and Sanjeev co-developed and contributed Pulsar Functions. They have
dedicated themselves to pulsar community, reviewing PRs, joining
discussions and continuing their awesome contributions.

Welcome Jerry and Sanjeev!

- Sijie

Re: Weirdness in jenkins build (tests not isolated as expected)

[Sijie Guo]

Gotcha!

- Sijie

On Mon, Jun 4, 2018 at 11:50 AM, Ivan Kelly  wrote:

> >> Two concurrent builds on the same machine will have different
> >> workspaces (@2 or @3 will be appended to $WORKSPACE).
> >
> > I am not sure about that. If taking a look at the jenkins job output, the
> > workspace is "
> > /home/jenkins/jenkins-slave/workspace/pulsar_precommit_integrationtests"
>
> It only adds the @ for the second concurrent build onwards.
>
> https://github.com/jenkinsci/jenkins/blob/master/core/src/
> main/java/hudson/slaves/WorkspaceList.java#L181
>
> -Ivan
>

Planning for release 2.1

[Sijie Guo]

Hi all,

It has been one month since we cutting 2.0 release. We have quite bunch of
changes completed in the past month. These changes include:

- Pulsar IO connector framework using pulsar functions
- CPP Client catch up tasks
- Tiered storage
- Functions integration with schema registry


I am starting this email thread discussing the plan for release 2.1, such
as what are other items need to be included in 2.1 and who would like to
drive the 2.1 release.

Thoughts?

- Sijie