Re: [VOTE] Graduation resolution proposal

[Rajan Dhabalia]

+1.

Thanks,
Rajan

On Fri, Sep 7, 2018 at 5:21 PM, Matteo Merli  wrote:

> Please vote on the proposal for Pulsar graduation to TLP to submit to
> the Incubator PMC.
>
> This vote will stay open for at least 72 hours.
>
>
> 
>
> Establish the Apache Pulsar Project
>
> WHEREAS, the Board of Directors deems it to be in the best interests of
> the Foundation and consistent with the Foundation's purpose to establish
> a Project Management Committee charged with the creation and maintenance
> of open-source software, for distribution at no charge to the public,
> related to a highly scalable, low latency messaging platform running on
> commodity hardware. It provides simple pub-sub and queue semantics over
> topics, lightweight compute framework, automatic cursor management for
> subscribers, and cross-datacenter replication.
>
> NOW, THEREFORE, BE IT RESOLVED, that a Project Management Committee
> (PMC), to be known as the "Apache Pulsar Project", be and hereby is
> established pursuant to Bylaws of the Foundation; and be it further
>
> RESOLVED, that the Apache Pulsar Project be and hereby is responsible
> for the creation and maintenance of software related to a highly
> scalable, low latency messaging platform running on commodity hardware.
> It provides simple pub-sub and queue semantics over topics, lightweight
> compute framework, automatic cursor management for subscribers, and
> cross-datacenter replication; and be it further
>
> RESOLVED, that the office of "Vice President, Apache Pulsar" be and
> hereby is created, the person holding such office to serve at the
> direction of the Board of Directors as the chair of the Apache Pulsar
> Project, and to have primary responsibility for management of the
> projects within the scope of responsibility of the Apache Pulsar
> Project; and be it further
>
> RESOLVED, that the persons listed immediately below be and hereby are
> appointed to serve as the initial members of the Apache Pulsar Project:
>
>  * Boyang Jerry Peng   
>  * Brad McMillen   
>  * David Fisher
>  * Francis Christopher Liu 
>  * Hiroyuki Sakai  
>  * Ivan Brendan Kelly  
>  * Jai Asher   
>  * Jia Zhai
>  * Jim Jagielski   
>  * Joe Francis 
>  * Ludwig Pummer   
>  * Masahiro Sakamoto   
>  * Masakazu Kitajo 
>  * Matteo Merli
>  * Nozomi Kurihara 
>  * P. Taylor Goetz 
>  * Rajan Dhabalia  
>  * Sahaya Andrews  
>  * Sanjeev Kulkarni
>  * Sebastián Schepens  
>  * Siddharth Boobna
>  * Sijie Guo   
>  * Yuki Shiga  
>
> NOW, THEREFORE, BE IT FURTHER RESOLVED, that Matteo Merli be appointed
> to the office of Vice President, Apache Pulsar, to serve in accordance
> with and subject to the direction of the Board of Directors and the
> Bylaws of the Foundation until death, resignation, retirement, removal
> or disqualification, or until a successor is appointed; and be it
> further
>
> RESOLVED, that the Apache Pulsar Project be and hereby is tasked with
> the migration and rationalization of the Apache Incubator Pulsar
> podling; and be it further
>
> RESOLVED, that all responsibilities pertaining to the Apache Incubator
> Pulsar podling encumbered upon the Apache Incubator PMC are hereafter
> discharged.
>
> 
>
>
>
> --
> Matteo Merli
> 
>

Re: [DRAFT] Podling report

[Rajan Dhabalia]

+1.

Thanks,
Rajan

Sent from my iPhone

> On Sep 5, 2018, at 5:36 PM, Matteo Merli  wrote:
> 
> Sorry for sending out at very last minute (deadline is today). Please take
> a look.
> 
> I have added a paragraph regarding the branding discussion that happen in
> early June (as discussed with Taylor then) since the previous draft was
> already out at that point.
> 
> 
> 
> ---
> 
> Pulsar
> Pulsar is a highly scalable, low latency messaging platform running on
> commodity hardware. It provides simple pub-sub semantics over topics,
> guaranteed at-least-once delivery of messages, automatic cursor management
> for
> subscribers, and cross-datacenter replication.
> 
> Pulsar has been incubating since 2017-06-01.
> 
> Most important issues to address in the move towards graduation:
> 
>  None
> 
> Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
> aware of?
> 
>  Earlier in June there have been few discussions on the private list
>  regarding communications regarding Pulsar that were not coming from
>  PPMC or that were not respecting the ASF policies.  Clarifications
>  followed between PPMC members, mentors and interested parties to
>  ensure the mistakes were made in good faith and, in particular, to
>  make sure everyone was fully has full understanding of ASF
>  policies. There was no other branding related issue after the first
>  occourence.
> 
> How has the community developed since the last report?
> 
>  The community added 8 new contributors that submitted pull-requests
>  which were merged into master.
> 
>  The number of users approaching the team on the Slack channel has
>  kept steadily increasing since the last report. Many users have
>  actively deployed. Pulsar for evaluation and production use cases.
> 
>  Different meetups were organized by project members and hosted by
>  Yahoo in Sunnyvale and Yahoo Japan in Tokyo. We have presented
>  Pulsar's introductions, updates on the state of the projects,
>  deep-dives and hands-on tutorial, including recorded podcasts.
> 
>  One talk on Pulsar was presented at one at OSCon in July and there
>  are several scheduled talks: 2 at ApacheCon in September, and 2
>  others at Strata New York in September.
> 
>  Since the last report the number of weekly-active-users on the Slack
>  channel has increased from 53 to 88.
> 
> 
> How has the project developed since the last report?
> 
>  28 authors have pushed 494 commits to master in the last 3 months.
> 
>  The project has made the its seventh release since joining the
>  Apache Incubator (2.1.0-incubating on Aug 2nd).
> 
>  This release introduced these new features:
> 
>   * Pulsar IO: A connector framework for moving data in and out of
> Apache Pulsar leveraging Pulsar Functions runtime.
>   * A number of builtin connectors: (Aerospike, Cassandra, Kafka,
> Kinesis, RabbitMQ, Twitter)
>   * Tiered Storage: An extension in Pulsar segment store to offload
> older segments into long term storage (e.g. HDFS, S3). S3 support
> is supported in 2.1 release.
>   * Stateful function: Pulsar Functions is able to use State API for
> storing state within Pulsar.
>   * Pulsar Go Client
>   * Avro and Protobuf Schema support
> 
>  Community is actively working on a bug-fix release
>  (2.1.1-incubating) and on the next milestone, 2.2 release for which
>  the biggest feature will be support for SQL within Pulsar.
> 
>  Since June, 5 new PIPs (Pulsar Improvement Proposals) for
>  major feature/changes, have been submitted to the wiki and
>  discussed in the mailing list.
> 
>PIP 23: Message Tracing By Interceptors
>PIP 22: Pulsar Dead Letter Topic
>PIP 21: Pulsar Edge Component
>PIP 20: Mechanism to revoke TLS authentication
>PIP 19: Pulsar SQL
> 
> 
> How would you assess the podling's maturity?
> Please feel free to add your own commentary.
> 
>  [ ] Initial setup
>  [ ] Working towards first release
>  [ ] Community building
>  [X] Nearing graduation
>  [ ] Other:
> 
> Date of last release:
>  2018-08-02, 2.1.0-incubating
> 
> When were the last committers or PPMC members elected?
> 
>  2018-06-11 - Ivan Kelly
>  2018-06-11 - Jia Zhai
> 
> 
> -- 
> Matteo Merli
> 

Re: [DRAFT] Graduation resolution proposal

[Rajan Dhabalia]

+1, would like to be part of PMC.

Thanks,
Rajan

On Fri, Aug 31, 2018 at 9:43 AM, Matteo Merli  wrote:

> I have prepared the draft for the graduation resolution using the
> Whimsy tool (https://whimsy.apache.org/roster/ppmc/pulsar).
>
> Maturity model self-assessment is at
> https://github.com/apache/incubator-pulsar/wiki/Apache-
> Maturity-Model-Assessment-for-Pulsar
> Contributing to Pulsar page is now available at
> http://pulsar.incubator.apache.org/contributing/
>
> Notes:
>
>  * I'm proposing myself as initial PMC chair -- Please comment if
> community is onboard with this or propose other persons as well
>
>  * This draft includes all existing PPMC members and mentors into the new
> PMC.
>
>  - For all: please indicate if you want to keep being part of the
> PMC or if you prefer to be removed.
>
>
> 
>
> Establish the Apache Pulsar Project
>
> WHEREAS, the Board of Directors deems it to be in the best interests of
> the Foundation and consistent with the Foundation's purpose to establish
> a Project Management Committee charged with the creation and maintenance
> of open-source software, for distribution at no charge to the public,
> related to a highly scalable, low latency messaging platform running on
> commodity hardware. It provides simple pub-sub and queue semantics over
> topics, lightweight compute framework, automatic cursor management for
> subscribers, and cross-datacenter replication.
>
> NOW, THEREFORE, BE IT RESOLVED, that a Project Management Committee
> (PMC), to be known as the "Apache Pulsar Project", be and hereby is
> established pursuant to Bylaws of the Foundation; and be it further
>
> RESOLVED, that the Apache Pulsar Project be and hereby is responsible
> for the creation and maintenance of software related to a highly
> scalable, low latency messaging platform running on commodity hardware.
> It provides simple pub-sub and queue semantics over topics, lightweight
> compute framework, automatic cursor management for subscribers, and
> cross-datacenter replication; and be it further
>
> RESOLVED, that the office of "Vice President, Apache Pulsar" be and
> hereby is created, the person holding such office to serve at the
> direction of the Board of Directors as the chair of the Apache Pulsar
> Project, and to have primary responsibility for management of the
> projects within the scope of responsibility of the Apache Pulsar
> Project; and be it further
>
> RESOLVED, that the persons listed immediately below be and hereby are
> appointed to serve as the initial members of the Apache Pulsar Project:
>
>  * Boyang Jerry Peng   
>  * Brad McMillen   
>  * David Fisher
>  * Francis Christopher Liu 
>  * Hiroyuki Sakai  
>  * Ivan Brendan Kelly  
>  * Jai Asher   
>  * Jia Zhai
>  * Jim Jagielski   
>  * Joe Francis 
>  * Ludwig Pummer   
>  * Masahiro Sakamoto   
>  * Masakazu Kitajo 
>  * Matteo Merli
>  * Nozomi Kurihara 
>  * P. Taylor Goetz 
>  * Rajan Dhabalia  
>  * Sahaya Andrews  
>  * Sanjeev Kulkarni
>  * Sebastián Schepens  
>  * Siddharth Boobna
>  * Sijie Guo   
>  * Yuki Shiga  
>
> NOW, THEREFORE, BE IT FURTHER RESOLVED, that Matteo Merli be appointed
> to the office of Vice President, Apache Pulsar, to serve in accordance
> with and subject to the direction of the Board of Directors and the
> Bylaws of the Foundation until death, resignation, retirement, removal
> or disqualification, or until a successor is appointed; and be it
> further
>
> RESOLVED, that the initial Apache Pulsar PMC be and hereby is tasked
> with the creation of a set of bylaws intended to encourage open
> development and increased participation in the Apache Pulsar Project;
> and be it further
>
> RESOLVED, that the Apache Pulsar Project be and hereby is tasked with
> the migration and rationalization of the Apache Incubator Pulsar
> podling; and be it further
>
> RESOLVED, that all responsibilities pertaining to the Apache Incubator
> Pulsar podling encumbered upon the Apache Incubator PMC are hereafter
> discharged.
>
> 
>
>
> --
> Matteo Merli
> 
> --
> Matteo Merli
> 
>

Re: PIP 20: Mechanism to revoke TLS authentication

[Rajan Dhabalia]

> I will consider both this and system topics and update the PIP.

I think we should avoid more complexity by introducing system topic to
store "subject key" and then process it. Instead we can store it to
global/config zk as it requires to store keys with few bytes. I think
dynamic-configuration might be enough for this configuration.

Thanks,
Rajan



On Mon, Aug 20, 2018 at 6:01 AM, Ivan Kelly  wrote:

> > Here, we would like to configure "subject-key-identifier" at every broker
> in the cluster dynamically. We also want to perform certain actions once
> this configuration-value has been changed.
>
> Ah, i didn't know this existed. Very useful. I will consider both this
> and system topics and update the PIP.
>
>
> -Ivan
>

Re: PIP 20: Mechanism to revoke TLS authentication

[Rajan Dhabalia]

Hi,

Here, we would like to configure "subject-key-identifier" at every broker
in the cluster dynamically. We also want to perform certain actions once
this configuration-value has been changed.

Broker's dynamic configuration is introduced for such usecases where
dynamic-config value gets stored into zk and change will be listened by all
brokers so, it can be applied to entire cluster immediately.

So, I think we should add dynamic configuration

"*tlsSubjectKeyIdentifier*" at "ServiceConfiguration" and we can also register
appropriate listener
to
perform subsequent action with newly changed value of the configuration.
So, I would not add one more REST end-point and it should be part of
dynamic configuration.

You can see "dispatchThrottlingRatePerSubscriptionInMsg
"
example for dynamic-configuration.

Thanks,
Rajan




On Tue, Aug 7, 2018 at 1:12 AM, Ivan Kelly  wrote:

> Hi folks,
>
> This is a PIP to add a mechanism to block TLS client certs from
> accessing Pulsar if they have been compromised.
>
> This is a relatively small change, but I thought it best to put it to
> the community before moving ahead with it, as people may have opinions
> on the approach.
>
> The PIP is here:
> https://github.com/apache/incubator-pulsar/wiki/PIP-20%
> 3A-Mechanism-to-revoke-TLS-authentication
>
> Cheers,
> Ivan
>

Re: [VOTE] Maintaining "Apache Pulsar" name

[Rajan Dhabalia]

+1 for "Apache Pulsar"

Thanks,
Rajan

On Wed, Jun 20, 2018 at 5:06 PM, Matteo Merli  wrote:

> Following the previous discussion at
> https://lists.apache.org/thread.html/fe60c21b7cdca00918fe25e7ddf977
> 2677342063b9da24c9b84b3329@%3Cdev.pulsar.apache.org%3E
> I am calling a formal vote for keep using "Apache Pulsar" name for the
> project.
>
> This vote will be open for at least 72 hours. After that we will
> communicate the decision to trademarks@apache
>
> +1 from me
>
> Matteo
> --
> Matteo Merli
> 
>

Re: pulsar-master build failures

[Rajan Dhabalia]

Hi Prono,

Thanks for letting us know. We had an issue earlier with one of our test:
PulsarSinkE2ETes which was hanging and not finishing up due to one of the
bug. Later we had fixed it and added the timeout to the test so, it should
not generating big logs anymore. We have fixed this issue on 06/09 with PR:
https://github.com/apache/incubator-pulsar/pull/1945
So, can you please confirm now, if we are still seeing log issue in the
pipeline?

Thanks,
Rajana

On Wed, Jun 13, 2018 at 11:24 AM, Daniel Pono Takamori 
wrote:

> Recently H34 ran out of disk and while looking for culprits I found 2
> log files which seem to have been the same: 52G
> /home/jenkins/jenkins-slave/workspace/pulsar-master/
> pulsar-broker/target/surefire-reports/org.apache.pulsar.io.
> PulsarSinkE2ETest-output.txt
> 52G /tmp/stdout1630697337623691382deferred
>
> I believe something went wrong which caused build 520 [0] to have
> taken so long as well as use 100gb of disk (most build nodes have
> ~500gb). Job 522 [1] also hung after about 7 hours running (and
> continued running for 21 hours total), so I had to kill it this
> morning.
>
> While I went ahead and nuked the log file, there is reason to think
> that your build should be reviewed to see if there are any oddities
> performed by the job, or evidence of our build nodes misbehaving (this
> happens from time to time).
>
> Thanks for taking a look!
> -Pono on behalf of Infra
>
> [0] - https://builds.apache.org/job/pulsar-master/520/
> [1] - https://builds.apache.org/job/pulsar-master/522/
>

Re: [DRAFT] Pulsar Podling report June 2018

[Rajan Dhabalia]

+1. Glad to see, Pulsar adaption and user growth is increasing and users
outside Yahoo! are using with positive experience. and yes, one more
milestone, we are nearing graduation.

Thanks,
Rajan


On Wed, Jun 6, 2018 at 8:57 AM, Matteo Merli  wrote:

> Here is the draft for the podling report. Please submit feedback soon, the
> deadline is today (sorry for sending draft at last moment).
>
>
> 
>
> Pulsar is a highly scalable, low latency messaging platform running on
> commodity hardware. It provides simple pub-sub semantics over topics,
> guaranteed at-least-once delivery of messages, automatic cursor management
> for
> subscribers, and cross-datacenter replication.
>
> Pulsar has been incubating since 2017-06-01.
>
> Most important issues to address in the move towards graduation:
>
>   1. Complete the Podling name search tasks. The task is in progress right
> now.
>
> Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be aware
> of?
>
>   None
>
> How has the community developed since the last report?
>
>   The community added 7 new contributors that submitted pull-requests which
>   were merged into master.
>
>   The number of users approaching the team on the Slack channel has kept
>   steadily increasing since the last report. Many users have actively
> deployed
>   Pulsar for evaluation and production use cases.
>
>   Project members from several companies have organized or participated in
>   several meetups, presenting Pulsar's introductions, deep-dives and
> hands-on
>   tutorial, including recorded podcasts. We have several scheduled talks on
>   Pulsar at various conferences, 2 at ApacheCon in September, one at OSCon
> in
>   July and 2 others at Strata New York in September. A Pulsar dedicated
> meetup
>   is being organized for next July.
>
>   Since the last report the number of weekly-active-users on the Slack
> channel
>   has increased from 53 to 88.
>
>   We have reached the 1 year mark since Pulsar entering the Apache
> Incubator.
>   Here is a summary of the community developments over the past year:
>
>
>   1. Pulsar community has done 5 Apache releases since entering
>  incubator. The release process is well documented and we have
>  had 4 different release managers from 3 different companies.
>
>   2. We have added 3 committers and PPMC members since incubation and
>  there are also other candidates who have already made significant
>  contributions to the project.
>
>   3. Community of users and people interested in Pulsar has expanded
>  considerably. Thanks to the months long work in improving ease of
>  use, documentation and blogs, many people became aware of Pulsar
>  and started playing with it, then evaluating it and finally
>  putting it in production for critical use cases.
>
>   4. We have tried to help users getting started through any
>  communication channel. Even though we keep trying to encourage
>  people to use the mailing list, most of the first interactions
>  have been happening through the Slack channel. We also did make
>  sure that:
>
>  a) No decisions are taken in Slack channel
>
>  b) Developers technical discussion happen mostly in Github
> issue/Pull-Request or in developers mailing list
>
>  c) Conversations in Slack are sent to dev/user mailing list in a
> daily digest form for archival and to be searchable
>
>  In any case Slack has been working fairly well in engaging with
>  users, by providing a tool to have very quick informal
>  question/answer interactions that were very appreciated by users.
>
>5. Overall, there were a lot of healthy discussions, with feedback
>   and collaborations from people from different companies and
>   different perspectives that resulted in much stronger design
>   decisions and ultimately a better system.
>
>6. We have taken several steps to increase awareness, like blog
>   posts, meetups (both dedicated to Pulsar or dedicated to similar
>   topics) and presentations to conferences, like Strata or
>   ApacheCon (where we have 2 talks scheduled for next September).
>
> How has the project developed since the last report?
>
>   23 authors have pushed 469 commits to master in the last 3 months.
>
>   The project has made the its fifth release since joining the
>   Apache Incubator (2.0.0-rc1-incubating on May 29th). This was a
>   major release that culminated several months of works and lays the
>   foundation for the next stage in Pulsar development. New major
>   features include:
>* Pulsar Functions (Lightweight compute framework)
>* Schema registry
>* Topic compaction
>
>   Community is actively working on next milestone, 2.1 release that
>   will include several new features including:
>* Pulsar IO connector framework
>* Tiered storage
>* Go client library
>
>   Since March, 3 new PIPs (Pulsar Improvement Proposals) for major
>   feature/changes

Re: [VOTE] Pulsar Release 2.0.1-incubating Candidate 1

[Rajan Dhabalia]

+1 (binding)

Environment: Mac-OS 10.12.6

Checked:
 - signatures and checksum
 - maven repository artifacts
 - start standalone service and client-test for producer/consumer (src/bin
distribution)
 - rat check on src distribution
 - compile and unit test on src distribution

Environment: REHL-6.7

Build fails on REHL-6.x with error:
"[ERROR]
/home/rdhabalia/pulsar/apache-pulsar-2.0.1-incubating/pulsar-functions/proto/src/main/proto/Request.proto
[0:0]:
/home/rdhabalia/pulsar/apache-pulsar-2.0.1-incubating/pulsar-functions/proto/target/protoc-plugins/protoc-3.5.1-linux-x86_64.exe:
/lib64/libc.so.6: version `GLIBC_2.14' not found (required by
/home/rdhabalia/pulsar/apache-pulsar-2.0.1-incubating/pulsar-functions/proto/target/protoc-plugins/protoc-3.5.1-linux-x86_64.exe)"

We had this issue in 2.0.0 as well  but now, we have  fixed this issue in
2.1.0. Build works fine with REHL7 so, we can ignore it for this release.

Thanks,
Rajan


On Mon, Jun 4, 2018 at 10:43 PM, Matteo Merli  wrote:

> This is the first release candidate for Apache Pulsar, version
> 2.0.1-incubating.
>
> This is a patch release that fixes few issues identified in previous
> release 2.0.0-rc1-incubating.
>
> It fixes the following issues:
> https://github.com/apache/incubator-pulsar/milestone/14?closed=1
>
> *** Please download, test and vote on this release. This vote will stay
> open
> for at least 72 hours ***
>
> Note that we are voting upon the source (tag), binaries are provided for
> convenience.
>
> Source and binary files:
> https://dist.apache.org/repos/dist/dev/incubator/pulsar/
> pulsar-2.0.1-incubating-candidate-1/
>
> SHA-1 checksums:
>
> ae1bd6f658ebff0745d5eb82961f400f9872f1da
> apache-pulsar-2.0.1-incubating-bin.tar.gz
> 2a22e40f290329ae5fac1d33e9754a67e01e882f
> apache-pulsar-2.0.1-incubating-src.tar.gz
>
> Maven staging repo:
> https://repository.apache.org/content/repositories/orgapachepulsar-1019
>
> The tag to be voted upon:
> v2.0.1-incubating-candidate-1 (b18a2689d592ac676ab70a5447cc023e378eda3b)
> https://github.com/apache/incubator-pulsar/releases/tag/
> v2.0.1-incubating-candidate-1
>
> Pulsar's KEYS file containing PGP keys we use to sign the release:
> https://dist.apache.org/repos/dist/release/incubator/pulsar/KEYS
>
> Please download the the source package, and follow the README to build
> and run the Pulsar standalone service.
> --
> Matteo Merli
> 
>

Re: [VOTE] Pulsar Release 2.0.0-rc1-incubating Candidate 4

[Rajan Dhabalia]

+1 (binding)

Environment: Mac-OS 10.12.6

Checked:
 - signatures and checksum
 - maven repository artifacts
 - start standalone service and client-test for producer/consumer (src/bin
distribution)
 - rat check on src distribution
 - compile and unit test on src distribution

Thanks,
Rajan

On Mon, May 14, 2018 at 4:33 PM, Matteo Merli  wrote:

> This is the fourth release candidate for Apache Pulsar, version
> 2.0.0-rc1-incubating.
>
> It fixes the following issues:
> https://github.com/apache/incubator-pulsar/milestone/12?closed=1
>
> *** Please download, test and vote on this release. This vote will stay
> open
> for at least 72 hours ***
>
> Note that we are voting upon the source (tag), binaries are provided for
> convenience.
>
> Source and binary files:
> https://dist.apache.org/repos/dist/dev/incubator/pulsar/
> pulsar-2.0.0-rc1-incubating-candidate-4/
>
> SHA-1 checksums:
> a959a33500508d0ba690ca177eb39cf83dddf022
> apache-pulsar-2.0.0-rc1-incubating-bin.tar.gz
> 5b9bedd2f284c5a8d95ba6dc3c70af8c266fbb3b
> apache-pulsar-2.0.0-rc1-incubating-src.tar.gz
>
> Maven staging repo:
> https://repository.apache.org/content/repositories/orgapachepulsar-1014/
>
> The tag to be voted upon:
> v2.0.0-rc1-incubating-candidate-4 (e5d3bda9607703c2e012983c7ef64f
> 5eb4118de7)
> https://github.com/apache/incubator-pulsar/releases/tag/
> v2.0.0-rc1-incubating-candidate-4
>
> Pulsar's KEYS file containing PGP keys we use to sign the release:
> https://dist.apache.org/repos/dist/release/incubator/pulsar/KEYS
>
> Please download the the source package, and follow the README to build
> and run the Pulsar standalone service.
>
> --
> Matteo Merli
> 
>

Re: [VOTE] Pulsar Release 2.0.0-rc1-incubating Candidate 3

[Rajan Dhabalia]

+1 (binding)

Environment: Mac-OS 10.12.6

Checked:
 - signatures and checksum
 - maven repository artifacts
 - start standalone service and client-test for producer/consumer (src/bin
distribution)
 - rat check on src distribution
 - compile and unit test on src distribution

Thanks,
Rajan

On Tue, May 8, 2018 at 3:22 PM, Matteo Merli  wrote:

> This is the third release candidate for Apache Pulsar, version
> 2.0.0-rc1-incubating.
>
> It fixes the following issues:
> https://github.com/apache/incubator-pulsar/milestone/12?closed=1
>
> *** Please download, test and vote on this release. This vote will stay
> open
> for at least 72 hours ***
>
> Note that we are voting upon the source (tag), binaries are provided for
> convenience.
>
> Source and binary files:
> https://dist.apache.org/repos/dist/dev/incubator/pulsar/
> pulsar-2.0.0-rc1-incubating-candidate-3/
>
> SHA-1 checksums:
> b1a392c5935448da195eb927520fe6deb20182fc
> apache-pulsar-2.0.0-rc1-incubating-bin.tar.gz
> 8c547a3e0afaee3446c770c076d5803e7e7da1c2
> apache-pulsar-2.0.0-rc1-incubating-src.tar.gz
>
> Maven staging repo:
> https://repository.apache.org/content/repositories/orgapachepulsar-1013/
>
> The tag to be voted upon:
> v2.0.0-rc1-incubating-candidate-3 (9b33a8788f171f98528aeaa9040d16
> fc702404ef)
> https://github.com/apache/incubator-pulsar/releases/tag/
> v2.0.0-rc1-incubating-candidate-3
>
> Pulsar's KEYS file containing PGP keys we use to sign the release:
> https://dist.apache.org/repos/dist/release/incubator/pulsar/KEYS
>
> Please download the the source package, and follow the README to build
> and run the Pulsar standalone service.
> --
> Matteo Merli
> 
>

Re: [VOTE] Pulsar 1.22.0-incubating Release Candidate 3

[Rajan Dhabalia]

+1 (binding)

Environment: Mac-OS 10.12.6

Checked:
 - signatures and checksum
 - start standalone service and producer/consumer client test (Src/Bin
distribution)
 - rat check on Src distribution
 - compile and unit test on src distribution
 - maven repository artifacts

Thanks,
Rajan

On Tue, Feb 20, 2018 at 8:38 PM, Jai Asher  wrote:

> This is the fourth release candidate for Apache Pulsar, version
> 1.22.0-incubating.
>
> It fixes the following issues:
> https://github.com/apache/incubator-pulsar/milestone/11?closed=1
>
> *** Please download, test and vote by Friday, Feb 23, 2018, 10:00 GMT.
>
> Note that we are voting upon the source (tag), binaries are provided for
> convenience.
>
> Source and binary files:
> https://dist.apache.org/repos/dist/dev/incubator/pulsar/
> pulsar-1.22.0-incubating-candidate-3/
>
> Maven staging repo:
> https://repository.apache.org/content/repositories/orgapachepulsar-1010/
>
> The tag to be voted upon:
> v1.22.0-incubating-candidate-3 (5d14788e510faec23fd8ed189ed343e93b489dda)
> https://github.com/apache/incubator-pulsar/releases/tag/
> v1.22.0-incubating-candidate-3
>
> Pulsar's KEYS file containing PGP keys we use to sign the release:
> https://dist.apache.org/repos/dist/release/incubator/pulsar/KEYS
>
> Please download the source package, and follow the README to build
> and run the Pulsar standalone service.
>

Re: [PROPOSAL] Planning for next releases

[Rajan Dhabalia]

Hi,

Yes, we can target 1.22.0-incubating release by reviewing and merging
pending PRs by end of this month and then we can prepare for 2.0 release.

>> Also, are there any volunteers for release managers for both releases?
Sure, I can be release manager for 1.22.0-incubating.

Thanks,
Rajan



On Fri, Jan 19, 2018 at 3:54 PM, Matteo Merli  wrote:

> Since it has already been a while since last release (at least since when
> we closed the branch for 1.21), I would like to start discussion and from
> my side propose the following:
>
> 1. Release 1.22.0-incubating at the end of this month
>
>We have already a considerable number of fixes and improvement over 1.21
> and we should release that, along with pending PRs. I would suggest anyone
> to mark current changes for either this or next release and help with
> reviewing them.
>
> 2. Prepare for 2.0 Pulsar release for end of February.
> We have been discussing a while back to take the opportunity to make
> some changes in API or tools that break binary compatibility, and do them
> all at once.
> I had created a project in Github to track the proposals:
> https://github.com/apache/incubator-pulsar/projects/4
> For this I would also propose to quickly create a branch for 1.22 so
> that the work that is targeting 2.0 can be merged into master before the
> release is done.
>
> Please anyone share your thoughts on this proposal.
>
> Also, are there any volunteers for release managers for both releases? I
> can also pick up one of them otherwise.
>
> Thanks,
> Matteo
>
>
>
> --
> Matteo Merli
> 
>

Re: [VOTE] Pulsar 1.21.0-incubating Release Candidate 3

[Rajan Dhabalia]

+1 (binding)

Environment: Mac-OS 10.12.6

Checked:
 - signatures and checksum
 - start standalone service and producer/consumer client test (Src/Bin
distribution)
 - rat check on Src distribution
 - compile and unit test on src distribution
 - maven repository artifacts

Thanks,
Rajan

On Mon, Dec 11, 2017 at 2:52 AM, Masahiro Sakamoto 
wrote:

> +1
>
> Environment: Linux (CentOS 6.7)
>
> Checked:
> - signatures and checksum
> - start standalone service and producer/consumer client test
> - rat check
> - compile and unit test
>
> --
> Masahiro Sakamoto
> Yahoo Japan Corp.
> E-mail: massa...@yahoo-corp.jp
> --
>
> > -Original Message-
> > From: Sahaya Andrews [mailto:andr...@apache.org]
> > Sent: Monday, December 11, 2017 5:11 PM
> > To: dev@pulsar.incubator.apache.org
> > Subject: Re: [VOTE] Pulsar 1.21.0-incubating Release Candidate 3
> >
> > +1 (binding)
> >
> > Environment: Mac 10.12.6
> >
> > - Verified the checksum and signature
> > - Ran perf producer/consumer test using standalone broker
> > - I was able to build the src repo and run tests
> >
> > I'm seeing test failure in RHEL 6.3, which appears to be related to my vm
> > not setup correctly. I'll update if I manage to get it working.
> >
> > Andrews.
> >
> > On Sun, Dec 10, 2017 at 10:42 PM, Hiroyuki Sakai 
> > wrote:
> > > +1
> > >
> > > Environment: Mac 10.12.6
> > >
> > > - Verified sha,md5 and signature
> > >
> > > * bin distribution
> > > - Start standalone service and producer/consumer test
> > >
> > > * src distribution
> > > - RAT check
> > > - Compile and unit tests
> > > - Start standalone service and producer/consumer test
> > >
> > > Hiroyuki Sakai
> > >
> > > on 2017/12/06 20:35, "Nozomi Kurihara"  wrote:
> > >
> > > This is the FOURTH release candidate for Apache Pulsar, version
> > 1.21.0-incubating.
> > >
> > > It fixes the following issues:
> > > https://github.com/apache/incubator-pulsar/milestone/10?closed=1
> > >
> > >
> > > *** Please download, test and vote by December 11th 2017, 12:00
> GMT.
> > >
> > > Note that we are voting upon the source (tag), binaries are
> provided
> > for convenience.
> > >
> > > Source and binary files:
> > >
> > >
> > https://dist.apache.org/repos/dist/dev/incubator/pulsar/pulsar-1.21.0-
> > > incubating-candidate-3/
> > >
> > > Maven staging repo:
> > >
> > >
> > https://repository.apache.org/content/repositories/orgapachepulsar-100
> > > 5/
> > >
> > > The tag to be voted upon:
> > > v1.21.0-incubating-candidate-3
> > >
> > >
> > >
> > https://github.com/apache/incubator-pulsar/releases/tag/v1.21.0-incuba
> > > ting-candidate-3
> > >
> > > Pulsar's KEYS file containing PGP keys we use to sign the release:
> > > https://dist.apache.org/repos/dist/release/incubator/pulsar/KEYS
> > >
> > > Please download the the source package, and follow the README to
> build
> > and run the Pulsar standalone service.
> > >
> > > Thanks,
> > > Nozomi
> > >
> > >
>

Re: [DRAFT] Podling report

[Rajan Dhabalia]

Looks good.

Thanks,
Rajan

On Wed, Dec 6, 2017 at 10:10 AM, Matteo Merli  wrote:

> Pulsar is a highly scalable, low latency messaging platform running on
> commodity hardware. It provides simple pub-sub semantics over topics,
> guaranteed at-least-once delivery of messages, automatic cursor management
> for
> subscribers, and cross-datacenter replication.
>
> Pulsar has been incubating since 2017-06-01.
>
> Three most important issues to address in the move towards graduation:
>
>   1. Grow the community with new Committers/PPMC members.
>   2. Set up a test cluster to be able to run system tests
>   3. Improve release process by having more committers taking part in
> releases
>
> Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
> aware of?
>
> None
>
> How has the community developed since the last report?
>
> The community added 4 contributors. There is a healthy discuss on issues
> related
> to development, tools and processes among the community members.
> There has been a good number of questions asked by users on the Slack
> channel
> where there are around 33 weekly active users.
>
> How has the project developed since the last report?
>
> 18 authors have pushed 150 commits to master in the last 3 months.
>
> The project has made the its second release in the Apache Incubator on
> October 13th.
> The next release (1.21-incubating) is being voted right now in the dev
> mailing list.
> Several design documents with improvement proposals have been submitted to
> the
> wiki and discussed in the mailing list.
>
> How would you assess the podling's maturity?
> Please feel free to add your own commentary.
>
>   [ ] Initial setup
>   [ ] Working towards first release
>   [X] Community building
>   [ ] Nearing graduation
>   [ ] Other:
>
> Date of last release:
>
>   2017-10-13, 1.20.0-incubating
>
> When were the last committers or PPMC members elected?
>
> 
>
> --
> Matteo Merli
> 
>

Re: [VOTE] Pulsar 1.21.0-incubating Release Candidate 2

[Rajan Dhabalia]

+1 (binding)

Environment: Mac-OS 10.12.6

Checked:
 - signatures and checksum
 - start standalone service and producer/consumer client test (Src/Bin
distribution)
 - rat check on Src distribution
 - compile and unit test on src distribution
 - maven repository artifacts

Thanks,
Rajan

On Wed, Nov 29, 2017 at 9:57 PM, Nozomi Kurihara 
wrote:

> This is the THIRD release candidate for Apache Pulsar, version
> 1.21.0-incubating.
>
> It fixes the following issues:
> https://github.com/apache/incubator-pulsar/milestone/10?closed=1
>
> *** Please download, test and vote by December 5th 2017, 10:00 GMT.
>
> Note that we are voting upon the source (tag), binaries are provided for
> convenience.
>
> Source and binary files:
> https://dist.apache.org/repos/dist/dev/incubator/pulsar/
> pulsar-1.21.0-incubating-candidate-2/
>
> Maven staging repo:
> https://repository.apache.org/content/repositories/orgapachepulsar-1004/
>
> The tag to be voted upon:
> v1.21.0-incubating-candidate-2 (be96bd38b1f97b1b0888565196d77419004abe88)
> https://github.com/apache/incubator-pulsar/releases/tag/
> v1.21.0-incubating-candidate-2
>
> Pulsar's KEYS file containing PGP keys we use to sign the release:
> https://dist.apache.org/repos/dist/release/incubator/pulsar/KEYS
>
> Please download the the source package, and follow the README to build and
> run the Pulsar standalone service.
>
> Thanks,
> Nozomi
>

Re: [DISCUSS][VOTE] Pulsar 1.21.0-incubating Release Candidate 0

[Rajan Dhabalia]

Yes, we should be good with all the merged changes.

Thanks,
Rajan

On Tue, Nov 28, 2017 at 10:49 PM, Matteo Merli  wrote:

> Hi Nozomi,
>
> from our side I think we have all changes merged.
>
> Rajan/Andrews can you confirm as well that nothing else is missing?
>
> Matteo
>
> On Wed, Nov 22, 2017 at 12:42 AM Nozomi Kurihara 
> wrote:
>
>> Hi Matteo,
>>
>>
>> > there's one last change I'd like to get in. There's a PR on
>> yahoo/bookkeeper repo. Will update in few hours whether we can make it
>> through or skip.
>>
>>
>> Thank you for your notice.
>>
>> Please let me know when it has done.
>>
>>
>> Thanks,
>>
>> Nozomi
>>
>> 
>> 差出人: Matteo Merli 
>> 送信日時: 2017年11月22日 1:56:27
>> 宛先: dev@pulsar.incubator.apache.org
>> 件名: Re: [DISCUSS][VOTE] Pulsar 1.21.0-incubating Release Candidate 0
>>
>> Hi Nozomi, there's one last change I'd like to get in. There's a PR on
>> yahoo/bookkeeper repo. Will update in few hours whether we can make it
>> through or skip.
>>
>> Matteo
>>
>> On Mon, Nov 20, 2017 at 7:45 PM Nozomi Kurihara 
>> wrote:
>>
>> > Hi all,
>> >
>> >
>> > #903 is merged, so I'm going to
>> >
>> > ・create new candidate
>> >
>> > ・re-tag 1.21.0-incubating issues/PRs which are still open to
>> > 1.22.0-incubating
>> >
>> >
>> > Please let me know by 11/22 17:00 GMT if there are more changes to be
>> > included.
>> >
>> > If there are no suggestions, I will open another vote.
>> >
>> >
>> > Thanks,
>> >
>> > Nozomi
>> >
>> > 
>> > 差出人: Dave Fisher 
>> > 送信日時: 2017年11月17日 5:00:12
>> > 宛先: dev@pulsar.incubator.apache.org
>> > 件名: Re: [DISCUSS][VOTE] Pulsar 1.21.0-incubating Release Candidate 0
>> >
>> > Hi Nozomi,
>> >
>> > Thanks for clarifying the situation.
>> >
>> > Regards,
>> > Dave
>> >
>> > > On Nov 16, 2017, at 11:46 AM, Nozomi Kurihara > >
>> > wrote:
>> > >
>> > > Hi, all
>> > >
>> > >
>> > > This vote has cancelled and after
>> > https://github.com/apache/incubator-pulsar/pull/903 is merged (around
>> > 17th), I will re-create new candidate and open another vote.
>> > >
>> > > I'm sorry to confuse you.
>> > >
>> > >
>> > > Please let me know if there are more changes to be included.
>> > >
>> > >
>> > > Thanks,
>> > >
>> > > Nozomi
>> > >
>> > > 
>> > > 差出人: Sahaya Andrews 
>> > > 送信日時: 2017年11月17日 3:29:03
>> > > 宛先: dev@pulsar.incubator.apache.org
>> > > 件名: Re: [DISCUSS][VOTE] Pulsar 1.21.0-incubating Release Candidate 0
>> > >
>> > > We would prefer to include
>> > > https://github.com/apache/incubator-pulsar/pull/903 in this release.
>> > > We should be done with this pull by 17th.
>> > >
>> > > Andrews.
>> > >
>> > > On Wed, Nov 15, 2017 at 3:56 PM, Dave Fisher 
>> > wrote:
>> > >> Hi -
>> > >>
>> > >> Has this VOTE and Candidate been canceled?
>> > >>
>> > >> Regards,
>> > >> Dave
>> > >>
>> > >>> On Nov 13, 2017, at 3:06 PM, Nozomi Kurihara <
>> nkuri...@yahoo-corp.jp>
>> > wrote:
>> > >>>
>> > >>> Hi all,
>> > >>>
>> > >>> There are still Issues/PRs labeled with milestone
>> '1.21.0-incubating’:
>> > >>> https://github.com/apache/incubator-pulsar/milestone/10
>> > >>>
>> > >>> Please let me know if there are changes that you think should get
>> > included, by 11/15.
>> > >>>
>> > >>> Thanks,
>> > >>> Nozomi
>> > >>>
>> > >>>
>> > >>> 
>> > >>> 差出人: Nozomi Kurihara 
>> > >>> 送信日時: 2017年11月9日 8:24:07
>> > >>> 宛先: dev@pulsar.incubator.apache.org
>> > >>> 件名: RE: [VOTE] Pulsar 1.21.0-incubating Release Candidate 0
>> > >>>
>> > >>> Hi all,
>> > >>>
>> > >>>
>> > >>> I created the first candidate for version 1.21.0.
>> > >>>
>> > >>>
>> > >>> If there are any changes that you think should get included,  please
>> > let me know and then I'll re-create the candidate.
>> > >>>
>> > >>>
>> > >>> Thanks,
>> > >>>
>> > >>> Nozomi
>> > >>>
>> > >>> 
>> > >>> 差出人: Nozomi Kurihara 
>> > >>> 送信日時: 2017年11月9日 8:05:52
>> > >>> 宛先: dev@pulsar.incubator.apache.org
>> > >>> 件名: [VOTE] Pulsar 1.21.0-incubating Release Candidate 0
>> > >>>
>> > >>> This is the first release candidate for Apache Pulsar, version
>> > 1.21.0-incubating.
>> > >>>
>> > >>> It fixes the following issues:
>> > >>> https://github.com/apache/incubator-pulsar/milestone/10?closed=1
>> > >>> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
>> > https://github.com/apache/incubator-pulsar/milestone/10?closed=1>
>> > >>>
>> > >>> apache/incubator-pulsar<
>> > https://github.com/apache/incubator-pulsar/milestone/10?closed=1>
>> > >>> github.com
>> > >>> incubator-pulsar - Pulsar - distributed pub-sub messaging system
>> > >>>
>> > >>>
>> > >>>
>> > >>> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
>> > https://github.com/apache/incubator-pulsar/milestone/10?closed=1>
>> > >>>
>> > >>> apache/incubator-pulsar<
>> > https://github.com/apache/incubator-pulsar/milestone/10?closed=1>
>> > >>> github.com
>> > >>> incubator-pulsar - Pulsar - distributed pub-sub messag

Re: [VOTE] Pulsar 1.20.0-incubating Release Candidate 0

[Rajan Dhabalia]

+1

Verified on
Environment: Mac 10.12.6 & RHEL 6.7
- Verified checksum & signature for binary and src distribution.
- verified build
- passed with regression test suit
- verified notice and license

Thanks,
Rajan

On Mon, Oct 2, 2017 at 4:06 PM, Sahaya Andrews  wrote:

> >We can find closed and open items for this milestone:
> > Closed: https://github.com/apache/incubator-pulsar/milestone/9?closed=1
> > Open: https://github.com/apache/incubator-pulsar/milestone/9
>
> Then we should change the link to the closed link since we are calling
> it out in the text as such.
>
> Environment: Mac 10.12.6
> - Verified checksum & signature for binary and src distribution.
> - Ran local build and unit test in Mac - passed
> - Ran perf producer and consumer against bin distribution and local
> build - passed
> - Verified Notice info related to cryptography in bin and src distribution
>
> Andrews.
>
> On Mon, Oct 2, 2017 at 2:06 PM, Rajan Dhabalia 
> wrote:
> > We can find closed and open items for this milestone:
> > Closed: https://github.com/apache/incubator-pulsar/milestone/9?closed=1
> > Open: https://github.com/apache/incubator-pulsar/milestone/9
> >
> > Thanks,
> > Rajan
> >
> > On Mon, Oct 2, 2017 at 2:03 PM, Sahaya Andrews 
> wrote:
> >
> >> I can't find the changes under
> >> https://github.com/apache/incubator-pulsar/milestone/9.
> >>
> >>
> >> On Sun, Oct 1, 2017 at 1:55 PM, Joe F  wrote:
> >> > This is the first release candidate for Apache Pulsar, version
> >> > 1.20.0-incubating.
> >> >
> >> > The complete changed in the release are listed here:
> >> > https://github.com/apache/incubator-pulsar/milestone/9
> >> >
> >> > *** Please download, test and vote by Oct 5 2017, 16:00 GMT.
> >> >
> >> > Note that we are voting upon the source (tag), binaries are provided
> for
> >> > convenience.
> >> >
> >> > Source and binary files:
> >> > https://dist.apache.org/repos/dist/dev/incubator/pulsar/
> >> > pulsar-1.20.0-incubating-candidate-0/
> >> >
> >> > Maven staging repo:
> >> > https://repository.apache.org/content/repositories/
> orgapachepulsar-1001/
> >> >
> >> > The tag to be voted upon:
> >> > v1.20.0-incubating-candidate-0 (539c8182037fe540433eeb4e7072e4
> >> bf2b3eae04)
> >> >
> >> > Pulsar's KEYS file containing PGP keys we use to sign the release:
> >> > https://dist.apache.org/repos/dist/release/incubator/pulsar/KEYS
> >> >
> >> > Please download the the source package, and follow the README to build
> >> > and run the Pulsar standalone service.
> >> >
> >> > Joe
> >>
>

Re: [VOTE] Pulsar 1.20.0-incubating Release Candidate 0

[Rajan Dhabalia]

We can find closed and open items for this milestone:
Closed: https://github.com/apache/incubator-pulsar/milestone/9?closed=1
Open: https://github.com/apache/incubator-pulsar/milestone/9

Thanks,
Rajan

On Mon, Oct 2, 2017 at 2:03 PM, Sahaya Andrews  wrote:

> I can't find the changes under
> https://github.com/apache/incubator-pulsar/milestone/9.
>
>
> On Sun, Oct 1, 2017 at 1:55 PM, Joe F  wrote:
> > This is the first release candidate for Apache Pulsar, version
> > 1.20.0-incubating.
> >
> > The complete changed in the release are listed here:
> > https://github.com/apache/incubator-pulsar/milestone/9
> >
> > *** Please download, test and vote by Oct 5 2017, 16:00 GMT.
> >
> > Note that we are voting upon the source (tag), binaries are provided for
> > convenience.
> >
> > Source and binary files:
> > https://dist.apache.org/repos/dist/dev/incubator/pulsar/
> > pulsar-1.20.0-incubating-candidate-0/
> >
> > Maven staging repo:
> > https://repository.apache.org/content/repositories/orgapachepulsar-1001/
> >
> > The tag to be voted upon:
> > v1.20.0-incubating-candidate-0 (539c8182037fe540433eeb4e7072e4
> bf2b3eae04)
> >
> > Pulsar's KEYS file containing PGP keys we use to sign the release:
> > https://dist.apache.org/repos/dist/release/incubator/pulsar/KEYS
> >
> > Please download the the source package, and follow the README to build
> > and run the Pulsar standalone service.
> >
> > Joe
>

Re: [GitHub] merlimat commented on issue #742: Avoid huge backlog on topic reloading: due to large gap between markDelete-offset and read-position of cursor.

[Rajan Dhabalia]

> and leave normal queue consumption out of this mechanism, (to reduce the
ZK writes)
>> To be precise, these are BookKeeper writes that would be happening anyway

Just to clarify: Broker also stores ack-holes in ZK along with BK
(Cursor-ledger). But Broker only writes it to ZK when broker unloads the
topic gracefully and deletes the cursor-ledger.


> We could also do that once you reach the max number of "holes" the
delivery stops
The only problem I see in restricting based on ack-holes metrics is
"Ack-hole doesn't follow any pattern and it might not be in sequence".
*For example:*
If we have that max-number of ack-hole is = 1K
and if consumer acks alternate consumed message then there will be 1K
ack-holes built with in 2K consumed messages, and broker will stop the
message-delivery. Consumer should not suffer in this usecase where consumer
is blocked after consuming only 2K messages.

Thanks,
Rajan



On Thu, Sep 7, 2017 at 11:29 AM, Matteo Merli 
wrote:

> On Thu, Sep 7, 2017 at 11:10 AM Joe F  wrote:
>
> > Sure. Setting N=0 (or 1),  and you will get this behavior. In a large (1M
> > topics) system, I  could, say set N = 1000, and leave normal queue
> > consumption out of this mechanism, (to reduce the ZK writes).   So this
> > approach is flexible enough to handle both situations.
> >
>
> To be precise, these are BookKeeper writes that would be happening anyway
> (when the cursor is updated). The only difference is that the size of that
> BK entry increases when there are "holes", up to 154Kb for 10K holes.
>
> Also these BK writes for cursor updates are already throttled (by default 1
> per sec)
>
>
> > Ack holes are not an useful user metric. They are  useful as a system
> > limit - only so many holes can be stored. and as a system metric we can
> use
> > that.
> >
>
> Absolutely agree that are not useful to user, though they are the best
> indicator for triggering a system level alert, so that someone can take a
> look and explain to user :)
>
>
> > But to me as an user, letting 1M messages go unacked is a problem - even
> > though Pulsar can store it super-efficient. I see the core user problem
> as
> > the unexpected duplicate delivery of a large set of messages  on certain
> > broker failures.  As distinct from how Pulsar can store a large set of
> > holes, and to what limit.
> >
>
> Even if we can store a large set of messages super-efficient, it does not
> > resolve the end user pain experienced today - which is a large set of old
> > messages getting delivered on broker failure. Why did you dump 1M
> messages
> > on me all of a sudden after a week? --  that is the question we have run
> > into many times.  So I would prefer that once a certain set limit is
> > crossed, delivery is stopped/alert generated and the the user fixes the
> > problem.  In essence, deal with unacked messages as a first class user
> > resource limit, like backlog quota.
> >
>
> We could also do that once you reach the max number of "holes" the delivery
> stops.
>
> --
> Matteo Merli
> 
>

Re: Add EventTime to pulsar messages

[Rajan Dhabalia]

> Didn't get this, can you clarify?

I think I misinterpreted previous reply where I thought we want to add
"eventTIme" field to avoid deserialization of  Message-Metadata so, I asked
question how to retrieve "eventTime" field without deserializing.

> It's certain that an application can use a (string -> string) property,
though it would be a custom field and it will need to be in string format
and also carry the field name.
But now, I understand actual reason for not using properties to avoid extra
"key" in the message and keeping int64 value instead string.

> It's also nice to expose both `getPublishTime()` and `getEventTime()`
directly in a consistent way.
Sure.

Thanks,
Rajan

On Wed, Aug 23, 2017 at 4:31 PM, Matteo Merli 
wrote:

> > So, I believe in your usecase, we want to skip metadata-deserialization
> and
> > idea is to read only "EventTime" field (field-12) without deserializing
> > message-metada using ByteBuf-offset (eg: read last int64 using end of
> > metadata-index).? Don't you think it will prevent us to add additional
> > String-field in future because then it will be hard to know "EventTime"
> > index?
>
> Didn't get this, can you clarify?
>
> In my view, the event time is a common concept that can be useful in many
> scenarios, every time there is the chance for the message being published
> to refer to some event that happened before the publishing time.
>
> It's also nice to expose both `getPublishTime()` and `getEventTime()`
> directly in a consistent way.
>
> Finally, there will be zero overhead when the field is not set and very
> little when it's set.
>
> It's certain that an application can use a (string -> string) property,
> though it would be a custom field and it will need to be in string format
> and also carry the field name. I think event time is general enough to be
> considered as a built-in field and exposed in API.
>
> Matteo
>
> On Wed, Aug 23, 2017 at 4:23 PM Rajan Dhabalia 
> wrote:
>
> > > because this field might be accessed when processing every message. the
> > overhead in key/value property is much higher than an int64 field.
> >
> > I have the same concern as Andrews mentioned.
> > But a very quick implementation question: properties
> > <
> > https://github.com/apache/incubator-pulsar/blob/master/
> pulsar-common/src/main/proto/PulsarApi.proto#L47
> > >
> > field is also part of message-metadata only.
> > So, I believe in your usecase, we want to skip metadata-deserialization
> and
> > idea is to read only "EventTime" field (field-12) without deserializing
> > message-metada using ByteBuf-offset (eg: read last int64 using end of
> > metadata-index).? Don't you think it will prevent us to add additional
> > String-field in future because then it will be hard to know "EventTime"
> > index?
> >
> > Thanks,
> > Rajan
> >
> >
> > On Wed, Aug 23, 2017 at 3:56 PM, Sijie Guo  wrote:
> >
> > > On Wed, Aug 23, 2017 at 3:19 PM, Sahaya Andrews 
> > > wrote:
> > >
> > > > If this is not going to be used by the broker for anything, why can't
> > > > this be set in message property key/value?
> > > >
> > >
> > > Currently it is not used by broker. However, we might leverage field
> > later
> > > on to provide advanced features like event-time based index. It is not
> > > covered by this proposal.
> > >
> > > Also there is another consideration - adding this field to message
> > metadata
> > > is more serialization friendly than putting it into key/value property.
> > > this is important for stream computing,
> > > because this field might be accessed when processing every message. the
> > > overhead in key/value property is much higher than an int64 field.
> > >
> > > Hope this make sense.
> > >
> > > - Sijie
> > >
> > >
> > > >
> > > > Andrews
> > > >
> > > > On Wed, Aug 23, 2017 at 3:03 PM, Matteo Merli 
> > wrote:
> > > > > Proposal looks good to me.
> > > > >
> > > > > Added to wiki:
> > > > > https://github.com/apache/incubator-pulsar/wiki/PIP-5:-Event-time
> > > > >
> > > > > On Wed, Aug 23, 2017 at 2:52 PM Sijie Guo 
> > wrote:
> > > > >
> > > > >> Hi pulsar folks,
> > > > >>
> > > > >> Currently Pulsar messages has fields for `publish_time`. Howeve

Re: Add EventTime to pulsar messages

[Rajan Dhabalia]

> because this field might be accessed when processing every message. the
overhead in key/value property is much higher than an int64 field.

I have the same concern as Andrews mentioned.
But a very quick implementation question: properties

field is also part of message-metadata only.
So, I believe in your usecase, we want to skip metadata-deserialization and
idea is to read only "EventTime" field (field-12) without deserializing
message-metada using ByteBuf-offset (eg: read last int64 using end of
metadata-index).? Don't you think it will prevent us to add additional
String-field in future because then it will be hard to know "EventTime"
index?

Thanks,
Rajan


On Wed, Aug 23, 2017 at 3:56 PM, Sijie Guo  wrote:

> On Wed, Aug 23, 2017 at 3:19 PM, Sahaya Andrews 
> wrote:
>
> > If this is not going to be used by the broker for anything, why can't
> > this be set in message property key/value?
> >
>
> Currently it is not used by broker. However, we might leverage field later
> on to provide advanced features like event-time based index. It is not
> covered by this proposal.
>
> Also there is another consideration - adding this field to message metadata
> is more serialization friendly than putting it into key/value property.
> this is important for stream computing,
> because this field might be accessed when processing every message. the
> overhead in key/value property is much higher than an int64 field.
>
> Hope this make sense.
>
> - Sijie
>
>
> >
> > Andrews
> >
> > On Wed, Aug 23, 2017 at 3:03 PM, Matteo Merli  wrote:
> > > Proposal looks good to me.
> > >
> > > Added to wiki:
> > > https://github.com/apache/incubator-pulsar/wiki/PIP-5:-Event-time
> > >
> > > On Wed, Aug 23, 2017 at 2:52 PM Sijie Guo  wrote:
> > >
> > >> Hi pulsar folks,
> > >>
> > >> Currently Pulsar messages has fields for `publish_time`. However in
> some
> > >> use cases for example stream computing, `event_time` is required. I'd
> > like
> > >> to start a proposal to add `event_time` support to pulsar messages.
> > >>
> > >> This proposal only covers adding `event_time` to pulsar messages and
> > make
> > >> it available to both producers and subscribers. It doesn't cover
> > advanced
> > >> features, such as event-time index or rewind subscriptions based on
> > >> event-time.
> > >>
> > >> I created the proposal here:
> > >> https://gist.github.com/sijie/60324cc892643961b923593a597109ab
> > >>
> > >> Please go over and provide your feedback/comments.
> > >>
> > >> Also, since I don't have any permissions on writing pulsar wiki pages,
> > can
> > >> any pulsar committers help me add this PIP to the wiki pages if it is
> > >> accepted?
> > >>
> > >> - Sijie
> > >>
> > > --
> > > Matteo Merli
> > > 
> >
>

Re: [VOTE] Pulsar 1.19.0-incubating Release Candidate 0

[Rajan Dhabalia]

+1. I ran the regression-tests and few of the load-tests with this version
and looks fine.

Thanks,
Rajan

On Wed, Aug 2, 2017 at 1:36 PM, Dave Fisher  wrote:

> Hi -
>
> I will cast an IPMC vote.
>
> +1 Release!
>
> I checked the source and binary package hashes.
> I built the project with Maven from the source package.
> I ran the rat check and checked the rat report.
> Discussed the Category B issues which look good for now.
>
> Regards,
> Dave
>
> > On Aug 2, 2017, at 12:47 PM, Matteo Merli  wrote:
> >
> > Remainder for all committers:
> >
> > please check the release artifacts and vote on this thread, before
> tomorrow
> > 3pm PDT.
> >
> > Vote as in :
> >
> > [ ] +1 Release this package
> > [ ] 0 I don't feel strongly about it, but don't object
> > [ ] -1 Do not release this package because...
> >
> >
> > The documentation on the release process and what needs to be included
> can
> > be found at:
> >   - https://incubator.apache.org/guides/releasemanagement.html
> >   - http://www.apache.org/legal/release-policy.html
> >   - http://www.apache.org/dev/release-distribution.html
> >   - http://www.apache.org/dev/release-publishing.html
> >   - http://www.apache.org/dev/publishing-maven-artifacts.html
> >
> > --
> > Matteo Merli
> > 
> >
> > On Mon, Jul 31, 2017 at 2:39 PM, Matteo Merli  wrote:
> >
> >> This is the first release candidate for Apache Pulsar, version
> >> 1.19.0-incubating.
> >>
> >> Major changes included in this release are:
> >> * Added stateless Pulsar proxy
> >> * Support for non-persistent topics
> >> * Upgraded RocksDB to comply with ASF policy
> >> * Instrumentation of ZooKeeper client to expose metrics
> >> * Fixes for TLS auth in WebSocket proxy
> >>
> >>
> >> Complete list of changes can be found at:
> >>
> >> https://github.com/apache/incubator-pulsar/milestone/8?closed=1
> >>
> >> *** Please download, test and vote by August 3th 2017, 15:00 PDT.
> >>
> >> Note that we are voting upon the source (tag), binaries are provided for
> >> convenience.
> >>
> >> Source and binary files:
> >> https://dist.apache.org/repos/dist/dev/incubator/pulsar/
> >> pulsar-1.19.0-incubating-candidate-0/
> >>
> >> Maven staging repo:
> >> https://repository.apache.org/content/repositories/
> orgapachepulsar-1000/
> >>
> >> The tag to be voted upon:
> >> v1.19.0-incubating-candidate-0 (5125279b50f7a3da8c211b698580e1
> d2f4dd65e2)
> >>
> >> Pulsar's KEYS file containing PGP keys we use to sign the release:
> >> https://dist.apache.org/repos/dist/release/incubator/pulsar/KEYS
> >>
> >> Please download the the source package, and follow the README to build
> >> and run the Pulsar standalone service.
> >>
> >>
> >>
> >> --
> >> Matteo Merli
> >> 
> >>
>
>

Re: CI builds on Jenkins

[Rajan Dhabalia]

> Another option would be to make the Jenkins build status non-mandatory for
merging the PRs.

Then we might want to be careful before merging because flaky tests might
be failing due to genuine reason or bug. Probably, we can keep it enable
for sometime and disable if we really think that flaky tests are giving lot
of pain for merging?

Thanks,
Rajan


On Wed, Aug 2, 2017 at 1:49 PM, Matteo Merli  wrote:

> Another option would be to make the Jenkins build status non-mandatory for
> merging the PRs.
>
> In this case, if we see any known flaky tests, we could proceed with
> merging. For any new failing test, we
> should re-run the build and create a new issue to track it.
>
> Pros:
>* We don't have to spend big amount of time rebuilding the same PRs to
> merge
>
> Cons:
>  * We'll get acquainted with the failing tests and possibly never fix them.
>
> Any thoughts or preferences?
>
>
>
> --
> Matteo Merli
> 
>
> On Wed, Aug 2, 2017 at 12:22 PM, Matteo Merli  wrote:
>
> > Reading on the Jenkins Github plugin documentation at
> > https://wiki.jenkins.io/display/JENKINS/GitHub+pull+
> request+builder+plugin
> >
> >
> > You can trigger the build of a PR by adding a comment "retest this
> please"
> > on the PR itself. This will kick a new build that will properly update
> the
> > PR status.
> >
> > This will work for all committers.
> >
> >
> >
> > --
> > Matteo Merli
> > 
> >
> > On Wed, Aug 2, 2017 at 10:24 AM, Matteo Merli  wrote:
> >
> >> As most of you have seen, we have a bunch of intermittent test failures.
> >>
> >> This, added to the difficulties in rebuilding a pull-request and
> updating
> >> the status lead to a bit of a cumbersome process to get a PR merged.
> >>
> >> I have created issues for the tests I've seen failing and marked them
> >> with label "component/test". The full list is at
> >> https://github.com/apache/incubator-pulsar/issues?q=is%3Aiss
> >> ue+is%3Aopen+label%3Acomponent%2Ftest
> >>
> >> Please anyone that has time to investigate these, take one, assign to
> >> yourself and try to reproduce. Fortunately, in new builds we now have
> >> individual INFO logs for each of the test, so that it might help in the
> >> debugging.
> >>
> >>
> >>
> >> --
> >> Matteo Merli
> >> 
> >>
> >> On Tue, Aug 1, 2017 at 9:40 AM, Matteo Merli  wrote:
> >>
> >>> As we discussed some time back, I have added CI builds in Jenkins.
> >>>
> >>> Jenkins seems to be much more stable compare to Travis and it's bit
> more
> >>> flexible in terms of what we can do. For example, for C++ build we can
> use
> >>> custom Docker image with all the dependencies we require to build the
> >>> Pulsar client lib.
> >>>
> >>> I have asked INFRA to disable the mandatory check on Travis (to merge
> >>> the PRs) and instead use Jenkins for the same purpose.
> >>>
> >>> The build page links:
> >>>
> >>> https://builds.apache.org/job/pulsar-master/
> >>>
> >>> https://builds.apache.org/job/pulsar-pull-request/
> >>>
> >>> The only minor issue that I can see is that the "Rebuild" button the on
> >>> the PR builder is not updating back the status on the Github pull
> request.
> >>> The workaround is to "repush" the branch on github. If there are no
> >>> changes, you can do a "git commit --amend" and it will just update the
> >>> timestamp of the commit and force a new sha hash. Then push --force to
> >>> update the PR.
> >>>
> >>> If anyone knows or wants to take a look for a better solution.. :)
> >>>
> >>>
> >>>
> >>>
> >>> --
> >>> Matteo Merli
> >>> 
> >>>
> >>
> >>
> >
>

Re: [PROPOSAL] Created a Slack channel for Pulsar

[Rajan Dhabalia]

> The ASF has already a Slack org, but it's restricted to committers only. I
think it would be beneficial to have it with open signup so that external
people can join as well.

Yes, that would be nice to have it. +1

Thanks,
Rajan

On Wed, Jul 12, 2017 at 6:37 PM, Matteo Merli  wrote:

> What do you all think about creating a Pulsar dedicated Slack channel for
> quick and informal discussion between developers and to eventually better
> support users and contributors ramping up on the code base?
>
> I know that a handful of Apache projects are doing that already, and there
> are ways to have a bot that sends a daily digest of the conversation to the
> mailing lists.
>
> The ASF has already a Slack org, but it's restricted to committers only. I
> think it would be beneficial to have it with open signup so that external
> people can join as well.
>
> Thoughts?
>
> Matteo
>
> --
> Matteo Merli
> 
>

Re: [DISCUSS] PIP-1 Pulsar proxy component

[Rajan Dhabalia]

>> or if people prefer we can create a GitHub issue to have the discussion
on the proposal
I think github issue would be more helpful for discussion. so, my vote to
github-issue.

Thanks,
Rajan

On Thu, Jun 29, 2017 at 1:51 PM, Matteo Merli  wrote:

> I have created the wiki page with a first proposal (as we discussed
> earlier).
>
> https://github.com/apache/incubator-pulsar/wiki/PIP-1
>
> Please add feedback in this thread (or if people prefer we can create a
> GitHub issue to have the discussion on the proposal).
>
> Other than the proposal itself, it would be good to have some feedback on
> the format as well.
>
> Matteo
>
>
> --
> Matteo Merli
> 
>

Re: [jira] [Commented] (INFRA-14376) Pulsar GitHub Integration

[Rajan Dhabalia]

Hi Dave,

> Everyone who has provided an iCLA and has setup their Apache ID to point
to their GitHub ID should have Write access as you see.
Actually, I have verified that my Apache ID is pointing to GitHub-ID
(username) and I have also submitted the iCLA earlier. I think it's same
for Andrews as well. So, not sure if write-access is not provided due to
other incomplete process for which we might have to wait for some more time.

Thanks,
Rajan


On Thu, Jun 22, 2017 at 5:49 PM, Dave Fisher  wrote:

> Hi Andrews,
>
> > On Jun 22, 2017, at 3:40 PM, Sahaya Andrews 
> wrote:
> >
> > When can we expect the setup to be finalized? We are still waiting to
> > get write access. Some of us who were part of the dev group in the old
> > repo got write access automatically. But the rest who were part of
> > admin group did not :)
>
> Everyone who has provided an iCLA and has setup their Apache ID to point
> to their GitHub ID should have Write access as you see.
>
> Daniel - would you comment on Admin privilege setup with Gitbox
> integration. What can the podling expect? How do we control the group? Do
> we do JIRA tickets? I know the Gitbox integration is new and the
> documentation that you pointed me to is out of date and not on point with
> the new.
>
> Regards,
> Dave
>
> >
> > Andrews.
> >
> > On Wed, Jun 21, 2017 at 4:36 PM, Matteo Merli  wrote:
> >> Andrews,
> >>
> >> once the setup is finalized, all the committers will have write access
> if
> >> the github account is linked to the Apache account.
> >>
> >>
> >>
> >> On Wed, Jun 21, 2017 at 3:32 PM Sahaya Andrews <
> sahaya.andr...@gmail.com>
> >> wrote:
> >>
> >>> Hi Daniel,
> >>>
> >>>  Could you add pulsar repo write access to the following GitHub
> account?
> >>> These folks were part of admin account before moving the repo.
> >>> *apache id - GitHub id *
> >>> andrews  - saandrews
> >>> joef  - joefk
> >>> rdhabalia - rdhabalia
> >>>
> >>> Thanks,
> >>> Andrews.
> >>>
> >>> On Wed, Jun 21, 2017 at 3:08 PM, Sahaya Andrews <
> sahaya.andr...@gmail.com>
> >>> wrote:
> >>>
>  I have updated my GitHub id in profile.
> 
>  What does the "reporeq" step meant to do?
> 
>  On Wed, Jun 21, 2017 at 2:56 PM, Matteo Merli 
> wrote:
> 
> > The repo location has now been transferred to
> > https://github.com/apache/incubator-pulsar
> >
> > I think the next step would be to do the "reporeq" at
> > https://reporeq.apache.org
> > And that should be done by someone in IPMC :)
> > I am not sure wether it was already done though.
> >
> > For all the committers: you should go to id.apache.org and enter
> your
> > github account there, so that you can link that account to the apache
> > account.
> >
> > Matteo
> >
> >
> > -- Forwarded message -
> > From: Daniel Takamori (JIRA) 
> > Date: Wed, Jun 21, 2017 at 2:11 PM
> > Subject: [jira] [Commented] (INFRA-14376) Pulsar GitHub Integration
> > To: 
> >
> >
> > Daniel Takamori
> > 
> > *commented* on [image: Github Integration] INFRA-14376
> > 
> >
> > Re: Pulsar GitHub Integration
> > 
> > Imported the repo to Apache and set the hooks, now your PMC can go to
> > https://reporeq.apache.org to clone the Apache repo and we can move
> on
> > from there :)
> >
> >  Add
> > Comment  comment>
> >
> > This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe)
> >
> 
> 
> >>>
>
>