Re: [Discussion] PIP 78: Replace brodocs with docsify?

[Dave Fisher]

Sent from my iPhone

> On Aug 18, 2022, at 4:55 AM, Yu  wrote:
> 
> Hi team,
> 
> To reduce manual work, we’re constantly implementing PIP 78: Generate Docs
> from Code Automatically [1].
> 
> Docs here refer to the following：
> - Connector configurations [2]
> - Client configurations [3]
> - Pulsar CLI tools [4]
> - Pulsar configurations [5]
> - pulsar-admin [6]
> 
> # Question
> 
> Currently, some of the docs above [7] are generated using brodocs [8].
> We’re wondering if it makes sense to replace brodocs with docsify?
> 
> # docsify benefits
> 
> ## For users
> - Enjoy new features (eg. search, copy to clipboard, language highlighting,
> tabs, Disqus, etc.)

How is this search organized on the server and how would this integrate with 
algolia?

> - More clear and organized layout
> 
> ## For doc maintainers
> - Simple and lightweight
> - No statically built HTML files. Instead, it smartly loads and parses .md
> files and displays them as a website.
> - Allow feature customization
> - Able to work with Google Analytics to track data

Google Analytics will be disallowed on ASF project sites very soon due to 
Privacy concerns with tracking data. I’ve warned about this several times. 
Please eliminate GA. If we need to track then we have options.

Also we never discuss GA anywhere within the PMC or developer community. What 
value does it have?

Dave

> 
> # Demo
> 
> We’ve created a demo here [9].
> This demo is only for demonstration purposes, so we just add a few
> improvements. It can be iterated later.
> 
> 
> 
> If docsify is better, we’ll use it for all the docs above to provide a
> consistent content experience.
> 
> Feel free to check and comment, thank you!
> 
> [1] https://lists.apache.org/thread/638q6kmq81z2qjmyhgkqzp00sllh4x0w
> [2] https://pulsar.apache.org/docs/next/io-connectors (Configurations for
> each connector)
> [3] https://pulsar.apache.org/docs/next/client-libraries-java
> (Configuration for each client. Here takes Java client as an example, it
> refers to configurations of client, producer, consumer, and reader)
> [4] https://pulsar.apache.org/docs/next/reference-cli-tools
> [5] https://pulsar.apache.org/docs/next/reference-configuration
> [6] https://pulsar.apache.org/tools/pulsar-admin/
> [7] https://pulsar.apache.org/tools/
> [8] https://github.com/Birdrock/brodocs
> [9] https://pulsar.apache.org/tools/pulsar-config/2.11.0-SNAPSHOT/#/
> 
> Yu, signormercurio, urfreespace

Re: [Discussion] PIP 78: Replace brodocs with docsify?

[Dave Fisher]

If page urls change then we should also add http redirect rules to help users 
who bookmarked these docs.

Sent from my iPhone

> On Aug 23, 2022, at 5:53 AM, Yu  wrote:
> 
> Hi team,
> 
> Now is 3 working days after the initial discussion
> 
> We'll start replacing brodocs with docsify for docs since there is no
> objection or technical suggestions.
> 
> Yu, signormercurio, urfreespace
> 
>> On Fri, Aug 19, 2022 at 5:09 PM Yu  wrote:
>> 
>> Thanks Dave!
>> 
>>> How is this search organized on the server and how would this integrate
>> with algolia?
>> 
>> Search is performed at the front end, and it does not need to integrate
>> with Algolia.
>> 
>>> Google Analytics will be disallowed on ASF project sites very soon due
>> to Privacy concerns with tracking data. I’ve warned about this several
>> times. Please eliminate GA. If we need to track then we have options.
>> 
>> Listing GA here just shows docsify's capabilities.
>> As we discussed before [1], GA will no longer process new data in standard
>> properties beginning July 1, 2023.
>> Matteo is being considered to replace it.
>> 
>> [1] https://github.com/apache/pulsar/issues/15664
>> 
>> Yu and signormercurio
>> 
>> 
>> 
>> 
>> 
>> 
>>> On Thu, Aug 18, 2022 at 9:46 PM Dave Fisher  wrote:
>>> 
>>> 
>>> 
>>> Sent from my iPhone
>>> 
>>>> On Aug 18, 2022, at 4:55 AM, Yu  wrote:
>>>> 
>>>> Hi team,
>>>> 
>>>> To reduce manual work, we’re constantly implementing PIP 78: Generate
>>> Docs
>>>> from Code Automatically [1].
>>>> 
>>>> Docs here refer to the following：
>>>> - Connector configurations [2]
>>>> - Client configurations [3]
>>>> - Pulsar CLI tools [4]
>>>> - Pulsar configurations [5]
>>>> - pulsar-admin [6]
>>>> 
>>>> # Question
>>>> 
>>>> Currently, some of the docs above [7] are generated using brodocs [8].
>>>> We’re wondering if it makes sense to replace brodocs with docsify?
>>>> 
>>>> # docsify benefits
>>>> 
>>>> ## For users
>>>> - Enjoy new features (eg. search, copy to clipboard, language
>>> highlighting,
>>>> tabs, Disqus, etc.)
>>> 
>>> How is this search organized on the server and how would this integrate
>>> with algolia?
>>> 
>>>> - More clear and organized layout
>>>> 
>>>> ## For doc maintainers
>>>> - Simple and lightweight
>>>> - No statically built HTML files. Instead, it smartly loads and parses
>>> .md
>>>> files and displays them as a website.
>>>> - Allow feature customization
>>>> - Able to work with Google Analytics to track data
>>> 
>>> Google Analytics will be disallowed on ASF project sites very soon due to
>>> Privacy concerns with tracking data. I’ve warned about this several times.
>>> Please eliminate GA. If we need to track then we have options.
>>> 
>>> Also we never discuss GA anywhere within the PMC or developer community.
>>> What value does it have?
>>> 
>>> Dave
>>> 
>>>> 
>>>> # Demo
>>>> 
>>>> We’ve created a demo here [9].
>>>> This demo is only for demonstration purposes, so we just add a few
>>>> improvements. It can be iterated later.
>>>> 
>>>> 
>>>> 
>>>> If docsify is better, we’ll use it for all the docs above to provide a
>>>> consistent content experience.
>>>> 
>>>> Feel free to check and comment, thank you!
>>>> 
>>>> [1] https://lists.apache.org/thread/638q6kmq81z2qjmyhgkqzp00sllh4x0w
>>>> [2] https://pulsar.apache.org/docs/next/io-connectors (Configurations
>>> for
>>>> each connector)
>>>> [3] https://pulsar.apache.org/docs/next/client-libraries-java
>>>> (Configuration for each client. Here takes Java client as an example, it
>>>> refers to configurations of client, producer, consumer, and reader)
>>>> [4] https://pulsar.apache.org/docs/next/reference-cli-tools
>>>> [5] https://pulsar.apache.org/docs/next/reference-configuration
>>>> [6] https://pulsar.apache.org/tools/pulsar-admin/
>>>> [7] https://pulsar.apache.org/tools/
>>>> [8] https://github.com/Birdrock/brodocs
>>>> [9] https://pulsar.apache.org/tools/pulsar-config/2.11.0-SNAPSHOT/#/
>>>> 
>>>> Yu, signormercurio, urfreespace
>>> 
>>> 

Re: [DISCUSS] Move PIPs to the codebase?

[Dave Fisher]

> On Aug 18, 2022, at 8:27 AM, PengHui Li  wrote:
> 
> Hi all,
> 
> Currently, the new proposal will be added to the issue list and then shared
> link in the email
> to request the proposal review. It's really hard to review a long proposal
> if you want to comment
> in detail.
> 
> Here is an example:
> https://github.com/apache/pulsar/issues/16763#issuecomment-1219606491
> This seems very unintuitive.
> 
> I think we can move all the PIPs to the codebase and the new proposal and
> proposal without
> any reviews should happen with a PR first. So that we can review and
> comment easily.
> Certainly, all the votes should happen on the mailing list. And we can also
> discuss the
> proposal on the mailing list.
> 
> Following this way, we don't need to sync the PIPs from the issue to the
> wiki page.
> We can just add a link that points to the PIPs dir to the contribution
> guide or README.
> 
> We have another pain point about the duplicated PIP number. We can maintain
> a file, a list of
> all the proposal contains the approved, in-review, drafting. Before
> creating a proposal, we should
> have a discussion first on the mailing list, just get feedback on the
> motivation. If there are no objections,
> the proposal owner can add a line to the file with the PIP number through a
> PR, like PIP-123: xxx (Under Discussion).
> So that we can prevent the duplicated PIP number(which will conflict if
> someone merged first).
> After the PR is merged, we can send out a new PR to add the proposal.

Should we track the status of PIPs as a project?

Also we have old PIPs that have been abandoned or that never completed VOTEs 
(there was one for pulsarctl that I would -1 if it were brought back.)

Regards,
Dave
> 
> Thanks,
> Penghui

Re: [Discuss][PIP-164] Support split bundle by flow or qps

[Dave Fisher]

While not a comment about this proposal I have a comment another split bundle 
concept.

Manually split out a topic into its own bundle.

Say a bundle is 0x to 0x0200 with 5 topics.

t1 at 0x0010
t2 at 0x0030
t3 at 0x0110
t4 at 0x0123
t5 at 0x01AE

Let’s split out t3 to it’s own bundle and then have three new bundles:

0x:0x010F - t1, t2
0x0110:0x0110 - t3
0x0111:0x0200 - t4, t5

This can be useful when t3 has most of the traffic

> On Aug 23, 2022, at 2:13 AM, lordcheng10 <1572139...@qq.com.INVALID> wrote:
> 
> "It's a good idea to improve the bundle split for the case that the traffic
> of the topic doesn't change drastically
> Otherwise, we should not use this policy. or can we use it for all cases?"
> 
> 
> 1.It is suitable for scenarios where topic traffic is relatively stable. In 
> addition, we can also adjust the flowOrQpsDifferenceThresholdPercentage 
> configuration to adapt to traffic fluctuations;
> 2.The current strategy is to split bundles based on the configuration 
> loadBalancerNamespaceBundleMaxMsgRate or 
> loadBalancerNamespaceBundleMaxBandwidthMbytes;
> 3.If the qps or traffic of a bundle is less than 
> loadBalancerNamespaceBundleMaxMsgRate*(1+ 
> flowOrQpsDifferenceThresholdPercentage) or 
> loadBalancerNamespaceBundleMaxBandwidthMbytes*(1+flowOrQpsDifferenceThresholdPercentage),
>  the policy will no longer trigger split;
> 
> 
> "do we need to consider the consumer rate
> the `flow or qps` is based on the entries or messages?"
> 
> 
> 1. The consumer rate has been considered;
> 2. The strategy has been based on the entries and throughput;
> 
> 
> 
> 
> Thanks,
> lordcheng10
> 
> 
> Reply for Penghui

Re: [VOTE] [PIP-201] Extensions mechanism for Pulsar Admin CLI tools

[Dave Fisher]

+1 (binding)

Best,
Dave

Sent from my iPhone

> On Aug 24, 2022, at 12:43 AM, Yunze Xu  wrote:
> 
> +1 (non binding)
> 
> Thanks,
> Yunze
> 
> 
> 
> 
>> 2022年8月24日 15:38，Nicolò Boschi  写道：
>> 
>> +1 (non binding)
>> Nicolò Boschi
>> 
>> 
>>> Il giorno mer 24 ago 2022 alle ore 09:11 Enrico Olivelli <
>>> eolive...@gmail.com> ha scritto:
>>> 
>>> Hello,
>>> this is the official thread VOTE for PIP-201 Extensions mechanism for
>>> Pulsar Admin CLI tools
>>> 
>>> This is the PIP link https://github.com/apache/pulsar/issues/17155
>>> This is the discussion:
>>> https://lists.apache.org/thread/287ft8twc11cp4s1y4qkcx5nmh451cyo
>>> 
>>> I am still working on the PR, that is the subject of the VOTE.
>>> 
>>> Best regards
>>> Enrico Olivelli
>>> 
> 

Re: [DISCUSS] Move PIPs to the codebase?

[Dave Fisher]

> On Aug 23, 2022, at 10:22 AM, Rajan Dhabalia  wrote:
> 
> Hi,
> 
 I think we can move all the PIPs to the codebase and the new proposal
> and proposal without any reviews should happen with a PR first. So that we
> can review and comment easily.
> 
> I didn't understand this part. You want one to create a PR before
> submitting a proposal? That's clearly not a good idea because if the PIP
> approach will change then the entire development effort will be wasted and
> that's the whole purpose of PIP. I guess creating PIP into an issue and
> discussing the issue is definitely working and it's an easier way to
> discuss quickly rather than discussing over email threads.
> 
> Let's not change this practice without good discussion and agreement from
> the community.

Agreed let’s have a PIP Discussion here to carefully outline how the PIP 
process will change. I don’t think that a new PIP should be overly planned or 
implemented before the idea is more fully discussed and accepted. The Apache 
Way always works best with small incremental and reversible steps.

Let’s outline how PIPs are currently working and then discuss changes. I’m not 
sure what is meant by putting the PIP into the “codebase”.

Is the proposal to create PIPs here? 
https://github.com/apache/pulsar/tree/master/wiki

I think that a directory structure / convention could be used for pip status:

1. /wiki/pip/discussion - for PIPs being discussed and specified.
2. /wiki/pip/proposed - for PIPs ready to be formally DISCUSSED and VOTED
3. /wiki/pip/accepted - for PIPs that have been accepted and are works in 
progress
4. /wiki/pip/complete - for PIPs that have been completed.
5. /wiki/pip/rejected - for PIPs that were proposed, but then rejected or 
abandoned.

Regards,
Dave

> 
> Thanks,
> Rajan
> 
> On Thu, Aug 18, 2022 at 8:27 AM PengHui Li  wrote:
> 
>> Hi all,
>> 
>> Currently, the new proposal will be added to the issue list and then shared
>> link in the email
>> to request the proposal review. It's really hard to review a long proposal
>> if you want to comment
>> in detail.
>> 
>> Here is an example:
>> https://github.com/apache/pulsar/issues/16763#issuecomment-1219606491
>> This seems very unintuitive.
>> 
>> I think we can move all the PIPs to the codebase and the new proposal and
>> proposal without
>> any reviews should happen with a PR first. So that we can review and
>> comment easily.
>> Certainly, all the votes should happen on the mailing list. And we can also
>> discuss the
>> proposal on the mailing list.
>> 
>> Following this way, we don't need to sync the PIPs from the issue to the
>> wiki page.
>> We can just add a link that points to the PIPs dir to the contribution
>> guide or README.
>> 
>> We have another pain point about the duplicated PIP number. We can maintain
>> a file, a list of
>> all the proposal contains the approved, in-review, drafting. Before
>> creating a proposal, we should
>> have a discussion first on the mailing list, just get feedback on the
>> motivation. If there are no objections,
>> the proposal owner can add a line to the file with the PIP number through a
>> PR, like PIP-123: xxx (Under Discussion).
>> So that we can prevent the duplicated PIP number(which will conflict if
>> someone merged first).
>> After the PR is merged, we can send out a new PR to add the proposal.
>> 
>> Thanks,
>> Penghui
>> 

Re: [VOTE] Pulsar Release 2.7.5 Candidate 3

[Dave Fisher]

This is an interesting result. It would be great if the download page after 
this release is made would indicate JDK versions.

Thanks!
Dave

Sent from my iPhone

> On Aug 30, 2022, at 10:46 PM, Michael Marshall  wrote:
> 
> Thank you both for verifying it on your end. After debugging this for
> a way too long tonight, I discovered my mistake was using Java 11
> instead of Java 8. It didn't occur to me that the different versions
> could/would result in a different set of dependencies.
> 
> When I use Java 8, this command passes now: src/check-binary-license
> distribution/server/target/apache-pulsar-2.7.5-bin.tar.gz. I plan to
> finish my release validation tomorrow.
> 
> Thanks!
> Michael
> 
> 
> 
>> On Tue, Aug 30, 2022 at 4:21 AM Anon Hxy  wrote:
>> 
>> Hi Michael,
>> 
>> I also couldn't reproduce it  with the same steps Haiting provided.
>> 
>> Here is my local environment info with `mvn -v`
>> 
>> Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
>> Maven home: /Users/didi/Documents/apache-maven-3.6.3
>> Java version: 1.8.0_291, vendor: Oracle Corporation, runtime:
>> /Library/Java/JavaVirtualMachines/jdk1.8.0_291.jdk/Contents/Home/jre
>> Default locale: zh_CN, platform encoding: UTF-8
>> OS name: "mac os x", version: "10.16", arch: "x86_64", family: "mac"
>> 
>> Thanks,
>> Xiaoyu Hou
>> 
>> 
>> Haiting Jiang  于2022年8月30日周二 15:23写道：
>> 
>>> Hi Michael,
>>> Thanks for your verification.
>>> 
>>> I tried to recreate the issue, but I can't reproduce this.
>>> Here is my steps:
>>> - wget
>>> 
>>> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.7.5-candidate-3/apache-pulsar-2.7.5-src.tar.gz
>>> - tar -xvf apache-pulsar-2.7.5-src.tar.gz
>>> - cd apache-pulsar-2.7.5
>>> - mvn clean install -DskipTests
>>> - src/check-binary-license
>>> distribution/server/target/apache-pulsar-2.7.5-bin.tar.gz
>>> 
>>> And nothing appears in `check-binary-license` output.
>>> 
>>> Here is my local environment info with `mvn -v`
>>> 
>>> Apache Maven 3.6.1 (d66c9c0b3152b2e69ee9bac180bb8fcc8e6af555;
>>> 2019-04-05T03:00:29+08:00)
>>> Maven home: /Users/jason/apache-maven-3.6.1
>>> Java version: 1.8.0_291, vendor: Oracle Corporation, runtime:
>>> /Library/Java/JavaVirtualMachines/jdk1.8.0_291.jdk/Contents/Home/jre
>>> Default locale: en_CN, platform encoding: UTF-8
>>> OS name: "mac os x", version: "10.15.7", arch: "x86_64", family: "mac"
>>> 
>>> Please check the settings again and I also will ask others to help verify
>>> the issue you mention.
>>> 
>>> Thanks,
>>> Haiting
>>> 
>>> 
>>> On Tue, Aug 30, 2022 at 12:37 PM Michael Marshall 
>>> wrote:
>>> 
 When I build from the source (both for the git tag
 `v2.7.5-candidate-3` and the extracted src tarball
 `apache-pulsar-2.7.5-src.tar.gz`), I am getting an error when I run
 the license check:
 
 ```
 $ src/check-binary-license
 distribution/server/target/apache-pulsar-2.7.5-bin.tar.gz
 com.sun.activation-jakarta.activation-1.2.2.jar unaccounted for in
>>> LICENSE
 jakarta.activation-jakarta.activation-api-1.2.2.jar unaccounted for in
 LICENSE
 jakarta.xml.bind-jakarta.xml.bind-api-2.3.3.jar unaccounted for in
>>> LICENSE
 jakarta.activation-jakarta.activation-api-1.2.1.jar mentioned in
 LICENSE, but not bundled
 jakarta.xml.bind-jakarta.xml.bind-api-2.3.2.jar mentioned in LICENSE,
 but not bundled
 jakarta.activation-1.2.2.jar unaccounted for in lib/presto/LICENSE
 jakarta.activation-1.2.2.jar unaccounted for in lib/presto/LICENSE
 jakarta.activation-api-1.2.2.jar unaccounted for in lib/presto/LICENSE
 jakarta.xml.bind-api-2.3.3.jar unaccounted for in lib/presto/LICENSE
 jetty-alpn-java-client-9.4.27.v20200227.jar unaccounted for in
 lib/presto/LICENSE
 
 It looks like there are issues with the LICENSE/NOTICE.
 ```
 
 My understanding is that the missing license attributions are a
 blocker for the release candidate. Is someone able to confirm that
 this is an issue? I ran this command as the release process doc
 indicates here [0].
 
 Before finding the above issue, I successfully ran the following checks:
 - Verified checksums and signatures on all 45 artifacts
 - Compiled from source (apache-pulsar-2.7.5-src.tar.gz) using `mvn
 clean install -DskipTest`
 - Ran `mvn apache-rat:check` successfully
 
 Thanks,
 Michael
 
 [0]
 
>>> https://github.com/apache/pulsar/blob/db16e440622c3eed225c4da5380c70b16a583787/wiki/release/release-process.md#3-build-and-inspect-the-artifacts
 
 On Mon, Aug 29, 2022 at 9:20 AM Nicolò Boschi 
 wrote:
> 
> +1 (non binding)
> 
> Checks:
> 
> - Checksum and signatures
> 
> - Apache Rat check passes
> 
> - Compile from source
> 
> - Run Pulsar standalone and produce-consume from CLI
> 
> 
> Nicolò Boschi
> 
> 
> Il giorno sab 27 ago 2022 alle ore 17:12 Qiang Huang <
> qiang.huang1

Re: [DISCUSS] Enable message deduplication for replicator by default

[Dave Fisher]

Excellent point Rajan!

We had a significant decrease in performance with consumption in 2.8.3 which 
was not fixed until 2.10.1.

Let’s take care to solve problems without surprising users and impacting 
performance.

Best,
Dave

Sent from my iPhone

> On Sep 5, 2022, at 5:11 PM, Rajan Dhabalia  wrote:
> 
> Message deduplication always comes with memory and CPU cost and making it
> default means charging this penalty to every user without having this
> requirement.
> 
> Enabling by default means you are impacting every user who is not aware
> about this feature after upgrading the release. This is purely requirement
> bases and we should avoid enabling it by default.
> 
> Thanks,
> Rajan
> 
>> On Mon, Sep 5, 2022 at 2:50 AM lordcheng10  wrote:
>> 
>> +1
>> 
>> Haiting Jiang  于2022年8月26日周五 09:52写道：
>> 
>>> +1
>>> 
>>> Thanks,
>>> Haiting
>>> 
>>> On Thu, Aug 25, 2022 at 9:52 AM Baodi Shi 
>>> wrote:
>>> 
 +1
 
 Thanks,
 Baodi Shi
 
> On Aug 24, 2022, at 20:1312, Qiang Huang 
 wrote:
> 
> +1
> 
> Zike Yang  于2022年8月22日周一 15:32写道：
> 
>> +1
>> 
>> Thanks,
>> Zike Yang
>> 
>> On Mon, Aug 22, 2022 at 3:16 PM mattison chao <
>>> mattisonc...@apache.org>
>> wrote:
>>> 
>>> +1
>>> 
>>> Best,
>>> Mattison
>>> 
>>> On Fri, 19 Aug 2022 at 01:40, Enrico Olivelli >> 
>> wrote:
>>> 
 I agree
 
 Enrico
 
 Il Gio 18 Ago 2022, 18:23 PengHui Li  ha
>>> scritto:
 
> Hi all,
> 
> When I tried to fix a problem related to replicator
> https://github.com/apache/pulsar/pull/17154
> It surprised me that the message deduplication will not work by
>> default
> with the replicator.
> I always thought it was enabled for replicators by default.
>> Details
>> to
 see
> [0].
> 
> I think we should enable the deduplication for the replicator.
>> Otherwise,
> we will see duplicated
> messages on the remote cluster. And the producer of the
>> replicator
>> always
> has a fixed producer
> name, this will make the message deduplication work properly.
> 
> The test introduced in
>> https://github.com/apache/pulsar/pull/17154
>> will
> check the message
> replication ordering. Without the message deduplication enabled,
>>> the
>> test
> is flaky with received
> duplicated messages. After enabling, everything is fine.
> 
> Best,
> Penghui
> 
> [0]
>> https://github.com/apache/pulsar/pull/17154#discussion_r948736894
> 
 
>> 
> 
> 
> --
> BR,
> Qiang Huang
 
 
>>> 
>> 

Re: Pulsar CI congested, master branch build broken

[Dave Fisher]

We are going to need to take actions to fix our problems. See 
https://issues.apache.org/jira/browse/INFRA-23633?focusedCommentId=17600749&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17600749

Jarek has done a large amount of GitHub Action work with Apache Airflow and his 
suggestions might be helpful. One of his suggestions was Apache Yetus. I think 
he means using the Maven plugins - 
https://yetus.apache.org/documentation/0.14.0/yetus-maven-plugin/


> On Sep 6, 2022, at 4:48 AM, Lari Hotari  wrote:
> 
> The Apache Infra ticket is https://issues.apache.org/jira/browse/INFRA-23633 
> . 
> 
> -Lari
> 
> On 2022/09/06 11:36:46 Lari Hotari wrote:
>> I asked for an update on the Apache org GitHub Actions usage stats from 
>> Gavin McDonald on the-asf slack in this thread: 
>> https://the-asf.slack.com/archives/CBX4TSBQ8/p1662464113873539?thread_ts=1661512133.913279&cid=CBX4TSBQ8
>>  .
>> 
>> I hope we get this issue resolved since it delays PR processing a lot.
>> 
>> -Lari
>> 
>> On 2022/09/06 11:16:07 Lari Hotari wrote:
>>> Pulsar CI continues to be congested, and the build queue [1] is very long 
>>> at the moment. There are 147 build jobs in the queue and 16 jobs in 
>>> progress at the moment.
>>> 
>>> I would strongly advice everyone to use "personal CI" to mitigate the issue 
>>> of the long delay of CI feedback. You can simply open a PR to your own 
>>> personal fork of apache/pulsar to run the builds in your "personal CI". 
>>> There's more details in the previous emails in this thread.
>>> 
>>> -Lari
>>> 
>>> [1] - build queue: 
>>> https://github.com/apache/pulsar/actions?query=is%3Aqueued
>>> 
>>> On 2022/08/30 12:39:19 Lari Hotari wrote:
 Pulsar CI continues to be congested, and the build queue is long.
 
 I would strongly advice everyone to use "personal CI" to mitigate the 
 issue of the long delay of CI feedback. You can simply open a PR to your 
 own personal fork of apache/pulsar to run the builds in your "personal 
 CI". There's more details in the previous email in this thread.
 
 Some updates:
 
 There has been a discussion with Gavin McDonald from ASF infra on the-asf 
 slack about getting usage reports from GitHub to support the 
 investigation. Slack thread is the same one mentioned in the previous 
 email, https://the-asf.slack.com/archives/CBX4TSBQ8/p1661512133913279 . 
 Gavin already requested the usage report in GitHub UI, but it produced 
 invalid results.
 
 I made a change to mitigate a source of additional GitHub Actions 
 overhead. 
 In the past, each cherry-picked commit to a maintenance branch of Pulsar 
 has triggered a lot of workflow runs. 
 
 The solution for cancelling duplicate builds automatically is to add this 
 definition to the workflow definition:
 concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true
 
 I added this to all maintenance branch GitHub Actions workflows:
 
 branch-2.10 change:
 https://github.com/apache/pulsar/commit/5d2c9851f4f4d70bfe74b1e683a41c5a040a6ca7
 branch-2.9 change:
 https://github.com/apache/pulsar/commit/3ea124924fecf636cc105de75c62b3a99050847b
 branch-2.8 change:
 https://github.com/apache/pulsar/commit/48187bb5d95e581f8322a019b61d986e18a31e54
 branch-2.7:
 https://github.com/apache/pulsar/commit/744b62c99344724eacdbe97c881311869d67f630
 
 branch-2.11 already contains the necessary config for cancelling duplicate 
 builds.
 
 The benefit of the above change is that when multiple commits are 
 cherry-picked to a branch at once, only the build of the last commit will 
 get run eventually. The builds for the intermediate commits will get 
 cancelled. Obviously there's a tradeoff here that we don't get the 
 information if one of the earlier commits breaks the build. It's the cost 
 that we need to pay. Nevertheless our build is so flaky that it's hard to 
 determine whether a failed build result is only caused by bad flaky test 
 or whether it's an actual failure. Because of this we don't lose anything 
 by cancelling builds. It's more important to save build resources. In the 
 maintenance branches for 2.10 and older, the average total build time 
 consumed is around 20 hours which is a lot.
 
 At this time, the overhead of maintenance branch builds doesn't seem to be 
 the source of the problems. There must be some other issue which is 
 possibly related to exceeding a usage quota. Hopefully we get the CI 
 slowness issue solved asap.
 
 BR,
 
 Lari
 
 
 On 2022/08/26 12:00:20 Lari Hotari wrote:
> Hi,
> 
> GitHub Actions builds have been piling up in the build queue in the last 
> few days.
> I posted on bui...@apache.org 
> https://lists.apache.org/thread/6lbqr0f6mqt9s8ggollp5kj2nv7rlo9s a

Re: [DISCUSS] Homebrew libpulsar

[Dave Fisher]

Hi -

I’ve played with Homebrew before and from the git log I can see that Matteo 
created the initial version, but for the most part the Homebrew community has 
maintained the script.

Here is the current script:

Formula/libpulsar.rb
class Libpulsar < Formula
  desc "Apache Pulsar C++ library"
  homepage "https://pulsar.apache.org/";
  url 
"https://www.apache.org/dyn/closer.cgi?path=pulsar/pulsar-2.4.2/apache-pulsar-2.4.2-src.tar.gz";
  sha256 "4b543932db923aa135c4d54b9122bcbdfc67bd73de641f9fffbc9a4ddf3430ae"
  revision 1

  bottle do
cellar :any
sha256 "13c37b77dee18f4bba454484b4426a6f3dad27f902e0793a56a6358897ab4f3f" 
=> :catalina
sha256 "ff0090b77842ac6034c4a425438f8d5b401164da4bada7ac11c4963dcdaa3a28" 
=> :mojave
sha256 "d73f612edf73a351ab7f90776c24670c4c77b85fd95e37999e863ff27daf89ef" 
=> :high_sierra
  end

  depends_on "cmake" => :build
  depends_on "pkg-config" => :build
  depends_on "boost"
  depends_on "openssl@1.1"
  depends_on "protobuf"
  depends_on "snappy"
  depends_on "zstd"

  def install
cd "pulsar-client-cpp" do
  system "cmake", ".", *std_cmake_args,
  "-DBUILD_TESTS=OFF",
  "-DBUILD_PYTHON_WRAPPER=OFF",
  "-DBoost_INCLUDE_DIRS=#{Formula["boost"].include}",
  "-DProtobuf_INCLUDE_DIR=#{Formula["protobuf"].include}",
  
"-DProtobuf_LIBRARIES=#{Formula["protobuf"].lib}/libprotobuf.dylib"
  system "make", "pulsarShared", "pulsarStatic"
  system "make", "install"
end
  end

  test do
(testpath/"test.cc").write <<~EOS
  #include 

  int main (int argc, char **argv) {
pulsar::Client client("pulsar://localhost:6650");
return 0;
  }
EOS
system ENV.cxx, "test.cc", "-L#{lib}", "-lpulsar", "-o", "test"
system "./test"
  end
end

Here is the log w/o the bottling.

% git log master -- Formula/libpulsar.rb
commit e4ed6752b0cab4c98ae72f81b2bd7160412bb938
Author: FX Coudert 
Date:   Thu Dec 26 16:17:50 2019 +0100

libpulsar: revision for boost

commit 07bd5db82526c43a0e5004f690f3afbda4b89c7c
Author: Matteo Merli 
Date:   Thu Dec 19 11:38:56 2019 -0800

libpulsar 2.4.2

Closes #48088.

Signed-off-by: FX Coudert 

commit 3e6adba69f61da1b6fee5ba49199d299878f0db1
Author: Austin Shalit 
Date:   Wed Nov 27 23:32:07 2019 -0800

libpulsar: revision bump

commit fc6e1f2d6179379740466631eb0dffd56b404f5c
Author: Rui Chen 
Date:   Thu Oct 3 13:05:09 2019 -0400

libpulsar: revision for protobuf

commit 11b3aa3dbe8a76716784cda9f351a28b49e9dfd0
Author: Henry Fredrick Schreiner 
Date:   Mon Sep 2 20:47:07 2019 -0400

libpulsar 2.4.1: revision for boost 1.71.0

commit 0d14181d6cbdd044eeec9b78e1d4b7d766ff5f1d
Author: Henry Fredrick Schreiner 
Date:   Sun Sep 1 19:56:04 2019 -0400

libpulsar: Keep from linking to snappy

Closes #43761.

Signed-off-by: FX Coudert 

commit d9288916f84c02730c9ba800eb12550cb1b5b865
Author: FX Coudert 
Date:   Thu Aug 29 10:28:00 2019 +0200

libpulsar: move to OpenSSL 1.1

commit 84d41c55af723989ccfd61161fc014c69f62da5c
Author: Matteo Merli 
Date:   Wed Aug 28 11:26:50 2019 -0700

libpulsar 2.4.0

Closes #43555.

Signed-off-by: FX Coudert 

commit 7e24d72cbe366889612c2acc9d230d7263db4122
Author: Chongyu Zhu 
Date:   Wed May 29 15:24:44 2019 +0800

libpulsar: revision for protobuf

commit 42a18870f92e42a787692c840d00cb5731ae6a5c
Author: FX Coudert 
Date:   Fri Apr 12 21:19:13 2019 +0200

libpulsar: revision and patch for boost

commit 167f3474dd8ca620ab3e9d12513e88e499fe1ca2
Author: Steven Peters 
Date:   Sat Jun 1 15:36:17 2019 -0700

libpulsar 2.3.2

Closes #40591.

Signed-off-by: Steven Peters 

commit 1b3b8c15319503ff8a33e9627c7a46d5f8d2546f
Author: FX Coudert 
Date:   Mon Apr 15 17:40:36 2019 +0200

libpulsar 2.3.1

Closes #38947.

Signed-off-by: FX Coudert 

commit 3c0b62cc0288f6d5dfb0ed9f682d6e245d1e3cfe
Author: Steven Peters 
Date:   Mon Mar 4 12:41:16 2019 -0800

libpulsar: revision for boost

commit cac1d53d2254004b2082a2d501f772fcaf77b8c0
Author: Chongyu Zhu 
Date:   Sat Mar 2 23:50:28 2019 +0800

libpulsar: revision for protobuf

commit fcc845573eef8564c7fac9ddba8a9e660b2d06f7
Author: Matteo Merli 
Date:   Wed Feb 20 22:32:16 2019 -0800

libpulsar 2.3.0

Closes #37151.

Signed-off-by: FX Coudert 

commit 24c06c4c49374136d484ef8a000b1c1626f2bf79
Author: Viktor Szakats 
Date:   Fri Jan 18 01:50:31 2019 +

libpulsar: use common/short form of apache url

commit c2f34710ae2d378fbea076661016b87b31ce35af
Author: Matteo Merli 
Date:   Wed Oct 31 15:28:10 2018 -0700

libpulsar 2.2.1 (new formula)

Closes #35809.

Signed-off-by: FX Coudert 

> On Jan 20, 2020, at 1:19 PM, Sijie Guo  wrote:
> 
> Hi all,
> 
> It is my first time to realize that there is a libpulsar entry in homebrew.
> (https://github.com/apache/pulsar/issues/6091). A

Re: [VOTE] Pulsar Client Go Release 0.1.0 Candidate 1

[Dave Fisher]

-1 - What artifacts are we voting on? Where is the source package on the Apache 
servers?

Packages here https://github.com/apache/pulsar-client-go/releases are not 
immutable. There are no checksums or signatures. No update to the KEYS file.

Please put release candidates here: 
https://dist.apache.org/repos/dist/dev/pulsar/ in there own folder.

Regards,
Dave

> On Mar 26, 2020, at 9:00 AM, Matteo Merli  wrote:
> 
> +1
> 
> 
> --
> Matteo Merli
> 
> 
> On Thu, Mar 26, 2020 at 4:52 AM Yong Zhang  wrote:
>> 
>> +1
>> 
>> Thanks
>> Yong
>> 
>> On Thu, 26 Mar 2020 at 19:44, PengHui Li  wrote:
>> 
>>> +1
>>> 
>>> Sijie Guo  于2020年3月26日周四 上午8:28写道：
>>> 
 +1
 
 On Tue, Mar 24, 2020 at 11:26 PM anonymitaet _ >>> 
 wrote:
 
> +1
> 
> Thanks xiaolong for your great work
> 
> On 2020/3/25, 13:43, "xiaolong ran"  wrote:
> 
>Hi everyone,
> 
>Please review and vote on the release candidate #1 for the version
> 0.1.0, as follows:
>[ ] +1, Approve the release
>[ ] -1, Do not approve the release (please provide specific
>>> comments)
> 
>This is the first release candidate for Apache Pulsar Go client,
> version 0.1.0.
> 
>It fixes the following issues:
> 
>https://github.com/apache/pulsar-client-go/milestone/1?closed=1 <
> https://github.com/apache/pulsar-client-go/milestone/1?closed=1>
> 
>Please download the source packages and review this release
 candidate:
> 
>- Review release notes
>- Download the source package and follow the README.md to build and
> run the pulsar-client-go.
> 
>The vote will be open for at least 72 hours. It is adopted by
 majority
> approval, with at least 3 PMC affirmative votes.
> 
>Source file:
>https://github.com/apache/pulsar-client-go/tree/v0.1.0-candidate-1
>>> <
> https://github.com/apache/pulsar-client-go/tree/v0.1.0-candidate-1>
> 
> 
> 
> 
> 
 
>>> 

Re: [VOTE] Pulsar Client Go Release 0.1.0 Candidate 1

[Dave Fisher]

Hi -

There is an official release policy - 
http://www.apache.org/legal/release-policy.html#policy

It does not include GitHub as a source of official releases. 

Releases are acts of the Foundation through each PMC.

Perhaps that is something to be discussed on legal-discuss@, but that would 
mean a change to a well established Foundation Policy and would breach the 
legal shield which protects you as VP, the PMC and the Release Manager.

Regards,
Dave

> On Mar 26, 2020, at 10:41 AM, Matteo Merli  wrote:
> 
> Hi Dave,
> 
> we discussed in past to only release a Git tag for Go. All the users
> of this library will just fetch the library directly from Github,
> specifying a tag in their Go dependencies tool.
> 
> While we could publish a source tar.gz, it would be of little
> practical utility to users.
> 
> Having said that, we'd need to specify the Git tag hash, both here in
> the vote thread as in the release notes.
> 
> Dave, do you thing that this would be an acceptable way to "release" a
> blessed tag?
> --
> Matteo Merli
> 
> 
> 
> On Thu, Mar 26, 2020 at 9:26 AM Dave Fisher  wrote:
>> 
>> -1 - What artifacts are we voting on? Where is the source package on the 
>> Apache servers?
>> 
>> Packages here https://github.com/apache/pulsar-client-go/releases are not 
>> immutable. There are no checksums or signatures. No update to the KEYS file.
>> 
>> Please put release candidates here: 
>> https://dist.apache.org/repos/dist/dev/pulsar/ in there own folder.
>> 
>> Regards,
>> Dave
>> 
>>> On Mar 26, 2020, at 9:00 AM, Matteo Merli  wrote:
>>> 
>>> +1
>>> 
>>> 
>>> --
>>> Matteo Merli
>>> 
>>> 
>>> On Thu, Mar 26, 2020 at 4:52 AM Yong Zhang  
>>> wrote:
>>>> 
>>>> +1
>>>> 
>>>> Thanks
>>>> Yong
>>>> 
>>>> On Thu, 26 Mar 2020 at 19:44, PengHui Li  wrote:
>>>> 
>>>>> +1
>>>>> 
>>>>> Sijie Guo  于2020年3月26日周四 上午8:28写道：
>>>>> 
>>>>>> +1
>>>>>> 
>>>>>> On Tue, Mar 24, 2020 at 11:26 PM anonymitaet _ >>>>> 
>>>>>> wrote:
>>>>>> 
>>>>>>> +1
>>>>>>> 
>>>>>>> Thanks xiaolong for your great work
>>>>>>> 
>>>>>>> On 2020/3/25, 13:43, "xiaolong ran"  wrote:
>>>>>>> 
>>>>>>>   Hi everyone,
>>>>>>> 
>>>>>>>   Please review and vote on the release candidate #1 for the version
>>>>>>> 0.1.0, as follows:
>>>>>>>   [ ] +1, Approve the release
>>>>>>>   [ ] -1, Do not approve the release (please provide specific
>>>>> comments)
>>>>>>> 
>>>>>>>   This is the first release candidate for Apache Pulsar Go client,
>>>>>>> version 0.1.0.
>>>>>>> 
>>>>>>>   It fixes the following issues:
>>>>>>> 
>>>>>>>   https://github.com/apache/pulsar-client-go/milestone/1?closed=1 <
>>>>>>> https://github.com/apache/pulsar-client-go/milestone/1?closed=1>
>>>>>>> 
>>>>>>>   Please download the source packages and review this release
>>>>>> candidate:
>>>>>>> 
>>>>>>>   - Review release notes
>>>>>>>   - Download the source package and follow the README.md to build and
>>>>>>> run the pulsar-client-go.
>>>>>>> 
>>>>>>>   The vote will be open for at least 72 hours. It is adopted by
>>>>>> majority
>>>>>>> approval, with at least 3 PMC affirmative votes.
>>>>>>> 
>>>>>>>   Source file:
>>>>>>>   https://github.com/apache/pulsar-client-go/tree/v0.1.0-candidate-1
>>>>> <
>>>>>>> https://github.com/apache/pulsar-client-go/tree/v0.1.0-candidate-1>
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>> 

Re: [VOTE] Pulsar Client Go Release 0.1.0 Candidate 1

[Dave Fisher]

Hi Sijie,

Good plan. Pulsar should keep doing what it is doing, but don’t call these 
clients “official”.

The answer may vary with the client technology. For example maven archives do 
go through an official repository of Apache’s. NPMs are different and are 
distributed in different ways. Still a release package is required in the 
Apache Distribution Repos and Archives. This is so that these can always be 
found - even 21 years later.

Regards,
Dave

> On Mar 26, 2020, at 11:47 AM, Sijie Guo  wrote:
> 
> Hi Dave,
> 
> IMO there are no "artifacts" to release for a Golang client. Go client
> works with git tags.
> 
> I think the policy should be updated to reflect the process for releasing
> clients that don't have "artifacts".
> 
> I created a LEGAL issue and starting an email discussion on legal-discuss@
> 
> https://issues.apache.org/jira/browse/LEGAL-512
> 
> - Sijie
> 
> 
> On Thu, Mar 26, 2020 at 11:29 AM Dave Fisher  wrote:
> 
>> Hi -
>> 
>> There is an official release policy -
>> http://www.apache.org/legal/release-policy.html#policy
>> 
>> It does not include GitHub as a source of official releases.
>> 
>> Releases are acts of the Foundation through each PMC.
>> 
>> Perhaps that is something to be discussed on legal-discuss@, but that
>> would mean a change to a well established Foundation Policy and would
>> breach the legal shield which protects you as VP, the PMC and the Release
>> Manager.
>> 
>> Regards,
>> Dave
>> 
>>> On Mar 26, 2020, at 10:41 AM, Matteo Merli  wrote:
>>> 
>>> Hi Dave,
>>> 
>>> we discussed in past to only release a Git tag for Go. All the users
>>> of this library will just fetch the library directly from Github,
>>> specifying a tag in their Go dependencies tool.
>>> 
>>> While we could publish a source tar.gz, it would be of little
>>> practical utility to users.
>>> 
>>> Having said that, we'd need to specify the Git tag hash, both here in
>>> the vote thread as in the release notes.
>>> 
>>> Dave, do you thing that this would be an acceptable way to "release" a
>>> blessed tag?
>>> --
>>> Matteo Merli
>>> 
>>> 
>>> 
>>> On Thu, Mar 26, 2020 at 9:26 AM Dave Fisher  wrote:
>>>> 
>>>> -1 - What artifacts are we voting on? Where is the source package on
>> the Apache servers?
>>>> 
>>>> Packages here https://github.com/apache/pulsar-client-go/releases are
>> not immutable. There are no checksums or signatures. No update to the KEYS
>> file.
>>>> 
>>>> Please put release candidates here:
>> https://dist.apache.org/repos/dist/dev/pulsar/ in there own folder.
>>>> 
>>>> Regards,
>>>> Dave
>>>> 
>>>>> On Mar 26, 2020, at 9:00 AM, Matteo Merli  wrote:
>>>>> 
>>>>> +1
>>>>> 
>>>>> 
>>>>> --
>>>>> Matteo Merli
>>>>> 
>>>>> 
>>>>> On Thu, Mar 26, 2020 at 4:52 AM Yong Zhang 
>> wrote:
>>>>>> 
>>>>>> +1
>>>>>> 
>>>>>> Thanks
>>>>>> Yong
>>>>>> 
>>>>>> On Thu, 26 Mar 2020 at 19:44, PengHui Li  wrote:
>>>>>> 
>>>>>>> +1
>>>>>>> 
>>>>>>> Sijie Guo  于2020年3月26日周四 上午8:28写道：
>>>>>>> 
>>>>>>>> +1
>>>>>>>> 
>>>>>>>> On Tue, Mar 24, 2020 at 11:26 PM anonymitaet _ <
>> anonymita...@hotmail.com
>>>>>>>> 
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> +1
>>>>>>>>> 
>>>>>>>>> Thanks xiaolong for your great work
>>>>>>>>> 
>>>>>>>>> On 2020/3/25, 13:43, "xiaolong ran" 
>> wrote:
>>>>>>>>> 
>>>>>>>>>  Hi everyone,
>>>>>>>>> 
>>>>>>>>>  Please review and vote on the release candidate #1 for the
>> version
>>>>>>>>> 0.1.0, as follows:
>>>>>>>>>  [ ] +1, Approve the release
>>>>>>>>>  [ ] -1, Do not approve the release (please provide specific
>>>>>>> comments)
>>>>>>>>> 
>>>>>>>>>  This is the first release candidate for Apache Pulsar Go client,
>>>>>>>>> version 0.1.0.
>>>>>>>>> 
>>>>>>>>>  It fixes the following issues:
>>>>>>>>> 
>>>>>>>>>  https://github.com/apache/pulsar-client-go/milestone/1?closed=1
>> <
>>>>>>>>> https://github.com/apache/pulsar-client-go/milestone/1?closed=1>
>>>>>>>>> 
>>>>>>>>>  Please download the source packages and review this release
>>>>>>>> candidate:
>>>>>>>>> 
>>>>>>>>>  - Review release notes
>>>>>>>>>  - Download the source package and follow the README.md to build
>> and
>>>>>>>>> run the pulsar-client-go.
>>>>>>>>> 
>>>>>>>>>  The vote will be open for at least 72 hours. It is adopted by
>>>>>>>> majority
>>>>>>>>> approval, with at least 3 PMC affirmative votes.
>>>>>>>>> 
>>>>>>>>>  Source file:
>>>>>>>>> 
>> https://github.com/apache/pulsar-client-go/tree/v0.1.0-candidate-1
>>>>>>> <
>>>>>>>>> https://github.com/apache/pulsar-client-go/tree/v0.1.0-candidate-1
>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>> 
>> 
>> 

Re: [VOTE] Pulsar Client Go Release 0.1.0 Candidate 1

[Dave Fisher]

Sent from my iPhone

> On Mar 26, 2020, at 1:28 PM, Sijie Guo  wrote:
> 
> Hi Dave,
> 
> Thank you for your response!
> 
> I know there would be no immediate conclusion about the release policy of
> golang client since it seems that it requires some discussions on the legal
> side.
> 
> We can publish a tarball of the released source code and distribute it to
> the ASF svn dist repo for each release.
> 
> Does that make sense?

Yes. That is exactly what should be done for all the clients. The download page 
can point to both that package as well as the expected language approved way. 
This gives Pulsar’s users the ability to validate what’s on the official 
channel vs. the platform channel. The source is open forever.

Thanks,
Dave

> 
> - Sijie
> 
> 
>> On Thu, Mar 26, 2020 at 11:59 AM Dave Fisher  wrote:
>> 
>> Hi Sijie,
>> 
>> Good plan. Pulsar should keep doing what it is doing, but don’t call these
>> clients “official”.
>> 
>> The answer may vary with the client technology. For example maven archives
>> do go through an official repository of Apache’s. NPMs are different and
>> are distributed in different ways. Still a release package is required in
>> the Apache Distribution Repos and Archives. This is so that these can
>> always be found - even 21 years later.
>> 
>> Regards,
>> Dave
>> 
>>>> On Mar 26, 2020, at 11:47 AM, Sijie Guo  wrote:
>>> 
>>> Hi Dave,
>>> 
>>> IMO there are no "artifacts" to release for a Golang client. Go client
>>> works with git tags.
>>> 
>>> I think the policy should be updated to reflect the process for releasing
>>> clients that don't have "artifacts".
>>> 
>>> I created a LEGAL issue and starting an email discussion on
>> legal-discuss@
>>> 
>>> https://issues.apache.org/jira/browse/LEGAL-512
>>> 
>>> - Sijie
>>> 
>>> 
>>>> On Thu, Mar 26, 2020 at 11:29 AM Dave Fisher  wrote:
>>> 
>>>> Hi -
>>>> 
>>>> There is an official release policy -
>>>> http://www.apache.org/legal/release-policy.html#policy
>>>> 
>>>> It does not include GitHub as a source of official releases.
>>>> 
>>>> Releases are acts of the Foundation through each PMC.
>>>> 
>>>> Perhaps that is something to be discussed on legal-discuss@, but that
>>>> would mean a change to a well established Foundation Policy and would
>>>> breach the legal shield which protects you as VP, the PMC and the
>> Release
>>>> Manager.
>>>> 
>>>> Regards,
>>>> Dave
>>>> 
>>>>> On Mar 26, 2020, at 10:41 AM, Matteo Merli  wrote:
>>>>> 
>>>>> Hi Dave,
>>>>> 
>>>>> we discussed in past to only release a Git tag for Go. All the users
>>>>> of this library will just fetch the library directly from Github,
>>>>> specifying a tag in their Go dependencies tool.
>>>>> 
>>>>> While we could publish a source tar.gz, it would be of little
>>>>> practical utility to users.
>>>>> 
>>>>> Having said that, we'd need to specify the Git tag hash, both here in
>>>>> the vote thread as in the release notes.
>>>>> 
>>>>> Dave, do you thing that this would be an acceptable way to "release" a
>>>>> blessed tag?
>>>>> --
>>>>> Matteo Merli
>>>>> 
>>>>> 
>>>>> 
>>>>> On Thu, Mar 26, 2020 at 9:26 AM Dave Fisher  wrote:
>>>>>> 
>>>>>> -1 - What artifacts are we voting on? Where is the source package on
>>>> the Apache servers?
>>>>>> 
>>>>>> Packages here https://github.com/apache/pulsar-client-go/releases are
>>>> not immutable. There are no checksums or signatures. No update to the
>> KEYS
>>>> file.
>>>>>> 
>>>>>> Please put release candidates here:
>>>> https://dist.apache.org/repos/dist/dev/pulsar/ in there own folder.
>>>>>> 
>>>>>> Regards,
>>>>>> Dave
>>>>>> 
>>>>>>> On Mar 26, 2020, at 9:00 AM, Matteo Merli  wrote:
>>>>>>> 
>>>>>>> +1
>>>>>>> 
>>>>>>> 
>>>>>>> --
>>>>>>&

Re: [DISCUSS] Protobuf 2.4.1

[Dave Fisher]

> On May 22, 2020, at 10:52 AM, Sijie Guo  wrote:
> 
> Hi all,
> 
> I would like to kick off a discussion about what we should do for protobuf
> 2.4.1 and figure out what is the long term plan.
> 
> Currently, Pulsar is using a customized version of protobuf 2.4.1 for wire
> protocol. The customization was made to leverage netty object pooling to
> reduce object generation and cause bad JVM GC behavior. However, protobuf
> 2.4.1 is a many-years old release and there is also a vulnerability issue
> reported for protobuf-java 2.4.1 (
> https://nvd.nist.gov/vuln/detail/CVE-2015-5237). We need to figure out a
> plan to upgrade this dependency.
> 
> Here are some thoughts. Happy to see what others think.
> 
> a) We upgrade to protobuf 3.x without customization.
> 
> - Pros: be able to catch up with changes in protobuf and avoid any
> vulnerabilities.
> - Cons: we might have to suffer the pains from object generations. But it
> might not be too bad since we don't store the message payloads in protobuf.
> 
> b) We upgrade to protobuf 3.x with new customization.
> 
> - Pros: We are able to control object generations.
> - Cons: Maintaining a customization makes harder to catch up changes in
> protobuf community and avoid vulnerabilities.

Is it possible that the protobuf community would want us to upstream the 
customizations?

Regards,
Dave

> 
> c) Look for other alternatives to protobuf such as flatbuffers.
> 
> - Cons: it is not a backward-compatible solution with existing Pulsar
> clients.
> 
> - Sijie

Re: [DISCUSS] PIP-67: Contribute Pulsarctl to Apache Pulsar

[Dave Fisher]

Very cool!

This contribution may need to go through IP clearance. 
https://incubator.apache.org/ip-clearance/

Regards?
Dave

Sent from my iPhone

> On Jun 18, 2020, at 8:16 PM, Yong Zhang  wrote:
> 
> It seems that there are no objections to accepting this repo.
> 
> What should I do in the next step? Should I open a vote in a separate email?
> 
> Thanks
> 
> ---
>> On Mon, 15 Jun 2020 at 14:30, Guangning E  wrote:
>> 
>> Great job, +1, Thanks Yong and Xiaolong.
>> 
>> anonymitaet _  于2020年6月15日周一 上午9:56写道：
>> 
>>> Thanks Yong for this great job, +1
>>> 
>>> On 2020/6/15, 09:34, "PengHui Li"  wrote:
>>> 
>>>+1
>>>On Jun 15, 2020, 9:24 AM +0800, xiaolong ran <
>> ranxiaolong...@gmail.com>,
>>> wrote:
 Thanks Yong Zhang work for this, LGTM+1
 
 --
 Thanks
 Xiaolong Ran
 
> 在 2020年6月12日，上午11:38，Yong Zhang  写道：
> 
> Hi all,
> 
> We have developed a new tool Pulsarctl for managing Pulsar
>>> resources. It’s
> written in Golang
> and is an alternative tool of pulsar-admin. We’d like to
>>> contribute the
> project back to the Pulsar
> community.
> 
> ## Motivation
> 
> Pulsarctl is an alternative tool of pulsar-admin, used to manage
>>> resources
> in Apache Pulsar.
> Pulsarctl is written in Go and based on Pulsar REST API. It
>>> provides Go
> developers with API
> interface and user-friendly commands, making it easier to
>> interact
>>> with
> Pulsar brokers.
> 
> Compared with pulsar-admin, Pulsarctl is more user-friendly.
>>> Pulsarctl
> requires fewer dependencies
> to use commands and provides more comprehensive description and
>>> usage for
> commands.
> With Pulsarctl, users can find and resolve issues faster when
>>> errors occur.
> 
> ## Features
> 
> Pulsarctl not only integrates the Pulsar commands and BookKeeper
>>> commands
> but also
> provides some useful tools like output format and context
>>> management.
> 
> 
> ### Pulsar operations
> 
> For Pulsar operations, Pulsarctl integrates almost all the
>>> commands of
> pulsar-admin, including
> but not limited to the following operations:
> 
> - broker operations
> - cluster operations
> - tenant operations
> - namespace operations
> - topic operations
> - function operations
> - sink operations
> - source operations
> 
> Also, Pulsarctl supports the JWT(JSON Web Token) authentication
>>> and TLS
> authentication.
> 
> The following are command flags of Pulsarctl. You can use
>>> `--auth-params`
> to specify the auth
> params configured in pulsar-client. Or you can just specify
>>> `--token` to
> use that token to connect the broker.
> 
> ```
> Common flags:
> -s, --admin-service-url string The admin web service url that
> pulsarctl connects to. (default "http://localhost:8080";)
> --auth-params string Authentication parameters are used to
> configure the public and private key files required by tls
> For example:
> "tlsCertFile:val1,tlsKeyFile:val2"
> -C, --color string toggle colorized logs
> (true,false,fabulous) (default "true")
> -h, --help help for this command
> --tls-allow-insecure Allow TLS insecure connection
> --tls-trust-cert-path string Allow TLS trust cert file path
> --token string Using the token to authentication
> --token-file string Using the token file to authentication
> -v, --verbose int set log level, use 0 to silence, 4 for
> debugging (default 3)
> ```
> 
> ### BookKeeper operations
> 
> For BookKeeper operations, Pulsarctl integrates the commands
>>> listed in the
> [REST API](https://bookkeeper.apache.org/docs/4.10.0/admin/http/
>> )
> 
> - auto-recovery operations
> - bookie operations
> - bookies operations
> - ledger operations
> 
> 
> ### Save your configuration in the different contexts
> 
> Pulsarctl provides a context command which lets you manage your
>>> Pulsar
> cluster easier.
> The ‘context’ can save the different configurations of your
>> Pulsar
>>> cluster.
> And you can easily
> change to another cluster by the command `pulsarctl context use`.
>>> For more
> details please
> refer to
> 
>>> 
>> https://github.com/streamnative/pulsarctl/blob/master/docs/en/how-to-use-context.md
> 
> 
> ### Get different formats of the output
> 
> Pulsarctl provides an output flag `--output` to make the output
>>> transform
> into different
> formats, such as text, JSON, and YAML. The default view is the
>>> text which
> allows you
> to check the resources in your cluster directly. Also, you can
>> get
>>> the JSON
> or YAML
> format to do what you want to.
> 
> 
> ### Extend Pulsarctl using plugins easily
> 
> You can easily extend the Pulsarctl with a p

Fwd: [IMPORTANT] - Migration to ci-builds.a.o

[Dave Fisher]

Hi -

There is a critical migration for Jenkins builds that the project is missing.

Several PMC members ought to be subscribed to bui...@apache.org.

Please act quickly.

Regards,
Dave

> Begin forwarded message:
> 
> From: Gavin McDonald 
> Subject: [IMPORTANT] - Migration to ci-builds.a.o
> Date: July 16, 2020 at 9:33:08 AM PDT
> To: builds 
> Reply-To: bui...@apache.org
> Reply-To: gmcdon...@apache.org
> 
> Hi All,
> 
> This NOTICE is for everyone on builds.apache.org. We are migrating to a new
> Cloudbees based Client Master called https://ci-builds.apache.org. The
> migrations of all jobs needs to be done before the switch off date of 15th
> August 2020, so you have a maximum of 4 weeks.
> 
> There is no tool or automated way of migrating your jobs, the
> differences in the platforms, the plugins and the setup makes it impossible
> to do in a safe way. So, you all need to start creating new jobs on
> ci-infra.a.o and then , when you are happy, turn off your old builds on
> builds.a.o.
> 
> There are currently 4 agents over there ready to take jobs, plus a floating
> agent which is shared amongst many masters (more to come). I will migrate
> away 2 more agents from builds.a.o to ci-builds.a.o every few days, and
> will keep an eye of load across both and adjust accordingly.
> 
> If needed, create a ticket on INFRA jira for any issues that crop up, or
> email here on builds@a.o. there may be one or two plugins we need to
> install/tweak etc.
> 
> We will be not using 'Views' at the top level, but rather we will take
> advantage of 'Folders Plus' - each project will get its own Folder and have
> authorisation access to create/edit jobs etc within that folder. I will
> create these folders as you ask for them to start with. This setup allows
> for credentials isolation amongst other benefits, including but not limited
> to exclusive agents (Controlled Agents) for your own use , should you have
> any project targeted donations of agents.
> 
> As with other aspects of the ASF, projects can choose to just enable all
> committers access to their folder, just ask.
> 
> We will re-use builds.apache.org as a CNAME to ci-builds.a.o but will not
> be setting up any forwarding rules or anything like that.
> 
> So, please, get started *now *on this so you can be sure we are all
> completed before the final cutoff date of 15th August 2020.
> 
> Any questions - I expect a few (dozen :) ) - ask away and/or file INFRA
> tickets.
> 
> Hadoop and related projects have their own migration path to follow, same
> cut off date, Cassandra, Beam, CouchDB have already migrated and are doing
> very well in their new homes.
> 
> Lets get going ...
> 
> -- 
> 
> *Gavin McDonald*
> Systems Administrator
> ASF Infrastructure Team

Re: [IMPORTANT] - Migration to ci-builds.a.o

[Dave Fisher]

Migration technique: 
https://cwiki.apache.org/confluence/display/INFRA/Migrating+jobs+from+Jenkins+to+Cloudbees

Helpful info: 
https://cwiki.apache.org/confluence/display/INFRA/Kicking+off+a+Jenkins+build+with+a+GitHub+PR


> On Aug 14, 2020, at 12:48 PM, Dave Fisher  wrote:
> 
> Hi -
> 
> There is a critical migration for Jenkins builds that the project is missing.
> 
> Several PMC members ought to be subscribed to bui...@apache.org.
> 
> Please act quickly.
> 
> Regards,
> Dave
> 
>> Begin forwarded message:
>> 
>> From: Gavin McDonald 
>> Subject: [IMPORTANT] - Migration to ci-builds.a.o
>> Date: July 16, 2020 at 9:33:08 AM PDT
>> To: builds 
>> Reply-To: bui...@apache.org
>> Reply-To: gmcdon...@apache.org
>> 
>> Hi All,
>> 
>> This NOTICE is for everyone on builds.apache.org. We are migrating to a new
>> Cloudbees based Client Master called https://ci-builds.apache.org. The
>> migrations of all jobs needs to be done before the switch off date of 15th
>> August 2020, so you have a maximum of 4 weeks.
>> 
>> There is no tool or automated way of migrating your jobs, the
>> differences in the platforms, the plugins and the setup makes it impossible
>> to do in a safe way. So, you all need to start creating new jobs on
>> ci-infra.a.o and then , when you are happy, turn off your old builds on
>> builds.a.o.
>> 
>> There are currently 4 agents over there ready to take jobs, plus a floating
>> agent which is shared amongst many masters (more to come). I will migrate
>> away 2 more agents from builds.a.o to ci-builds.a.o every few days, and
>> will keep an eye of load across both and adjust accordingly.
>> 
>> If needed, create a ticket on INFRA jira for any issues that crop up, or
>> email here on builds@a.o. there may be one or two plugins we need to
>> install/tweak etc.
>> 
>> We will be not using 'Views' at the top level, but rather we will take
>> advantage of 'Folders Plus' - each project will get its own Folder and have
>> authorisation access to create/edit jobs etc within that folder. I will
>> create these folders as you ask for them to start with. This setup allows
>> for credentials isolation amongst other benefits, including but not limited
>> to exclusive agents (Controlled Agents) for your own use , should you have
>> any project targeted donations of agents.
>> 
>> As with other aspects of the ASF, projects can choose to just enable all
>> committers access to their folder, just ask.
>> 
>> We will re-use builds.apache.org as a CNAME to ci-builds.a.o but will not
>> be setting up any forwarding rules or anything like that.
>> 
>> So, please, get started *now *on this so you can be sure we are all
>> completed before the final cutoff date of 15th August 2020.
>> 
>> Any questions - I expect a few (dozen :) ) - ask away and/or file INFRA
>> tickets.
>> 
>> Hadoop and related projects have their own migration path to follow, same
>> cut off date, Cassandra, Beam, CouchDB have already migrated and are doing
>> very well in their new homes.
>> 
>> Lets get going ...
>> 
>> -- 
>> 
>> *Gavin McDonald*
>> Systems Administrator
>> ASF Infrastructure Team
> 

Re: [IMPORTANT] - Migration to ci-builds.a.o

[Dave Fisher]

Hi -

It is critical to complete this migration THIS WEEK.

I’ve submitted https://issues.apache.org/jira/browse/INFRA-20720 to request the 
folder.

Pulsar has the following “projects” on builds.a.o.

pulsar-manager-build
pulsar-master
pulsar-pull-request-c++-test
pulsar-release-binaries
pulsar-website-build
pulsar_precommit_client_node
pulsar_release_nightly_snapshot

I can migrate the projects so that are not lost, but I have no idea about the 
integrations with GitHub and any missing Jenkins Plugins.

These all need to be migrated by Friday EOD.

Regards,
Dave


> On Aug 14, 2020, at 1:35 PM, Dave Fisher  wrote:
> 
> Migration technique: 
> https://cwiki.apache.org/confluence/display/INFRA/Migrating+jobs+from+Jenkins+to+Cloudbees
> 
> Helpful info: 
> https://cwiki.apache.org/confluence/display/INFRA/Kicking+off+a+Jenkins+build+with+a+GitHub+PR
> 
> 
>> On Aug 14, 2020, at 12:48 PM, Dave Fisher  wrote:
>> 
>> Hi -
>> 
>> There is a critical migration for Jenkins builds that the project is missing.
>> 
>> Several PMC members ought to be subscribed to bui...@apache.org.
>> 
>> Please act quickly.
>> 
>> Regards,
>> Dave
>> 
>>> Begin forwarded message:
>>> 
>>> From: Gavin McDonald 
>>> Subject: [IMPORTANT] - Migration to ci-builds.a.o
>>> Date: July 16, 2020 at 9:33:08 AM PDT
>>> To: builds 
>>> Reply-To: bui...@apache.org
>>> Reply-To: gmcdon...@apache.org
>>> 
>>> Hi All,
>>> 
>>> This NOTICE is for everyone on builds.apache.org. We are migrating to a new
>>> Cloudbees based Client Master called https://ci-builds.apache.org. The
>>> migrations of all jobs needs to be done before the switch off date of 15th
>>> August 2020, so you have a maximum of 4 weeks.
>>> 
>>> There is no tool or automated way of migrating your jobs, the
>>> differences in the platforms, the plugins and the setup makes it impossible
>>> to do in a safe way. So, you all need to start creating new jobs on
>>> ci-infra.a.o and then , when you are happy, turn off your old builds on
>>> builds.a.o.
>>> 
>>> There are currently 4 agents over there ready to take jobs, plus a floating
>>> agent which is shared amongst many masters (more to come). I will migrate
>>> away 2 more agents from builds.a.o to ci-builds.a.o every few days, and
>>> will keep an eye of load across both and adjust accordingly.
>>> 
>>> If needed, create a ticket on INFRA jira for any issues that crop up, or
>>> email here on builds@a.o. there may be one or two plugins we need to
>>> install/tweak etc.
>>> 
>>> We will be not using 'Views' at the top level, but rather we will take
>>> advantage of 'Folders Plus' - each project will get its own Folder and have
>>> authorisation access to create/edit jobs etc within that folder. I will
>>> create these folders as you ask for them to start with. This setup allows
>>> for credentials isolation amongst other benefits, including but not limited
>>> to exclusive agents (Controlled Agents) for your own use , should you have
>>> any project targeted donations of agents.
>>> 
>>> As with other aspects of the ASF, projects can choose to just enable all
>>> committers access to their folder, just ask.
>>> 
>>> We will re-use builds.apache.org as a CNAME to ci-builds.a.o but will not
>>> be setting up any forwarding rules or anything like that.
>>> 
>>> So, please, get started *now *on this so you can be sure we are all
>>> completed before the final cutoff date of 15th August 2020.
>>> 
>>> Any questions - I expect a few (dozen :) ) - ask away and/or file INFRA
>>> tickets.
>>> 
>>> Hadoop and related projects have their own migration path to follow, same
>>> cut off date, Cassandra, Beam, CouchDB have already migrated and are doing
>>> very well in their new homes.
>>> 
>>> Lets get going ...
>>> 
>>> -- 
>>> 
>>> *Gavin McDonald*
>>> Systems Administrator
>>> ASF Infrastructure Team
>> 
> 

Re: [IMPORTANT] - Migration to ci-builds.a.o

[Dave Fisher]

Hi Sijie,

They are still running on builds.apahce.org.

Regards,
Dave

> On Aug 18, 2020, at 10:50 AM, Sijie Guo  wrote:
> 
> Hi Dave,
> 
> I believe all the Pulsar projects have migrated to use Github Actions for
> CI. The jobs might already not be used anymore. Other committers can
> confirm if that is the case.
> 
> - Sijie
> 
> 
> On Tue, Aug 18, 2020 at 11:39 AM Dave Fisher  wrote:
> 
>> Hi -
>> 
>> It is critical to complete this migration THIS WEEK.
>> 
>> I’ve submitted https://issues.apache.org/jira/browse/INFRA-20720 to
>> request the folder.
>> 
>> Pulsar has the following “projects” on builds.a.o.
>> 
>> pulsar-manager-build
>> pulsar-master
>> pulsar-pull-request-c++-test
>> pulsar-release-binaries
>> pulsar-website-build
>> pulsar_precommit_client_node
>> pulsar_release_nightly_snapshot
>> 
>> I can migrate the projects so that are not lost, but I have no idea about
>> the integrations with GitHub and any missing Jenkins Plugins.
>> 
>> These all need to be migrated by Friday EOD.
>> 
>> Regards,
>> Dave
>> 
>> 
>>> On Aug 14, 2020, at 1:35 PM, Dave Fisher  wrote:
>>> 
>>> Migration technique:
>> https://cwiki.apache.org/confluence/display/INFRA/Migrating+jobs+from+Jenkins+to+Cloudbees
>>> 
>>> Helpful info:
>> https://cwiki.apache.org/confluence/display/INFRA/Kicking+off+a+Jenkins+build+with+a+GitHub+PR
>>> 
>>> 
>>>> On Aug 14, 2020, at 12:48 PM, Dave Fisher  wrote:
>>>> 
>>>> Hi -
>>>> 
>>>> There is a critical migration for Jenkins builds that the project is
>> missing.
>>>> 
>>>> Several PMC members ought to be subscribed to bui...@apache.org.
>>>> 
>>>> Please act quickly.
>>>> 
>>>> Regards,
>>>> Dave
>>>> 
>>>>> Begin forwarded message:
>>>>> 
>>>>> From: Gavin McDonald 
>>>>> Subject: [IMPORTANT] - Migration to ci-builds.a.o
>>>>> Date: July 16, 2020 at 9:33:08 AM PDT
>>>>> To: builds 
>>>>> Reply-To: bui...@apache.org
>>>>> Reply-To: gmcdon...@apache.org
>>>>> 
>>>>> Hi All,
>>>>> 
>>>>> This NOTICE is for everyone on builds.apache.org. We are migrating to
>> a new
>>>>> Cloudbees based Client Master called https://ci-builds.apache.org. The
>>>>> migrations of all jobs needs to be done before the switch off date of
>> 15th
>>>>> August 2020, so you have a maximum of 4 weeks.
>>>>> 
>>>>> There is no tool or automated way of migrating your jobs, the
>>>>> differences in the platforms, the plugins and the setup makes it
>> impossible
>>>>> to do in a safe way. So, you all need to start creating new jobs on
>>>>> ci-infra.a.o and then , when you are happy, turn off your old builds on
>>>>> builds.a.o.
>>>>> 
>>>>> There are currently 4 agents over there ready to take jobs, plus a
>> floating
>>>>> agent which is shared amongst many masters (more to come). I will
>> migrate
>>>>> away 2 more agents from builds.a.o to ci-builds.a.o every few days, and
>>>>> will keep an eye of load across both and adjust accordingly.
>>>>> 
>>>>> If needed, create a ticket on INFRA jira for any issues that crop up,
>> or
>>>>> email here on builds@a.o. there may be one or two plugins we need to
>>>>> install/tweak etc.
>>>>> 
>>>>> We will be not using 'Views' at the top level, but rather we will take
>>>>> advantage of 'Folders Plus' - each project will get its own Folder and
>> have
>>>>> authorisation access to create/edit jobs etc within that folder. I will
>>>>> create these folders as you ask for them to start with. This setup
>> allows
>>>>> for credentials isolation amongst other benefits, including but not
>> limited
>>>>> to exclusive agents (Controlled Agents) for your own use , should you
>> have
>>>>> any project targeted donations of agents.
>>>>> 
>>>>> As with other aspects of the ASF, projects can choose to just enable
>> all
>>>>> committers access to their folder, just ask.
>>>>> 
>>>>> We will re-use builds.apache.org as a CNAME to ci-builds.a.o but will
>> not
>>>>> be setting up any forwarding rules or anything like that.
>>>>> 
>>>>> So, please, get started *now *on this so you can be sure we are all
>>>>> completed before the final cutoff date of 15th August 2020.
>>>>> 
>>>>> Any questions - I expect a few (dozen :) ) - ask away and/or file INFRA
>>>>> tickets.
>>>>> 
>>>>> Hadoop and related projects have their own migration path to follow,
>> same
>>>>> cut off date, Cassandra, Beam, CouchDB have already migrated and are
>> doing
>>>>> very well in their new homes.
>>>>> 
>>>>> Lets get going ...
>>>>> 
>>>>> --
>>>>> 
>>>>> *Gavin McDonald*
>>>>> Systems Administrator
>>>>> ASF Infrastructure Team
>>>> 
>>> 
>> 
>> 

Re: [IMPORTANT] - Migration to ci-builds.a.o

[Dave Fisher]

Hi -

Infra has created a folder in case it is needed for one of those “projects”.

https://ci-builds.apache.org/job/Pulsar/

Let me know if any more help is needed.

Regards,
Dave

> On Aug 18, 2020, at 11:15 AM, Sijie Guo  wrote:
> 
> Okay. We can look into it to see if we can just delete them.
> 
> 
> - Sijie
> 
> On Tue, Aug 18, 2020 at 11:52 AM Dave Fisher  wrote:
> 
>> Hi Sijie,
>> 
>> They are still running on builds.apahce.org.
>> 
>> Regards,
>> Dave
>> 
>>> On Aug 18, 2020, at 10:50 AM, Sijie Guo  wrote:
>>> 
>>> Hi Dave,
>>> 
>>> I believe all the Pulsar projects have migrated to use Github Actions for
>>> CI. The jobs might already not be used anymore. Other committers can
>>> confirm if that is the case.
>>> 
>>> - Sijie
>>> 
>>> 
>>> On Tue, Aug 18, 2020 at 11:39 AM Dave Fisher  wrote:
>>> 
>>>> Hi -
>>>> 
>>>> It is critical to complete this migration THIS WEEK.
>>>> 
>>>> I’ve submitted https://issues.apache.org/jira/browse/INFRA-20720 to
>>>> request the folder.
>>>> 
>>>> Pulsar has the following “projects” on builds.a.o.
>>>> 
>>>> pulsar-manager-build
>>>> pulsar-master
>>>> pulsar-pull-request-c++-test
>>>> pulsar-release-binaries
>>>> pulsar-website-build
>>>> pulsar_precommit_client_node
>>>> pulsar_release_nightly_snapshot
>>>> 
>>>> I can migrate the projects so that are not lost, but I have no idea
>> about
>>>> the integrations with GitHub and any missing Jenkins Plugins.
>>>> 
>>>> These all need to be migrated by Friday EOD.
>>>> 
>>>> Regards,
>>>> Dave
>>>> 
>>>> 
>>>>> On Aug 14, 2020, at 1:35 PM, Dave Fisher  wrote:
>>>>> 
>>>>> Migration technique:
>>>> 
>> https://cwiki.apache.org/confluence/display/INFRA/Migrating+jobs+from+Jenkins+to+Cloudbees
>>>>> 
>>>>> Helpful info:
>>>> 
>> https://cwiki.apache.org/confluence/display/INFRA/Kicking+off+a+Jenkins+build+with+a+GitHub+PR
>>>>> 
>>>>> 
>>>>>> On Aug 14, 2020, at 12:48 PM, Dave Fisher  wrote:
>>>>>> 
>>>>>> Hi -
>>>>>> 
>>>>>> There is a critical migration for Jenkins builds that the project is
>>>> missing.
>>>>>> 
>>>>>> Several PMC members ought to be subscribed to bui...@apache.org.
>>>>>> 
>>>>>> Please act quickly.
>>>>>> 
>>>>>> Regards,
>>>>>> Dave
>>>>>> 
>>>>>>> Begin forwarded message:
>>>>>>> 
>>>>>>> From: Gavin McDonald 
>>>>>>> Subject: [IMPORTANT] - Migration to ci-builds.a.o
>>>>>>> Date: July 16, 2020 at 9:33:08 AM PDT
>>>>>>> To: builds 
>>>>>>> Reply-To: bui...@apache.org
>>>>>>> Reply-To: gmcdon...@apache.org
>>>>>>> 
>>>>>>> Hi All,
>>>>>>> 
>>>>>>> This NOTICE is for everyone on builds.apache.org. We are migrating
>> to
>>>> a new
>>>>>>> Cloudbees based Client Master called https://ci-builds.apache.org.
>> The
>>>>>>> migrations of all jobs needs to be done before the switch off date of
>>>> 15th
>>>>>>> August 2020, so you have a maximum of 4 weeks.
>>>>>>> 
>>>>>>> There is no tool or automated way of migrating your jobs, the
>>>>>>> differences in the platforms, the plugins and the setup makes it
>>>> impossible
>>>>>>> to do in a safe way. So, you all need to start creating new jobs on
>>>>>>> ci-infra.a.o and then , when you are happy, turn off your old builds
>> on
>>>>>>> builds.a.o.
>>>>>>> 
>>>>>>> There are currently 4 agents over there ready to take jobs, plus a
>>>> floating
>>>>>>> agent which is shared amongst many masters (more to come). I will
>>>> migrate
>>>>>>> away 2 more agents from builds.a.o to ci-builds.a.o every few days,
>> and
>>>>>>> will keep an eye of load across both and 

Re: Proposal for Consumer Filtering in Pulsar brokers

[Dave Fisher]

Sorry, for the top post.

I was reading what you wrote and knew where you were heading.

Couldn’t these filtering functions / dispatchers be a special configuration of 
functions?

(1) Topic
(2) Filtered function
(3) Filtered topic

Isn’t the whole point of Functions to compute, filter, and distribute at any 
load needed?

And don’t these patterns have other projects that implement solutions like 
Apache Storm or Heron(Incubating)?

Regards,
Dave

PS. You remind of why I never liked massive procedures (particularly Java) in 
Oracle RDBMS.


> On Nov 16, 2020, at 10:40 AM, Joe F  wrote:
> 
> We have had discussions in the community list on server side logic
> previously. I would like to keep the specific proposal in this PIP aside,
> and address what this PIP is  implicitly changing in core Pulsar design.  I
> want to have an explicit discussion on that topic: what is the path for
> server-side business logic in Pulsar?
> 
> Pulsar has been designed to do a few things very well.  It is designed to
> be run as a hosted service, meaning it can be scaled horizontally by adding
> storage or compute hardware, as traffic or tenants on the service grows. It
> is optimized for data streaming at  throughput and scale,  and does
> multi-tenancy extremely well.  Part of that design is that there is no
> business logic that is in the data flow path. Since  business logic lives
> outside of the core data flow path in Pulsar, the core is optimized for
> data flow. Do plain byte movement - no ser/de, no byte copy, no
> computations - and do it extremely well. Other systems, like Kafka and
> Kinesis have taken the same approach;  no to server side business logic.
> 
> This particular PIP  may be  expensive on the server, or not. The next PIP
> could be, and there is no rationale to stop adding any kind of business
> logic into the broker, once this concept is allowed.
> 
> Selective consumers are an anti-pattern for data flow systems. There are
> systems out there that support implementation of business logic in the data
> flow path, and they don't scale.   Take the example of AMQ.   AMQ allows
> JMS/SQL-92 expressions server side. Once the door to this anti-pattern  is
> opened, there is no rhyme or reason to deny anything, upto  including a
> full-blown SQL query evaluation in the dispatch path.
> 
> So why not allow that? Why not allow a full blown expression evaluation in
> the data flow path?
> 
> Unfortunately there  is no way to answer this without bringing up the
> conflict of interest between small users vs. large scale users running
> multi-tenant hosted Pulsar, at huge traffic volumes.
> 
> For low scale, single (or few) tenant installations, efficiency of flow,
> latency and throughput are not the driving concern. In a small cluster,
> the implications of cost and scale, is minimal in absolute terms,  when
> server side business logic is executed.
> 
> For large scale users (like me) this is a no go. There are many problems
> with this,  that makes it very difficult to run a hosted platform with
> predictable  SLAs, once users can introduce business logic into the broker.
> These are on top of the performance and cost  implications
> 
> First, broker throughput and performance becomes unpredictable.  The
> current Pulsar load model (and it is used in the load manager for load
> balancing) becomes unusable. Not only that, there will be no pre-computed
> model that can be used in the load manager. Since  the producer and
> consumer randomly decide on what is the business logic,and the computation
> can change based on the data,  the model itself becomes dynamic and the
> load manager has to rebuild the model anytime an user updates the business
> logic. That is a tall order, worth years of work to implement.
> 
> Second, this introduces the noisy neighbor issue. Two tenants will happily
> run on the same broker, till one of them decides to change the logic on the
> subscription, and suddenly the  quality for the other tenant is degraded
> because the broker is impacted.  The system operator of the cluster has now
> to get involved out of the blue, because one tenant did a change.
> Basically  any tenant can disrupt the system by triggering additional
> business logic in the server, or by specific data patterns that can make
> the business logic expensive on the server
> 
> Third, this makes provisioning capacity impossible. Today Pulsar users can
> be provisioned on flow - bw in/out. Msgs in/out.  With server side business
> logic, there is some random overhead that needs to be accounted in the
> capacity calculation.
> 
> We, who run Pulsar as a hosted service, do not want any of our tenants to
> introduce server side logic into the service.  Because,  to do it well
> requires a load balancer that can continuously and dynamically adjust its
> load model and capacity model (based on ML on the traffic maybe).  The
> scope of building such a system will convert Pulsar  from a  data streaming
> project  to a load ba

Re: [PROPOSAL] PIP 75: Replace protobuf code generator

[Dave Fisher]

Hi Matteo,

I looked at the dependencies in Splunk LightPro and the following are not 
acceptable in an Apache Binary:


org.openjdk.jmh
jmh-core
${jmh.version}


org.openjdk.jmh
jmh-generator-annprocess
${jmh.version}
provided


These are GPL according to mvnrepository.com.

The other dependencies are fine and all class A or B.

Have a look at https://www.apache.org/legal/resolved.html

Regards,
Dave

> On Dec 23, 2020, at 9:35 AM, Matteo Merli  wrote:
> 
> ## Motivation
> 
> In the Pulsar wire protocol, we are using Google Protobuf in order to perform
> serialization/deserialization of the commands that are exchanged between
> clients and brokers.
> 
> Because of the overhead involved with the regular Protobuf implementation, 
> since
> very early on, we have been using a modified version of Protobuf 2.4.1.
> The modifications were done to ensure a more efficient serialization code that
> used thread local caches for the objects used in the process.
> 
> There are few issues with the current approach:
> 1. The patch to the Protobuf code generator is only based on version 2.4.1 and
>cannot be upgraded to newer Protobuf versions
> 2. The new Protobuf version, 3.xx, do have the same performance issues as the
>2.x versions.
> 3. The thread-local approach for reusing objects is not ideal. Thread-local
>access is not free and it would be better to instead cache the root objects
>only.
> 
> ## Goal
> 
> Have an efficient and maintainable way to perform 
> serialization/deserialization
> of Pulsar protocol.
> 
> The current proposal is to switch from the patched Protobuf 2.4.1 and use a
> different code generator, Splunk LightProto:
> https://github.com/splunk/lightproto.
> 
> This code generator has the following features/goals:
> 
> 1. Generate the fastest possible Java code for Protobuf SerDe
> 2. 100% Compatible with proto2 definition and wire protocol
> 3. Zero-copy deserialization using Netty ByteBuf
> 4. Deserialize from direct memory
> 5. Zero heap allocations in serialization / deserialization
> 6. Lazy deserialization of strings and bytes
> 7. Reusable mutable objects
> 8. No runtime dependency library
> 9. Java based code generator with Maven plugin
> 
> There is extensive testing to ensure the generated code serializes and parses
> the same bytes in the same way as the Google Protobuf does.
> 
> 
> --
> Matteo Merli
> 

Re: [PROPOSAL] PIP 75: Replace protobuf code generator

[Dave Fisher]

> On Dec 23, 2020, at 10:55 AM, Matteo Merli  wrote:
> 
> Hi Dave,
> 
> these are not dependencies that are needed (or would be pulled) from
> the Pulsar build.
> 
> The JMH is only used if someone wants to run a benchmark on the
> LightProto code. Pulsar does not depend on that.

Excellent. You’ll need to note that for developers if benchmarking is exposed 
in Pulsar.

> 
> From Pulsar build we would need:
> 1. At build time -> The LightProto Maven Plugin with the generator module
> 2. At runtime (and in Apache release) -> No dependencies
> 
> 
> Matteo
> 
> 
> 
> --
> Matteo Merli
> 
> 
> On Wed, Dec 23, 2020 at 10:20 AM Dave Fisher  wrote:
>> 
>> Hi Matteo,
>> 
>> I looked at the dependencies in Splunk LightPro and the following are not 
>> acceptable in an Apache Binary:
>> 
>>
>>org.openjdk.jmh
>>jmh-core
>>${jmh.version}
>>
>>
>>org.openjdk.jmh
>>jmh-generator-annprocess
>>${jmh.version}
>>provided
>>
>> 
>> These are GPL according to mvnrepository.com.
>> 
>> The other dependencies are fine and all class A or B.
>> 
>> Have a look at https://www.apache.org/legal/resolved.html
>> 
>> Regards,
>> Dave
>> 
>>> On Dec 23, 2020, at 9:35 AM, Matteo Merli  wrote:
>>> 
>>> ## Motivation
>>> 
>>> In the Pulsar wire protocol, we are using Google Protobuf in order to 
>>> perform
>>> serialization/deserialization of the commands that are exchanged between
>>> clients and brokers.
>>> 
>>> Because of the overhead involved with the regular Protobuf implementation, 
>>> since
>>> very early on, we have been using a modified version of Protobuf 2.4.1.
>>> The modifications were done to ensure a more efficient serialization code 
>>> that
>>> used thread local caches for the objects used in the process.
>>> 
>>> There are few issues with the current approach:
>>> 1. The patch to the Protobuf code generator is only based on version 2.4.1 
>>> and
>>>   cannot be upgraded to newer Protobuf versions
>>> 2. The new Protobuf version, 3.xx, do have the same performance issues as 
>>> the
>>>   2.x versions.
>>> 3. The thread-local approach for reusing objects is not ideal. Thread-local
>>>   access is not free and it would be better to instead cache the root 
>>> objects
>>>   only.
>>> 
>>> ## Goal
>>> 
>>> Have an efficient and maintainable way to perform 
>>> serialization/deserialization
>>> of Pulsar protocol.
>>> 
>>> The current proposal is to switch from the patched Protobuf 2.4.1 and use a
>>> different code generator, Splunk LightProto:
>>> https://github.com/splunk/lightproto.
>>> 
>>> This code generator has the following features/goals:
>>> 
>>> 1. Generate the fastest possible Java code for Protobuf SerDe
>>> 2. 100% Compatible with proto2 definition and wire protocol
>>> 3. Zero-copy deserialization using Netty ByteBuf
>>> 4. Deserialize from direct memory
>>> 5. Zero heap allocations in serialization / deserialization
>>> 6. Lazy deserialization of strings and bytes
>>> 7. Reusable mutable objects
>>> 8. No runtime dependency library
>>> 9. Java based code generator with Maven plugin
>>> 
>>> There is extensive testing to ensure the generated code serializes and 
>>> parses
>>> the same bytes in the same way as the Google Protobuf does.
>>> 
>>> 
>>> --
>>> Matteo Merli
>>> 
>> 

Re: Haskell Client: transfer of ownership

[Dave Fisher]

One important consideration is if there are enough community members who can 
support Haskiel.

Another is if Gabriel will stick around and support the client code.

Best Regards,
Dave

Sent from my iPhone

> On Jan 4, 2021, at 6:25 PM, Sijie Guo  wrote:
> 
> Hi Gabriel,
> 
> Thank you for putting up the proposal! I have moved your gist to the Pulsar
> wiki page as PIP-77 (
> https://github.com/apache/pulsar/wiki/PIP-77:-Contribute-Supernova-to-Apache-Pulsar
> ).
> 
> Because there is no objection in this thread, I will start an official vote
> thread for it. Once the community approves that, I will work with you on
> the following steps.
> 
> - Sijie
> 
>> On Mon, Jan 4, 2021 at 5:32 PM anonymitaet _ 
>> wrote:
>> 
>> Thanks for your inputs, Gabriel.
>> 
>> Anyone could help proceed with the next step?
>> --
>> *From:* Gabriel Volpe 
>> *Sent:* Saturday, January 2, 2021 19:47
>> *To:* dev@pulsar.apache.org 
>> *Subject:* Re: Haskell Client: transfer of ownership
>> 
>> Hi Sijie,
>> 
>> I wrote it as a gist here:
>> https://gist.github.com/gvolpe/2d7c171600f4e821d3c354516b7aedf2
>> 
>> Cheers,
>> Gabriel.
>> 
>> Sent with ProtonMail Secure Email.
>> 
>> ‐‐‐ Original Message ‐‐‐
>> 
>> On Wednesday, December 30th, 2020 at 11:12 AM, Sijie Guo <
>> guosi...@gmail.com> wrote:
>> 
>>> Gabriel,
>>> 
>>> Thank you for initiating the discussion.
>>> 
>>> Can you write a PIP for this? You can check
>>> 
>>> 
>> https://github.com/apache/pulsar/wiki/PIP-53%3A-Contribute-DotPulsar-to-Apache-Pulsar
>>> 
>>> as an example.
>>> 
>>> -   Sijie
>>> 
>>>On Wed, Dec 30, 2020 at 2:01 AM Gabriel Volpe he...@gvolpe.com
>> wrote:
>>> 
 Hi,
 
 I'm the author of Supernova, a Haskell client for Pulsar, and I have
>> total
 
 ownership on thecurrent repository.
 
 I'd be happy to donate it over to the Apache organization.
 
 Let me know how we should proceed.
 
 Best,
 
 Gabriel
 
 Sent with ProtonMail Secure Email.
>> 

Re: Five minute interview blog posts

[Dave Fisher]

> 
>> Clean up the header by folding Clients, REST APIs, and Cli into Docs
> 
> One of the reasons that why "Clients, REST APIs, and CLI" are added to the
> menu, not the sidebar is due to the limitation of the documentation
> framework we are using. If there is a way to improve this, that would be
> great.

You can find the website source here:

https://github.com/apache/pulsar/tree/master/site2

It is a combination of Crowdin and Docusaurus.

It looks like the top navigation is configured in Javascript while sidebars are 
json.

My personal experience is recently in JBake (incubator.apache.org and 
www.openoffice.org) and Pelican (petri.apache.org and openoffice.apache.org). 
Pelican is Apache Infra’s preferred site builder.

Regards,
Dave

Re: Five minute interview blog posts

[Dave Fisher]

Inline

Sent from my iPhone

> On Feb 12, 2021, at 9:36 AM, Jonathan Ellis  wrote:
> 
> 
> To make a specific proposal, I've attached a mockup of what a streamlined 
> home page would look like.
> 
> Commentary:
> - Introduction and Quickstart will follow Kafka’s model 
> (http://kafka.apache.org/intro, http://kafka.apache.org/quickstart)
> - Use cases will be a blog of use cases (Docusaurus supports multiple blogs)
> - Announcements is the current blog renamed
> - REST APIs and Admin CLI have sub-dropdowns in the current site.  We can 
> either move these into a simple list in a new page, or we can just move the 
> sub-dropdowns into the menu (“Admin REST API, Functions REST API, Sources 
> REST API,” etc)
> - Apache menu is removed.  If we want, we can add the most important parts 
> (security issues?) to the page footer for visibility.

Pulsar already has some work to do to meet the ASF’s website expectations:

https://whimsy.apache.org/site/

Regards,
Dave

> - Clients link is also removed, we can add those to Downloads page (“client 
> guides” are already referenced there)
> 
> We can do more but I don’t want to boil the ocean, I think this is a good 
> step that we can take without adding extra dependencies.
> 
>> On Thu, Feb 11, 2021 at 5:28 PM Joshua Odmark  
>> wrote:
>> Sijie,
>> 
>> What is the next best step here? How should we present this so that it can 
>> get a vote if it makes sense to proceed on?
>> 
>> I’d be happy to contribute as a technical writer as well.
>> 
>> I think we have all the contributors we need listed here in this thread to 
>> make this happen quickly once it is approved.
>> 
>> Thanks!
>> 
>> > On Feb 10, 2021, at 9:24 AM, Jonathan Ellis  wrote:
>> > 
>> > I'd also like to figure out how we can get the TGI Pulsar videos featured
>> > more prominently.  Those are really well done.
>> > 
>> > On Wed, Feb 10, 2021 at 3:58 AM Sijie Guo  wrote:
>> > 
>> >> Thanks, everyone for the input!
>> >> 
>> >> I think there are two different things mixing together here. One is the
>> >> improvement on the documentation which provides more information about use
>> >> cases and makes the Pulsar documentation site more searchable; the other
>> >> one is what content to be hosted on the project side.
>> >> 
>> >> I would suggest separating these two things in the discussion.
>> >> 
>> >> For the first part of the improvement, I think it is a great idea to have 
>> >> a
>> >> better "Get Started" section to include "Use Cases". Jennifer, Huanli, and
>> >> Yu are the main committers driving the development of the documentation.
>> >> They can provide some of the insights from a technical writing 
>> >> perspective.
>> >> 
>> >>> Clean up the header by folding Clients, REST APIs, and Cli into Docs
>> >> 
>> >> One of the reasons that why "Clients, REST APIs, and CLI" are added to the
>> >> menu, not the sidebar is due to the limitation of the documentation
>> >> framework we are using. If there is a way to improve this, that would be
>> >> great.
>> >> 
>> >> - Sijie
>> >> 
>> >> 
>> >> On Tue, Feb 9, 2021 at 3:02 PM Jonathan Ellis  wrote:
>> >> 
>> >>> You're right, Kafka does a really good job here.  Here's a proposal along
>> >>> those lines:
>> >>> 
>> >>> 1. Add a Get Started section to the site modeled on Kafka's
>> >>>   a. Introduction
>> >>>   b. Quickstart
>> >>>   c. Use cases -- this would be a blog-like section
>> >>> 
>> >>> Not included:
>> >>>   d. books&papers and podcasts we can ignore for now
>> >>>   e. videos we could do but I feel like it's hard to create an objective
>> >>> measure for what should be included, Kafka uses ratings from Kafka Summit
>> >>> and I don't think we have something similar.  As an alternative we could
>> >>> just link the most recent full Pulsar Summit archive.
>> >>> 
>> >>> 2. Clean up the header by folding Clients, REST APIs, and Cli into Docs
>> >>> (Clients is already just a deep link to a docs page)
>> >>> 
>> >>> 3. Rename Blog to Announcements
>> >>> 
>> >>> 4. Remove Community -> Resources as obsoleted by the new Get Started
>> >>> 
>> >>> I'm happy to volunteer to draft content for the new Get Started sections.
>> >>> 
>> >>> 
>> >>> 
>> >>> On Tue, Feb 9, 2021 at 3:45 PM Joshua Odmark 
>> >>> wrote:
>> >>> 
>>  For me this gets into a bigger issue. That issue for me is that right
>> >>> now,
>>  I need to visit half a dozen sites to get the full picture of Pulsar.
>>  
>>  Putting content like this on a separate site and linking to it from
>> >>> Pulsar
>>  doesn’t create the proper journey in my mind.
>>  
>>  I don’t know if it should be necessarily in the blog or not, but I
>> >> think
>>  it makes sense to create an entirely new section of the Pulsar website,
>>  similar to how Kafka’s does it. Take the concept of Resources, but turn
>> >>> it
>>  into a journey and put the content right on the Pulsar website.
>> >> Possibly
>>  get rid of the blog and name it something more precise if 

Re: [DISCUSS] Propose More Formal Policy for Security Patches and EOL of Versions

[Dave Fisher]

> On May 27, 2021, at 2:49 PM, Michael Marshall  wrote:
> 
> Hi Pulsar Community,
> 
> 
> I would like to discuss defining and documenting a process for an official
> Pulsar version EOL policy. This process will help users know when the
> version they are running will no longer be supported with security patches.
> 
> After the recent announcement of CVE-2021-22160, I looked on the official
> pulsar website for a documented process describing which branches will
> receive security patches when vulnerabilities are discovered. I could not
> find a process on the website. I did find a policy for handling version EOL
> described in PIP-47, though (
> https://github.com/apache/pulsar/wiki/PIP-47%3A-Time-Based-Release-Plan#what-is-our-eol-policy).
> Based on the policy, I would have expected a release of 2.6.4 with the
> security patch for CVE-2021-22160, but that patch was not cherry-picked to
> branch-2.6 until this week after the publication of the CVE, and we’re only
> just now starting the process to release 2.6.4.
> 
> I think it’s also relevant to consult the ASF guide for vulnerability
> handling. The process is outlined here:
> https://www.apache.org/security/committers.html. Regarding releasing fixes,
> it only mentions the following:
> 
>> 15. The project team creates a release that includes the fix.
> 
>> 16. The project team announces the vulnerability. The vulnerability
> announcement should be sent after, or at the same time as, the release
> announcement to the following destinations.
> 
> Given the above information, I think it would be appropriate to have a
> policy that ensures all active/supported branches receive security patches
> before a CVE is announced.
> 
> I propose we adopt a policy similar to Apache Spark’s policy, which is to
> provide security fixes for feature branches for at least 18 months after
> the initial minor version release:
> https://spark.apache.org/versioning-policy.html. If we apply this policy to
> Pulsar, 2.6.x will receive security patches until December 2021 (2.6.0 was
> released in June 2020).
> 
> When I brought this up at the community meeting today, Matteo mentioned
> supporting releases for a year. I am open to that time frame as well. My
> main objective is to have a policy that is documented and easily discovered
> by our users.

Looking at this as a PMC member who has had to triage security for a very 
widely downloaded and old project codebase (OpenOffice) there is some record 
keeping that the PMC should do in private to track vulnerabilities before they 
are CVEs.

The PMC can also assign members to a secur...@pulsar.apache.org mailing list.

The PMC can request a private SVN repository and/or private Confluence Wiki for 
keeping records and assuring that such missed back ports are less likely. 
(Private Git limited to the PMC is not currently possible (it is an Infra 
wish)) Doing this allows even “non-technical” PMC members to help manage the 
CVE process.

All The Best,
Dave

> 
> I look forward to your thoughts and suggestions.
> 
> Thanks,
> 
> Michael Marshall

Re: Proposing a meetup organizing committee

[Dave Fisher]

BCC: private@

The bulk of this discussion should proceed with good intent on dev@ with a 
separate thread on private@ only if needed.

I have prepared a set of links to documentation about the Apache Way that is 
worth reviewing:

https://petri.apache.org/way 

Hierarchy in an Apache project should be as flat as possible.

On dev@ an open discussion about the Apache Pulsar Community should be proceed 
about:

(1) Communications
- How do the members of the community let the community know what they have 
been doing?
- How should that be done to show vendor neutrality?

The standard answer is to mention progress on dev@. Other solutions are 
possible and anyone who is interested should be able to contribute and earn 
merit in the community. If community volunteers produce a Newsletter that 
should not bother the PMC.

(2) Meetups - There are a number of questions here and I am likely to miss a 
few.
- There has been a semi-official Meetup organized by Sree Vadi since Incubation.
- Others wish to his Meetups.
- It can only benefit the whole of the Pulsar Community to communicate all 
meetups where the commits can know about it.

While it might appear that everyone is far apart I think that people should 
make one or more proposals. Then there can be a discussion without summary veto 
from PMC members. Sometimes Consensus is hard.

All The Best,
Dave

> On Aug 20, 2021, at 12:46 AM, Rajan Dhabalia  wrote:
> 
> *I think the Apache project should avoid vendor-specific branding, in that
> way users will have a clear understanding about what they can receive from
> the free Apache project. In support of that, the Apache project should be
> driven by the community/PMC who are the real contributors to the project
> and let every member participate in each decision process. Therefore, I
> feel subcommittees will not give justice to every PMC member of the project
> and may create confusion to the users about the ownership and branding of
> the project. Therefore, we should address all those concerns that are
> misleading users about project branding or make users think that the Apache
> project is driven by a specific vendor. I think ASF branding policy
>  summarises it very
> well: “Third-party software product must be clearly branded such that,
> users clearly understand the different sources for software products such
> as Apache Foo (from the ASF) and BigCo SuperThing, Powered By Apache Foo
> (from BigCo).”Thanks,Rajan*
> 
> On Thu, Aug 19, 2021 at 4:49 PM Matteo Merli  wrote:
> 
>> On Thu, Aug 19, 2021 at 5:35 AM Jonathan Ellis  wrote:
>>> 
>>> Moving back to dev.
>>> 
>>> Since it seems like there's some confusion on this point, it's perfectly
>>> normal for PMC discussions around new proposals with decision-making
>>> authority by the PMC to take place on the public dev list.  The private
>>> list is only necessary when confidentiality is required, and the dev list
>>> allows non-PMC voices to be heard more readily as well as promoting
>>> transparency on how consensus was reached.
>>> 
>>> I am not aware of any ASF policy that would prohibit subcommittees like
>>> this.  (I'm not aware of precedent in starting one either, but as Aaron
>>> pointed out, this *is* common at similar foundations with similar
>>> governance goals to the ASF, and there's no reason we can't
>> cross-pollinate
>>> good ideas.)
>> 
>> Jonathan,
>> 
>> If you keep switching from private@ to dev@ list, most of the people
>> on dev@ list will be out of context of the replies from other PMC
>> members on the private list.
>> 
>> Your proposal of subcommittees has already raised strong concerns from
>> 4 PMC members, with no one speaking in favour of it.
>> 
>>> I am not aware of any ASF policy that would prohibit subcommittees like
>>> this
>> 
>> The part of your proposal of including representatives of vendors in
>> these subcommittees goes directly against the spirit of these
>> guidelines: https://community.apache.org/projectIndependence.html
>> 
>> Matteo
>> 
>> --
>> Matteo Merli
>> 
>> 

Re: [Proposal] Creating an Apache Pulsar Outreach Working Group

[Dave Fisher]

Hi -

Even without an official group it should be possible to discuss improvements to 
outreach here on the dev@ mailing list.

There is a page full of logos (some missing) at 
https://pulsar.apache.org/powered-by/  
While that is a good starting place there is no information on that page about 
how to request addition to that page. I’m not clear that the PMC has a policy 
or rule.

> On Aug 24, 2021, at 5:30 PM, Aaron Williams  wrote:
> 
> Hello Apache Pulsar Community,
> 
> I would like to propose that the PMC create an Outreach Working Group.
> 
> Here’s why it is needed:
> 
> The Apache Pulsar community currently doesn't have community-driven content
> channels that are unaffiliated with any vendor. We also don't have an
> active effort to help identify people/organizations who are using Pulsar,
> nor do we have resources who are helping to tell their stories and educate
> the broader industry about these users and organizations where Pulsar is
> being used successfully.  In my experience, this is the single most
> important factor in helping a new infrastructure “cross the chasm” from
> early adopters who are willing to roll up their sleeves, do their own
> research and help find bugs, to “early majority” where people feel
> confident in adopting it because peer groups that they trust have already
> done that research and hardening.
> 
> In other words, there are many great things happening around the Apache
> Pulsar community, but there is no unified effort to push that information
> out.
> 
> For example, we could do a LOT more to publicize the Pulsar Summit Europe.
> Instead of a website going live a month before the event with no awareness,
> the working group would be responsible for having the webpage go live well
> in advance, for contacting the relevant press to set up interviews with
> Pulsar Summit speakers, for helping the PMC define a theme of the event,
> for setting up social media & targeted ad campaigns, and for engaging with
> content creators like podcast interviews.  Blogs and interviews would be
> completed in advance and set to roll out on a predetermined cadence.
> 
> During the event, social media posts will be ready to go and augmented with
> news made the day of the event.  Afterwards, there needs to be follow up
> blogs, pointing people to the content that they missed and how to get to
> it.  Then this same content can be placed in the newsletter and recycled in
> other posts later on.  This is a lot of work that doesn’t “just happen”; it
> takes a lot of volunteers to get everything done.  And right now it doesn’t
> look like it is getting done, or at least the community has no insight into
> the current status of the event.  Thus, community members who want to
> volunteer, don’t know how to start.
> 
> I hope this helps illustrate how an Outreach Working Group could help
> organize and promote the community.
> 
> I look forward to hearing from the community about this proposal.
> 
> Thank you,
> 
> Aaron Williams

All The Best,
Dave

Re: Pulsar website concepts -- please vote

[Dave Fisher]

None of these concepts meets Apache project branding requirements (and neither 
does the current site)

https://whimsy.apache.org/site/project/pulsar

Sent from my iPhone

> On Sep 14, 2021, at 3:43 PM, Melissa Logan  wrote:
> 
> Reminder to cast your vote for the Pulsar website design concept by
> tomorrow, end of day your time.
> 
> Option 2 is in the lead so far.
> 
> 
>> On Wed, Sep 8, 2021 at 2:07 PM Melissa Logan  wrote:
>> 
>> Pulsar community:
>> 
>> As discussed on this thread (1), my team offered to develop new website
>> design concepts for the community that we would implement in collaboration
>> with others on that thread. We are pleased to share three concepts for your
>> consideration below.
>> 
>> Notes:
>> * This vote is for overall design concept only -- not content or copy
>> updates. Those will be tackled later.
>> * ASF recently requested no corporate logos be displayed on the homepage,
>> which is reflected in these designs.
>> * The navigation is based on the recommendation put forth in the previous
>> thread.
>> 
>> Project milestones:
>> * Choose design concept
>> * Meet with Pulsar infrastructure/website committers
>> * Begin website content review
>> * Begin template designs (8-10 estimated)
>> * Confirm IA
>> * Finalize content for community input/approval
>> * Finalize template designs for community input/approval
>> * Ensure ASF compliance
>> * Launch
>> 
>> Design concepts here:
>> https://projects.invisionapp.com/freehand/document/jY5pEt6jQ?
>> 
>> Please make your selection here by Sept. 15:
>> https://constantialabs.typeform.com/to/AJbOYCp3
>> 
>> Any questions, just let me know. Thanks!
>> 
>> Melissa Logan
>> 
>> (1)
>> https://lists.apache.org/thread.html/ra618c666d52f518741f64313969a9f16f20b8f06cd444f499f77cd93%40%3Cdev.pulsar.apache.org%3E
>> 
> 
> 
> -- 
> Melissa Logan (she/her)
> Principal, Constantia.io
> meli...@constantia.io
> Cell: 503-317-8498
> LinkedIn  | Twitter
> 

Re: Pulsar website concepts -- please vote

[Dave Fisher]

This is the Apache Pulsar project, not Pulsar.

There are numerous compliant project sites. Goto www.Apache.org and link to one 
of 200 project sites from the bottom.

Sent from my iPhone

> On Sep 14, 2021, at 5:05 PM, Dave Fisher  wrote:
> 
> None of these concepts meets Apache project branding requirements (and 
> neither does the current site)
> 
> https://whimsy.apache.org/site/project/pulsar
> 
> Sent from my iPhone
> 
>>> On Sep 14, 2021, at 3:43 PM, Melissa Logan  wrote:
>>> 
>> Reminder to cast your vote for the Pulsar website design concept by
>> tomorrow, end of day your time.
>> 
>> Option 2 is in the lead so far.
>> 
>> 
>>> On Wed, Sep 8, 2021 at 2:07 PM Melissa Logan  wrote:
>>> 
>>> Pulsar community:
>>> 
>>> As discussed on this thread (1), my team offered to develop new website
>>> design concepts for the community that we would implement in collaboration
>>> with others on that thread. We are pleased to share three concepts for your
>>> consideration below.
>>> 
>>> Notes:
>>> * This vote is for overall design concept only -- not content or copy
>>> updates. Those will be tackled later.
>>> * ASF recently requested no corporate logos be displayed on the homepage,
>>> which is reflected in these designs.
>>> * The navigation is based on the recommendation put forth in the previous
>>> thread.
>>> 
>>> Project milestones:
>>> * Choose design concept
>>> * Meet with Pulsar infrastructure/website committers
>>> * Begin website content review
>>> * Begin template designs (8-10 estimated)
>>> * Confirm IA
>>> * Finalize content for community input/approval
>>> * Finalize template designs for community input/approval
>>> * Ensure ASF compliance
>>> * Launch
>>> 
>>> Design concepts here:
>>> https://projects.invisionapp.com/freehand/document/jY5pEt6jQ?
>>> 
>>> Please make your selection here by Sept. 15:
>>> https://constantialabs.typeform.com/to/AJbOYCp3
>>> 
>>> Any questions, just let me know. Thanks!
>>> 
>>> Melissa Logan
>>> 
>>> (1)
>>> https://lists.apache.org/thread.html/ra618c666d52f518741f64313969a9f16f20b8f06cd444f499f77cd93%40%3Cdev.pulsar.apache.org%3E
>>> 
>> 
>> 
>> -- 
>> Melissa Logan (she/her)
>> Principal, Constantia.io
>> meli...@constantia.io
>> Cell: 503-317-8498
>> LinkedIn <https://www.linkedin.com/in/mklogan/> | Twitter
>> <https://twitter.com/Melissa_B2B>

Re: Cutting releases for 2.7 and 2.6 branches

[Dave Fisher]

No better place than the download page to communicate the lifecycle.

Sent from my iPhone

> On Oct 4, 2021, at 9:19 PM, Michael Marshall  wrote:
> 
> Hi Enrico,
> 
> Thank you for starting this conversation. I have been meaning to
> follow up on this topic for some time, so I already have some thoughts to 
> share.
> 
> Based on my interpretation of PIP-47 [0], we committed to support each
> minor release for a year after its initial release. We created tag
> "v2.6.0" on June, 17th 2020, so we can technically stop releasing
> patch updates for 2.6. There has only been one commit (a bug fix) to
> branch-2.6 since the 2.6.4 release. My preference is to say that 2.6
> has already reached its EOL, but if we think the EOL wasn't
> communicated clearly enough, we could of course do one final patch
> release. We created tag "v2.7.0" on November, 30th 2020, so I think it
> deserves at least one more patch release.
> 
> Moving forward, I would like to see our versioning policy added to the
> website to ensure that users can plan accordingly. Our current policy
> is only published in a PIP, and it isn't explicitly defined for each
> minor version. In my opinion, the best way to communicate our EOL
> policy is to add a table with specific version EOL dates to our Pulsar
> website. I will follow up this week with a PR to add a versioning
> policy table to the website.
> 
> Thanks,
> Michael
> 
> [0] - 
> https://github.com/apache/pulsar/wiki/PIP-47%3A-Time-Based-Release-Plan#what-is-our-eol-policy
> [1] - 
> https://lists.apache.org/x/thread.html/ra2db06e8da85bff67d8d588dc1e93d965f2e1d70c95bda2f08d14138@%3Cdev.pulsar.apache.org%3E
> [2] - https://github.com/apache/pulsar/pull/10829
> 
> 
> 
> 
> 
>> On Mon, Oct 4, 2021 at 3:26 AM Enrico Olivelli  wrote:
>> 
>> Hello folks,
>> It has been quite some time since the latest releases of branch 2.6 and 2.7.
>> 
>> We should cut a release.
>> 
>> Probably we should also start a discussion about sending 2.6 to EOL as we
>> are releasing 2.9.0.
>> 
>> I am following the release for 2.9,
>> is there any volunteer to drive a release for 2.7 and one for 2.6 ?
>> 
>> Enrico

Apache Releases - Current Lines

[Dave Fisher]

This note will cover several points about Apache Release Distribution Policy. 
[1][2] All of the official Apache Pulsar release artifacts should be on the 
project’s download page. [3]
What are the currently supported release artifacts?
Where must all source release artifacts be on Apache Distribution Channels?
How must download pages refer to current source release artifacts?
Where must release VOTEs place the artifacts?
Currently supported release artifacts.
From the download page it looks like the current main release series is 2.8.1. 
From the urls from all of the older versions indicate that they are archives. 
This implies that the 2.8.x is the only current version with 2.9.x coming up. 
Is this what the community wants?

Apache Distribution Channels
Artifacts that are current must be uploaded via svn to 
https://dist.apache.org/repos/dist/release/pulsar/
This location tells a different story about what is current: 2.7.2, 2.7.3, 
2.8.0, 2.8.1, adaptors-2.8.0, pulsar-manager, and client-go (but the other 
clients are missing). It is fine to distribute to other channels like PIP, NPM, 
and Maven Central and point users to them. It is required that all source 
releases made by the PMC also be made on dist.apache.org.

Download page links to release artifacts
The ASF is moving away from its mirror network and is now using a CDN. 
Consequently preferred links have changed slightly. See the current 
requirements here. [4] There are a lot of adjustments to make.

Client packages
While the separate page approach of describing the use and using appropriate 
channels for the client language is fine. A download of a separate source 
package for each client must be offered.

Location of packages for VOTEs
These artifacts should be uploaded via svn to 
https://dist.apache.org/repos/dist/dev/pulsar/. If that is done then it is easy 
to svn to the distribution channel ounce the VOTE is completed. [5]

In synopsis there are three action items
Determine the current development versions.
Bringing the client package release process into policy.
Updating the download page.

All The Best,
Dave

[1] https://www.apache.org/legal/release-policy.html#release-distribution
[2] https://infra.apache.org/release-distribution.html
[3] https://pulsar.apache.org/en/download/
[4] https://infra.apache.org/release-download-pages.html
[5] https://www.apache.org/legal/release-policy.html#stage

Re: [VOTE] Pulsar Node.js Client Release 1.3.2 Candidate 1

[Dave Fisher]

At least 3 +1 (binding) votes are required. I suggest that you hold this open 
until you get the 3rd vote otherwise this vote fails.

Regards,
Dave

Sent from my iPhone

> On Oct 11, 2021, at 10:19 PM, Masahiro Sakamoto  
> wrote:
> 
> Closing the vote with 2 +1 and no -1.
> 
> 2 +1 bindings are:
> * Yuto
> * Nozomi
> 
> Masahiro Sakamoto
> Yahoo Japan Corp.
> E-mail: massa...@yahoo-corp.jp
> 
> -Original Message-
> From: Nozomi Kurihara  
> Sent: Monday, October 11, 2021 12:20 PM
> To: dev@pulsar.apache.org
> Subject: Re: [VOTE] Pulsar Node.js Client Release 1.3.2 Candidate 1
> 
> +1 (binding)
> 
> * check the license headers
> * install the npm package
> * run producer/consumer
> 
> Thanks,
> Nozomi
> 
> 2021年10月11日(月) 11:28 Yuto Furuta :
> 
>> +1 (binding)
>> 
>> * build the source code
>> * check the license headers
>> * install the package
>> * run producer/consumer/reader/consumer_listener
>> 
>> Regards,
>> 
>> Yuto Furuta
>> 
>> 差出人: Masahiro Sakamoto 
>> 送信日時: 2021年10月8日 19:20
>> 宛先: dev@pulsar.apache.org 
>> 件名: [VOTE] Pulsar Node.js Client Release 1.3.2 Candidate 1
>> 
>> This is the first release candidate for Apache Pulsar Node.js client,
>> version 1.3.2.
>> 
>> It fixes the following issues:
>> 
>> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar-client-node%2Fmilestone%2F9%3Fclosed%3D1&data=04%7C01%7Cmassakam%40yahoo-corp.jp%7Cbca6a64874724d2ccbc608d98c6618d5%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637695192391453552%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=5e4sO1aomQBNhS4o1NVtyFZtszhQeFgq3nTXdOJUUCk%3D&reserved=0
>> 
>> *** Please download, test and vote on this release. This vote will stay
>> open
>> for at least 72 hours ***
>> 
>> Note that we are voting upon the source (tag), the npm package is provided
>> for convenience.
>> 
>> The tag to be voted upon:
>> v1.3.2-rc.1
>> 
>> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar-client-node%2Freleases%2Ftag%2Fv1.3.2-rc.1&data=04%7C01%7Cmassakam%40yahoo-corp.jp%7Cbca6a64874724d2ccbc608d98c6618d5%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637695192391453552%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=jpkbjSmYPx4TaWupGOaUUaFtiEyn8eLfOFSfmC3b%2Byo%3D&reserved=0
>> 
>> The npm package:
>> 
>> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.npmjs.com%2Fpackage%2Fpulsar-client%2Fv%2F1.3.2-rc.1&data=04%7C01%7Cmassakam%40yahoo-corp.jp%7Cbca6a64874724d2ccbc608d98c6618d5%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637695192391453552%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=vBNjZ4wsu64m2cPLiubMhipkFNBG%2BHInfaw6buNWe5k%3D&reserved=0
>> 
>> Please download the source files, and follow the README to build
>> and run the Pulsar Node.js client.
>> 
>> Masahiro Sakamoto
>> Yahoo Japan Corp.
>> E-mail: massa...@yahoo-corp.jp
>> 

Re: [VOTE] Pulsar Node.js Client Release 1.4.1 Candidate 1

[Dave Fisher]

You need at least 3 +1 (binding) votes for this release vote to succeed.

Sent from my iPhone

> On Oct 11, 2021, at 10:22 PM, Masahiro Sakamoto  
> wrote:
> 
> Closing the vote with 3 +1 and no -1.
> 
> 2 +1 bindings are:
> * Yuto
> * Nozomi
> 1 +1 non-binding is:
> * Guangning
> 
> Masahiro Sakamoto
> Yahoo Japan Corp.
> E-mail: massa...@yahoo-corp.jp
> 
> -Original Message-
> From: Yuto Furuta  
> Sent: Monday, October 11, 2021 12:42 PM
> To: dev@pulsar.apache.org
> Subject: Re: [VOTE] Pulsar Node.js Client Release 1.4.1 Candidate 1
> 
> +1 (binding)
> 
> * build the source code
> * check the license headers
> * install the package
> * run producer/consumer/reader/consumer_listener/reader_listener
> 
> Regards,
> 
> Yuto Furuta
> 
> 
> 差出人: Nozomi Kurihara 
> 送信日時: 2021年10月11日 12:21
> 宛先: dev@pulsar.apache.org 
> 件名: Re: [VOTE] Pulsar Node.js Client Release 1.4.1 Candidate 1
> 
> +1 (binding)
> 
> * check the license headers
> * install the npm package
> * run producer/consumer
> 
> Thanks,
> Nozomi
> 
> 2021年10月8日(金) 19:23 Masahiro Sakamoto :
> 
>> This is the first release candidate for Apache Pulsar Node.js client,
>> version 1.4.1.
>> 
>> It fixes the following issues:
>> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar-client-node%2Fmilestone%2F10%3Fclosed%3D1&data=04%7C01%7Cmassakam%40yahoo-corp.jp%7C03396cf613c44cde41b108d98c691412%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637695205196173583%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=fm6kbNwaC7phuIFmjjClHlMDvWPMbE3qu7ZpE7G6abs%3D&reserved=0
>> 
>> *** Please download, test and vote on this release. This vote will stay
>> open
>> for at least 72 hours ***
>> 
>> Note that we are voting upon the source (tag), the npm package is provided
>> for convenience.
>> 
>> The tag to be voted upon:
>> v1.4.1-rc.1
>> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar-client-node%2Freleases%2Ftag%2Fv1.4.1-rc.1&data=04%7C01%7Cmassakam%40yahoo-corp.jp%7C03396cf613c44cde41b108d98c691412%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637695205196173583%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=eVaD2ba66xPmrKujJFIY4h9g83P6u6NZWyRZ7oE7Bp0%3D&reserved=0
>> 
>> The npm package:
>> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.npmjs.com%2Fpackage%2Fpulsar-client%2Fv%2F1.4.1-rc.1&data=04%7C01%7Cmassakam%40yahoo-corp.jp%7C03396cf613c44cde41b108d98c691412%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637695205196173583%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=50YKnYQ6kzPOP%2FV00bOp8Ppk0%2FTDEqr7Fc6iTS3Bjhc%3D&reserved=0
>> 
>> Please download the source files, and follow the README to build
>> and run the Pulsar Node.js client.
>> 
>> Masahiro Sakamoto
>> Yahoo Japan Corp.
>> E-mail: massa...@yahoo-corp.jp
>> 

Re: [VOTE] Pulsar Node.js Client Release 1.3.2 Candidate 1

[Dave Fisher]

> On Oct 8, 2021, at 3:20 AM, Masahiro Sakamoto  wrote:
> 
> This is the first release candidate for Apache Pulsar Node.js client, version 
> 1.3.2.
> 
> It fixes the following issues:
> https://github.com/apache/pulsar-client-node/milestone/9?closed=1
> 
> *** Please download, test and vote on this release. This vote will stay open
> for at least 72 hours ***
> 
> Note that we are voting upon the source (tag), the npm package is provided
> for convenience.
> 
> The tag to be voted upon:
> v1.3.2-rc.1
> https://github.com/apache/pulsar-client-node/releases/tag/v1.3.2-rc.1

-1 (binding) we don’t vote on tags. We vote on signed packages that also have 
checksums.

https://www.apache.org/legal/release-policy.html#policy
https://infra.apache.org/release-publishing.html#definition

> 
> The npm package:
> https://www.npmjs.com/package/pulsar-client/v/1.3.2-rc.1
> 
> Please download the source files, and follow the README to build
> and run the Pulsar Node.js client.
> 
> Masahiro Sakamoto
> Yahoo Japan Corp.
> E-mail: massa...@yahoo-corp.jp

Re: [VOTE] Pulsar Node.js Client Release 1.4.1 Candidate 1

[Dave Fisher]

> On Oct 8, 2021, at 3:23 AM, Masahiro Sakamoto  wrote:
> 
> This is the first release candidate for Apache Pulsar Node.js client, version 
> 1.4.1.
> 
> It fixes the following issues:
> https://github.com/apache/pulsar-client-node/milestone/10?closed=1
> 
> *** Please download, test and vote on this release. This vote will stay open
> for at least 72 hours ***
> 
> Note that we are voting upon the source (tag), the npm package is provided
> for convenience.
> 
> The tag to be voted upon:
> v1.4.1-rc.1
> https://github.com/apache/pulsar-client-node/releases/tag/v1.4.1-rc.1

-1 (binding) we don’t vote on tags. We vote on signed packages that also have 
checksums.

https://www.apache.org/legal/release-policy.html#policy
https://infra.apache.org/release-publishing.html#definition

> 
> The npm package:
> https://www.npmjs.com/package/pulsar-client/v/1.4.1-rc.1
> 
> Please download the source files, and follow the README to build
> and run the Pulsar Node.js client.
> 
> Masahiro Sakamoto
> Yahoo Japan Corp.
> E-mail: massa...@yahoo-corp.jp

Re: [VOTE] Pulsar Node.js Client Release 1.3.2 Candidate 1

[Dave Fisher]

Hi Masahiro,

If you VOTE then there will be 3 +1 (binding) Votes.

Best Regards,
Dave

> On Oct 12, 2021, at 3:20 AM, Masahiro Sakamoto  wrote:
> 
> This is the first release candidate for Apache Pulsar Node.js client, version 
> 1.3.2.
> 
> The artifacts are exactly the same as the first vote, but I didn't upload 
> them and the signature
> to the staging repository of dist, so I'll run the vote again.
> 
> It fixes the following issues:
> https://github.com/apache/pulsar-client-node/milestone/9?closed=1
> 
> *** Please download, test and vote on this release. This vote will stay open
> for at least 72 hours ***
> 
> Note that we are voting upon the source (tag), the npm package is provided
> for convenience.
> 
> Source files:
> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-node-1.3.2-candidate-1/
> 
> SHA-512 checksum:
> 5f6c7e1a096a3ae66eee71c552af89532ed86bf94da3f3d49836c080426ee5dcaabeda440a989d51772d2e67e2dca9539ea83cfbc7c2a0847a0ec04b310f
>   v1.3.2-rc.1.tar.gz
> 
> The tag to be voted upon:
> v1.3.2-rc.1
> https://github.com/apache/pulsar-client-node/releases/tag/v1.3.2-rc.1
> 
> The npm package:
> https://www.npmjs.com/package/pulsar-client/v/1.3.2-rc.1
> 
> Pulsar's KEYS file containing PGP keys we use to sign the release:
> https://dist.apache.org/repos/dist/dev/pulsar/KEYS
> 
> Please download the source files, and follow the README to build
> and run the Pulsar Node.js client.
> 
> Masahiro Sakamoto
> Yahoo Japan Corp.
> E-mail: massa...@yahoo-corp.jp

Re: Apache Releases - Current Lines

[Dave Fisher]

Bringing this forward.

> On Oct 9, 2021, at 2:00 PM, Enrico Olivelli  wrote:
> 
> Dave,
> Thanks for pointing this out.
> I had sent a similar message a couple of weeks ago regarding the helm chart.
> I am not sure that we can recover past releases.
> But I believe that we can enforce a better process for the next releases of
> any artifact/sub project of Pulsar.
> 
> 
> 
> Il Ven 8 Ott 2021, 18:50 Dave Fisher  ha scritto:
> 
>> This note will cover several points about Apache Release Distribution
>> Policy. [1][2] All of the official Apache Pulsar release artifacts should
>> be on the project’s download page. [3]
>> What are the currently supported release artifacts?
>> Where must all source release artifacts be on Apache Distribution Channels?
>> How must download pages refer to current source release artifacts?
>> Where must release VOTEs place the artifacts?
>> Currently supported release artifacts.
>> From the download page it looks like the current main release series is
>> 2.8.1. From the urls from all of the older versions indicate that they are
>> archives. This implies that the 2.8.x is the only current version with
>> 2.9.x coming up. Is this what the community wants?
>> 
> 
> AFAIK we have 2.6, 2.7 and 2.8 as active release lines

We need Release Manager volunteers to roll dot releases for each of the current 
lines with required PRs applied.

> 
> In my opinion we should send 2.6. To EOL, possibly after cutting a final
> release.

Judging from what’s in the current official location 2.6 is already EOL unless 
an RM steps forward.

> 
> We should sent some EOL policy for every sub project that we release,
> otherwise we should maintain everything (at least provide security fixes)

Exactly. Whatever is not EOL needs maintenance releases.

> 
> 
> 
>> Apache Distribution Channels
>> Artifacts that are current must be uploaded via svn to
>> https://dist.apache.org/repos/dist/release/pulsar/
>> This location tells a different story about what is current: 2.7.2, 2.7.3,
>> 2.8.0, 2.8.1, adaptors-2.8.0, pulsar-manager, and client-go (but the other
>> clients are missing). It is fine to distribute to other channels like PIP,
>> NPM, and Maven Central and point users to them. It is required that all
>> source releases made by the PMC also be made on dist.apache.org.
>> 
>> Download page links to release artifacts
>> The ASF is moving away from its mirror network and is now using a CDN.
>> Consequently preferred links have changed slightly. See the current
>> requirements here. [4] There are a lot of adjustments to make.
>> 
> 
> Any volunteers?

I can look into the download page.

> 
> 
>> Client packages
>> While the separate page approach of describing the use and using
>> appropriate channels for the client language is fine. A download of a
>> separate source package for each client must be offered.
>> 
> 
> Agreed

I think that this should be done with the website redesign.

> 
>> 
>> Location of packages for VOTEs
>> These artifacts should be uploaded via svn to
>> https://dist.apache.org/repos/dist/dev/pulsar/. If that is done then it
>> is easy to svn to the distribution channel ounce the VOTE is completed. [5]
>> 
>> In synopsis there are three action items
>> Determine the current development versions.

Is 2.6 current or EOL?

>> Bringing the client package release process into policy.

The Node clients are in the proper channels.

>> Updating the download page.

Regards,
Dave

>> 
> 
> Thanks
> Enrico
> 
>> 
>> All The Best,
>> Dave
>> 
>> [1] https://www.apache.org/legal/release-policy.html#release-distribution
>> [2] https://infra.apache.org/release-distribution.html
>> [3] https://pulsar.apache.org/en/download/
>> [4] https://infra.apache.org/release-download-pages.html
>> [5] https://www.apache.org/legal/release-policy.html#stage

Re: [DISCUSS] cut pulsar-client-go 0.7.0 release

[Dave Fisher]

Are you volunteering to roll this release? If so then you should look at the PR 
and decide if you want to include it.

Sent from my iPhone

> On Oct 21, 2021, at 6:52 PM, Chris Kellogg  wrote:
> 
> What's the policy for waiting on PRs to merge? I was hoping to cut a
> release sometime next week if that works.
> 
>> On Thu, Oct 21, 2021 at 3:36 AM r...@apache.org 
>> wrote:
>> 
>> LGTM +1
>> 
>> This week I have tried to merge and process part of the code, And the pr
>> need to merge to master.
>> https://github.com/apache/pulsar-client-go/pull/528
>> 
>> --
>> Thanks
>> Xiaolong Ran
>> 
>> Enrico Olivelli  于2021年10月21日周四 下午2:33写道：
>> 
>>> Chris,
>>> 
>>> Il Gio 21 Ott 2021, 04:09 Chris Kellogg  ha
>> scritto:
>>> 
 Hello,
 
 It's been a while since the last go client release (July 25th). There
>>> have
 been many fixes that have gone in since then and I think it's time for
>> a
 new release.
 
>>> 
>>> Good idea
>>> +1
>>> 
>>> Enrico
>>> 
>>> 
>>> 
 I would like to volunteer to coordinate the next go client release.
>>> Please
 let me know if that works and If there are things that need to go in
>>> before
 a new release is cut.
 
 Thanks.
 Chris
 
>>> 
>> 

Re: Revote: Pulsar website concepts

[Dave Fisher]

Hi -

Great choices. Thank you Melissa.

Here are my preferences in order: 1, 3B, 3A, and 2.

I think that the body text should be slightly larger - roughly 90% of the 
heading in size.

I agree about larger icons. We can “bikeshed” about other adjustments later 
after picking the design.

All The Best,
Dave

> On Oct 22, 2021, at 8:37 AM, Devin Bost  wrote:
> 
> +1 for Option 1, though I'd made the icons bigger and increase the color
> contrast. Preferably, I'd like to see some additional minor color
> variations of Option 1 (primarily to increase readability on small
> devices), though I like the blue theme.
> 
> Also, these pictures only cover the desktop display. What do they look like
> on mobile devices? As over half of web browsing happens on mobile devices
> now, we should consider the mobile view.
> 
> Devin G. Bost
> 
> 
> On Fri, Oct 22, 2021 at 8:02 AM Melissa Logan  wrote:
> 
>> Hi all,
>> 
>> I'm starting a new thread to vote on Pulsar website designs, per request in
>> previous thread. This vote is being put forth through lazy consensus with
>> 72 hours to vote (1). Please reply here with your preferred site design --
>> option 1, 2, 3a or 3b.
>> 
>> Background:
>> As discussed on this thread (2), my team offered to develop new website
>> design concepts for the community that we would implement in collaboration
>> with others on that thread. We are pleased to share three concepts for your
>> consideration below.
>> 
>> Notes:
>> * This vote is for ONLY design concept at this stage -- not content or copy
>> updates. Those will be tackled later.
>> * ASF recently requested no corporate logos be displayed on the homepage,
>> which is reflected in these designs.
>> * The navigation is based on the recommendation put forth in the previous
>> thread.
>> 
>> Project milestones:
>> * Choose design concept
>> * Meet with Pulsar infrastructure/website committers
>> * Begin website content review
>> * Begin template designs (8-10 estimated)
>> * Confirm IA
>> * Finalize content for community input/approval
>> * Finalize template designs for community input/approval
>> * Ensure ASF compliance
>> * Launch
>> 
>> Design concepts here:
>> https://www.dropbox.com/sh/rwvnb36g7zmxptz/AAAIKHqfkWD6OwdY-OSG7Ovoa?dl=0
>> 
>> Any questions, just let me know. Thanks!
>> 
>> Melissa Logan
>> 
>> (1) https://community.apache.org/committers/lazyConsensus.html
>> (2)
>> 
>> https://lists.apache.org/thread.html/ra618c666d52f518741f64313969a9f16f20b8f06cd444f499f77cd93%40%3Cdev.pulsar.apache.org%3E
>> 
>> 
>> -- Forwarded message -
>> From: Melissa Logan 
>> Date: Tue, Sep 21, 2021 at 10:01 AM
>> Subject: Re: Pulsar website concepts -- please vote
>> To: Dev 
>> 
>> 
>> Hi all, if you have questions or comments about the design concepts, please
>> ask here by tomorrow (Wednesday Sep. 22) your time and we will
>> answer/incorporate suggestions as needed.
>> 
>> A re-vote will be done on the ML following. Thank you!
>> 
>> On Fri, Sep 17, 2021 at 5:53 PM Melissa Logan 
>> wrote:
>>> 
>>> Hi Sijie, happy to discuss design concepts. Please share any questions
>> here.
>>> 
>>> Per discussions on the previously mentioned thread, we offered to create
>> concepts based on the suggested navigation. Any of these can be tweaked to
>> match what the community wants.
>>> 
>>> Option 1 was the favorite. You can see results of the voting here:
>>> https://constantialabs.typeform.com/report/AJbOYCp3/EnObAQV70pg3tdZd
>>> 
>>> We can hold a revote and ask people to vote on the ML instead, no
>> problem. If folks would like to input on design concepts, please do here by
>> next Wednesday and we can incorporate suggestions into updated concepts as
>> needed -- then hold the revote.
>>> 
>>> The design concepts are available here:
>>> https://app.box.com/s/klmht8qohc8bnncq98v6k323992adi5a
>>> 
>>> Note for option 1: The waves will not be static, but would animate based
>> on the Codepen: https://codepen.io/miketravers/full/KKqwdVw
>>> 
>>> 
>>> On Thu, Sep 16, 2021 at 11:28 PM Sijie Guo  wrote:
 
 Hi Melissa,
 
 Thank you for your efforts for putting this together!
 
 I have a few comments:
 
 1) Adding "vote" in the email title is a bit confusing. The design
>> should
 be brought up for discussion before going into a vote process.
 2) The discussion (or "vote") should happen in the mailing list rather
>> than
 go into a form.
 3) It seems that you have to create an account in order to view the
>> design
 concepts. Can you put them in a public available location for people to
 access?
 
 - Sijie
 
 
 
 On Wed, Sep 8, 2021 at 2:08 PM Melissa Logan 
>> wrote:
 
> Pulsar community:
> 
> As discussed on this thread (1), my team offered to develop new
>> website
> design concepts for the community that we would implement in
>> collaboration
> with others on that thread. We are pleased to share three concepts for
>> your
> 

Re: Size of Asf-site branch in Pulsar repo

[Dave Fisher]

Hi -

> On Oct 29, 2021, at 12:02 PM, Matteo Merli  wrote:
> 
> The Pulsar website is getting published through a CI job that updates
> the generated HTML files and commits them in the Pulsar repo, in a
> separate branch ('asf-site'). From there the site is immediately
> visible on the web.
> 
> One of the issues with this process is that we have a lot of updates
> of generated HTML files that are growing the size of the Pulsar Git
> repo. Each time we clone, the entire repo has to be fetched by
> developers and users.
> 
> This is somewhat made worse by having daily updates in many HTML files
> to update timestamps. I just merged a fix for that
> https://github.com/apache/pulsar/pull/12538 .
> 
> The size of the clone git repo is already at 1.4 GB. 90% of this size
> is due to the 'asf-site' branch.
> 
> Ideally, we should try to find a solution to use an ad-hoc repo for
> the website deployment, outside the main Pulsar repo.

We can have as many apache/pulsar-* repos as the PMC wants

If we create a pulsar-site repos we can publish from multiple branches.

See GitHub.com/apache/openjpa-site

The main branch could contain website sources.
The asf-site branch would have the built website.
.asf.yaml
publish:
  profile: ~
  whoami: asf-site
A builds branch could have api docs that seldom change. OpenJPA keeps every 
release…
.asf.yaml
publish:
  profile: ~
  subdir: output/builds
  whoami: builds



> 
> In the meantime, I propose to truncate the history of the "asf-site"
> branch and squash all commits into a single one, in order to reduce
> the repo size.

+1

> 
> Let me know what you think.
> 
> Matteo
> 
> --
> Matteo Merli
> 

[DISCUSS] New repository for website - pulsar-site

[Dave Fisher]

Hi -

There are two efforts happening in the community around website refresh.

(1) Docusaurus upgrades.
(2) New web design.

There is an effort to eliminate all the extra commits in the asf-site branch of 
the main repository. In that thread I proposed a new repository for the website.

We can then discuss migration and development both on this mailing list and as 
PRs and Issues in that repository.

Do we want to have a PIP process here or can we be less formal? I think that 
PRs. Issues, and simple commits can be sufficient.

Unless there are objections I will create a new repository - pulsar-site on 
Friday in 72 hours.
‘
Regards,
Dave

Re: [DISCUSS] New repository for website - pulsar-site

[Dave Fisher]

Show me where the code is that commits to the asf-site branch.

> On Nov 17, 2021, at 12:25 PM, Matteo Merli  wrote:
> 
> I agree with that.
> 
> I understand that there are tradeoffs for each approach, though the
> original intention was to allow for doc changes to be committed in the
> same PR as the code change. That doesn't have to be the case always,
> especially for larger multi-PR changes, but it makes it easier to do
> quick corrections to the docs.
> 
> I think the bigger problem here is to get rid of the generated site
> HTML stuff from the main pulsar repo.
> 
> --
> Matteo Merli
> 
> 
> On Wed, Nov 17, 2021 at 12:16 PM Enrico Olivelli  wrote:
>> 
>> Dave,
>> Having a new repo will make it harder for developers to contribute
>> documentation.
>> 
>> Usually engineers do  it like and do not have time to write docs.
>> 
>> If we ask them to create two PRs only to add, for instance, a new
>> configuration option, then it would be somehow a pain.
>> 
>> I am not saying that we shouldn't go this way, but it would be kind of a
>> pain for someone and we need to ear more voices.
>> 
>> Enrico
>> 
>> Il Mer 17 Nov 2021, 19:28 Sijie Guo  ha scritto:
>> 
>>> I think we should have a PIP for this. Because this impacts all the
>>> developers who are making documentation changes.
>>> 
>>> - Sijie
>>> 
>>> On Tue, Nov 16, 2021 at 8:46 AM Dave Fisher  wrote:
>>> 
>>>> Hi -
>>>> 
>>>> There are two efforts happening in the community around website refresh.
>>>> 
>>>> (1) Docusaurus upgrades.
>>>> (2) New web design.
>>>> 
>>>> There is an effort to eliminate all the extra commits in the asf-site
>>>> branch of the main repository. In that thread I proposed a new repository
>>>> for the website.
>>>> 
>>>> We can then discuss migration and development both on this mailing list
>>>> and as PRs and Issues in that repository.
>>>> 
>>>> Do we want to have a PIP process here or can we be less formal? I think
>>>> that PRs. Issues, and simple commits can be sufficient.
>>>> 
>>>> Unless there are objections I will create a new repository - pulsar-site
>>>> on Friday in 72 hours.
>>>> ‘
>>>> Regards,
>>>> Dave
>>>> 
>>>> 
>>> 

Re: [DISCUSS] New repository for website - pulsar-site

[Dave Fisher]

If we change ORIGIN_REPO[1] to point to a new pulsar-site repos.
Then with the correct .asf.yaml file changes we can remove the asf-site branch.
I see that the publish is run from this workflow [2]
Let me think about a PR to make the move.

Regards,
Dave

[1] 
https://github.com/apache/pulsar/blob/7a34cebca25e6e584e8b758e6bd58c1c4fe8a58e/site2/tools/publish-website.sh#L25
[2] 
https://github.com/apache/pulsar/blob/master/.github/workflows/ci-pulsar-website-build.yaml


> On Nov 17, 2021, at 12:31 PM, Matteo Merli  wrote:
> 
> https://github.com/apache/pulsar/blob/master/site2/tools/publish-website.sh
> 
> 
> --
> Matteo Merli
> 
> 
> On Wed, Nov 17, 2021 at 12:29 PM Dave Fisher  wrote:
>> 
>> Show me where the code is that commits to the asf-site branch.
>> 
>>> On Nov 17, 2021, at 12:25 PM, Matteo Merli  wrote:
>>> 
>>> I agree with that.
>>> 
>>> I understand that there are tradeoffs for each approach, though the
>>> original intention was to allow for doc changes to be committed in the
>>> same PR as the code change. That doesn't have to be the case always,
>>> especially for larger multi-PR changes, but it makes it easier to do
>>> quick corrections to the docs.
>>> 
>>> I think the bigger problem here is to get rid of the generated site
>>> HTML stuff from the main pulsar repo.
>>> 
>>> --
>>> Matteo Merli
>>> 
>>> 
>>> On Wed, Nov 17, 2021 at 12:16 PM Enrico Olivelli  
>>> wrote:
>>>> 
>>>> Dave,
>>>> Having a new repo will make it harder for developers to contribute
>>>> documentation.
>>>> 
>>>> Usually engineers do  it like and do not have time to write docs.
>>>> 
>>>> If we ask them to create two PRs only to add, for instance, a new
>>>> configuration option, then it would be somehow a pain.
>>>> 
>>>> I am not saying that we shouldn't go this way, but it would be kind of a
>>>> pain for someone and we need to ear more voices.
>>>> 
>>>> Enrico
>>>> 
>>>> Il Mer 17 Nov 2021, 19:28 Sijie Guo  ha scritto:
>>>> 
>>>>> I think we should have a PIP for this. Because this impacts all the
>>>>> developers who are making documentation changes.
>>>>> 
>>>>> - Sijie
>>>>> 
>>>>> On Tue, Nov 16, 2021 at 8:46 AM Dave Fisher  wrote:
>>>>> 
>>>>>> Hi -
>>>>>> 
>>>>>> There are two efforts happening in the community around website refresh.
>>>>>> 
>>>>>> (1) Docusaurus upgrades.
>>>>>> (2) New web design.
>>>>>> 
>>>>>> There is an effort to eliminate all the extra commits in the asf-site
>>>>>> branch of the main repository. In that thread I proposed a new repository
>>>>>> for the website.
>>>>>> 
>>>>>> We can then discuss migration and development both on this mailing list
>>>>>> and as PRs and Issues in that repository.
>>>>>> 
>>>>>> Do we want to have a PIP process here or can we be less formal? I think
>>>>>> that PRs. Issues, and simple commits can be sufficient.
>>>>>> 
>>>>>> Unless there are objections I will create a new repository - pulsar-site
>>>>>> on Friday in 72 hours.
>>>>>> ‘
>>>>>> Regards,
>>>>>> Dave
>>>>>> 
>>>>>> 
>>>>> 
>> 

Re: [DISCUSS] New repository for website - pulsar-site

[Dave Fisher]

I’ve updated my fork of apache/pulsar

I’m not seeing how to run the workflow "CI - Pulsar Website build”. Any ideas?

If not then I’m going to need to test locally and it will take some time to 
ready it.


> On Nov 17, 2021, at 1:15 PM, Matteo Merli  wrote:
> 
> Yes, that should work.
> 
> After that we can go ahead and remove `asf-site` from the main repo,
> although we need to make it "unprotected" to be able to do so.

Yes once we have moved over to the new then we can ask Infra to take care of 
the branch protection along with deleting it.

When I create the new repository I will copy all of the asf-site branch which 
will take care of transferring the parts of the site not actively being built.

Regards,
Dave

> 
> 
> --
> Matteo Merli
> 
> 
> On Wed, Nov 17, 2021 at 12:46 PM Dave Fisher  wrote:
>> 
>> If we change ORIGIN_REPO[1] to point to a new pulsar-site repos.
>> Then with the correct .asf.yaml file changes we can remove the asf-site 
>> branch.
>> I see that the publish is run from this workflow [2]
>> Let me think about a PR to make the move.
>> 
>> Regards,
>> Dave
>> 
>> [1] 
>> https://github.com/apache/pulsar/blob/7a34cebca25e6e584e8b758e6bd58c1c4fe8a58e/site2/tools/publish-website.sh#L25
>> [2] 
>> https://github.com/apache/pulsar/blob/master/.github/workflows/ci-pulsar-website-build.yaml
>> 
>> 
>>> On Nov 17, 2021, at 12:31 PM, Matteo Merli  wrote:
>>> 
>>> https://github.com/apache/pulsar/blob/master/site2/tools/publish-website.sh
>>> 
>>> 
>>> --
>>> Matteo Merli
>>> 
>>> 
>>> On Wed, Nov 17, 2021 at 12:29 PM Dave Fisher  wrote:
>>>> 
>>>> Show me where the code is that commits to the asf-site branch.
>>>> 
>>>>> On Nov 17, 2021, at 12:25 PM, Matteo Merli  wrote:
>>>>> 
>>>>> I agree with that.
>>>>> 
>>>>> I understand that there are tradeoffs for each approach, though the
>>>>> original intention was to allow for doc changes to be committed in the
>>>>> same PR as the code change. That doesn't have to be the case always,
>>>>> especially for larger multi-PR changes, but it makes it easier to do
>>>>> quick corrections to the docs.
>>>>> 
>>>>> I think the bigger problem here is to get rid of the generated site
>>>>> HTML stuff from the main pulsar repo.
>>>>> 
>>>>> --
>>>>> Matteo Merli
>>>>> 
>>>>> 
>>>>> On Wed, Nov 17, 2021 at 12:16 PM Enrico Olivelli  
>>>>> wrote:
>>>>>> 
>>>>>> Dave,
>>>>>> Having a new repo will make it harder for developers to contribute
>>>>>> documentation.
>>>>>> 
>>>>>> Usually engineers do  it like and do not have time to write docs.
>>>>>> 
>>>>>> If we ask them to create two PRs only to add, for instance, a new
>>>>>> configuration option, then it would be somehow a pain.
>>>>>> 
>>>>>> I am not saying that we shouldn't go this way, but it would be kind of a
>>>>>> pain for someone and we need to ear more voices.
>>>>>> 
>>>>>> Enrico
>>>>>> 
>>>>>> Il Mer 17 Nov 2021, 19:28 Sijie Guo  ha scritto:
>>>>>> 
>>>>>>> I think we should have a PIP for this. Because this impacts all the
>>>>>>> developers who are making documentation changes.
>>>>>>> 
>>>>>>> - Sijie
>>>>>>> 
>>>>>>> On Tue, Nov 16, 2021 at 8:46 AM Dave Fisher  wrote:
>>>>>>> 
>>>>>>>> Hi -
>>>>>>>> 
>>>>>>>> There are two efforts happening in the community around website 
>>>>>>>> refresh.
>>>>>>>> 
>>>>>>>> (1) Docusaurus upgrades.
>>>>>>>> (2) New web design.
>>>>>>>> 
>>>>>>>> There is an effort to eliminate all the extra commits in the asf-site
>>>>>>>> branch of the main repository. In that thread I proposed a new 
>>>>>>>> repository
>>>>>>>> for the website.
>>>>>>>> 
>>>>>>>> We can then discuss migration and development both on this mailing list
>>>>>>>> and as PRs and Issues in that repository.
>>>>>>>> 
>>>>>>>> Do we want to have a PIP process here or can we be less formal? I think
>>>>>>>> that PRs. Issues, and simple commits can be sufficient.
>>>>>>>> 
>>>>>>>> Unless there are objections I will create a new repository - 
>>>>>>>> pulsar-site
>>>>>>>> on Friday in 72 hours.
>>>>>>>> ‘
>>>>>>>> Regards,
>>>>>>>> Dave
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>> 
>> 

Re: [DISCUSS] New repository for website - pulsar-site

[Dave Fisher]

I’m going to work through 
https://github.com/apache/pulsar/blob/master/site2/README.md

I’ll make sure that any changes related to the asf-site branch don’t have issue 
with that.

We may want to be able to publish alternative web designs to a staging sites.

> On Nov 17, 2021, at 3:02 PM, Dave Fisher  wrote:
> 
> I’ve updated my fork of apache/pulsar
> 
> I’m not seeing how to run the workflow "CI - Pulsar Website build”. Any ideas?
> 
> If not then I’m going to need to test locally and it will take some time to 
> ready it.
> 
> 
>> On Nov 17, 2021, at 1:15 PM, Matteo Merli  wrote:
>> 
>> Yes, that should work.
>> 
>> After that we can go ahead and remove `asf-site` from the main repo,
>> although we need to make it "unprotected" to be able to do so.
> 
> Yes once we have moved over to the new then we can ask Infra to take care of 
> the branch protection along with deleting it.
> 
> When I create the new repository I will copy all of the asf-site branch which 
> will take care of transferring the parts of the site not actively being built.

I have created the new repository and populated the asf-site branch: 
https://github.com/apache/pulsar-site/tree/asf-site

It publishes to a staging url which you can see here: 
https://pulsar.staged.apache.org

Once we are ready we alter: 
https://github.com/apache/pulsar-site/blob/asf-site/.asf.yaml

Per: https://cwiki.apache.org/confluence/display/INFRA/git+-+.asf.yaml+features

> 
> Regards,
> Dave
> 
>> 
>> 
>> --
>> Matteo Merli
>> 
>> 
>> On Wed, Nov 17, 2021 at 12:46 PM Dave Fisher  wrote:
>>> 
>>> If we change ORIGIN_REPO[1] to point to a new pulsar-site repos.
>>> Then with the correct .asf.yaml file changes we can remove the asf-site 
>>> branch.
>>> I see that the publish is run from this workflow [2]
>>> Let me think about a PR to make the move.
>>> 
>>> Regards,
>>> Dave
>>> 
>>> [1] 
>>> https://github.com/apache/pulsar/blob/7a34cebca25e6e584e8b758e6bd58c1c4fe8a58e/site2/tools/publish-website.sh#L25
>>> [2] 
>>> https://github.com/apache/pulsar/blob/master/.github/workflows/ci-pulsar-website-build.yaml
>>> 
>>> 
>>>> On Nov 17, 2021, at 12:31 PM, Matteo Merli  wrote:
>>>> 
>>>> https://github.com/apache/pulsar/blob/master/site2/tools/publish-website.sh
>>>> 
>>>> 
>>>> --
>>>> Matteo Merli
>>>> 
>>>> 
>>>> On Wed, Nov 17, 2021 at 12:29 PM Dave Fisher  wrote:
>>>>> 
>>>>> Show me where the code is that commits to the asf-site branch.
>>>>> 
>>>>>> On Nov 17, 2021, at 12:25 PM, Matteo Merli  
>>>>>> wrote:
>>>>>> 
>>>>>> I agree with that.
>>>>>> 
>>>>>> I understand that there are tradeoffs for each approach, though the
>>>>>> original intention was to allow for doc changes to be committed in the
>>>>>> same PR as the code change. That doesn't have to be the case always,
>>>>>> especially for larger multi-PR changes, but it makes it easier to do
>>>>>> quick corrections to the docs.
>>>>>> 
>>>>>> I think the bigger problem here is to get rid of the generated site
>>>>>> HTML stuff from the main pulsar repo.
>>>>>> 
>>>>>> --
>>>>>> Matteo Merli
>>>>>> 
>>>>>> 
>>>>>> On Wed, Nov 17, 2021 at 12:16 PM Enrico Olivelli  
>>>>>> wrote:
>>>>>>> 
>>>>>>> Dave,
>>>>>>> Having a new repo will make it harder for developers to contribute
>>>>>>> documentation.
>>>>>>> 
>>>>>>> Usually engineers do  it like and do not have time to write docs.
>>>>>>> 
>>>>>>> If we ask them to create two PRs only to add, for instance, a new
>>>>>>> configuration option, then it would be somehow a pain.
>>>>>>> 
>>>>>>> I am not saying that we shouldn't go this way, but it would be kind of a
>>>>>>> pain for someone and we need to ear more voices.
>>>>>>> 
>>>>>>> Enrico
>>>>>>> 
>>>>>>> Il Mer 17 Nov 2021, 19:28 Sijie Guo  ha scritto:
>>>>>>> 
>>>>>>>> I think we should have a PIP for this. Because this impacts all the
>>>>>>>> developers who are making documentation changes.
>>>>>>>> 
>>>>>>>> - Sijie
>>>>>>>> 
>>>>>>>> On Tue, Nov 16, 2021 at 8:46 AM Dave Fisher  wrote:
>>>>>>>> 
>>>>>>>>> Hi -
>>>>>>>>> 
>>>>>>>>> There are two efforts happening in the community around website 
>>>>>>>>> refresh.
>>>>>>>>> 
>>>>>>>>> (1) Docusaurus upgrades.
>>>>>>>>> (2) New web design.
>>>>>>>>> 
>>>>>>>>> There is an effort to eliminate all the extra commits in the asf-site
>>>>>>>>> branch of the main repository. In that thread I proposed a new 
>>>>>>>>> repository
>>>>>>>>> for the website.
>>>>>>>>> 
>>>>>>>>> We can then discuss migration and development both on this mailing 
>>>>>>>>> list
>>>>>>>>> and as PRs and Issues in that repository.
>>>>>>>>> 
>>>>>>>>> Do we want to have a PIP process here or can we be less formal? I 
>>>>>>>>> think
>>>>>>>>> that PRs. Issues, and simple commits can be sufficient.
>>>>>>>>> 
>>>>>>>>> Unless there are objections I will create a new repository - 
>>>>>>>>> pulsar-site
>>>>>>>>> on Friday in 72 hours.
>>>>>>>>> ‘
>>>>>>>>> Regards,
>>>>>>>>> Dave
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>> 
>>> 
> 

Re: [DISCUSS] New repository for website - pulsar-site

[Dave Fisher]

I’m making progress here, but I need help getting the pulsarbot GH secret into 
the pulsar-site repository.

If that secret can be shared directly to me then I can fully test before adding 
my PR.

Thanks,
Dave

> On Nov 17, 2021, at 3:57 PM, Dave Fisher  wrote:
> 
> I’m going to work through 
> https://github.com/apache/pulsar/blob/master/site2/README.md
> 
> I’ll make sure that any changes related to the asf-site branch don’t have 
> issue with that.
> 
> We may want to be able to publish alternative web designs to a staging sites.
> 
>> On Nov 17, 2021, at 3:02 PM, Dave Fisher  wrote:
>> 
>> I’ve updated my fork of apache/pulsar
>> 
>> I’m not seeing how to run the workflow "CI - Pulsar Website build”. Any 
>> ideas?
>> 
>> If not then I’m going to need to test locally and it will take some time to 
>> ready it.
>> 
>> 
>>> On Nov 17, 2021, at 1:15 PM, Matteo Merli  wrote:
>>> 
>>> Yes, that should work.
>>> 
>>> After that we can go ahead and remove `asf-site` from the main repo,
>>> although we need to make it "unprotected" to be able to do so.
>> 
>> Yes once we have moved over to the new then we can ask Infra to take care of 
>> the branch protection along with deleting it.
>> 
>> When I create the new repository I will copy all of the asf-site branch 
>> which will take care of transferring the parts of the site not actively 
>> being built.
> 
> I have created the new repository and populated the asf-site branch: 
> https://github.com/apache/pulsar-site/tree/asf-site
> 
> It publishes to a staging url which you can see here: 
> https://pulsar.staged.apache.org
> 
> Once we are ready we alter: 
> https://github.com/apache/pulsar-site/blob/asf-site/.asf.yaml
> 
> Per: 
> https://cwiki.apache.org/confluence/display/INFRA/git+-+.asf.yaml+features
> 
>> 
>> Regards,
>> Dave
>> 
>>> 
>>> 
>>> --
>>> Matteo Merli
>>> 
>>> 
>>> On Wed, Nov 17, 2021 at 12:46 PM Dave Fisher  wrote:
>>>> 
>>>> If we change ORIGIN_REPO[1] to point to a new pulsar-site repos.
>>>> Then with the correct .asf.yaml file changes we can remove the asf-site 
>>>> branch.
>>>> I see that the publish is run from this workflow [2]
>>>> Let me think about a PR to make the move.
>>>> 
>>>> Regards,
>>>> Dave
>>>> 
>>>> [1] 
>>>> https://github.com/apache/pulsar/blob/7a34cebca25e6e584e8b758e6bd58c1c4fe8a58e/site2/tools/publish-website.sh#L25
>>>> [2] 
>>>> https://github.com/apache/pulsar/blob/master/.github/workflows/ci-pulsar-website-build.yaml
>>>> 
>>>> 
>>>>> On Nov 17, 2021, at 12:31 PM, Matteo Merli  wrote:
>>>>> 
>>>>> https://github.com/apache/pulsar/blob/master/site2/tools/publish-website.sh
>>>>> 
>>>>> 
>>>>> --
>>>>> Matteo Merli
>>>>> 
>>>>> 
>>>>> On Wed, Nov 17, 2021 at 12:29 PM Dave Fisher  wrote:
>>>>>> 
>>>>>> Show me where the code is that commits to the asf-site branch.
>>>>>> 
>>>>>>> On Nov 17, 2021, at 12:25 PM, Matteo Merli  
>>>>>>> wrote:
>>>>>>> 
>>>>>>> I agree with that.
>>>>>>> 
>>>>>>> I understand that there are tradeoffs for each approach, though the
>>>>>>> original intention was to allow for doc changes to be committed in the
>>>>>>> same PR as the code change. That doesn't have to be the case always,
>>>>>>> especially for larger multi-PR changes, but it makes it easier to do
>>>>>>> quick corrections to the docs.
>>>>>>> 
>>>>>>> I think the bigger problem here is to get rid of the generated site
>>>>>>> HTML stuff from the main pulsar repo.
>>>>>>> 
>>>>>>> --
>>>>>>> Matteo Merli
>>>>>>> 
>>>>>>> 
>>>>>>> On Wed, Nov 17, 2021 at 12:16 PM Enrico Olivelli  
>>>>>>> wrote:
>>>>>>>> 
>>>>>>>> Dave,
>>>>>>>> Having a new repo will make it harder for developers to contribute
>>>>>>>> documentation.
>>>>>>>> 
>>>>>>>> Usually en

Re: [DISCUSS] New repository for website - pulsar-site

[Dave Fisher]

How might the planned move of various components and adaptors out of the main 
repository impact the choice to have the docs in the main repository?

> On Nov 17, 2021, at 12:25 PM, Matteo Merli  wrote:
> 
> I agree with that.
> 
> I understand that there are tradeoffs for each approach, though the
> original intention was to allow for doc changes to be committed in the
> same PR as the code change. That doesn't have to be the case always,
> especially for larger multi-PR changes, but it makes it easier to do
> quick corrections to the docs.
> 
> I think the bigger problem here is to get rid of the generated site
> HTML stuff from the main pulsar repo.
> 
> --
> Matteo Merli
> 
> 
> On Wed, Nov 17, 2021 at 12:16 PM Enrico Olivelli  wrote:
>> 
>> Dave,
>> Having a new repo will make it harder for developers to contribute
>> documentation.
>> 
>> Usually engineers do  it like and do not have time to write docs.
>> 
>> If we ask them to create two PRs only to add, for instance, a new
>> configuration option, then it would be somehow a pain.
>> 
>> I am not saying that we shouldn't go this way, but it would be kind of a
>> pain for someone and we need to ear more voices.
>> 
>> Enrico
>> 
>> Il Mer 17 Nov 2021, 19:28 Sijie Guo  ha scritto:
>> 
>>> I think we should have a PIP for this. Because this impacts all the
>>> developers who are making documentation changes.
>>> 
>>> - Sijie
>>> 
>>> On Tue, Nov 16, 2021 at 8:46 AM Dave Fisher  wrote:
>>> 
>>>> Hi -
>>>> 
>>>> There are two efforts happening in the community around website refresh.
>>>> 
>>>> (1) Docusaurus upgrades.
>>>> (2) New web design.
>>>> 
>>>> There is an effort to eliminate all the extra commits in the asf-site
>>>> branch of the main repository. In that thread I proposed a new repository
>>>> for the website.
>>>> 
>>>> We can then discuss migration and development both on this mailing list
>>>> and as PRs and Issues in that repository.
>>>> 
>>>> Do we want to have a PIP process here or can we be less formal? I think
>>>> that PRs. Issues, and simple commits can be sufficient.
>>>> 
>>>> Unless there are objections I will create a new repository - pulsar-site
>>>> on Friday in 72 hours.
>>>> ‘
>>>> Regards,
>>>> Dave
>>>> 
>>>> 
>>> 

Re: Creating Good Release notes

[Dave Fisher]

> On Dec 2, 2021, at 3:11 AM, Enrico Olivelli  wrote:
> 
> Hello community,
> 
> There is an open discussion on the Pulsar 2.9.0 release notes PR:
> https://github.com/apache/pulsar/pull/12425

In reading through the comments there I’m noticing these main themes.

(1) Wanting to standardize / rewrite PR titles. I don’t think that Developers 
from around the globe are going to do that. If this needs to be done then a 
volunteer who finds it important to them should review and update the PRs as 
they are being created.

(2) The other issue on consistently marking PR ids and PIPs should be possible 
with tooling.

I would suggest that the PMC needs to provide more support to whoever the RM 
for a release is. RMs are perhaps the most valuable committers we have and 
let’s try not to overburden them with extra tasks.

- POI uses automation and accepts whatever title are on the commits.
- For OpenOffice the RM doesn’t create the Release Notes, other members do into 
English and then various community members translate them. 

So, let’s not slow the process, but let’s go for eventual consistency.

Regards,
Dave

> 
> I have created the block of release notes by downloading the list of PR
> using some GitHub API.
> Then I have manually classified:
> - News and Noteworthy: cool things in the Release
> - Breaking Changes: things you MUST know when you upgrade
> - Java Client, C++ Client, Python Client, Functions/Pulsar IO
> 
> The goal is to provide useful information for people who want to upgrade
> Pulsar.
> 
> My problems are:
> - PR titles are often badly written, but I don't want to fix all of them
> (typos,  tenses of verbs, formatting)
> - There are more than 300 PRs, I don't want to classify them manually, I
> just highlighted the most important from my point of view
> 
> If for 2.9.0 we still keep a list of PR, then I believe that the current
> status of the patch is good.
> 
> If we want to do it another way, then I am now asking if there is someone
> who can volunteer in fixing and classifying the list of 300 PRs, it is a
> huge task.
> 
> There is already much more work to do to get 2.9.0 completely released (and
> also PulsarAdapters) and we have to cut 2.9.1 as soon as possible due to a
> bad regression found in 2.9.0.
> 
> Thanks
> Enrico

Re: [DISCUSS] Pulsar Protocol For Client Timeouts and Creating Producers

[Dave Fisher]

> On Dec 1, 2021, at 10:21 AM, Michael Marshall  wrote:
> 
> Following up here, I am still in need of reviews for PR [0]. It
> introduces an important clarification to the Pulsar Protocol Spec.
> Please take a look, if you are able.
> 
> [0] - https://github.com/apache/pulsar/pull/12948

I added some suggested reviewers to the PR.

Regards,
Dave

> 
> Thanks!
> Michael
> 
> On Tue, Nov 23, 2021 at 1:10 PM Michael Marshall  
> wrote:
>> 
>> I created a PR to update the protocol's documentation:
>> https://github.com/apache/pulsar/pull/12948. Please take a look, if
>> you're able.
>> 
>> Once this PR is accepted/merged, I will follow up with an update to
>> the Java client.
>> 
>> - Michael
>> 
>> On Thu, Nov 18, 2021 at 1:29 PM Michael Marshall  
>> wrote:
>>> 
>>> I view this as an edge case of the Pulsar Protocol that requires
>>> clarification. Once we clarify the spec, we can update the clients to
>>> conform to the spec. I don't think we need to give end users control
>>> over this small part of the protocol.
>>> 
>>> After reviewing the design a bit more, I think we should update the
>>> Java client to send the `CloseProducer` command, as well as update the
>>> spec to recommend this implementation. While the `ServerCnx` class
>>> "works" both ways, the current Java client implementation leads to
>>> unnecessary calls, unnecessary errors, and a longer time to recovery.
>>> 
>>> Specifically, if the client fails to send a `CloseProducer` command,
>>> it ends up getting into a sequence of retries where each new
>>> `Producer` command receives an immediate `ErrorResponse` because the
>>> `ServerCnx` already has a pending producer. By sending a
>>> `CloseProducer` command, the client gives the broker permission to
>>> stop keeping track of the original create producer request. It also
>>> means that if the topic eventually loads, the broker will respond to
>>> the right request id with a `ProducerSuccessResponse` command.
>>> 
>>> I will follow up with an update to the client and the protocol spec,
>>> unless there are any objections.
>>> 
>>> Thanks,
>>> Michael
>>> 
>>> On Thu, Nov 18, 2021 at 12:25 PM Neng Lu  wrote:
 
 How about making the behavior when timeout configurable? And by default, 
 it will send the `CloseProducer` cmd?
 
 On 2021/11/17 05:52:21 Michael Marshall wrote:
> Hi All,
> 
> I noticed that the `ServerCnxTest#testCreateProducerTimeout` test
> indicates that a producer will send a `CloserProducer` command when
> producer creation times out for the client.
> 
> The Java client does not follow this protocol. When the producer
> creation times out, it just schedules another attempt to create the
> producer.
> 
> The C++ client does follow this protocol and is annotated with the
> following comment:
> 
>> // Creating the producer has timed out. We need to ensure the broker 
>> closes the producer
>> // in case it was indeed created, otherwise it might prevent new create 
>> producer operation,
>> // since we are not closing the connection
> 
> I don't see anything in our official protocol spec indicating the
> "right" behavior. Given the test coverage, it seems like the initial
> design was to expect a `CloseProducer` command. However, I also see that
> the broker's `ServerCnx` class will reply to a redundant `Producer`
> command with a `ProducerSuccess` command, as long as the producer
> is already created.
> 
> Should I submit a PR to update the Java client to send a
> `CloseProducer` command when a `Producer` command times out?
> 
> Thanks,
> Michael
> 

[Website] CI - Pulsar Website Build takes a long time

[Dave Fisher]

Hi -

I noticed some things about the website build. 
https://github.com/apache/pulsar/actions/workflows/ci-pulsar-website-build.yaml

(1) It runs for nearly 2 hours.
(2) If it exceeds 2 hours then it is canceled.
(3) It runs every 6 hours.

We share runners with all of the ASF.

I think we need to change from 4 runs per day to either 2 or 3 runs.

We can also consider switching to the ASF’s Jenkins at 
https://builds.apache.org/job/Pulsar/ - it might give better performance.

Regards,
Dave

Re: [DISCUSS] How to handle stale PRs

[Dave Fisher]

I think that any Pulsar committer ought to close any PR that is more than one 
year old. That would clear about 75 from the backlog. The OP should be informed 
and if they are still interested then they can discuss it here.

So when a stale PR is closed we should suggest that the OP subscribe to and 
email dev@pulsar.apache.org to discuss the PR.

All the Best,
Dave
.
> On Dec 3, 2021, at 9:17 AM, tison  wrote:
> 
> From my experience, any process won't work. The only way is to inspire more
> reviewers act on PRs.
> 
> Instead of talking about how to do it, reviewing one PR now can help the
> case.
> Also, it's reasonable to close inactive PR if there is a successor. But do
> not let
> a bot do it, which will create many corner (bad) cases.
> 
> Best,
> tison.
> 
> 
> Michael Marshall  于2021年12月4日周六 00:57写道：
> 
>> Hi Pulsar Community,
>> 
>> I am excited to start contributing as a committer! I have a question
>> about our process for closing stale PRs.
>> 
>> We have ~300 open PRs right now. Do we have any guidelines on closing
>> stale PRs? Of course we don't want to ignore important bug fixes, but
>> we also don't want to clutter our repo with open PRs that won't get merged.
>> 
>> For example, I reviewed this PR [0] about 3 months ago. The
>> contributor has not yet responded to my feedback and it doesn't seem
>> to fix an actual bug, so I think it is a candidate for closure. Here
>> is another example [1]. I closed this one because it had merge
>> conflicts with a commit that fixed the same underlying issue.
>> 
>> Note that our committer guidelines [2] do not provide guidance on this
>> subject.
>> 
>> [0] - https://github.com/apache/pulsar/pull/11237
>> [1] - https://github.com/apache/pulsar/pull/11162
>> [2] - https://github.com/apache/pulsar/wiki/Committer-Guide
>> 
>> Thanks,
>> Michael
>> 

Re: Missing check on .jar files committed to the source repo

[Dave Fisher]

> On Dec 3, 2021, at 11:45 AM, Michael Marshall  wrote:
> 
>> Automated checks are useful because we are human and we usually miss to
>> validate this kind of boring stuff.
> 
> +1 I think it sounds appropriate to add an automated check. If a use
> case arises where we need to add compiled files, we'll also need a way
> to bypass/override this check.

For example, have a look at why grade wrapper jars cannot be included - 
https://issues.apache.org/jira/browse/LEGAL-570

Regards,
Dave

> 
> Michael
> 
> On Fri, Dec 3, 2021 at 7:10 AM Enrico Olivelli  wrote:
>> 
>> Il giorno ven 3 dic 2021 alle ore 10:36 ZhangJian He 
>> ha scritto:
>> 
>>> I agree. I mean that the situation can be easily judged during the review
>>> process. So I think the automated check sames not so valuable.
>>> If you prefer, I have no objection.
>>> 
>> 
>> Automated checks are useful because we are human and we usually miss to
>> validate this kind of boring stuff.
>> It is very like LICENSE headers, NOTICE files, checkstyle
>> :-)
>> 
>> Enrico
>> 
>>> 
>>> 
>>> Thanks
>>> ZhangJian He
>>> 
>>> Enrico Olivelli  于2021年12月3日周五 17:22写道：
>>> 
 Il giorno ven 3 dic 2021 alle ore 10:20 ZhangJian He >>> 
 ha scritto:
 
> Gradle has `gradle-wrapper.jar` too. I think we don't need an automated
> check, the reviewers can find if it's reasonable.
> 
 
 For some files there are specific acceptance rules.
 But we cannot commit other files that are not needed.
 
 
 Enrico
 
 
> 
> Enrico Olivelli  于2021年11月10日周三 16:47写道：
> 
>> ping
>> 
>> 
>> Il giorno ven 5 nov 2021 alle ore 09:23 Enrico Olivelli <
>> eolive...@gmail.com>
>> ha scritto:
>> 
>>> Hello,
>>> This patch [1] contains a .jar file that must not be committed to
>>> the
>>> source repository.
>>> We generally cannot commit binary files to the source repo, as it
 won't
>> be
>>> "open source" anymore.
>>> 
>>> When we are permitted to commit binary/compiled code, there must
>>> be a
>> very
>>> good reason and it must be approved explicitly and noted somewhere
>> (NOTICE
>>> file?)
>>> 
>>> My point here is that we are missing some automated check to
>>> prevent
> such
>>> accidental commits.
>>> I believe that we have to add a check to be run at every PR
 validation
>> and
>>> during the release process that our source distribution does not
> contain
>>> compiled code.
>>> 
>>> Thoughts ?
>>> 
>>> Enrico
>>> 
>>> [1] https://github.com/apache/pulsar/pull/12535
>>> 
>>> 
>> 
> 
 
>>> 

[ANNOUCEMENT] Welcome to our newest PMC Member - Yu Liu!

[Dave Fisher]

Hi everyone,

The Apache Pulsar Project Management Committee (PMC) has invited Yu Liu 
(https://github.com/anonymitaet) as a member of the PMC and we are pleased to 
announce that she has accepted.

She is very active in the community and organizes the documentation-related 
work of the community

- Pulsar Bot
- Upgrade Pulsar Website
- Redesign Pulsar Doc Information Architecture
- Code review for the document changes

And made great contributions 

https://github.com/apache/pulsar/issues?q=assignee%3AAnonymitaet+is%3Aclosed

She has presented at many meetings to share Apache Pulsar and Apache Pulsar 
Community.

Here are some of her talks.

https://apachecon.com/acasia2021/sessions/1102.html
https://www.slideshare.net/streamnative/cos-con19-pulsaryuliu
https://www.tcworld-china.cn/en/program/program-tcworld-china-2021
https://www.slideshare.net/streamnative/code-the-docsyu-liu

Welcome and Congratulations, You Liu!

Best Regards,
Dave on behalf of the Pulsar PMC

Re: [Great News] Pulsar Hits 10,000 GitHub Stars Milestone!

[Dave Fisher]

This is great news!

However the ending of the blog post [1] does not follow the Apache Way

This bullet point is misleading and serves to split the community.

> • Join the 2022 Pulsar Ambassador Program and work directly with Pulsar PMCs 
> and top contributors to co-host events, promote new project updates, and 
> build the Pulsar community in your city. Contact Us: 
> ambassa...@streamnative.io

(1) This mailing list (dev@pulsar.apache.org) is the one and only place to work 
directly with the whole of the Apache Pulsar community.
(2) If you engage in this unofficial, vendor-specific “Ambassador” program you 
are not growing the whole and miss connecting with most of the true Apache 
Pulsar community.

Please rewrite or remove this divisive bullet point.

Here is a blog post about how Apache is supposed to work. [2] And more about 
the Apache Way [3]

FYI - PMCs is plural for PMC. Apache Pulsar is one of 200+ Apache PMCs. PMC’s 
have members who are individuals. Here are definitions of roles [4]. The 
community consists of people who are in all of these roles. A project is 
supposed to be managed as described here. [5]

All The Best,
Dave

[1] 
https://streamnative.io/blog/community/2021-12-07-pulsar-hits-1-github-stars-milestone/
[2] https://blogs.apache.org/foundation/entry/the-apache-way-to-sustainable
[3] https://www.apache.org/theapacheway/index.html
[4] https://www.apache.org/foundation/how-it-works.html#roles
[5] https://www.apache.org/foundation/how-it-works.html#management

> On Dec 7, 2021, at 9:36 PM, Yu  wrote:
> 
> Hi Pulsar enthusiasts,
> 
> Our community has been growing ever-faster since it became a top-level 
> project of the ASF in 2018. 
> 
> Recently, Pulsar hits 10, 000 Github stars milestone! We would like to thank 
> every stargazer for joining us on the journey. More importantly, thank you to 
> every Pulsar user, contributor, and committer for making this happen!
> 
> Here is a glance at the vibrancy of the Pulsar community:
> 
> 
> For more details about Pulsar's community growth, recent updates, and future 
> plans, see here.
> 
> Cheers,
> Anonymitaet   

Re: [Great News] Pulsar Hits 10,000 GitHub Stars Milestone!

[Dave Fisher]

Hi Sijie,

Thanks for handling this issue. I’ll check in my morning. There was a reference 
on the Community Newsletter as well. Please check that too.

Best Regards,
Dave

Sent from my iPhone

> On Dec 8, 2021, at 9:24 PM, Sijie Guo  wrote:
> 
> Dave,
> 
> I think that is a mistake. We already caught it. Dianjin is fixing it.
> 
> - Sijie
> 
>> On Wed, Dec 8, 2021 at 11:41 AM Dave Fisher  wrote:
>> 
>> This is great news!
>> 
>> However the ending of the blog post [1] does not follow the Apache Way
>> 
>> This bullet point is misleading and serves to split the community.
>> 
>>> • Join the 2022 Pulsar Ambassador Program and work directly with Pulsar
>> PMCs and top contributors to co-host events, promote new project updates,
>> and build the Pulsar community in your city. Contact Us:
>> ambassa...@streamnative.io
>> 
>> (1) This mailing list (dev@pulsar.apache.org) is the one and only place
>> to work directly with the whole of the Apache Pulsar community.
>> (2) If you engage in this unofficial, vendor-specific “Ambassador” program
>> you are not growing the whole and miss connecting with most of the true
>> Apache Pulsar community.
>> 
>> Please rewrite or remove this divisive bullet point.
>> 
>> Here is a blog post about how Apache is supposed to work. [2] And more
>> about the Apache Way [3]
>> 
>> FYI - PMCs is plural for PMC. Apache Pulsar is one of 200+ Apache PMCs.
>> PMC’s have members who are individuals. Here are definitions of roles [4].
>> The community consists of people who are in all of these roles. A project
>> is supposed to be managed as described here. [5]
>> 
>> All The Best,
>> Dave
>> 
>> [1]
>> https://streamnative.io/blog/community/2021-12-07-pulsar-hits-1-github-stars-milestone/
>> [2]
>> https://blogs.apache.org/foundation/entry/the-apache-way-to-sustainable
>> [3] https://www.apache.org/theapacheway/index.html
>> [4] https://www.apache.org/foundation/how-it-works.html#roles
>> [5] https://www.apache.org/foundation/how-it-works.html#management
>> 
>>>> On Dec 7, 2021, at 9:36 PM, Yu  wrote:
>>> 
>>> Hi Pulsar enthusiasts,
>>> 
>>> Our community has been growing ever-faster since it became a top-level
>> project of the ASF in 2018.
>>> 
>>> Recently, Pulsar hits 10, 000 Github stars milestone! We would like to
>> thank every stargazer for joining us on the journey. More importantly,
>> thank you to every Pulsar user, contributor, and committer for making this
>> happen!
>>> 
>>> Here is a glance at the vibrancy of the Pulsar community:
>>> 
>>> 
>>> For more details about Pulsar's community growth, recent updates, and
>> future plans, see here.
>>> 
>>> Cheers,
>>> Anonymitaet
>> 
>> 

Re: [Great News] Pulsar Hits 10,000 GitHub Stars Milestone!

[Dave Fisher]

OK. I see the new name.

My concern is about SN posting blogs that are about community news without 
telling readers where the whole Pulsar community can be found.


> On Dec 9, 2021, at 6:18 AM, Dianjin Wang  
> wrote:
> 
> I want to sync the blog status, the update has been done.
> 
> Best,
> Dianjin Wang
> 
> 
> On Thu, Dec 9, 2021 at 2:34 PM Sijie Guo  wrote:
> 
>> Hi Dave,
>> 
>> Will follow up tomorrow morning (Pacific Time).
>> 
>> - Sijie
>> 
>> On Wed, Dec 8, 2021 at 9:35 PM Dave Fisher  wrote:
>> 
>>> Hi Sijie,
>>> 
>>> Thanks for handling this issue. I’ll check in my morning. There was a
>>> reference on the Community Newsletter as well. Please check that too.
>>> 
>>> Best Regards,
>>> Dave
>>> 
>>> Sent from my iPhone
>>> 
>>>> On Dec 8, 2021, at 9:24 PM, Sijie Guo  wrote:
>>>> 
>>>> Dave,
>>>> 
>>>> I think that is a mistake. We already caught it. Dianjin is fixing it.
>>>> 
>>>> - Sijie
>>>> 
>>>>> On Wed, Dec 8, 2021 at 11:41 AM Dave Fisher  wrote:
>>>>> 
>>>>> This is great news!
>>>>> 
>>>>> However the ending of the blog post [1] does not follow the Apache Way
>>>>> 
>>>>> This bullet point is misleading and serves to split the community.
>>>>> 
>>>>>> • Join the 2022 Pulsar Ambassador Program and work directly with
>> Pulsar
>>>>> PMCs and top contributors to co-host events, promote new project
>>> updates,
>>>>> and build the Pulsar community in your city. Contact Us:
>>>>> ambassa...@streamnative.io
>>>>> 
>>>>> (1) This mailing list (dev@pulsar.apache.org) is the one and only
>> place
>>>>> to work directly with the whole of the Apache Pulsar community.
>>>>> (2) If you engage in this unofficial, vendor-specific “Ambassador”
>>> program
>>>>> you are not growing the whole and miss connecting with most of the
>> true
>>>>> Apache Pulsar community.
>>>>> 
>>>>> Please rewrite or remove this divisive bullet point.
>>>>> 
>>>>> Here is a blog post about how Apache is supposed to work. [2] And more
>>>>> about the Apache Way [3]
>>>>> 
>>>>> FYI - PMCs is plural for PMC. Apache Pulsar is one of 200+ Apache
>> PMCs.
>>>>> PMC’s have members who are individuals. Here are definitions of roles
>>> [4].
>>>>> The community consists of people who are in all of these roles. A
>>> project
>>>>> is supposed to be managed as described here. [5]
>>>>> 
>>>>> All The Best,
>>>>> Dave
>>>>> 
>>>>> [1]
>>>>> 
>>> 
>> https://streamnative.io/blog/community/2021-12-07-pulsar-hits-1-github-stars-milestone/
>>>>> [2]
>>>>> 
>> https://blogs.apache.org/foundation/entry/the-apache-way-to-sustainable
>>>>> [3] https://www.apache.org/theapacheway/index.html
>>>>> [4] https://www.apache.org/foundation/how-it-works.html#roles
>>>>> [5] https://www.apache.org/foundation/how-it-works.html#management
>>>>> 
>>>>>>> On Dec 7, 2021, at 9:36 PM, Yu  wrote:
>>>>>> 
>>>>>> Hi Pulsar enthusiasts,
>>>>>> 
>>>>>> Our community has been growing ever-faster since it became a
>> top-level
>>>>> project of the ASF in 2018.
>>>>>> 
>>>>>> Recently, Pulsar hits 10, 000 Github stars milestone! We would like
>> to
>>>>> thank every stargazer for joining us on the journey. More importantly,
>>>>> thank you to every Pulsar user, contributor, and committer for making
>>> this
>>>>> happen!
>>>>>> 
>>>>>> Here is a glance at the vibrancy of the Pulsar community:
>>>>>> 
>>>>>> 
>>>>>> For more details about Pulsar's community growth, recent updates, and
>>>>> future plans, see here.
>>>>>> 
>>>>>> Cheers,
>>>>>> Anonymitaet
>>>>> 
>>>>> 
>>> 
>>> 
>> 

Moving to a TLP

[Dave Fisher]

Hi Everyone,

Congratulations on Graduation!

Matteo and the PMC have some tasks which are described here to move the project 
out of the Incubator and to Pulsar!

https://incubator.apache.org/guides/transferring.html 


I will look into "Update the Incubator status records” tomorrow.

I have been moderating the mailing lists. Other volunteers are welcome.

Regards,
Dave

Re: Pulsar release 2.2

[Dave Fisher]

BTW - the Infra team just now moved the repository out of the incubator!

Check the release scripts!

Sent from my iPhone

> On Sep 21, 2018, at 4:29 PM, Sijie Guo  wrote:
> 
> I would suggest holding off a bit. trying to get those PR merged.
> 
> - Sijie
> 
>> On Fri, Sep 21, 2018 at 4:26 PM Joe F  wrote:
>> 
>> I was planning to cut the release now, but seems like we still have a few
>> PRs almost ready, but not committed. Should I hold off till tomorrow
>> morning?
>> 
>> Joe
>> 
>>> On Mon, Sep 17, 2018 at 6:48 PM 李鹏辉gmail  wrote:
>>> 
>>> Please contain this PR(
>>> https://github.com/apache/incubator-pulsar/pull/2543 <
>>> https://github.com/apache/incubator-pulsar/pull/2543>)  as far as
>>> possible.
>>> 
>>> Thanks.
>>> 
 在 2018年9月18日，02:30，Joe F  写道：
 
 Now that 2.1.1 is completed, I intend to start the Pulsar 2.2 release
 process in a few days, and would like to freeze the code this Friday.
 
 Please ensure that all your committed PR's are merged and complete. I
>>> still
 see 11 outstanding items.
 https://github.com/apache/incubator-pulsar/milestone/16
 
 Are we OK with pushing some of these to the next release, if not
>>> completed
 by Friday ?
 
 Joe
>>> 
>>> 
>> 

Re: Pulsar release 2.2

[Dave Fisher]

Hurray for rolling the first TLP release!

Regards,
Dave

Sent from my iPhone

> On Sep 27, 2018, at 11:43 PM, Joe Francis  wrote:
> 
> Good. I will get this going.
> 
> Joe
> 
>> On Thu, Sep 27, 2018 at 8:30 PM, Sijie Guo  wrote:
>> 
>> Hi Joe,
>> 
>> I think all the issues for 2.2 are merged :) we are ready to go with the
>> release :)
>> 
>> - Sijie
>> 
>> 
>> On Fri, Sep 21, 2018 at 5:54 PM Matteo Merli 
>> wrote:
>> 
>>> Hopefully there should be no issues because github transparently
>> redirects
>>> the repository after renames, though... of course there will be some
>> things
>>> to polish here and there :)
>>> 
>>> 
>>> 
>>> On Fri, Sep 21, 2018 at 5:49 PM Dave Fisher 
>> wrote:
>>> 
>>>> BTW - the Infra team just now moved the repository out of the
>> incubator!
>>>> 
>>>> Check the release scripts!
>>>> 
>>>> Sent from my iPhone
>>>> 
>>>>> On Sep 21, 2018, at 4:29 PM, Sijie Guo  wrote:
>>>>> 
>>>>> I would suggest holding off a bit. trying to get those PR merged.
>>>>> 
>>>>> - Sijie
>>>>> 
>>>>>> On Fri, Sep 21, 2018 at 4:26 PM Joe F  wrote:
>>>>>> 
>>>>>> I was planning to cut the release now, but seems like we still have
>> a
>>>> few
>>>>>> PRs almost ready, but not committed. Should I hold off till tomorrow
>>>>>> morning?
>>>>>> 
>>>>>> Joe
>>>>>> 
>>>>>>> On Mon, Sep 17, 2018 at 6:48 PM 李鹏辉gmail 
>>>> wrote:
>>>>>>> 
>>>>>>> Please contain this PR(
>>>>>>> https://github.com/apache/incubator-pulsar/pull/2543 <
>>>>>>> https://github.com/apache/incubator-pulsar/pull/2543>)  as far as
>>>>>>> possible.
>>>>>>> 
>>>>>>> Thanks.
>>>>>>> 
>>>>>>>> 在 2018年9月18日，02:30，Joe F  写道：
>>>>>>>> 
>>>>>>>> Now that 2.1.1 is completed, I intend to start the Pulsar 2.2
>>> release
>>>>>>>> process in a few days, and would like to freeze the code this
>>> Friday.
>>>>>>>> 
>>>>>>>> Please ensure that all your committed PR's are merged and
>> complete.
>>> I
>>>>>>> still
>>>>>>>> see 11 outstanding items.
>>>>>>>> https://github.com/apache/incubator-pulsar/milestone/16
>>>>>>>> 
>>>>>>>> Are we OK with pushing some of these to the next release, if not
>>>>>>> completed
>>>>>>>> by Friday ?
>>>>>>>> 
>>>>>>>> Joe
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>> 
>>>> --
>>> Matteo Merli
>>> 
>>> 
>> 

Re: [DISCUSS] Graduate Apache Pulsar (incubating) as a TLP

[Dave Fisher]

Sent from my iPhone

> On Oct 3, 2018, at 5:45 PM, Matteo Merli  wrote:
> 
> Justin, I was doing the change on the website and I went through the policy
> again and I was a bit confused.
> 
> In the "Who we are" section:
> https://www.apache.org/foundation/marks/linking#whoweare
> the text says:
> ```
> Should not include corporate affiliations of actual contributors.
> Committers are expected to participate in Apache projects as individuals,
> and not as representatives of any employers. Including corporate
> affiliations on pages that are primarily about the community of individuals
> on a project can give the wrong impression to new community members about
> the fact that projects are managed by PMCs of individuals, and not managed
> by outside organizations. PMCs are free to allow including corporate
> affiliations, but should be consistent in their policy for all committers.
> ```
> 
> Doesn't the last paragraph means that a PMC is free to decide the policy?
> (as long as it's either all affiliations or none).

I view this as a choice. If these are exposed then the community really has to 
be careful to be operating for the benefit of the whole community even if that 
conflicts with their company’s plans. Vendor neutrality in project development 
is the goal. Showing this information makes the question more transparent. In 
general I think this is good. That said should development ever appear more 
vendor directed that would be more likely to be questioned. So, it is a choice 
with a strong recommendation, but not all members agree.

If we keep affiliation then mine is currently Dave Fisher Technology, LLC

Regards,
Dave

> 
> Thanks,
> Matteo
> 
> 
>> On Wed, Oct 3, 2018 at 2:54 PM Matteo Merli  wrote:
>> 
>> Justin, we'll get the website fixed today and I'll ping back on the thread.
>> 
>> Thanks,
>> 
>> Matteo
>> 
>>> On Wed, Oct 3, 2018 at 2:51 PM Justin Mclean  wrote:
>>> 
>>> Hi,
>>> 
>>>> Regarding affiliations for committers, I would say that it's not that we
>>>> have "copied" from the only TLP that had it in their "Team" page.
>>>> I just did a spot check and it seems to be a quite common practice
>>> across
>>>> the board. Few examples (taken ramdomly):
>>>> http://hive.apache.org/people.html
>>>> http://storm.apache.org/contribute/People.html
>>>> http://activemq.apache.org/team.html
>>>> http://camel.apache.org/team.html
>>>> http://zookeeper.apache.org/credits.html
>>>> http://avro.apache.org/credits.html
>>> 
>>> It makes it hard when TLP set bad examples but the policy IMO is quite
>>> clear here. [1]
>>> 
>>> Thanks,
>>> Justin
>>> 
>>> 1. https://www.apache.org/foundation/marks/linking
>>> 
>> --
>> Matteo Merli
>> 
>> 
> -- 
> Matteo Merli
> 

Re: All commits to the Pulsar repository

[Dave Fisher]

Oops! I meant to reject this.

Sent from my iPhone

> On Oct 8, 2018, at 8:13 PM, wangkai1  wrote:
> 
> 
> 王凯
> 北京东方国信科技股份有限公司
> 电信事业部
> 技术中心－大数据技术部
> 手机：17503461125
> 邮箱：wangk...@bonc.com.cn
> 
> 地址：北京市朝阳区来广营创达三路1号院1号楼东方国信大厦
> 邮编：100102
> 网址：http://www.bonc.com.cn

Apache Edgent

[Dave Fisher]

Hi -

I saw a JIRA issue for Apache Edgent (Edgent-450 Create an Edge connector for 
Apache Pulsar). Is this of interest to the Pulsar comminutity?

If so then Edgent is going to need help from other developers joining their 
efforts. If interested then please engage on d...@edgent.apache.org 


Regards,
Dave

Re: Apache Edgent

[Dave Fisher]

Hi David,

That’s ok. I just wanted to note that Edgent is struggling to maintain their 
developer community. If Ian Edgent connector is important to the Pulsar 
community it would be good to help sooner than later.

Regards,
Dave

> On Nov 5, 2018, at 10:47 AM, David Kjerrumgaard  wrote:
> 
> Hi Dave,
> 
> I created the JIRA issue and am trying to follow up now on getting that
> JIRA implemented. When I contacted them, they basically said to commit a PR
> for them to review, but I haven't had the time to do so.
> 
> Regards,
> David Kjerrumgaard
> 
> On Mon, Nov 5, 2018 at 8:55 AM Dave Fisher  wrote:
> 
>> Hi -
>> 
>> I saw a JIRA issue for Apache Edgent (Edgent-450 Create an Edge connector
>> for Apache Pulsar). Is this of interest to the Pulsar comminutity?
>> 
>> If so then Edgent is going to need help from other developers joining
>> their efforts. If interested then please engage on d...@edgent.apache.org
>> <mailto:d...@edgent.apache.org>
>> 
>> Regards,
>> Dave
> 
> 
> 
> -- 
> Regards,
> 
> David Kjerrumgaard
> Director - Solution Architecture
> Streamlio
> Cell: 330-437-5467

Re: [DISCUSS] PIP 26: Delayed Message Delivery

[Dave Fisher]

I think Joe left a clue in the idea of using Pulsar Functions.

Sent from my iPhone

> On Nov 10, 2018, at 6:26 PM, Jia Zhai  wrote:
> 
> Thanks Joe for the comments.
> To implementation this feature in client side is great, but from my view,
> to achieve "Delayed Message", we may have to touch the dispatch path.
> Is there any idea or suggestion of how to do it from client side?
> 
> 
>> On Sat, Nov 10, 2018 at 3:41 PM wenfeng wang  wrote:
>> 
>> Joe +1
>> 
>> Joe F  于2018年11月10日周六 下午12:47写道：
>> 
>>> I am not a fan of adding complexity to the dispatch path, and I will
>> always
>>> have serious concerns about proposals that do so, including this one.
>>> 
>>> In general, I would prefer Pulsar to keep the dispatch path simple and
>>> efficient, and avoid server side implementations of business logic.
>>> Streaming at scale, at low latency is what  I think Pulsar should do.  I
>> am
>>> biased here, because that is one of the reasons Pulsar got created
>>> originally, at a time when there were many other message brokers out
>> there
>>> ( and many under the Apache umbrella too)
>>> 
>>> All those other message brokers do all kinds of server-side logic -
>>> filtering, transforming, scheduling, and so on. All of those systems have
>>> more or less ended up with bottlenecks and  complexity.  And  this is not
>>> without reason. Message queues are queues, and most of the server side
>>> logic implementations are attempts to make a queue into a database. A
>>> system that is optimized for flow as a queue, will not be good as a
>>> database, and vice-versa.
>>> 
>>> I think the right way to do this kind of business logic is in the client
>> or
>>> leverage Pulsar functions, and the core broker dispatch path and process
>>> space should just deal with performance and flow at scale
>>> 
> 
>> Joe
>>> 
>>> 
>>> 
>>> 
 On Thu, Nov 8, 2018 at 1:39 PM 李鹏辉gmail  wrote:
 
 Dear all
 
 This is a PIP to add feature of delayed message delivery.
 
 ## Motivation
 Scheduled and delayed message delivery is a very common feature to
>>> support
 in a message system. Basically individual message can have a header
>> which
 will be set by publisher and based on the header value the broker
>> should
 hold on delivering messages until the configured delay or the scheduled
 time is met.
 
 ## Usage
 The delayed message delivery feature is enabled per message at producer
 side.
 
 Delayed messages publish example in client side:
 
 ```java
 // message to be delivered at the configured delay interval
 producer.newMessage().delayAt(3L, TimeUnit.Minute).value("Hello
 Pulsar!").send();
 
 // message to be delivered at the configure time.
 producer.newMessage().scheduleAt(new Date(2018, 10, 31, 23, 00, 00))
 ```
 
 To enable or disable delay message feature:
 
 ```shell
 pulsar-admin namespaces
 
 enable-delayed-message   Enable delayed message for all topics of
>> the
 namespace
  Usage: enable-delayed-message [options] tenant/namespace
 
Options:
  -p --time-partition-granularity
Granularities of time will be partitioned, every time
 partition will be
stored into legders and current time partition will be
 load in memory
and organized in a TimeWheel.(eg: 30s, 5m, 1h, 3d, 2w)
Default: 5m
  -t --tick-duration
The duration between tick in TimeWheel. Calculate ticks
 per wheel
using time-partition-granularity / tick-duration before
 load time
partition into a TimeWheel.(eg: 500ms, 1s, 5m)
Default: 1s
 
 disable-delayed-message  Disable delayed message for all topics
>>> of
 the namespace
  Usage: disable-delayed-message tenant/namespace
 ```
 
 ## Design
 
 ### Delayed Message Index
 
 The “DelayedMessageIndex” will be implemented using a [TimeWheel
>>> approach](
 
>> http://www.cs.columbia.edu/~nahum/w6998/papers/sosp87-timing-wheels.pdf
>>> ).
 We will be maintaining a delayed index, indexing the delayed message by
>>> its
 time and actual message id.
 
 The index is partitioned by the delayed time. Each time partition will
>> be
 stored using one (or few) ledger(s). For example, if we are configuring
 the index to be partitioned by 5 minutes, we will store the index data
>>> for
 every 5 minutes by its delayed time. The latest time partition will be
 loaded in memory and organized in a TimeWheel.
 
 The TimeWheel is indexed by ticks. For example, if we configured the
>> tick
 to be 1 second, we will be maintaining 300 ticks for 5 minutes’ index.
>> A
 timer task is scheduled every tick, and it will pick the indexed
>> message
 from the TimeWheel and dispatch them to the real co

Re: Limitations on No.Of Producers and Consumers

[Dave Fisher]

Message was moderated onto the list, including the OP

Sent from my iPhone

> On Nov 12, 2018, at 2:28 AM, Ivan Kelly  wrote:
> 
> Hi Koushik,
> 
>> I just got introduced to Apache pulsar and studying it to solve a problem at 
>> hand.
>> As I understand that Pulsar supports 1M of topics, Is there any limitation 
>> on no.of producers and subscribers that brokers can handle? I have the 
>> following use cases.
> 
> The limit of how many producers and consumers a broker can serve is a
> function of the network bandwidth and CPU of that broker. However, the
> way Pulsar is designed, if the number of producers/consumers becomes
> too large for the number of brokers you have, you can add broker
> nodes, and some topics will move to the new nodes, balancing your load
> on all the machines.
> 
> In terms of consumers, depending on which kind of consumer you use,
> these may count towards the 1M topic limit. The real limit in pulsar
> is the amount of metadata that can be stored in zookeeper. Everything
> else can be scaled horizontally. Each topic has one piece of metadata
> that belongs to the topic itself. However, each durable subscription
> (as is the default) also has one piece of metadata. So in the examples
> you give below, if you are creating consumers using
> PulsarClient#newSubscriber(), you will have ~200k pieces of metadata
> stored in ZK. ZK can easily hold 200k pieces of metadata. I think the
> highest I've seen it go is 2M pieces of metadata.
> 
>> Usecase 1 :  Few hundred producers producing to 100k topics, and each topic 
>> has corresponding consumer (i.e 100k consumers) trying to consume messages 
>> from its topic. To summarize ~100k clients interacting with brokers here.
>> 
>> Usecase 2 :  100k producing to corresponding topic (i.e 100k topics), and 
>> each topic has corresponding consumer (i.e 100k consumers) trying to consume 
>> messages from its topic. To summarize ~200k clients interacting with brokers 
>> here.
> 
> Both these usecases look very doable with Pulsar.
> 
> Regards,
> Ivan

Re: [ANNOUNCE] Apache Pulsar

[Dave Fisher]

Hi -

Sent from my iPhone

> On Jan 6, 2019, at 8:55 PM, Sijie Guo  wrote:
> 
> That’s a good suggestion. I also think that provides a better way to
> organize sub modules for now or even mid-term.
> 
> However in long term, we should try to move out them out of Pulsar repo, or
> even to push to corresponding computing engine repo.

Are you suggesting that these be kept by the Pulsar project or donated to the 
other projects?

> This would 1) allow
> those modules having their own release schedules without interfering with
> pulsar core release, 2) allow more exposures or publicity of Pulsar if they
> are pushed to the computing engine’s repo. E.g. pulsar-storm goes to storm
> repo, pulsar-flink goes to flink repo and pulsar-spark goes to spark repo.

I think these connectors are a significant value to Pulsar and should remain in 
Pulsar. Having separate releases for connectors might be an advantage.

Regards,
Dave

> 
> Hope this makes sense!
> 
> Sijie
> 
>> On Mon, Jan 7, 2019 at 1:43 PM Pingle Wang <1269223...@qq.com> wrote:
>> 
>> sorry, the photo not show in content. The structure is as follows：
>> /pulsar
>> /pulsar-external
>> /pulsar-flink
>> /pulsar-spark
>> /pulsar-storm etc..
>> 
>> -- 原始邮件 --
>> *发件人:* "Pingle Wang"<1269223...@qq.com>;
>> *发送时间:* 2019年1月7日(星期一) 中午12:38
>> *收件人:* "dev";
>> *主题:* 回复： [ANNOUNCE] Apache Pulsar
>> 
>> like:
>> 
>> 
>> -- 原始邮件 --
>> *发件人:* "Pingle Wang"<1269223...@qq.com>;
>> *发送时间:* 2019年1月7日(星期一) 中午12:36
>> *收件人:* "dev";
>> *主题:* 回复： [ANNOUNCE] Apache Pulsar
>> 
>> Dear all:
>>   when I PR found, we can have pulsar-external(sub-module) under /pulsar,
>> as long-term, other computing engine can be rolled under /pulsar-external
>> as well:
>> /pulsar-external  |-- pulsar-flink  |--
>> pulsar-spark  |-- pulsar-storm etc..
>> 
>> WDYT?
>> 
>> 
>> 
>> 
>> -- 原始邮件 --
>> 发件人: "zhaijia03";
>> 发送时间: 2019年1月4日(星期五) 上午8:12
>> 收件人: "dev";
>> 抄送: "users"; "announce";
>> 主题: Re: [ANNOUNCE] Apache Pulsar 2.2.1 released
>> 
>> 
>> 
>> cong~
>> 
>>> On Tue, Jan 1, 2019 at 5:14 AM Sijie Guo  wrote:
>>> 
>>> 👍 congrats!🎉🎈
>>> 
>>> Thanks everyone for making this happen!
>>> 
>>> Sijie
>>> 
 On Mon, Dec 31, 2018 at 12:37 PM Matteo Merli  wrote:
 
 The Apache Pulsar team is proud to announce Apache Pulsar version
>> 2.2.1.
 
 Pulsar is a highly scalable, low latency messaging platform running on
 commodity hardware. It provides simple pub-sub semantics over topics,
 guaranteed at-least-once delivery of messages, automatic cursor
>>> management
 for
 subscribers, and cross-datacenter replication.
 
 For Pulsar release details and downloads, visit:
 
 https://pulsar.apache.org/download
 
 Release Notes are at:
 http://pulsar.apache.org/release-notes
 
 We would like to thank the contributors that made the release possible.
 
 Regards,
 
 The Pulsar Team
 
>>> 
>> 

Re: About explicitly choose partition topic

[Dave Fisher]

Adding the OP who I moderated onto the list.

(You subscribe by sending an email to dev-subscr...@pulsar.apache.org)

Sent from my iPhone

> On Jan 10, 2019, at 9:26 PM, Matteo Merli  wrote:
> 
> Hi Ruoruo,
> 
> In order to consume messages in order on a partitioned topic, you should
> use the “Failover” subscription type.
> 
> That will ensure that each partition will be asssigned to only 1 consumer,
> and you can add consumers to scale up your processsing. (And each consumer
> will get assigned multiple partitions).
> 
> http://pulsar.apache.org/docs/en/standalone/#failover
> 
> 
>> On Thu, Jan 10, 2019 at 9:18 PM Ruoruo Wong  wrote:
>> 
>> Hi Pulsar team
>>I am really pleasant that Pulsar system helps me to improve my business
>> performance.
>>And I have a problem about topic partition that I suppose the same hash
>> to be consumed by the same consumer, like Kafka.
>>After I check the document, only resolution is "use the
>> $TOPIC-partition-N as the topic name to subscribe to a specific
>> subscription",
>> but this is not proper for elastic system.
>>Is there any other option could I choose?
>> 
>> Thanks
>> 
> -- 
> Matteo Merli
> 

Re: [DISCUSS] Skip tests for documentation related changes

[Dave Fisher]

Hi -

Inline

Sent from my iPhone

> On Jan 28, 2019, at 1:12 PM, Sijie Guo  wrote:
> 
> both adding tag or checkbox are developer driven. but it is easier to
> achieve with current github pull request builder in Jenkins.
> so I think the responsibility of this approach is kind of spreading across
> contributors and committers. It is also a way to educate
> pulsar contributors to understand better pulsar codebase and its build
> system.
> 
> However if we want to automate the process, one way I can think of is
> introducing some kind of `pulsarbot`.

One suggestion is to bring this question to bui...@apache.org mailing list and 
see if others have solved this problem in some way.

> 
> - DONT run precommit jobs automatically (java, cpp, integration and docs)
> - use Github hook to listen on activities of a pull request
> - pulsarbot check the Github diff when a pull request is created or updated
> - pulsarbot trigger corresponding precommit jobs to run
> 
> But that is going to be a huge task to implement this mechanism. I am not
> sure it is worth to do initially and who has enough time dedicated to this.

Regards,
Dave

> 
> - Sijie
> 
>> On Mon, Jan 28, 2019 at 12:26 PM Joe F  wrote:
>> 
>> +1 to Sanjeev's suggestion
>> 
>> On Mon, Jan 28, 2019 at 12:15 PM Sanjeev Kulkarni 
>> wrote:
>> 
>>> If developers are in charge of checking the checkbox, it might lead to
>>> errors. Any way to make it automatic? Since docs are restricted to
>> certain
>>> areas of repo, maybe we can have some rules around that?
>>> 
 On Mon, Jan 28, 2019 at 12:12 PM Sijie Guo  wrote:
 
 Hi all,
 
 Currently for every documentation change, we have run 3 precommit jobs,
 java, c++ and integrationt tests. None of them is actually testing the
 documentation change and it is wasting jenkins resources and make the
>>> merge
 process for documentation changes take much longer time.
 
 So I am proposing :
 
 - add a separate precommit job for documentation-only changes. e.g.
 `Jenkins: Documentation Tests`
 
 - provide a checkbox in description
  * [ ] documentation-only change
  * [ ] code-only change
 
 - if `[x] documentation-only` is checked, then java, c++ and
>> integration
 tests will be skipped.
 - if `[x] code-only` is checked, then documentation tests will be
>>> skipped.
 
 
 I would like to see what other people think about this.
 
 - Sijie
 
>>> 
>> 

Re: PIP 28: Improve pulsar proxy log messages

[Dave Fisher]

Hi Sun Yao,

When you started this thread writing that the proxy is almost like a gateway. 
You were correct. I think that what is wanted is for Proxies to usually be 
lightweight / highly performant, and optionally become a highly resilient 
Gateway. 

Let’s call this new feature Gateway mode.

Regards,
Dave

Sent from my iPhone

> On Jan 31, 2019, at 8:34 PM, sun yao  wrote:
> 
> 
> 
> Hi Matteo,
>   You are right, will give it a shot by using "netty".
>  Thanks again.
> 
> Sam 
> 
>On Thursday, January 31, 2019, 6:53:03 AM PST, Matteo Merli 
>  wrote:  
> 
> Sure, it would be good to have that as an option.
> 
> It's just that it would need to be a completely different "operating
> mode" for the proxy,
> since more than the logging, is the parsing of each frame / command
> that will be
> "expensive".
> 
> It should be possible to do that by controlling the handlers in the
> netty channel pipeline,
> by having different handlers based on the config.
> 
> 
> --
> Matteo Merli
> 
> 
>> On Thu, Jan 31, 2019 at 6:38 AM Sijie Guo  wrote:
>> 
>> I think if this feature is controlled under some flags/settings. It should 
>> be a good tradeoff between performance concerns and debuggability/visibility.
>> 
>> As what Yao said, people can choose what level of details that they would 
>> like to see. It is similar to log4j's debug level.
>> 
>> Thanks,
>> Sijie
>> 
>>> On Thu, Jan 31, 2019 at 2:30 PM sun yao  wrote:
>>> 
>>> Hi Sijie, Matteo and Joe ,
>>> 
>>> Thanks for your kind and professional opinions .
>>> 
>>> For performance concern from Joe, what I think is like this, different 
>>> log levels for different detail outputs and let dev decide which is good 
>>> for them, and I also mentioned in PIP about we can disable this feature by 
>>> parameter in some critical scenarios.
>>> 
>>>   For protocol implement from Matteo and Joe,  we could try to 
>>> implement it for we need to get from the proxy, like what it did in other 
>>> proxies, proxysql and twemproxy , or any others.
>>> 
>>> The point for this feature is we could have more details for our 
>>> traffic, not only for performances, and also resources usage and msg 
>>> details for debug and audit, like client issues, proxy issue or broker 
>>> issues or ...
>>> 
>>>   Thanks.
>>> 
>>> 
>>> 
>>> 
>>> 
>>> On Wednesday, January 30, 2019, 11:26:07 AM PST, Joe F 
>>>  wrote:
>>> 
>>> 
>>> I run Pulsar proxy in production, and the same concern here.
>>> 
>>> I don't think we can get any of these metrics unless we start parsing
>>> protocol, and definitely its going to make everything slower, and create
>>> additional memory and GC pressures.
>>> 
>>> Joe
>>> 
>>> 
>>> On Wed, Jan 30, 2019 at 11:08 AM Matteo Merli 
>>> wrote:
>>> 
 Missed to comment on this :)
 
 One issue might arise from the fact that proxy is not actually parsing
 each and every request.
 
 The proxy only "speaks" Pulsar protocol initial Connect/Connected
 handshake, in which the proxy forwards the client credentials and
 route it through the appropriate broker.
 
 After the initial handshake, the proxy is essentially degrading itself
 into a 1-1 TCP proxy, patching 2 TCP connections through
 without checking the commands anymore. That's the reason we only have
 metrics around bytes/s and "operation/s" where
 operation maps to a "buffer" we're getting from a socket (with no
 direct relation to IP packets).
 
 
 --
 Matteo Merli
 
 
> On Wed, Jan 30, 2019 at 9:19 AM Sijie Guo  wrote:
> 
> Yao,
> 
> Thank you for your proposal! The proposal looks good to me +1.
> 
> In general, the ASF is using lazy consensus for a lot of things, like
> adopting PIPs. basically, if there is no objection coming up within a
> period (typically 1~2 days), you are good to pull the trigger and send
 PRs
> :-)
> 
> - Sijie
> 
> On Sun, Jan 27, 2019 at 11:39 AM sun yao >>> .invalid>
> wrote:
> 
>> 
>> 
>> Hi folks,
>>   Pulsar Proxy is almost a gateway for all pulsar requests, it
 would
>> be helpful if it can record more details for the traffic, like source,
>> target, session id, response time(different stage) for each request,
 even
>> for message body.  I am proposing an improve for pulsar proxy, for
>> more information :PIP:
>> 
 https://github.com/apache/pulsar/wiki/PIP-28%3A-Improve-pulsar-proxy-log-messages
>> Feel free to let me know any ideas or suggestions for
 this
>> feature.  Thanks.
>> 

Fwd: Tying Dockerhub into development and release management

[Dave Fisher]

Hi Folks,

The project might want to move at least some of the DockerHub images published 
under /u/apachepulsar over to u/apache

At the very least update to the project URL and include a comment that these 
are “UNOFFICIAL RELEASES from the Apache Pulsar community”.

Regards,
Dave

> Begin forwarded message:
> 
> From: Chris Lambertus 
> Subject: Re: Tying Dockerhub into development and release management
> Date: February 7, 2019 at 5:22:36 PM PST
> To: gene...@incubator.apache.org
> Reply-To: gene...@incubator.apache.org
> 
> 
> 
>> On Feb 7, 2019, at 4:54 PM, Justin Mclean  wrote:
>> 
>> Hi,
>> 
>>> I assume that "https://hub.docker.com/u/apache 
>>> <https://hub.docker.com/u/apache>" is an Apache controlled
>>> location, so publishing Apache images from there is fine provided they obey
>>> our policies (release policy, website policy etc).
>> 
>> I believe INFA has something to do with setting that up, but I think we 
>> should extend what be consider “official" to anything with apache in it’s 
>> user name, like apachepulsar.
> 
> 
> Hi, 
> 
> Infra manages the u/apache dockerhub org. We provide this service with the 
> express caveat that projects note that these are UNOFFICIAL releases, and 
> CONVENIENCE BINARIES ONLY. How much projects comply with this is unknown to 
> me. It is a common thread in discussions of this nature because the party 
> line is “the ASF releases code” but the reality is that “people consume 
> binaries.”
> 
> Until that juxtaposition is well and truly addressed, these discussions will 
> continue to happen ad nauseam. I have not reviewed the threads on legal, I 
> just popped in over here because Dave Fisher pointed it out to me. 
> 
> We (Infra) always point out http://www.apache.org/legal/release-policy.html 
> as the defining document that describes official release channels. 
> 
> -Chris
> ASF Infra
> 
> 
> 
> -
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
> 

Re: [DISCUSSION] Delayed message delivery

[Dave Fisher]

Hi -

My thoughts here may be completely useless but I wonder if clients can add an 
optional argument to the broker call when pulling events. The argument would be 
the amount of delay. Any messages younger than the delay are not returned by 
the broker.

Regards,
Dave

> On Feb 19, 2019, at 11:47 AM, Ezequiel Lovelle  
> wrote:
> 
>> The recent changes made to support DLQ caused major problems with garbage
> collection
> 
> If garbage collection is a big concern maybe we could add some config
> parameter on the broker to disable the usage of this feature and return
> BrokerMetadataException in this situation, giving the power to the
> administrator whether to offer this feature or not.
> 
>> is it acceptable to do it at broker side?
> 
> I think this is the big question that needs to be answered.
> 
>> can we just have a separated dispatcher for fixed delayed subscription?
> 
> I will try to do a completely new approach, simpler, and more isolated
> from broker logic. Maybe some way to plug to the broker some logic without
> interfering with its core?
> 
> In our business fixed delay at consumer level regardless of any producer
> configuration is a big win due to easy implementation and usage.
> 
> --
> *Ezequiel Lovelle*
> 
> 
> On Wed, 13 Feb 2019 at 23:25, Sijie Guo  wrote:
> 
>> Agreed that dispatcher is a performance sensitive piece of code. Feel bad
>> to hear that DLQ causes GC. Are there any issues tracking those items you
>> guys identified with DLQ changes?
>> 
>>> How is this different from a subscription running behind?
>> 
>> As far as I understand form the discussion at #3155, I don't think there is
>> a fundamental difference from a backlogged subscriber.
>> The discussion point will mainly be - if a delayed subscription can be
>> implemented with a simpler approach at broker side without changing other
>> dispatcher logic,
>> is it acceptable to do it at broker side? So we don't have to reimplement
>> the same mechanism at different language clients. I think that's the same
>> tradeoff we were discussing for generic delayed messages.
>> 
>> My thought would be - can we just have a separated dispatcher for fixed
>> delayed subscription? The logic can be ISOLATED from other normal
>> dispatchers. if users don't enable delayed subscription, they will not
>> exercise that dispatcher. This can be a good direction to explore for
>> future changes that are related to dispatchers.
>> 
>> - Sijie
>> 
>> 
>> On Thu, Feb 14, 2019 at 8:43 AM Joe F  wrote:
>> 
>>> Delayed subscription is simpler, and probably worth doing in the broker
>> IF
>>> done right.
>>> 
>>> How is this different from a subscription running behind?  Why does
>>> supporting that require this complex a change in the dispatcher, when we
>>> already support backlogged subscribers?
>>> 
>>> I am extremely wary of changes in the dispatcher. The recent changes made
>>> to support DLQ caused major problems with garbage collection, broker
>>> failure  and service interruptions for us. Even though we ARE NOT using
>> the
>>> DLQ feature. Not a pleasant experience.
>>> 
>>> This is a very performance sensitive piece of code, and it should be
>>> treated as such.
>>> 
>>> Joe
>>> 
>>> 
>>> 
>>> On Wed, Feb 13, 2019 at 3:58 PM Sijie Guo  wrote:
>>> 
 Hi all,
 
 I am going to wrap up the discussion regarding delayed delivery use
>>> cases.
 
 For arbitrary delayed delivery, there are a few +1s to doing PIP-26 in
 functions. I am assuming that we will go down this path, unless there
>> are
 other proposals.
 
 However there is a use case Lovelle pointed out about "Fixed Delayed
 Message". More specifically it is
 https://github.com/apache/pulsar/pull/3155
 (The caption in #3155 is a bit misleading). IMO it is a "delayed
 subscription", basically all messages in the subscription is delayed to
 dispatch in a given time interval. The consensus of this feature is not
>>> yet
 achieved. Basically, there will be two approaches for this:
 
 a) DONT treat "fixed delayed message" as a different case. Just use the
 same approach as in PIP-26.
 b) treat "fixed delayed message" as a different case, e.g. we can
>> better
 call it "delayed subscription" or whatever can distinguish it from
>>> general
 arbitrary delayed delivery. Use the approach proposed/discussed in
>> #3155.
 
 I would like the community to discuss this and also come to an
>> agreement.
 So Lovelle can move forward with the approach agreed by the community.
 
 Thanks,
 Sijie
 
 On Tue, Jan 29, 2019 at 6:30 AM Ezequiel Lovelle <
 ezequiellove...@gmail.com>
 wrote:
 
> "I agree, but that is *not what #3155 tries to achieve."
> 
> This typo made this phrase nonsense, sorry!
> 
> On Mon, 28 Jan 2019, 16:44 Ezequiel Lovelle <
>> ezequiellove...@gmail.com
> wrote:
> 
>>> What exactly is the delayed delivery use case?
>> 
>> This

Re: [DISCUSSION] Delayed message delivery

[Dave Fisher]

Hi -

Well, it does, but can this be implemented without building a delayQueue? It 
seems to me that a delayQueue both breaks resiliency if the broker goes down 
and would certainly add overhead. Perhaps my idea to discard responses that are 
too new and then retrieve once they are out of the delayed timeframe would be 
simpler?

Again I am somewhat naive to the details. I’m not sure that the path through 
the code is kept to an absolute minimum when you have a Consumer with a nonzero 
delay?

Regards,
Dave

> On Feb 19, 2019, at 12:39 PM, Ezequiel Lovelle  
> wrote:
> 
> Hi Dave!
> 
>> I wonder if clients can add an optional argument to the broker call when
> pulling events. The argument would be the amount of delay. Any messages
> younger than the delay are not returned by the broker.
> 
> This is exactly what https://github.com/apache/pulsar/pull/3155 does :).
> We still need to decide if we want to add this feature at client side or
> broker side, the pull request does it on the broker.
> 
> --
> *Ezequiel Lovelle*
> 
> 
> On Tue, 19 Feb 2019 at 17:06, Dave Fisher  wrote:
> 
>> Hi -
>> 
>> My thoughts here may be completely useless but I wonder if clients can add
>> an optional argument to the broker call when pulling events. The argument
>> would be the amount of delay. Any messages younger than the delay are not
>> returned by the broker.
>> 
>> Regards,
>> Dave
>> 
>>> On Feb 19, 2019, at 11:47 AM, Ezequiel Lovelle <
>> ezequiellove...@gmail.com> wrote:
>>> 
>>>> The recent changes made to support DLQ caused major problems with
>> garbage
>>> collection
>>> 
>>> If garbage collection is a big concern maybe we could add some config
>>> parameter on the broker to disable the usage of this feature and return
>>> BrokerMetadataException in this situation, giving the power to the
>>> administrator whether to offer this feature or not.
>>> 
>>>> is it acceptable to do it at broker side?
>>> 
>>> I think this is the big question that needs to be answered.
>>> 
>>>> can we just have a separated dispatcher for fixed delayed subscription?
>>> 
>>> I will try to do a completely new approach, simpler, and more isolated
>>> from broker logic. Maybe some way to plug to the broker some logic
>> without
>>> interfering with its core?
>>> 
>>> In our business fixed delay at consumer level regardless of any producer
>>> configuration is a big win due to easy implementation and usage.
>>> 
>>> --
>>> *Ezequiel Lovelle*
>>> 
>>> 
>>> On Wed, 13 Feb 2019 at 23:25, Sijie Guo  wrote:
>>> 
>>>> Agreed that dispatcher is a performance sensitive piece of code. Feel
>> bad
>>>> to hear that DLQ causes GC. Are there any issues tracking those items
>> you
>>>> guys identified with DLQ changes?
>>>> 
>>>>> How is this different from a subscription running behind?
>>>> 
>>>> As far as I understand form the discussion at #3155, I don't think
>> there is
>>>> a fundamental difference from a backlogged subscriber.
>>>> The discussion point will mainly be - if a delayed subscription can be
>>>> implemented with a simpler approach at broker side without changing
>> other
>>>> dispatcher logic,
>>>> is it acceptable to do it at broker side? So we don't have to
>> reimplement
>>>> the same mechanism at different language clients. I think that's the
>> same
>>>> tradeoff we were discussing for generic delayed messages.
>>>> 
>>>> My thought would be - can we just have a separated dispatcher for fixed
>>>> delayed subscription? The logic can be ISOLATED from other normal
>>>> dispatchers. if users don't enable delayed subscription, they will not
>>>> exercise that dispatcher. This can be a good direction to explore for
>>>> future changes that are related to dispatchers.
>>>> 
>>>> - Sijie
>>>> 
>>>> 
>>>> On Thu, Feb 14, 2019 at 8:43 AM Joe F  wrote:
>>>> 
>>>>> Delayed subscription is simpler, and probably worth doing in the broker
>>>> IF
>>>>> done right.
>>>>> 
>>>>> How is this different from a subscription running behind?  Why does
>>>>> supporting that require this complex a change in the dispatcher, when
>> we
>>>>> already support backlogged subscr

Re: Benchmarking Pulsar using OpenMessaging Benchmark

[Dave Fisher]

Since I moderated this email onto the list I am cc’ing the OP to assure that 
any replies are seen.

Regards,
Dave

> On Feb 19, 2019, at 4:34 PM, Tyler Landle  wrote:
> 
> Hey Guys,
> 
> I have been trying to benchmark pulsar and get some E2E latency data from
> it, and try to stress it in regards to End to End Latency. Namely, we have
> been trying to stress it in a way that we would see End to End Latency
> increase in some sort of way between say, 5ms and 50ms.
> 
> We have encountered some strange behavior doing this, and was wondering if
> you guys had any insights on how to generate the information we are trying
> to gather.
> 
> Weird behavior we are seeing:
> 
> 1. The clients actually run out of resource space before the broker does
> for a non persistent workload.
> 
> Using Openmessaging benchmark, we run workloads with number of topics
> varying from 3-243 topics, and up to 600,000 msg/s, non persistent topics,
> one broker. We are finding with under 6 local workers working as producers
> and consumers, that the clients usually run out of resources first, skewing
> our End to End latency statistics. Is this...in line with what you see? Is
> it normal for clients to run out of resources before the broker starts
> seeing latency gain?
> 
> 2. If you raise rate on single topic, it will have higher end to end
> latency. But if you add another topic at a lower rate, that end to end
> latency does not increase until resource utilization is constrained.
> 
> We ran the following experiments(1 broker, non persistent topics)
> 
> 2 local workers run a workload on 50 topics with 200,000 msg/s aggregate
> message rate. 2 Other local workers run a workload on 1 topic at 10,000
> msg/s, with 1 broker on non persistent topic. The single topic at 10,000
> msg/s sees latency around 2ms, while the "background" workload of 50 topics
> with 200,000 msg/s sees latency in the 20ms range. Do you have any idea why
> we would see this behavior?
> 
> 
> 
> Overall, we are looking to stress the brokers without stressing the clients
> first to see how number of topics and message rate affects pulsar as a
> whole. Rather than seeing any linear or explainable increase in latency, we
> have been seeing a pretty flat latency curve(2-5ms) followed by a huge
> spike at some workload level(somewhere around 100ms). Is there some way
> that you know of to see some kind of normalized latency increase that is
> not due to resource utilization(our first guess as to why we see such high
> spikes in latency).
> 
> Would this be better for the Users group? I wasn't really sure which one to
> send to, but the devs may have a better idea for some of the behavior that
> we are seeing.
> 
> Thanks,
> Tyler Landle

Re: Major and Minor Release Management

[Dave Fisher]

> On Feb 20, 2019, at 6:47 PM, Sijie Guo  wrote:
> 
> On Wed, Feb 20, 2019 at 9:12 PM Ivan Kelly  wrote:
> 
>>> Can anyone propose a change to current release management? Otherwise it
>> is
>>> not a scalable solution because not every committer knows how to handle
>>> this situation clearly.
>> 
>> I agree that not every committer knows this.
>> What has worked well for me in the past is to have _every_ commit go
>> to master, and cherry-pick nothing until it is decided that a bugfix
>> release is needed.
> 
> Then the release manager goes through the list of commits that
>> occurred on the master branch since the last release on the bugfix
>> branch, select those that are required for the bugfix release, and
>> then cherry-pick those, in order, to the bugfix branch.
>> 
> 
> That works well for me as well. I think the community needs to come up with
> a clear workflow for this.
> 
> At minimum, a simple rule should be:
> - if a change is merged to master, it should be "tagged" for a major
> release;
> - if a change is merged directly to a branch, it should be "tagged" for a
> minor release;
> - if a change is merged to master, and then cherry-picked to a branch, it
> should be "tagged" for a major release, and then tagged for a minor release
> after cherry-pick. otherwise if we merge a change to master but directly
> label it for a minor release, IMO this is an "inconsistent" state in a
> source repo.

Depending on how close to a minor release the branch is on many projects the 
Release Manager decides whether or not a change should be cherry picked.

A “tag” could be used to request a cherry pick.

> 
> How to tag the release, and how to cherry-pick is a question to the
> community. The community should decide and vote a workflow for tagging and
> cherry-pick changes, and *DOCUMENT* it in Pulsar website. So everyone in
> the community (including committers, contributors, and users) know exactly
> how the release is done.

+1

Regards,
Dave

> 
> 
>> 
>> -Ivan
>> 

Re: PIP-31: Add support for transactional messaging

[Dave Fisher]

Hi Richard,

I moderated your email onto the list. To participate please subscribe by 
sending an email to dev-subscr...@pulsar.apache.org and respond to the 
confirmation email as it instructs.

Regards,
Dave

Sent from my iPhone

> On Feb 27, 2019, at 8:35 PM, Richard Yu  wrote:
> 
> Hi all,
> 
> I would like to create a PIP for issue #2664 on Github. The details of the
> PIP are below.
> I hope we could discuss this thoroughly.
> 
> Cheers,
> Richard
> 
> PIP-31: Add support for transactional messaging
> 
> Motivation: Pulsar currently could improve upon their system of sending
> packets of data by implementing transactional messaging. This system
> enforces eventual consistency within the system, and allows operations to
> be performed atomically.
> 
> Proposal:
> 
> As described in the issue, we would implement the following policy in
> Producer and Pulsar Broker:
> 1. The producer produces the pre-processing transaction message. At this
> point, the broker will set the status of this message to unknown.
> 2. After the local transaction is successfully executed, the commit message
> is sent, otherwise the rollback message is sent.
> 3. The broker receives the message. If it is a commit message, it modifies
> the transaction status to commit, and then sends an actual message to the
> consumer queue. At this time, the consumer can consume the message.
> Otherwise, the transaction status is modified to rollback. The message will
> be discarded.
> 4. If at step 2, the producer is down or abnormal, at this time, the broker
> will periodically ask the specific producer for the status of the message,
> and update the status according to the producer's response, and process it
> according to step 3, the action that comes down.
> 
> Specific concerns:
> There are a number of things we will improve upon or add:
> - A configuration called ```maxMessageUnknownTime```. Consider this
> scenario: the pre-processing transaction message is sent, but the commit or
> rollback message is never received, which could mean that the status of a
> message would be permanently unknown. To avoid this from happening, we
> would need a config which limits the amount of time the status of a message
> could be unknown (i.e. ```maxMessageUnknownTime```) After that, the message
> would be discarded.
> - Logging would be updated to log the status of a message i.e. UNKNOWN,
> ROLLBACK, or COMMITTED. This would allow the user to know whether or not a
> message had failed or fallen through.
> 
> Possible Additional API:
> - We would add a method which allows the user to query the state of the
> message i.e. ```getStateOfMessage(long id)```

Re: PIP-31: Add support for transactional messaging

[Dave Fisher]

Hi -

Is this a case where a Pulsar function base class for transactions would help?

Regards,
Dave

Sent from my iPhone

> On Mar 2, 2019, at 2:39 AM, Sijie Guo  wrote:
> 
> Pravega's model is a better model than Kafka - it addressed the
> interleaving problems. However Pravega's model is based on a giant
> replicated log and rewrite the data to a second tiered storage for
> persistence, which basically re-implemented bookkeeper's logic in broker. A
> fundamental drawback of Pravega is write amplifications. The amplifications
> of both network and IO bandwidth are huge. If you use bookkeeper both for
> its first-and-second tier storage and assume the bookkeeper replication
> factor is 3, pravega requires 6x network bandwidth and 12x IO bandwidth.
> For a given message, it needs to write 3 times into the journal, and 3
> times for persistent. The amplifications hugely limit the throughput at
> pravega "brokers".
> 
> - Sijie
> 
> 
> 
>> On Sat, Mar 2, 2019 at 6:13 PM Ali Ahmed  wrote:
>> 
>> I agree we many want to review pravega's past efforts in this area also.
>> 
>> 
>> https://github.com/pravega/pravega/blob/master/documentation/src/docs/transactions.md
>> 
>> https://github.com/pravega/pravega/blob/master/client/src/main/java/io/pravega/client/stream/Transaction.java
>> 
>> -Ali
>> 
>>> On Sat, Mar 2, 2019 at 1:56 AM Sijie Guo  wrote:
>>> 
>>> Kafka's implementation is interleaving committed messages with
>> uncommitted
>>> messages at storage. Personally I think it is a very ugly design and
>>> implementation.
>>> 
>>> Pulsar is a segment centric system, where we have a shared segment
>> storage
>>> - bookkeeper. I think a better direction is to leverage the segments (aka
>>> ledgers)
>>> for buffering uncommitted messages and commit the whole segment when the
>>> whole transaction is committed.
>>> 
>>> A rough idea would be:
>>> 
>>> 1) for any transaction, write the messages to a separate ledger (or
>>> multiple separate ledger).
>>> 2) during the transaction, accumulates the messages in those ledgers.
>>> 3) when commit, merge the txn ledgers back to the main data ledger. the
>>> merge can be done either adding a meta message where data is stored in
>> the
>>> txn ledger or actually copying the data to data ledger (depending on the
>>> size of data accumulate in the transaction).
>>> 4) when abort, delete the txn ledger. No other additional work to be
>> done.
>>> 
>>> This would be producing a much clear design than Kafka.
>>> 
>>> On Ivan's comments:
>>> 
 Transactional acknowledgement also needs to be taken into account
>>> 
>>> I don't think we have to treat `transactional acknowledgement` as a
>> special
>>> case. currently `acknowledgment` are actually "append" operations into
>>> cursor ledgers.
>>> So the problem set can be reduced as `atomic append` to both data ledgers
>>> and cursor ledgers. in that way, we can use one solution for handling
>>> appending data and updating cursors.
>>> 
>>> Additionally, I think a related topic about transactions would be
>>> supporting large sized message (e.g. >= 5MB). If we take the approach I
>>> described above using a separated ledger for accumulating messages for a
>>> transaction, that we are easy to model a large size message as a
>>> transaction of chunked messages.
>>> 
>>> @Richard, @Ivan let me know what do you think. If you guys think the
>>> direction I raised is a good one to go down, I am happy to write them
>> down
>>> into details, and drive the design and coordinate the implementations in
>>> the community.
>>> 
>>> - Sijie
>>> 
>>> On Sat, Mar 2, 2019 at 9:45 AM Richard Yu 
>>> wrote:
>>> 
 Hi all,
 
 We might be able to get some ideas on implementing this from Kafka:
 
 
>>> 
>> https://cwiki.apache.org/confluence/display/KAFKA/Transactional+Messaging+in+Kafka
 
 Obviously, there is some differences in Kafka and Pulsar internals but
>> at
 some level, the implementation would be similar.
 It should help.
 
 
 
 On Thu, Feb 28, 2019 at 4:29 PM Richard Yu >> 
 wrote:
 
> Hi,
> 
> Per request, I've created a doc so we could get some more input in an
> organized manner:
> 
> 
 
>>> 
>> https://docs.google.com/document/d/1mSUsJvPgThnWizeQqljKU5244BMEiYHabA6OP6QEHmQ/edit?usp=sharing
> 
> And for Ivan's questions, I would answer accordingly.
> 
>> By "set the message to unknown", do you mean the broker will cache
>> the
>> message, not writing it to any log?
> 
> We wouldn't cache the message from my interpretation of the steps.
>> What
> the producer is first sending is a pre-processing message, not the
>> real
> message itself. This step basically notifies the broker that the
>>> message
 is
> on its way. So all we have to do is store the message id and its
> corresponding status in a map, and depending on the producer's
>>> response,
> the status will change accordingly.
> 
>> In desig

Re: PIP-31: Add support for transactional messaging

[Dave Fisher]

Hi -

> On Mar 2, 2019, at 6:39 PM, Sijie Guo  wrote:
> 
> Dave,
> 
> You mean implementing the transactions in pulsar function?

Yes, that way there is no additional broker overhead and whatever happens when 
a commit happens is under the control of those making the transaction.

I’m not sure if it would work, but it seems that functions, spouts, and 
connectors make sense as opposed to burdening the highly performant brokers.

Regards,
Dave

> 
> - Sijie
> 
>> On Sun, Mar 3, 2019 at 1:52 AM Dave Fisher  wrote:
>> 
>> Hi -
>> 
>> Is this a case where a Pulsar function base class for transactions would
>> help?
>> 
>> Regards,
>> Dave
>> 
>> Sent from my iPhone
>> 
>>> On Mar 2, 2019, at 2:39 AM, Sijie Guo  wrote:
>>> 
>>> Pravega's model is a better model than Kafka - it addressed the
>>> interleaving problems. However Pravega's model is based on a giant
>>> replicated log and rewrite the data to a second tiered storage for
>>> persistence, which basically re-implemented bookkeeper's logic in
>> broker. A
>>> fundamental drawback of Pravega is write amplifications. The
>> amplifications
>>> of both network and IO bandwidth are huge. If you use bookkeeper both for
>>> its first-and-second tier storage and assume the bookkeeper replication
>>> factor is 3, pravega requires 6x network bandwidth and 12x IO bandwidth.
>>> For a given message, it needs to write 3 times into the journal, and 3
>>> times for persistent. The amplifications hugely limit the throughput at
>>> pravega "brokers".
>>> 
>>> - Sijie
>>> 
>>> 
>>> 
>>>> On Sat, Mar 2, 2019 at 6:13 PM Ali Ahmed  wrote:
>>>> 
>>>> I agree we many want to review pravega's past efforts in this area also.
>>>> 
>>>> 
>>>> 
>> https://github.com/pravega/pravega/blob/master/documentation/src/docs/transactions.md
>>>> 
>>>> 
>> https://github.com/pravega/pravega/blob/master/client/src/main/java/io/pravega/client/stream/Transaction.java
>>>> 
>>>> -Ali
>>>> 
>>>>> On Sat, Mar 2, 2019 at 1:56 AM Sijie Guo  wrote:
>>>>> 
>>>>> Kafka's implementation is interleaving committed messages with
>>>> uncommitted
>>>>> messages at storage. Personally I think it is a very ugly design and
>>>>> implementation.
>>>>> 
>>>>> Pulsar is a segment centric system, where we have a shared segment
>>>> storage
>>>>> - bookkeeper. I think a better direction is to leverage the segments
>> (aka
>>>>> ledgers)
>>>>> for buffering uncommitted messages and commit the whole segment when
>> the
>>>>> whole transaction is committed.
>>>>> 
>>>>> A rough idea would be:
>>>>> 
>>>>> 1) for any transaction, write the messages to a separate ledger (or
>>>>> multiple separate ledger).
>>>>> 2) during the transaction, accumulates the messages in those ledgers.
>>>>> 3) when commit, merge the txn ledgers back to the main data ledger. the
>>>>> merge can be done either adding a meta message where data is stored in
>>>> the
>>>>> txn ledger or actually copying the data to data ledger (depending on
>> the
>>>>> size of data accumulate in the transaction).
>>>>> 4) when abort, delete the txn ledger. No other additional work to be
>>>> done.
>>>>> 
>>>>> This would be producing a much clear design than Kafka.
>>>>> 
>>>>> On Ivan's comments:
>>>>> 
>>>>>> Transactional acknowledgement also needs to be taken into account
>>>>> 
>>>>> I don't think we have to treat `transactional acknowledgement` as a
>>>> special
>>>>> case. currently `acknowledgment` are actually "append" operations into
>>>>> cursor ledgers.
>>>>> So the problem set can be reduced as `atomic append` to both data
>> ledgers
>>>>> and cursor ledgers. in that way, we can use one solution for handling
>>>>> appending data and updating cursors.
>>>>> 
>>>>> Additionally, I think a related topic about transactions would be
>>>>> supporting large sized message (e.g. >= 5MB). If we take the approach I
>>>>> described ab

Re: Google Season of Docs 2019

[Dave Fisher]

> On Mar 12, 2019, at 7:01 PM, Huxing Zhang  wrote:
> 
> Hi Dave,
> 
> On Wed, Mar 13, 2019 at 2:14 AM Dave Fisher  wrote:
>> 
>> Hi -
>> 
>> Looks pretty cool.
> 
> Yes, all the ASF projects could benefit from it. I think ASF should
> apply it on behalf of all ASF projects, just like how the GSoC work.
> Am I right?

Yes! And the time is pretty quick. I’m hoping a someone over here will 
volunteer to help.

Regards,
Dave


> 
>> 
>> Cc: to Apache Community Development.
>> 
>> Regards,
>> Dave
>> 
>> Sent from my iPhone
>> 
>>> On Mar 12, 2019, at 9:22 AM, Huxing Zhang  wrote:
>>> 
>>> Hi,
>>> 
>>> Google Season of Docs 2019[1] seems to be an interesting project,
>>> which bring open source project and technical writer communities
>>> together, just Like Google summer of code.
>>> 
>>> I think the Dubbo can benefit from the project, especially the English
>>> version of documentation could be improved.
>>> 
>>> How do you think?
>>> 
>>> [1] https://developers.google.com/season-of-docs/docs/timeline
>>> 
>>> --
>>> Best Regards！
>>> Huxing
>> 
> 
> 
> -- 
> Best Regards！
> Huxing
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
> For additional commands, e-mail: dev-h...@community.apache.org
> 

Re: subscribe request

[Dave Fisher]

You subscribe by sending an email to dev-subscr...@pulsar.apache.org and 
replying to the verification email.

Regards,
Dave

Sent from my iPhone

> On Mar 25, 2019, at 12:43 AM, jian he  wrote:
> 
> subscribe request

Re: PIP-40: Contribute Pulsar Manager

[Dave Fisher]

Hi -

I see that MySQL is used. This would be a GPL dependency. Does the Pulsar 
Manager require MySQL, or can other Databases that are compliant with Apache 
Release Policy be used instead?

Regards,
Dave

> On Aug 29, 2019, at 7:49 AM, Guangning E  wrote:
> 
> Hi all,
> 
> We have developed a new Pulsar web UI - pulsar-manager, aiming at
> supporting managing Pulsar clusters running in different environments
> (on-premise data centers, cloud, and so on). We’d like to contribute the
> project back to the Pulsar community.
> 
> ---
> 
> ## Motivation
> 
> Currently, Pulsar has a monitoring tool - [Pulsar Dashboard]( [
> http://pulsar.apache.org/docs/en/administration-dashboard/](http://pulsar.apache.org/docs/en/administration-dashboard/)
> ).
> However, it only focuses on the simple monitoring of Pulsar - collecting
> and displaying information such as statistics of tenants, namespaces,
> topics, subscriptions, and so on. It doesn’t provide any management
> operations such as add, delete and update tenants, namespaces, topics, and
> so on. When the scale of Pulsar cluster increases or the number of clusters
> grows, using `pulsar-admin` to manage Pulsar can not satisfy demands.
> Therefore, Pulsar needs a simple and easy-to-use management console for
> administrators.
> 
> Pulsar Manager is a web-based GUI management and monitoring tool that helps
> administrators and users manage and monitor tenants, namespaces, topics,
> subscriptions, brokers, clusters, and so on, and supports dynamic
> configuration of multiple environments.
> 
> ## Features
> 
> Pulsar manager provides two main features: management and monitoring.
> 
> ### Management
> 
> 1. Environment : (operations on the environment)
> Create / Delete / Update / List / Get
> 2. Cluster : (operations on clusters)
> Create / Delete / Update / List / Get
> 3. Brokers : (operations on brokers)
> Heartbeat / Unload
> 4. Tenants:
> Create / Delete / Update / List
> 5. Namespaces:
> Create / Delete / List
> Manage Namespace Policy
> Unload
> Operations on Namespace Bundles
> 6. Topics
> Create / Delete / List
> Unload / Terminate / Offload / Compact
> Topic Details
> 7. Subscriptions
> Create / Delete / List
> Reset Cursor / Skip / Clear backlog / Unsubscribe
> 8. Namespace Isolation Policies
> Create / Delete / Update / Get / List
> 9. Failure Domains
> Create / Delete / Update / Get / List
> 
> ### Monitoring: (display *aggregated* stats at different levels)
> 
> 1. Tenants
> List the total number of namespaces per tenant (both aggregated and
> per-cluster basis)
> 
> 2. Namespaces
> List the total number of topics per namespace
> List the aggregated stats per namespace (such as rate-in, rate-out,
> throughput-in, and throughput-out)
> List the distributions of namespace bundles
> 
> 3. Topics
> List the total number of partitions per topic
> List the aggregated stats per topic (such as rate-in, rate-out,
> throughput-in, and throughput-out)
> Detailed stats per topic partition
> Detailed stats of storage per topic partition
> Detailed stats of subscription per topic and per partition
> Detailed stats of producers
> Detailed stats of consumers
> 
> The detailed design proposal is in
> [
> https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#](https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#)
> 
> Looking forward to any feedback.
> 
> Thanks,
> Guangning

Is Slack Blocked in China?

[Dave Fisher]

Hi -

I heard somewhere that Slack is blocked in China. Is this so?

Regards,
Dave

Re: PIP-40: Contribute Pulsar Manager

[Dave Fisher]

BTW - Once the VOTE is complete to accept the codebase there is some paperwork.

http://incubator.apache.org/ip-clearance/

While this is the Incubator the Board has requested that the IPMC track IP 
Clearance. Everything is in SVN and the pages are updated every evening PDT. If 
help is needed then let me know.

Regards,
Dave

> On Sep 4, 2019, at 9:01 AM, Enrico Olivelli  wrote:
> 
> For whom who is interested I am working on adding support for HerdDB in
> Pulsar Manager.
> 
> The pull request is still very raw, but Pulsar Manager seems to work
> https://github.com/streamnative/pulsar-manager/pull/183
> 
> 
> 
> Il giorno mar 3 set 2019 alle ore 08:37 Yuva raj  ha
> scritto:
> 
>> On Fri, 30 Aug 2019 at 07:09, Sijie Guo  wrote:
>> 
>>>> I see there are dependencies on websockets, I would a prefer a simple
>>> polling model of the http, also it's enable by default in pulsar.
>>> 
>>> I don't think we depend don pulsar websocket. All are http restful
>>> requests.
>>> 
>>>> For the ui the default persistence should be sqlite. Potentially
>> packaged
>>> by default.
>>> 
>>> Initially the default is sqlite. But sqlite doesn't work if there are a
>> lot
>>> of topic metrics.
>>> We switched to MySQL as default for supporting production traffic. We can
>>> switch default to PostgresSQL (as pulsar dashboard).
>>> 
>> +1
>> Yes, Switching to Postgres Would be great. Because Postgres license is more
>> liberal and works well with Apache license ecosystem.
>> 
>>> 
>>> - Sijie
>>> 
>>> On Thu, Aug 29, 2019 at 6:23 PM Ali Ahmed  wrote:
>>> 
>>>> I see there are dependencies on websockets, I would a prefer a simple
>>>> polling model of the http, also it's enable by default in pulsar.
>>>> 
>>>> For the ui the default persistence should be sqlite. Potentially
>> packaged
>>>> by default.
>>>> 
>>>> -Ali
>>>> 
>>>> On Thu, Aug 29, 2019 at 2:47 PM Enrico Olivelli 
>>>> wrote:
>>>> 
>>>>> Il gio 29 ago 2019, 23:28 Sijie Guo  ha scritto:
>>>>> 
>>>>>> That sounds an interesting idea!
>>>>> 
>>>>> 
>>>>> Awesome
>>>>> 
>>>>> Does HerdDB support JDBC? If so, it should
>>>>>> be pretty straightforward to enable HerdDB.
>>>>>> 
>>>>> 
>>>>> Sure, as far as I know, the JDBC client is the only client really
>> used
>>> in
>>>>> production.
>>>>> 
>>>>> In a replicated environment it uses Zookeeper for metadata and
>> service
>>>>> discovery and Bookkeeper as WAL and if you have a pulsar cluster you
>>>>> already have both of them (zk cluster an bookies)
>>>>> 
>>>>> I will take a look to how Pulsar Manager   uses JDBC, maybe it will
>> be
>>>> very
>>>>> easy.
>>>>> 
>>>>> I will be back with news
>>>>> 
>>>>> 
>>>>> Enrico
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>>> Thanks,
>>>>>> Sijie
>>>>>> 
>>>>>> On Thu, Aug 29, 2019 at 12:08 PM Enrico Olivelli <
>>> eolive...@gmail.com>
>>>>>> wrote:
>>>>>> 
>>>>>>> Maybe you can try to use HerdDB (1), it is a replicated SQL
>>> database
>>>>> that
>>>>>>> can be run embedded in the JVM.
>>>>>>> It is an SQL database that we developed in my company, in order
>> to
>>>>>>> explicitly replace MySQL.
>>>>>>> It uses Bookkeeper to store the WAL and implement replication,
>> but
>>> it
>>>>> can
>>>>>>> run in standalone mode or in memory (for tests/dev).
>>>>>>> 
>>>>>>> I can help in setting up a demo.
>>>>>>> 
>>>>>>> Enrico
>>>>>>> 
>>>>>>> (1) https://github.com/diennea/herddb
>>>>>>> 
>>>>>>> Il gio 29 ago 2019, 16:59 Sijie Guo  ha
>>> scritto:
>>>>>>> 
>>>>>>>> Hi Dave,
>>>>>>>> 
>>>>>>>> It doesn't requi

Re: PIP-40: Contribute Pulsar Manager

[Dave Fisher]

> On Sep 4, 2019, at 10:10 AM, Sijie Guo  wrote:
> 
> Hi Dave,
> 
> Thank you for the reminder. I was about to ping you yesterday about this.
> The IP clearance is the last piece that I wasn’t sure about. Just to make
> sure I understand correctly, so the Pulsar PMC has to fill out the IP
> clearance template, check in the file to the SVN repo and start a thread in
> general@incubator to get the approval from IPMC. Is that correct?

Yes. The approval is via LAZY CONSENSUS of 3 days.

StreamNative ought to provide an SGA to secretary@ unless all of the IP is 
already covered by ICLAs.

Infra can then help transfer the repository. 

Regards,
Dave

> 
> Thanks,
> Sijie
> 
> On Wed, Sep 4, 2019 at 9:41 AM Dave Fisher  wrote:
> 
>> BTW - Once the VOTE is complete to accept the codebase there is some
>> paperwork.
>> 
>> http://incubator.apache.org/ip-clearance/
>> 
>> While this is the Incubator the Board has requested that the IPMC track IP
>> Clearance. Everything is in SVN and the pages are updated every evening
>> PDT. If help is needed then let me know.
>> 
>> Regards,
>> Dave
>> 
>>> On Sep 4, 2019, at 9:01 AM, Enrico Olivelli  wrote:
>>> 
>>> For whom who is interested I am working on adding support for HerdDB in
>>> Pulsar Manager.
>>> 
>>> The pull request is still very raw, but Pulsar Manager seems to work
>>> https://github.com/streamnative/pulsar-manager/pull/183
>>> 
>>> 
>>> 
>>> Il giorno mar 3 set 2019 alle ore 08:37 Yuva raj  ha
>>> scritto:
>>> 
>>>> On Fri, 30 Aug 2019 at 07:09, Sijie Guo  wrote:
>>>> 
>>>>>> I see there are dependencies on websockets, I would a prefer a simple
>>>>> polling model of the http, also it's enable by default in pulsar.
>>>>> 
>>>>> I don't think we depend don pulsar websocket. All are http restful
>>>>> requests.
>>>>> 
>>>>>> For the ui the default persistence should be sqlite. Potentially
>>>> packaged
>>>>> by default.
>>>>> 
>>>>> Initially the default is sqlite. But sqlite doesn't work if there are a
>>>> lot
>>>>> of topic metrics.
>>>>> We switched to MySQL as default for supporting production traffic. We
>> can
>>>>> switch default to PostgresSQL (as pulsar dashboard).
>>>>> 
>>>> +1
>>>> Yes, Switching to Postgres Would be great. Because Postgres license is
>> more
>>>> liberal and works well with Apache license ecosystem.
>>>> 
>>>>> 
>>>>> - Sijie
>>>>> 
>>>>> On Thu, Aug 29, 2019 at 6:23 PM Ali Ahmed 
>> wrote:
>>>>> 
>>>>>> I see there are dependencies on websockets, I would a prefer a simple
>>>>>> polling model of the http, also it's enable by default in pulsar.
>>>>>> 
>>>>>> For the ui the default persistence should be sqlite. Potentially
>>>> packaged
>>>>>> by default.
>>>>>> 
>>>>>> -Ali
>>>>>> 
>>>>>> On Thu, Aug 29, 2019 at 2:47 PM Enrico Olivelli 
>>>>>> wrote:
>>>>>> 
>>>>>>> Il gio 29 ago 2019, 23:28 Sijie Guo  ha scritto:
>>>>>>> 
>>>>>>>> That sounds an interesting idea!
>>>>>>> 
>>>>>>> 
>>>>>>> Awesome
>>>>>>> 
>>>>>>> Does HerdDB support JDBC? If so, it should
>>>>>>>> be pretty straightforward to enable HerdDB.
>>>>>>>> 
>>>>>>> 
>>>>>>> Sure, as far as I know, the JDBC client is the only client really
>>>> used
>>>>> in
>>>>>>> production.
>>>>>>> 
>>>>>>> In a replicated environment it uses Zookeeper for metadata and
>>>> service
>>>>>>> discovery and Bookkeeper as WAL and if you have a pulsar cluster you
>>>>>>> already have both of them (zk cluster an bookies)
>>>>>>> 
>>>>>>> I will take a look to how Pulsar Manager   uses JDBC, maybe it will
>>>> be
>>>>>> very
>>>>>>> easy.
>>>>>>> 
>>>>>>> I will be back with news
>>>>>>&

Re: ApacheCon NA

[Dave Fisher]

Hi -

In addition to finding a time to meet there are two “official” ways to do this:

(1) Birds of a Feather in the evening which may miss other events: 
https://www.apachecon.com/acna19/bof.html

(2) Hackathon - https://www.apachecon.com/acna19/hackathon.html

Regards,
Dave

> On Sep 4, 2019, at 10:49 AM, David Kjerrumgaard  wrote:
> 
> Count me in!!  What day / time are you thinking?
> 
> On Tue, Sep 3, 2019 at 5:32 PM Sijie Guo  wrote:
> 
>> Hi,
>> 
>> ApacheCon NA is coming next week. There are about 5 pulsar talks this year.
>> I guess quite a lot of Pulsar folks and users will be attending it. For the
>> people who is attending ApacheCon, maybe we should gather together for a
>> drink :-) That can be fun. Grab me if you are in ApacheCon and interested
>> in meeting for a drink :-)
>> 
>> Thanks,
>> Sijie
>> 
> 
> 
> -- 
> Regards,
> 
> David Kjerrumgaard
> Director - Solution Architecture
> Streamlio
> Cell: 330-437-5467

Re: ApacheCon NA

[Dave Fisher]

Hi -

Send an email to rbo...@apache.org once you’ve picked the day.

Regards,
Dave

> On Sep 4, 2019, at 11:00 AM, David Kjerrumgaard  wrote:
> 
> Dave,
> 
> That sounds great, can we add a BoF session for Apache Pulsar?
> 
> Regards,
> David Kjerrumgaard
> 
> 
> 
> On Wed, Sep 4, 2019 at 10:55 AM Dave Fisher  wrote:
> 
>> Hi -
>> 
>> In addition to finding a time to meet there are two “official” ways to do
>> this:
>> 
>> (1) Birds of a Feather in the evening which may miss other events:
>> https://www.apachecon.com/acna19/bof.html
>> 
>> (2) Hackathon - https://www.apachecon.com/acna19/hackathon.html
>> 
>> Regards,
>> Dave
>> 
>>> On Sep 4, 2019, at 10:49 AM, David Kjerrumgaard 
>> wrote:
>>> 
>>> Count me in!!  What day / time are you thinking?
>>> 
>>> On Tue, Sep 3, 2019 at 5:32 PM Sijie Guo  wrote:
>>> 
>>>> Hi,
>>>> 
>>>> ApacheCon NA is coming next week. There are about 5 pulsar talks this
>> year.
>>>> I guess quite a lot of Pulsar folks and users will be attending it. For
>> the
>>>> people who is attending ApacheCon, maybe we should gather together for a
>>>> drink :-) That can be fun. Grab me if you are in ApacheCon and
>> interested
>>>> in meeting for a drink :-)
>>>> 
>>>> Thanks,
>>>> Sijie
>>>> 
>>> 
>>> 
>>> --
>>> Regards,
>>> 
>>> David Kjerrumgaard
>>> Director - Solution Architecture
>>> Streamlio
>>> Cell: 330-437-5467
>> 
>> 
> 
> -- 
> Regards,
> 
> David Kjerrumgaard
> Director - Solution Architecture
> Streamlio
> Cell: 330-437-5467

Re: [DISCUSS] Improve website publish process in release procedure

[Dave Fisher]

A logical conundrum! You can’t get it index until its published and you can’t 
control when the updated site is indexed. Is there anyway to trigger their 
index? If so, do it to minimize the trouble.

Infra just announced a new .asf.yaml service. I can see how we could add Apache 
Solr indexing to it. A thought to discuss with Infra.

Regards,
Dave

Sent from my iPhone

> On Sep 5, 2019, at 11:26 AM, Sijie Guo  wrote:
> 
> Hi all,
> 
> I am starting a discussion to gather ideas on how to improve the website
> publish process in our release procedure. Searchbox doesn't work
> immediately after we publish the documentation for new release. It happened
> both 2.4.0 and 2.4.1 release.
> 
> - The documentation is indexed using Algolia DocSearch.
> - Algolia only indexes the documentation when the documentation is alive.
> - Apparently the index sometimes take longer time to complete.
> 
> So it seems to me that we should consider marking the release as
> active/default only after confirming that Algolia already indexes the
> documentation and search box is working properly.
> 
> Thoughts?
> 
> - Sijie

Re: [DISCUSS] PIP: Producer Send Message with Different Schema

[Dave Fisher]

So true Sijie!  That said these discussions are critical!

Regards,
Dave

Sent from my iPhone

> On Sep 6, 2019, at 1:12 AM, Sijie Guo  wrote:
> 
> We haven't defined a PIP process. However the community has run PIP in the
> lazy consensus mode.
> 
> Lazy consensus means you don't have to insist people discuss and/or approve
> your plan, and you certainly don't
> need to call a vote to get approval. You just assume you have the
> communities support unless someone says otherwise.
> 
> So that means go ahead and send the pull requests. If people has
> suggestions or objections, they  will raise in the discussion or in your
> pull requests.
> 
> Thanks,
> Sijie
> 
>> On Thu, Sep 5, 2019 at 11:14 PM Yi Tang  wrote:
>> 
>> Thanks, Sijie, Penghui. So what's next? I am not familiar with the whole
>> process.
>> 
>> Sijie Guo  于 2019年9月6日周五 02:15写道：
>> 
>>> Hi Yi,
>>> 
>>> The  proposal looks pretty good. +1 from me.
>>> 
>>> Looking forward to the implementation.
>>> 
>>> - Sijie
>>> 
 On Thu, Sep 5, 2019 at 7:37 AM Yi Tang  wrote:
 
 Hi, Pulsar folks,any more feedbacks?
 
 PengHui Li  于 2019年9月3日周二 14:53写道：
 
> 👍 Looks good to me +1.
> 
> Sijie Guo  于2019年9月3日周二 下午2:12写道：
> 
>> Thank you Yi.
>> 
>> I have copied your gist to PIP-43
>> 
>> 
> 
 
>>> 
>> https://github.com/apache/pulsar/wiki/PIP-43%3A-producer-send-message-with-different-schema
>> .
>> 
>> Thanks,
>> Sijie
>> 
>>> On Mon, Sep 2, 2019 at 10:35 PM 唐谊  wrote:
>>> 
>>> Here are some previous discussions,
>>> https://github.com/apache/pulsar/issues/4806
>>> 
>> 
> 
 
>>> 
>> 

Re: [DISCUSS] Create the first pulsar-manager apache release

[Dave Fisher]

> On Oct 24, 2019, at 1:16 PM, Enrico Olivelli  wrote:
> 
> Guangnin,
> The documents looks good.
> Just one clarification:
> 
> When you say
> 
> Source and binary files:
> 
> 
> You don't provide a link to the staging area but only to the github tag
> 
> Is this expected?
> IMHO we should have a link to the svn repository

That’s not just an opinion, it really is a requirement. That way we can be sure 
that what is voted to be released is what is actually released.

Also, Github tags are not immutable.

Here is the release distribution policy: 
https://www.apache.org/dev/release-distribution.html

Regards,
Dave

> 
> 
> Hope that helps
> Enrico
> 
> Il gio 24 ott 2019, 08:10 Sijie Guo  ha scritto:
> 
>> Guangning,
>> 
>> Thank you very much for putting all these together!
>> 
>> Overall looks good to me. A few comments:
>> 
>> 1) Initially we started with version 0.0.1. I would suggest us bump the
>> version to 0.1.0 to follow semantic versioning (https://semver.org/) for
>> managing our release versions. So we can manage both major releases and
>> minor/bugfix releases.
>> 
>> 2) I didn't see a process of updating pulsar website. I would suggest
>> adding a section in pulsar download (http://pulsar.apache.org/en/download/
>> )
>> page for hosting the releases for pulsar manager, and add a section of
>> updating download page as part of release process.
>> 
>> 3) It would be good to add a documentation page about pulsar manager to
>> pulsar website under "Admin" section. So people know there is a pulsar
>> manager can be used for administering the Pulsar clusters.
>> 
>> - Sijie
>> 
>> On Wed, Oct 23, 2019 at 11:10 AM Guangning E  wrote:
>> 
>>> Hi, all
>>> 
>>> I have written some release and verification steps and recorded them as
>>> follows:
>>> 
>>> release:
>>> https://gist.github.com/tuteng/86a75e3421794a6b2048d5d2aca2c790
>>> 
>>> validation:
>>> https://gist.github.com/tuteng/41c5c6489e27864f9cd4c2ac13dc2925
>>> 
>>> What do you think about this?
>>> If you have any comments on this, please reply to this email.
>>> 
>>> Next, I will apply for pulsar-manager to enable a wiki page and record
>> the
>>> content on the wiki.
>>> 
>>> Thanks,
>>> Guangning
>>> 
>>> Guangning E  于2019年9月25日周三 下午8:51写道：
>>> 
 Ok, I will draft a release document next.
 
 
 Thanks,
 Guangning
 
 Sijie Guo  于2019年9月25日周三 下午3:13写道：
 
> Thank you guys!
> 
> Guangning, do you want to draft a doc for the release process and
>> share
>>> it
> in the mailing list? So we can move the discussion forward.
> 
> Thanks,
> Sijie
> 
> On Tue, Sep 24, 2019 at 6:18 AM Nozomi Kurihara <
>> nkuri...@yahoo-corp.jp
 
> wrote:
> 
>> +1
>> 
>> Sounds nice to create the first release as soon as possible.
>> As Guangning says, deciding the release process is necessary.
>> 
>> Regards,
>> Nozomi
>> 
>> 差出人: Guangning E 
>> 送信日時: 2019年9月22日 11:18
>> 宛先: dev@pulsar.apache.org 
>> 件名: Re: [DISCUSS] Create the first pulsar-manager apache release
>> 
>> That sounds very good. Before that, we may need to write a release
>> and
>> verification process.
>> 
>> 
>> 
>> Thanks,
>> Guangning
>> 
>> Sijie Guo  于2019年9月22日周日 上午9:35写道：
>> 
>>> Hi all,
>>> 
>>> Since the pulsar-manager repo has been transferred to the ASF
>> under
>> Pulsar
>>> PMC, I would suggest us creating the first pulsar-manager apache
> release
>> as
>>> soon as possible. So that we can identify what to release and what
>>> are
>> the
>>> potential issues to fix to follow the Apache way.
>>> 
>>> This is the email thread for discussing the first pulsar-manager
> release.
>>> Please chime in with your thoughts.
>>> 
>>> Thanks,
>>> Sijie
>>> 
>> 
> 
 
>>> 
>> 

Re: [VOTE] PIP-47: Time Based Release Plan

[Dave Fisher]

-1 (binding)

This proposal has NOT been properly discussed in the Apache Way on the mailing 
list. There has been absolutely no discussion of this policy. There has been no 
attempt to reach consensus.

Apache projects do NOT proceed like this. Please rescind this VOTE!

Start a proper discussion thread.

Thank you,
Dave

Sent from my iPhone

> On Nov 6, 2019, at 2:18 AM, xiaolong ran  wrote:
> 
> Hi everyone
> 
> Please review and vote on the PIP-47 propose,
> 
> https://github.com/apache/pulsar/wiki/PIP-47%3A-Time-Based-Release-Plan 
> 
> 
> as follows:
> [ ] +1, Approve the propose
> [ ] -1, Do not approve the propose (please provide specific comments)
> 
> --
> 
> Thanks
> Xiaolong Ran
> 

Re: PIP 49: Permission levels and inheritance

[Dave Fisher]

I’m not diving in but thinking about the logical implication of this dichotomy. 
For any object’s attributes some ought to be controlled by object level 
permissions and others by sysadmin permissions. How can developers tell?

Best Regards,
Dave


Sent from my iPhone

> On Nov 7, 2019, at 8:02 PM, Joe F  wrote:
> 
> This proposal has the same issues as the previous one . In general it is
> not correct to think of commands  as being controlled by owner of the
> object  on which the command operates. Eg: It will be an error to assing
> control of all namespace commands to the namespace owner
> 
> For eg:  set subscription-dispatch-rate operates on a subscription rate and
> set-max-producers-per-topic These operate on namespaces. A naive approach
> is to think that this would be controlled by the namespace owner.   But
> essentially both these are resource management  operations. That should be
> controlled by the system admin, unless you want to deal with every
> namespace owner having the power to destroy your SLAs fo the cluster.
> 
> I just picked 2 examples to indicate the drawback of approaching
> permissions in the proposed  manner.
> 
> There are many such cases in the proposal, too many to list out here. I
> suggest completely ditching this approach of equating objects with  admin
> capability of object owner
> 
> Joe
> 
> 
>> On Wed, Nov 6, 2019 at 2:14 AM xiaolong ran 
>> wrote:
>> 
>> Hello all committers:
>> 
>> About this PIP, do you have any other good ideas or propose.
>> 
>> 
>> --
>> 
>> Thanks
>> Xioalong Ran
>> 
>> 
 在 2019年10月30日，下午7:49，xiaolong ran  写道：
>>> 
>>> Hello all:
>>> 
>>> When using pulsar-admin, I found that the current permission
>> verification mechanism has some problems.
>>> 
>>> We are starting a proposal about permission levels and inheritance in
>> Apache Pulsar.
>>> 
>>> The proposal is in:
>>> 
>> https://github.com/apache/pulsar/wiki/PIP-49%3A-Permission-levels-and-inheritance
>> <
>> https://github.com/apache/pulsar/wiki/PIP-49:-Permission-levels-and-inheritance
>>> 
>>> 
>>> To ensure compatibility, these changes will be made in v4's admin api.
>> About the v4 admin API, see:
>>> 
>>> https://github.com/apache/pulsar/wiki/PIP-48%3A-hierarchical-admin-api <
>> https://github.com/apache/pulsar/wiki/PIP-48:-hierarchical-admin-api>
>>> 
>>> Looking forward to any feedback.
>> 
>> 

Re: [DISCUSS] Consistent code style (esp. ws/indent) and autotools

[Dave Fisher]

-0. I think it will introduce too much change. We have classes that are quite 
large and having to fix code style after making a small correction seems like a 
waste of volunteer energy.

Best,
Dave

Sent from my iPhone

> On Aug 31, 2023, at 9:05 PM, Zixuan Liu  wrote:
> 
> This idea looks good to me, but we need to format all codebase to
> avoid conflicts during cherry picking.
> 
> +1
> 
> Asaf Mesika  于2023年8月31日周四 20:39写道：
>> 
>> Opentelemetry-java employs both spotless for coding style
>> You run "./gradlew spotlessCheck" and it shows the problems.
>> You run "./gradlew spotlessApply" to automatically fix it.
>> 
>> It also uses errorprone to detect bugs.
>> 
>> I wonder if we can run it only in GitHub PRs, so we can instruct it to run
>> only on files you have changed / added. This is we "throttle" the style
>> through files touched.
>> 
>> 
>> 
>>> On Tue, Aug 15, 2023 at 8:31 PM tison  wrote:
>>> 
>>> These have been discussed that -
>>> 
>>> 1.  Cherrypick: we will reformat for all maintained branches.
>>> 2. Commit noise: metadata to filter out during blame.
>>> 3. PR rebased: invincible, while we don't have a large backlog or active
>>> large pending PR.
>>> 
>>> But if our critical mass agree this is not a good tradeoff, there is no
>>> issue to "resolve".
>>> 
>>> Enrico Olivelli 于2023年8月16日 周三00:50写道：
>>> 
 Tison,
 
 Il Mar 15 Ago 2023, 16:56 tison  ha scritto:
 
> A demostration for change set -
> https://github.com/apache/pulsar/pull/20974
 
 
 
 While I think it is great to start with Spotless for little projects or
 when you start from scratch,
 appling a patch that changes 3.000+ files will make it very hard to
>>> rebase
 all the pending PRs and also to cherry pick changes to older branches
>>> that
 we support.
 
 I think that this is not a good option for Pulsar at this stage.
 
 Maybe if we had a configuration that doesn't change so many files
 
 Enrico
 
> 
> 
> If we go forward this direction, it should be picked to branch-3.0 and
> branch-3.1. Earlier versions can be ported on demand and I'm glad to
> volunteer doing that.
> 
> Best,
> tison.
> 
> 
> PengHui Li  于2023年7月10日周一 10:00写道：
> 
>> My concern is how much value it will add.
>> From my experience, it's fine. The code style
>> is not consistent but doesn't affect my code reading
>> and writing much. But it might introduce risks and we
>> need to pay much effort to the code review.
>> 
>> Regards,
>> Penghui
>> 
>> On Wed, Jul 5, 2023 at 7:39 PM tison  wrote:
>> 
>>> ... which seems a GitHub only extension -
>>> 
>>> 
>> 
> 
 
>>> https://github.blog/changelog/2022-03-24-ignore-commits-in-the-blame-view-beta/
>>> 
>>> Best,
>>> tison.
>>> 
>>> 
>>> tison  于2023年7月5日周三 19:38写道：
>>> 
 For the git blame issue, I found also this practice in
 StreamPark[1].
 
 cc @Yunze.
 
 Best,
 tison.
 
 [1]
 
>>> 
>> 
> 
 
>>> https://github.com/apache/incubator-streampark/blob/cac931ae289e0753892279336e1c4e70e5f7d7c6/.git-blame-ignore-revs
 
 
 Kiryl Valkovich  于2023年6月30日周五
 13:03写道：
 
> My mistake. It looks that for some reason Spotless supports
>>> .editorconfig
> only for ktlint.
> 
> Best,
> Kiryl
> 
> From: Kiryl Valkovich 
> Date: Friday, June 30, 2023 at 6:45 AM
> To: dev@pulsar.apache.org 
> Subject: Re: [DISCUSS] Consistent code style (esp. ws/indent)
>>> and
> autotools
> Hi,
> 
> tison, are you going to use .editorconfig for specifying indent
> rules?
> 
> https://editorconfig.org/
> 
> It looks like Spotless supports it.
> https://github.com/diffplug/spotless/issues/734
> 
> Flink and many other ASF projects use it.
> 
>>> 
>> 
> 
 
>>> https://github.com/apache/flink/blob/21eba4ca4cb235a2189c94cdbf3abcec5cde1e6e/.editorconfig
> 
 https://github.com/search?q=org%3Aapache%20.editorconfig&type=code
> 
> Unlike Spotless, the .editorconfig works out of the box in most
>>> of
> the
> modern code editors.
> 
> Best,
> Kiryl
> 
> From: tison 
> Date: Friday, June 30, 2023 at 3:58 AM
> To: Dev 
> Subject: [DISCUSS] Consistent code style (esp. ws/indent) and
>> autotools
> Hi,
> 
> I'd like to propose applying a consistent code style (especially
> whitespace
> and indent) with an autotool Spotless.
> 
> // Background
> 
> Over and over we argue contributors reformat their patch
>>> manually
> for
> 

Re: [DISCUSS] Consistent code style (esp. ws/indent) and autotools

[Dave Fisher]

While I can agree that a consistent style can help I don’t agree that it is 
necessary. If the compiler understands the code then IMO we are good. 

I am a bit of a dinosaur since I have keypunched code on cards in my career. 
I’ve played with writing interpreters and specialized languages.

But I’m -0 and if the project prefers strict code style then that is fine too!

If anyone agrees with me know that part of consensus building is to provide 
opinions and accept divergent results.

Best,
Dave

PS. If tisun wants to put on their superhero cape and convert the code base 
then let’s acknowledge that AND let’s consider all of the PRs that are now 
effectively closed.

Sent from my iPhone

> On Sep 1, 2023, at 8:57 PM, SiNan Liu  wrote:
> 
> Consistent code style is crucial for a large project. In order to make
> Pulsar better, I believe we shouldn't be afraid of "risks".
> By introducing Spotless, we can permanently resolve the issue of
> inconsistent code style and ensure that all contributors adhere to this
> rule moving forward.
> If we don't make these changes now, we might have to deal with changes in
> over 3000 files today and potentially over 5000 files tomorrow.
> 
> Thanks,
> sinan
> 
> 
> Dave Fisher  于2023年9月1日周五 12:19写道：
> 
>> -0. I think it will introduce too much change. We have classes that are
>> quite large and having to fix code style after making a small correction
>> seems like a waste of volunteer energy.
>> 
>> Best,
>> Dave
>> 
>> Sent from my iPhone
>> 
>>>> On Aug 31, 2023, at 9:05 PM, Zixuan Liu  wrote:
>>> 
>>> This idea looks good to me, but we need to format all codebase to
>>> avoid conflicts during cherry picking.
>>> 
>>> +1
>>> 
>>> Asaf Mesika  于2023年8月31日周四 20:39写道：
>>>> 
>>>> Opentelemetry-java employs both spotless for coding style
>>>> You run "./gradlew spotlessCheck" and it shows the problems.
>>>> You run "./gradlew spotlessApply" to automatically fix it.
>>>> 
>>>> It also uses errorprone to detect bugs.
>>>> 
>>>> I wonder if we can run it only in GitHub PRs, so we can instruct it to
>> run
>>>> only on files you have changed / added. This is we "throttle" the style
>>>> through files touched.
>>>> 
>>>> 
>>>> 
>>>>> On Tue, Aug 15, 2023 at 8:31 PM tison  wrote:
>>>>> 
>>>>> These have been discussed that -
>>>>> 
>>>>> 1.  Cherrypick: we will reformat for all maintained branches.
>>>>> 2. Commit noise: metadata to filter out during blame.
>>>>> 3. PR rebased: invincible, while we don't have a large backlog or
>> active
>>>>> large pending PR.
>>>>> 
>>>>> But if our critical mass agree this is not a good tradeoff, there is no
>>>>> issue to "resolve".
>>>>> 
>>>>> Enrico Olivelli 于2023年8月16日 周三00:50写道：
>>>>> 
>>>>>> Tison,
>>>>>> 
>>>>>> Il Mar 15 Ago 2023, 16:56 tison  ha scritto:
>>>>>> 
>>>>>>> A demostration for change set -
>>>>>>> https://github.com/apache/pulsar/pull/20974
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> While I think it is great to start with Spotless for little projects
>> or
>>>>>> when you start from scratch,
>>>>>> appling a patch that changes 3.000+ files will make it very hard to
>>>>> rebase
>>>>>> all the pending PRs and also to cherry pick changes to older branches
>>>>> that
>>>>>> we support.
>>>>>> 
>>>>>> I think that this is not a good option for Pulsar at this stage.
>>>>>> 
>>>>>> Maybe if we had a configuration that doesn't change so many files
>>>>>> 
>>>>>> Enrico
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> If we go forward this direction, it should be picked to branch-3.0
>> and
>>>>>>> branch-3.1. Earlier versions can be ported on demand and I'm glad to
>>>>>>> volunteer doing that.
>>>>>>> 
>>>>>>> Best,
>>>>>>> tison.
>>>>>>> 
>>>>>>> 
>>>>>>> PengHui Li  于2023年7月10日周一 10

Re: [Vote] PIP-281: Optimize Bundle Unload(Transfer) Protocol for ExtensibleLoadManager

[Dave Fisher]

There has certainly been a lot of discussion on the pip and also the email 
thread. It was a substantive debate about potential protocol changes. The 
result doesn’t look conclusive.

Where did you land regarding Michael’s ideas?

Best,
Dave

Sent from my iPhone

> On Sep 5, 2023, at 11:56 AM, Heesung Sohn 
>  wrote:
> 
> Hi dev,
> 
> It has been open for a month now. Could we have some votes here?
> 
> Thanks,
> Heesung
> 
>> On Mon, Aug 7, 2023 at 11:55 AM Heesung Sohn 
>> wrote:
>> 
>> Hi,
>> 
>> I'd like to start a vote thread for the PIP-281.
>> https://github.com/apache/pulsar/pull/20748
>> 
>> Discussion :
>> https://lists.apache.org/thread/bdmx4qhn6hkoxm0xbtf67tq4kt5r8jmy
>> 
>> Best,
>> Heesung
>> 
>> 
>> 

Re: [DISSCUSS] PIP-298: Consumer supports specifying consumption isolation level

[Dave Fisher]

My concern is that this pip allows consumers to change the processing rules for 
transactions in ways that a producer might find unexpected.

I think if this proceeds then the scope needs to be expanded to 
producers/admins needing to proactively allow transactions to be consumed 
uncommitted.

I am also interested in the use case that motivates this change.

Best,
Dave

Sent from my iPhone

> On Sep 17, 2023, at 8:50 AM, hzh0425  wrote:
> 
> Hi, all
> 
> This PR contributed to pip 298: https://github.com/apache/pulsar/pull/21114
> 
> 
> 
> 
> This pip is to implement Read Committed and Read Uncommitted isolation levels 
> for Pulsar transactions, allow consumers to configure isolation levels during 
> the building process.
> 
> For more details, please refer to pip-298.md
> 
> 
> 
> 
> I hope everyone can help review and discuss this pip and enter the discuss 
> stage.
> 
> Thanks
> Zhangheng Huang

Re: [DISSCUSS] PIP-298: Consumer supports specifying consumption isolation level

[Dave Fisher]

Thanks. So, this is to support exfiltration of uncommitted transaction data? 
This is IMO wrong and a security risk.

Pulsar already supports CDC through IO Connectors.

Kafka can be wrong about these isolation levels.

There is really no information in those Paimon issues. How is Paimon’s ability 
to support Pulsar broken by this edge case?

Best,
Dave

Sent from my iPhone

> On Sep 18, 2023, at 7:26 AM, Xiangying Meng  wrote:
> 
> Hi Dave,
> This is an external request. Paimon has added support for Kafka but
> has not yet incorporated support for Pulsar. Therefore, the Paimon
> community desires to integrate Pulsar.
> 
> Furthermore, when integrating Pulsar into Paimon, it is desired to
> enable the ability to configure isolation levels, similar to Kafka, to
> support reading uncommitted transaction logs.
> 
> Additional context can be found in the following link:
> https://github.com/apache/incubator-paimon/issues/765
> 
> Sincerely,
> Xiangying
> 
>> On Mon, Sep 18, 2023 at 10:30 AM Dave Fisher  wrote:
>> 
>> My concern is that this pip allows consumers to change the processing rules 
>> for transactions in ways that a producer might find unexpected.
>> 
>> I think if this proceeds then the scope needs to be expanded to 
>> producers/admins needing to proactively allow transactions to be consumed 
>> uncommitted.
>> 
>> I am also interested in the use case that motivates this change.
>> 
>> Best,
>> Dave
>> 
>> Sent from my iPhone
>> 
>>>> On Sep 17, 2023, at 8:50 AM, hzh0425  wrote:
>>> 
>>> Hi, all
>>> 
>>> This PR contributed to pip 298: https://github.com/apache/pulsar/pull/21114
>>> 
>>> 
>>> 
>>> 
>>> This pip is to implement Read Committed and Read Uncommitted isolation 
>>> levels for Pulsar transactions, allow consumers to configure isolation 
>>> levels during the building process.
>>> 
>>> For more details, please refer to pip-298.md
>>> 
>>> 
>>> 
>>> 
>>> I hope everyone can help review and discuss this pip and enter the discuss 
>>> stage.
>>> 
>>> Thanks
>>> Zhangheng Huang
>> 

Re: [DISSCUSS] PIP-298: Consumer supports specifying consumption isolation level

[Dave Fisher]

Let me try a different approach. Please see the definition of a Pulsar 
Transaction - https://pulsar.apache.org/docs/3.1.x/transactions/

If messages that are uncommitted are consumed that definition is no longer 
true. If breaking the definition is going to be allowed to a consumer then the 
producer who may abort the transaction ought to have given permission.

You never know what content is in that transaction and it might be aborted at a 
user’s choice for any reason.

If Kafka follows a different policy with transactions then perhaps you should 
look into Kafka on Pulsar / Starlight for Kafka protocol handlers.

If we want Pulsar to allow for Kafka style then let’s be clear about the 
implications along with expectations.

Best,
Dave

Sent from my iPhone

> On Sep 18, 2023, at 6:41 PM, Xiangying Meng  wrote:
> 
> Hi Dave,
> 
> I greatly appreciate your perspective, yet it leaves me with some
> uncertainties that I am eager to address. Why would the introduction
> of isolation levels constitute an insecure action?
> 
>> I think if this proceeds then the scope needs to be expanded to 
>> producers/admins needing to proactively allow transactions to be consumed 
>> uncommitted.
> 
> We are merely presenting an option to the users. The notion of
> establishing isolation levels for producers and administrators is, in
> my view, devoid of necessity, and I am not inclined to implement it,
> for it is devoid of substance.
> 
> Sincerely,
> Xiangying
> 
>> On Mon, Sep 18, 2023 at 10:58 PM Dave Fisher  wrote:
>> 
>> Thanks. So, this is to support exfiltration of uncommitted transaction data? 
>> This is IMO wrong and a security risk.
>> 
>> Pulsar already supports CDC through IO Connectors.
>> 
>> Kafka can be wrong about these isolation levels.
>> 
>> There is really no information in those Paimon issues. How is Paimon’s 
>> ability to support Pulsar broken by this edge case?
>> 
>> Best,
>> Dave
>> 
>> Sent from my iPhone
>> 
>>>> On Sep 18, 2023, at 7:26 AM, Xiangying Meng  wrote:
>>> 
>>> Hi Dave,
>>> This is an external request. Paimon has added support for Kafka but
>>> has not yet incorporated support for Pulsar. Therefore, the Paimon
>>> community desires to integrate Pulsar.
>>> 
>>> Furthermore, when integrating Pulsar into Paimon, it is desired to
>>> enable the ability to configure isolation levels, similar to Kafka, to
>>> support reading uncommitted transaction logs.
>>> 
>>> Additional context can be found in the following link:
>>> https://github.com/apache/incubator-paimon/issues/765
>>> 
>>> Sincerely,
>>> Xiangying
>>> 
>>>> On Mon, Sep 18, 2023 at 10:30 AM Dave Fisher  wrote:
>>>> 
>>>> My concern is that this pip allows consumers to change the processing 
>>>> rules for transactions in ways that a producer might find unexpected.
>>>> 
>>>> I think if this proceeds then the scope needs to be expanded to 
>>>> producers/admins needing to proactively allow transactions to be consumed 
>>>> uncommitted.
>>>> 
>>>> I am also interested in the use case that motivates this change.
>>>> 
>>>> Best,
>>>> Dave
>>>> 
>>>> Sent from my iPhone
>>>> 
>>>>>> On Sep 17, 2023, at 8:50 AM, hzh0425  wrote:
>>>>> 
>>>>> Hi, all
>>>>> 
>>>>> This PR contributed to pip 298: 
>>>>> https://github.com/apache/pulsar/pull/21114
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> This pip is to implement Read Committed and Read Uncommitted isolation 
>>>>> levels for Pulsar transactions, allow consumers to configure isolation 
>>>>> levels during the building process.
>>>>> 
>>>>> For more details, please refer to pip-298.md
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> I hope everyone can help review and discuss this pip and enter the 
>>>>> discuss stage.
>>>>> 
>>>>> Thanks
>>>>> Zhangheng Huang
>>>> 
>> 

Re: [DISCUSS] Release Pulsar 3.1.1

[Dave Fisher]

Please use the correct name of the project - Apache Pulsar.

Thanks,
Dave

Sent from my iPhone

> On Sep 17, 2023, at 10:32 PM, guo jiwei  wrote:
> 
> Hi all,
> 
> I would like to propose releasing the Pulsar 3.1.1.
> 
> It's over one month since the release of 3.1.0 and there are 56 new commits
> in branch-3.1:
> https://github.com/apache/pulsar/compare/v3.1.0...branch-3.1
> 
> We need to cut a new release. Please let me know if you have any
> important fixes that need to be included in Pulsar 3.1.1.
> 
> 
> Regards
> Jiwei Guo (Tboy)