date:20230511



michaeljmarshall commented on code in PR #20286:
URL: https://github.com/apache/pulsar/pull/20286#discussion_r1191900752


##
pulsar-broker/src/test/java/org/apache/pulsar/broker/service/BrokerServiceTest.java:
##
@@ -172,8 +172,6 @@ public void testShutDownWithMaxConcurrentUnload() throws 
Exception {
 assertTrue(e instanceof PulsarClientException.TimeoutException);
 }
 
-pulsar = null;

Review Comment:
   Thanks for your explanation. Do we run these tests in parallel? It seems 
like we shouldn't if they are sharing state.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] nodece commented on a diff in pull request #19432: [improve] Allow to build and push multi-arch Docker images



nodece commented on code in PR #19432:
URL: https://github.com/apache/pulsar/pull/19432#discussion_r1191893474


##
docker/pulsar/pom.xml:
##
@@ -72,17 +71,27 @@
 
   
 
-  pulsar
+  ${docker.organization}/pulsar

Review Comment:
   This issue has been fixed by #20305, could you check that?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] nodece merged pull request #20305: [fix][build] Fix publish image script



nodece merged PR #20305:
URL: https://github.com/apache/pulsar/pull/20305


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] nodece commented on a diff in pull request #20303: [fix] [meta]Switch to the metadata store thread after zk operation



nodece commented on code in PR #20303:
URL: https://github.com/apache/pulsar/pull/20303#discussion_r1191892578


##
pulsar-metadata/src/test/java/org/apache/pulsar/metadata/MetadataStoreTest.java:
##
@@ -425,6 +428,75 @@ public void testDeleteUnusedDirectories(String provider, 
Supplier urlSup
 assertFalse(store.exists(prefix).join());
 }
 
+@DataProvider(name = "conditionOfSwitchThread")
+public Object[][] conditionOfSwitchThread(){
+return new Object[][]{
+{false, false},
+{false, true},
+{true, false},
+{true, true}
+};
+}
+
+@Test(dataProvider = "conditionOfSwitchThread")
+public void testThreadSwitchOfZkMetadataStore(boolean hasSynchronizer, 
boolean enabledBatch) throws Exception {
+final String prefix = newKey();
+final String metadataStoreName = 
UUID.randomUUID().toString().replaceAll("-", "");
+MetadataStoreConfig.MetadataStoreConfigBuilder builder =
+
MetadataStoreConfig.builder().metadataStoreName(metadataStoreName);
+builder.fsyncEnable(false);
+builder.batchingEnabled(enabledBatch);
+if (!hasSynchronizer) {
+builder.synchronizer(null);
+}
+MetadataStoreConfig config = builder.build();
+@Cleanup
+ZKMetadataStore store = (ZKMetadataStore) 
MetadataStoreFactory.create(zks.getConnectionString(), config);
+
+final Runnable verify = () -> {
+String currentThreadName = Thread.currentThread().getName();
+String errorMessage = String.format("Expect to switch to thread 
%s, but currently it is thread %s",
+metadataStoreName, currentThreadName);
+if 
(!Thread.currentThread().getName().startsWith(metadataStoreName)){
+throw new RuntimeException(errorMessage);
+}
+};
+
+// put with node which has parent(but the parent node is not exists).
+store.put(prefix + "/a1/b1/c1", "value".getBytes(), 
Optional.of(-1L)).thenApply((ignore) -> {
+verify.run();
+return null;
+}).join();
+// put.
+store.put(prefix + "/b1", "value".getBytes(), 
Optional.of(-1L)).thenApply((ignore) -> {
+verify.run();
+return null;
+}).join();
+// get.
+store.get(prefix + "/b1").thenApply((ignore) -> {
+verify.run();
+return null;
+}).join();
+// get the node which is not exists.
+store.get(prefix + "/non").thenApply((ignore) -> {
+verify.run();
+return null;
+}).join();
+// delete.
+store.delete(prefix + "/b1", Optional.empty()).thenApply((ignore) -> {
+verify.run();
+return null;
+}).join();
+// delete the node which is not exists.
+store.delete(prefix + "/non", Optional.empty()).thenApply((ignore) -> {
+verify.run();
+return null;
+}).exceptionally(ex -> {
+verify.run();
+return null;
+}).join();
+}

Review Comment:
   Could you add a nested request case? So like:
   
   ```
store.get(prefix + "/b1").thenApply((ignore) -> {
   verify.run();
   store.get(prefix + "/b1").join();
   verify.run();
   return null;
   }).join();
   ```
   
   BTW, please add timeout flag to avoid block testsuite.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[pulsar] branch master updated (525dd2f3b54 -> 94c7bf3e57f)

2023-05-11 Thread zixuan

This is an automated email from the ASF dual-hosted git repository.

zixuan pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git


from 525dd2f3b54 [fix][fn] Correct TLS cert config translation from broker 
to fn worker (#20297)
 add 94c7bf3e57f [fix][build] Fix publish image script (#20305)

No new revisions were added by this update.

Summary of changes:
 docker/publish.sh | 8 
 1 file changed, 4 insertions(+), 4 deletions(-)

[GitHub] [pulsar] nodece commented on a diff in pull request #20303: [fix] [meta]Switch to the metadata store thread after zk operation



nodece commented on code in PR #20303:
URL: https://github.com/apache/pulsar/pull/20303#discussion_r1191888657


##
pulsar-metadata/src/test/java/org/apache/pulsar/metadata/MetadataStoreTest.java:
##
@@ -425,6 +428,75 @@ public void testDeleteUnusedDirectories(String provider, 
Supplier urlSup
 assertFalse(store.exists(prefix).join());
 }
 
+@DataProvider(name = "conditionOfSwitchThread")
+public Object[][] conditionOfSwitchThread(){
+return new Object[][]{
+{false, false},
+{false, true},
+{true, false},
+{true, true}
+};
+}
+
+@Test(dataProvider = "conditionOfSwitchThread")
+public void testThreadSwitchOfZkMetadataStore(boolean hasSynchronizer, 
boolean enabledBatch) throws Exception {
+final String prefix = newKey();
+final String metadataStoreName = 
UUID.randomUUID().toString().replaceAll("-", "");
+MetadataStoreConfig.MetadataStoreConfigBuilder builder =
+
MetadataStoreConfig.builder().metadataStoreName(metadataStoreName);
+builder.fsyncEnable(false);
+builder.batchingEnabled(enabledBatch);
+if (!hasSynchronizer) {
+builder.synchronizer(null);
+}
+MetadataStoreConfig config = builder.build();
+@Cleanup
+ZKMetadataStore store = (ZKMetadataStore) 
MetadataStoreFactory.create(zks.getConnectionString(), config);
+
+final Runnable verify = () -> {
+String currentThreadName = Thread.currentThread().getName();
+String errorMessage = String.format("Expect to switch to thread 
%s, but currently it is thread %s",
+metadataStoreName, currentThreadName);
+if 
(!Thread.currentThread().getName().startsWith(metadataStoreName)){

Review Comment:
   ```suggestion
   assertTrue(Thread.currentThread().getName().startsWith(metadataStoreName), 
errorMessage);
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] codecov-commenter commented on pull request #20305: [fix][build] Fix publish image script



codecov-commenter commented on PR #20305:
URL: https://github.com/apache/pulsar/pull/20305#issuecomment-1545081489

   ## 
[Codecov](https://app.codecov.io/gh/apache/pulsar/pull/20305?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)
 Report
   > Merging 
[#20305](https://app.codecov.io/gh/apache/pulsar/pull/20305?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)
 (7208e11) into 
[master](https://app.codecov.io/gh/apache/pulsar/commit/b07f2a9b85a0f22b89fed5b51ee8625b00b7262e?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)
 (b07f2a9) will **increase** coverage by `42.72%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://app.codecov.io/gh/apache/pulsar/pull/20305/graphs/tree.svg?width=650=150=pr=acYqCpsK9J_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)](https://app.codecov.io/gh/apache/pulsar/pull/20305?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)
   
   ```diff
   @@  Coverage Diff  @@
   ## master   #20305   +/-   ##
   =
   + Coverage 30.23%   72.96%   +42.72% 
   - Complexity  29531971+31676 
   =
 Files  1689 1868  +179 
 Lines129020   138601 +9581 
 Branches  1407015240 +1170 
   =
   + Hits  39013   101125+62112 
   + Misses8414829449-54699 
   - Partials   5859 8027 +2168 
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | inttests | `24.08% <ø> (-0.09%)` | :arrow_down: |
   | systests | `24.98% <ø> (+0.01%)` | :arrow_up: |
   | unittests | `72.24% <ø> (?)` | |
   
   Flags with carried forward coverage won't be shown. [Click 
here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment)
 to find out more.
   
   [see 1490 files with indirect coverage 
changes](https://app.codecov.io/gh/apache/pulsar/pull/20305/indirect-changes?src=pr=tree-more_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] Samreay edited a comment on the discussion: [Python Client] How to subscribe to two topics - both with different schemas?



GitHub user Samreay edited a comment on the discussion: [Python Client] How to 
subscribe to two topics - both with different schemas?

What I've come up with doesn't seem optimal, but it works by hiding away 
multiple blocking synchronous code blocks into threads and then putting the 
messages received into an janus.Queue (asyncio.Queues are not thread safe, so 
we swapped to janus). This all works under FastAPI to support the existing REST 
endpoints we serve.

```python

class Consumer(Thread):
def __init__(self, queue: janus.Queue, topic: str, schema: 
pulsar.schema.AvroSchema):
super().__init__()
self.queue = queue
self.pulsar_client = pulsar.Client("pulsar://pulsar:6650")
self.topic = topic
self.schema = schema

def run(self):

reader = self.pulsar_client.create_reader(
self.topic, start_message_id=pulsar.MessageId.latest, 
schema=self.schema
)
while True:
msg = reader.read_next()
topic = msg.topic_name()
if "/" in topic:
topic = topic.split("/")[-1]
message = from_avro(msg.value())
   self.queue.sync_q.put_nowait(message)
```

Then when the FastAPI app starts you can kick off the threads, and also launch 
an `async` method to get from the `asyncio.Queue` passed into the Consumer 
threads.

```python
app = FastAPI()
app.queue = asyncio.Queue()

@app.on_event("startup")
async def startup_event():
consumer_preds = Consumer(app.queue, "predictions", Prediction.schema())
consumer_preds.start()
consumer_trades = Consumer(app.queue, "trades", Trade.schema())
consumer_trades.start()

asyncio.create_task(receive_message())
```

`Prediction.schema` and what not are just some wrappers we have to convert 
pydantic to avro schemas.

GitHub link: 
https://github.com/apache/pulsar/discussions/20263#discussioncomment-5845219


This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] Samreay added a comment to the discussion: [Python Client] How to subscribe to two topics - both with different schemas?



GitHub user Samreay added a comment to the discussion: [Python Client] How to 
subscribe to two topics - both with different schemas?

Oh yeah, if we just accept json and do the schema validation ourself, no 
issues. I was hoping to make use of Pulsars schema registry features and how it 
enforces things to guarantee a consumer with an incompatible version of the 
schema can't connect. Figured if the pulsar devs have put in the hard work to 
implement all of that, I should make use of it! (There's also no rush on our 
side, the nasty threading approach works, but its very nice.

GitHub link: 
https://github.com/apache/pulsar/discussions/20263#discussioncomment-5880316


This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] yaalsn added a comment to the discussion: [Python Client] How to subscribe to two topics - both with different schemas?



GitHub user yaalsn added a comment to the discussion: [Python Client] How to 
subscribe to two topics - both with different schemas?

Not JSON schema, I mean sending JSON string, just bytes. I think this can solve 
your problem quickly. 

GitHub link: 
https://github.com/apache/pulsar/discussions/20263#discussioncomment-5880284


This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org

[GitHub] [pulsar-site] Anonymitaet commented on pull request #565: Add CPP client table view doc.



Anonymitaet commented on PR #565:
URL: https://github.com/apache/pulsar-site/pull/565#issuecomment-1545063125

   > Thanks for your contribution!
   > 
   > 1. This PR ignores the 1st task in [[TableView-2] Implement all 
interfaces. pulsar-client-cpp#196 
(comment)](https://github.com/apache/pulsar-client-cpp/pull/196#issuecomment-1512418627)
 as we discussed before, any thoughts?
   > 2. Can you attach the related code PR in the doc PR description? So that 
reviewers will have more context, which can speed up the review process.
   
   Confirmed with @shibd,  the1st task can be skipped as compaction topics are 
enabled in TableView internally by default, and it is explained in 
https://pulsar.apache.org/docs/next/concepts-clients/#tableview  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] Samreay added a comment to the discussion: [Python Client] How to subscribe to two topics - both with different schemas?



GitHub user Samreay added a comment to the discussion: [Python Client] How to 
subscribe to two topics - both with different schemas?

Doesn't this have the same issue as Avro, which is if I want to enforce schema 
on the topics with a well defined producer and consumer, it looks like the 
synchronous single-schema Consumer in the API means we'd need to turn to 
multithreading or multiprocessing? Does the JSON schema allow us some way 
around this?

GitHub link: 
https://github.com/apache/pulsar/discussions/20263#discussioncomment-5880245


This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] yaalsn added a comment to the discussion: [Python Client] How to subscribe to two topics - both with different schemas?



GitHub user yaalsn added a comment to the discussion: [Python Client] How to 
subscribe to two topics - both with different schemas?

If pydantic, how about using `JSON` instead of `Avro`, you can convert the 
models to `JSON` easily 

https://docs.pydantic.dev/latest/usage/exporting_models/

GitHub link: 
https://github.com/apache/pulsar/discussions/20263#discussioncomment-5880223


This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] codecov-commenter commented on pull request #20286: [fix][test] Flaky-test: SimpleProducerConsumerTestStreamingDispatcherTest. rest



codecov-commenter commented on PR #20286:
URL: https://github.com/apache/pulsar/pull/20286#issuecomment-1545051242

   ## 
[Codecov](https://app.codecov.io/gh/apache/pulsar/pull/20286?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)
 Report
   > Merging 
[#20286](https://app.codecov.io/gh/apache/pulsar/pull/20286?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)
 (ea4f01c) into 
[master](https://app.codecov.io/gh/apache/pulsar/commit/00f17e88ad8fa9a4ead78d78a181c03fdee8e4df?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)
 (00f17e8) will **increase** coverage by `35.31%`.
   > The diff coverage is `100.00%`.
   
   [![Impacted file tree 
graph](https://app.codecov.io/gh/apache/pulsar/pull/20286/graphs/tree.svg?width=650=150=pr=acYqCpsK9J_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)](https://app.codecov.io/gh/apache/pulsar/pull/20286?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)
   
   ```diff
   @@  Coverage Diff  @@
   ## master   #20286   +/-   ##
   =
   + Coverage 37.61%   72.92%   +35.31% 
   - Complexity1258931958+19369 
   =
 Files  1691 1868  +177 
 Lines129028   138601 +9573 
 Branches  1406615240 +1174 
   =
   + Hits  48530   101080+52550 
   + Misses7418329484-44699 
   - Partials   6315 8037 +1722 
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | inttests | `24.13% <100.00%> (-0.05%)` | :arrow_down: |
   | systests | `24.96% <100.00%> (+0.18%)` | :arrow_up: |
   | unittests | `72.22% <100.00%> (+39.03%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click 
here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment)
 to find out more.
   
   | [Impacted 
Files](https://app.codecov.io/gh/apache/pulsar/pull/20286?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)
 | Coverage Δ | |
   |---|---|---|
   | 
[.../apache/bookkeeper/mledger/impl/MetaStoreImpl.java](https://app.codecov.io/gh/apache/pulsar/pull/20286?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-bWFuYWdlZC1sZWRnZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2Jvb2trZWVwZXIvbWxlZGdlci9pbXBsL01ldGFTdG9yZUltcGwuamF2YQ==)
 | `85.21% <100.00%> (+43.30%)` | :arrow_up: |
   
   ... and [1427 files with indirect coverage 
changes](https://app.codecov.io/gh/apache/pulsar/pull/20286/indirect-changes?src=pr=tree-more_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[pulsar-site] branch main updated: [improve][misc] add reminder to PR template (#572)

2023-05-11 Thread tison

This is an automated email from the ASF dual-hosted git repository.

tison pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/pulsar-site.git


The following commit(s) were added to refs/heads/main by this push:
 new b86dd218593 [improve][misc] add reminder to PR template (#572)
b86dd218593 is described below

commit b86dd21859350f533bc686070deb1bb2fcdf9653
Author: Anonymitaet <50226895+anonymit...@users.noreply.github.com>
AuthorDate: Fri May 12 11:08:15 2023 +0800

[improve][misc] add reminder to PR template (#572)
---
 .github/PULL_REQUEST_TEMPLATE.md | 16 +++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
index e66a1e4fdb3..61d9ac5f710 100644
--- a/.github/PULL_REQUEST_TEMPLATE.md
+++ b/.github/PULL_REQUEST_TEMPLATE.md
@@ -1,4 +1,18 @@
-### Documentation
+
+
+
+
+This PR adds doc for #xyz
+
+
+
+This PR fixes #xyz

[GitHub] [pulsar-site] tisonkun merged pull request #572: [improve][misc] add reminder to PR template



tisonkun merged PR #572:
URL: https://github.com/apache/pulsar-site/pull/572


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] hangc0276 opened a new pull request, #20312: Upgrade rocksDB version to 6.16.4 to keep sync with BookKeeper 4.14.7



hangc0276 opened a new pull request, #20312:
URL: https://github.com/apache/pulsar/pull/20312

   
   
   
   
   Fixes #xyz
   
   
   
   Master Issue: #xyz
   
   
   
   PIP: #xyz 
   
   
   
   ### Motivation
   
   Upgrade rocksDB version to 6.16.4 to keep sync with BookKeeper 4.14.7
   
   ### Modifications
   
   
   
   ### Verifying this change
   
   - [ ] Make sure that the change passes the CI checks.
   
   *(Please pick either of the following options)*
   
   This change is a trivial rework / code cleanup without any test coverage.
   
   *(or)*
   
   This change is already covered by existing tests, such as *(please describe 
tests)*.
   
   *(or)*
   
   This change added tests and can be verified as follows:
   
   *(example:)*
 - *Added integration tests for end-to-end deployment with large payloads 
(10MB)*
 - *Extended integration test for recovery after broker failure*
   
   ### Does this pull request potentially affect one of the following parts:
   
   
   
   *If the box was checked, please highlight the changes*
   
   - [ ] Dependencies (add or upgrade a dependency)
   - [ ] The public API
   - [ ] The schema
   - [ ] The default values of configurations
   - [ ] The threading model
   - [ ] The binary protocol
   - [ ] The REST endpoints
   - [ ] The admin CLI options
   - [ ] The metrics
   - [ ] Anything that affects deployment
   
   ### Documentation
   
   
   
   - [ ] `doc` 
   - [ ] `doc-required` 
   - [x] `doc-not-needed` 
   - [ ] `doc-complete` 
   
   ### Matching PR in forked repository
   
   PR in forked repository: 
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-site] Anonymitaet opened a new pull request, #572: [improve][misc] add reminder to PR template



Anonymitaet opened a new pull request, #572:
URL: https://github.com/apache/pulsar-site/pull/572

   ### Documentation
   
   
   
   - [ ] `doc` 
   - [ ] `doc-required` 
   - [x] `doc-not-needed` 
   - [ ] `doc-complete` 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-client-node] shibd merged pull request #324: Upgrade unit test depends on the pulsar version to the latest(currently 3.0.0)



shibd merged PR #324:
URL: https://github.com/apache/pulsar-client-node/pull/324


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[pulsar-client-node] branch master updated: Upgrade unit test depends on the pulsar version to the latest(currently 3.0.0) (#324)

2023-05-11 Thread baodi

This is an automated email from the ASF dual-hosted git repository.

baodi pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar-client-node.git


The following commit(s) were added to refs/heads/master by this push:
 new 6d89db4  Upgrade unit test depends on the pulsar version to the 
latest(currently 3.0.0) (#324)
6d89db4 is described below

commit 6d89db4b66d05caf52624c125e8da61b094449dd
Author: Baodi Shi 
AuthorDate: Fri May 12 10:26:53 2023 +0800

Upgrade unit test depends on the pulsar version to the latest(currently 
3.0.0) (#324)

* Upgrade unit test depend pulsar version to latest(3.0.0).

* Fix unit test.
---
 build-support/pulsar-test-service-start.sh | 2 +-
 tests/consumer.test.js | 4 ++--
 tests/producer.test.js | 4 ++--
 3 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/build-support/pulsar-test-service-start.sh 
b/build-support/pulsar-test-service-start.sh
index ded136c..c1b2d43 100755
--- a/build-support/pulsar-test-service-start.sh
+++ b/build-support/pulsar-test-service-start.sh
@@ -25,7 +25,7 @@ cd $SRC_DIR
 
 build-support/pulsar-test-service-stop.sh
 
-CONTAINER_ID=$(docker run -i -p 8080:8080 -p 6650:6650 -p 8443:8443 -p 
6651:6651 --rm --detach apachepulsar/pulsar:2.10.2 sleep 3600)
+CONTAINER_ID=$(docker run -i -p 8080:8080 -p 6650:6650 -p 8443:8443 -p 
6651:6651 --rm --detach apachepulsar/pulsar:latest sleep 3600)
 
 echo $CONTAINER_ID >.tests-container-id.txt
 
diff --git a/tests/consumer.test.js b/tests/consumer.test.js
index b13fac2..e41c638 100644
--- a/tests/consumer.test.js
+++ b/tests/consumer.test.js
@@ -102,7 +102,7 @@ const Pulsar = require('../index.js');
   subscription: 'sub1',
   ackTimeoutMs: 1,
   nAckRedeliverTimeoutMs: 6,
-})).rejects.toThrow('Failed to create consumer: BrokerMetadataError');
+})).rejects.toThrow('Failed to create consumer: TopicNotFound');
   });
 
   test('Not Exist Namespace', async () => {
@@ -111,7 +111,7 @@ const Pulsar = require('../index.js');
   subscription: 'sub1',
   ackTimeoutMs: 1,
   nAckRedeliverTimeoutMs: 6,
-})).rejects.toThrow('Failed to create consumer: BrokerMetadataError');
+})).rejects.toThrow('Failed to create consumer: TopicNotFound');
   });
 
   test('Not Positive NAckRedeliverTimeout', async () => {
diff --git a/tests/producer.test.js b/tests/producer.test.js
index 40712a5..fa7710b 100644
--- a/tests/producer.test.js
+++ b/tests/producer.test.js
@@ -55,7 +55,7 @@ const Pulsar = require('../index.js');
   topic: 'persistent://no-tenant/namespace/topic',
   sendTimeoutMs: 3,
   batchingEnabled: true,
-})).rejects.toThrow('Failed to create producer: BrokerMetadataError');
+})).rejects.toThrow('Failed to create producer: TopicNotFound');
   });
 
   test('Not Exist Namespace', async () => {
@@ -63,7 +63,7 @@ const Pulsar = require('../index.js');
   topic: 'persistent://public/no-namespace/topic',
   sendTimeoutMs: 3,
   batchingEnabled: true,
-})).rejects.toThrow('Failed to create producer: BrokerMetadataError');
+})).rejects.toThrow('Failed to create producer: TopicNotFound');
   });
 
   test('Automatic Producer Name', async () => {

[GitHub] [pulsar-site] shibd commented on a diff in pull request #571: Add client memory limits docs.



shibd commented on code in PR #571:
URL: https://github.com/apache/pulsar-site/pull/571#discussion_r1191842257


##
docs/client-libraries-clients.md:
##
@@ -51,4 +51,36 @@ The following example creates a Python client using multiple 
advertised listener
 
   
 
+
+
+## Set memory limits
+
+You can uses memory limits param to control client overall memory usage,
+the producers and consumers under this client will compete for the memory 
assigned. See [PIP 74: Pulsar client memory 
limits](https://github.com/apache/pulsar/wiki/PIP-74%3A-Pulsar-client-memory-limits)

Review Comment:
   I think nothing.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] lifepuzzlefun commented on issue #20274: Producer with JSON schema type with RecordSchemaBuilder doesn't throw an error if any extra parameter is passed in the message payload.



lifepuzzlefun commented on issue #20274:
URL: https://github.com/apache/pulsar/issues/20274#issuecomment-1545007083

   @vpeddada hi, can you provide the policy on the namespace like 
`SchemaCompatibilityStrategy`
   And does you demo upload an json schema to create a producer, send a message 
by runtime create avro record, am i understanding right ? just want to know the 
demo background and try to reproduce your finding : - )


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-site] shibd commented on pull request #571: Add client memory limits docs.



shibd commented on PR #571:
URL: https://github.com/apache/pulsar-site/pull/571#issuecomment-1545006140

   > Can you attach the related code PR in the doc PR description (as commented 
in https://github.com/apache/pulsar-site/pull/565#issuecomment-1541064214)? So 
that reviewers will have more context, which can speed up the review process.
   
   oh, Sorry. Maybe we can change template?
   
   ### Preview
   
   
   
   ### Relation PR
   
   
   ### Documentation
   
   
   
   - [ ] `doc` 
   - [ ] `doc-required` 
   - [ ] `doc-not-needed` 
   - [ ] `doc-complete` 
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] poorbarcode commented on a diff in pull request #20276: [fix][ml] Fix ledger left in OPEN state when enable `inactiveLedgerRollOverTimeMs`



poorbarcode commented on code in PR #20276:
URL: https://github.com/apache/pulsar/pull/20276#discussion_r1191839289


##
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java:
##
@@ -4353,7 +4353,22 @@ public void checkInactiveLedgerAndRollOver() {
 long currentTimeMs = System.currentTimeMillis();
 if (inactiveLedgerRollOverTimeMs > 0 && currentTimeMs > 
(lastAddEntryTimeMs + inactiveLedgerRollOverTimeMs)) {
 log.info("[{}] Closing inactive ledger, last-add entry {}", name, 
lastAddEntryTimeMs);
-ledgerClosed(currentLedger);
+if (STATE_UPDATER.compareAndSet(this, State.LedgerOpened, 
State.ClosingLedger)) {
+LedgerHandle currentLedger = this.currentLedger;
+currentLedger.asyncClose((rc, lh, o) -> {
+checkArgument(currentLedger.getId() == lh.getId(), 
"ledgerId %s doesn't match with "
++ "acked ledgerId %s", currentLedger.getId(), 
lh.getId());
+
+if (rc == Code.OK) {
+log.debug("Successfully closed ledger {}", lh.getId());
+} else {
+log.warn("Error when closing ledger {}. Status={}", 
lh.getId(), BKException.getMessage(rc));
+}
+
+ledgerClosed(lh);
+// we do not create ledger here, since topic is inactive 
for a long time.
+}, System.nanoTime());

Review Comment:
   The context `System.nanoTime()` is not used. Can we set this param to `null`?



##
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java:
##
@@ -4353,7 +4353,22 @@ public void checkInactiveLedgerAndRollOver() {
 long currentTimeMs = System.currentTimeMillis();
 if (inactiveLedgerRollOverTimeMs > 0 && currentTimeMs > 
(lastAddEntryTimeMs + inactiveLedgerRollOverTimeMs)) {
 log.info("[{}] Closing inactive ledger, last-add entry {}", name, 
lastAddEntryTimeMs);
-ledgerClosed(currentLedger);
+if (STATE_UPDATER.compareAndSet(this, State.LedgerOpened, 
State.ClosingLedger)) {
+LedgerHandle currentLedger = this.currentLedger;
+currentLedger.asyncClose((rc, lh, o) -> {
+checkArgument(currentLedger.getId() == lh.getId(), 
"ledgerId %s doesn't match with "
++ "acked ledgerId %s", currentLedger.getId(), 
lh.getId());
+
+if (rc == Code.OK) {
+log.debug("Successfully closed ledger {}", lh.getId());
+} else {
+log.warn("Error when closing ledger {}. Status={}", 
lh.getId(), BKException.getMessage(rc));

Review Comment:
   The log should include the name of this managed ledger object, such as 
   
   ```java
   log.warn("[{}] Error when closing ledger...", name,...);
   ```
   
   BTW, can you also change `rollCurrentLedgerIfFull`'s log in this PR? And it 
is better to make these two logs are not the same, so it is easier to 
troubleshoot problems based on logs.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] lifepuzzlefun commented on pull request #19561: [improve][meta] Support set metadata size threshold for compression.



lifepuzzlefun commented on PR #19561:
URL: https://github.com/apache/pulsar/pull/19561#issuecomment-1545000232

   https://github.com/apache/pulsar/issues/20307 pip-270 created. hopes for 
your comments and vote, thank you~ 
   
   the discuss mail list address is write at the bottom, feel free to join the 
discuss.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] lifepuzzlefun commented on pull request #19561: [improve][meta] Support set metadata size threshold for compression.



lifepuzzlefun commented on PR #19561:
URL: https://github.com/apache/pulsar/pull/19561#issuecomment-1544998480

   @congbobo184 hi, sorry for the late reply, i have just create a pip to 
discuss.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] github-actions[bot] commented on pull request #17485: PIP-211 [feat][tiered-storage] Introduce offload throttling, Part 1



github-actions[bot] commented on PR #17485:
URL: https://github.com/apache/pulsar/pull/17485#issuecomment-1544991655

   The pr had no activity for 30 days, mark with Stale label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] github-actions[bot] commented on pull request #17524: [fix][broker]One topic can be close multiple times concurrently



github-actions[bot] commented on PR #17524:
URL: https://github.com/apache/pulsar/pull/17524#issuecomment-1544991622

   The pr had no activity for 30 days, mark with Stale label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] lifepuzzlefun commented on a diff in pull request #19993: [improve][broker] Save createIfMissing in TopicLoadingContext



lifepuzzlefun commented on code in PR #19993:
URL: https://github.com/apache/pulsar/pull/19993#discussion_r1191833952


##
pulsar-broker/src/test/java/org/apache/pulsar/broker/service/BrokerServiceTest.java:
##
@@ -1159,6 +1160,75 @@ public void testTopicLoadingOnDisableNamespaceBundle() 
throws Exception {
 }
 }
 
+@Test
+public void testConcurrentLoadTopicExceedLimitShouldNotBeAutoCreated() 
throws Exception {
+boolean needDeleteTopic = false;
+final String namespace = "prop/concurrentLoad";
+try {
+// set up broker disable auto create and set concurrent load to 1 
qps.
+cleanup();
+conf.setMaxConcurrentTopicLoadRequest(1);
+conf.setAllowAutoTopicCreation(false);
+setup();
+
+try {
+admin.namespaces().createNamespace(namespace);
+} catch (PulsarAdminException.ConflictException e) {
+// Ok.. (if test fails intermittently and namespace is already 
created)
+}
+
+// create 3 topic
+
+String topicName = "persistent://" + namespace + "/my-topic";
+
+for (int i = 0; i < 3; i++) {
+admin.topics().createNonPartitionedTopic(topicName + "_" + i);
+}
+
+needDeleteTopic = true;
+
+
+// try to load 10 topic
+

Review Comment:
   yeah, the empty line removed, please take a look again : -)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] github-actions[bot] commented on pull request #19679: [improve][fn] PIP-237: Expose PulsarAdmin in SinkContext and SourceContext



github-actions[bot] commented on PR #19679:
URL: https://github.com/apache/pulsar/pull/19679#issuecomment-1544990621

   The pr had no activity for 30 days, mark with Stale label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] github-actions[bot] commented on pull request #19571: [improve][broker] Reduce object creation of EntryBatchIndexesAcks



github-actions[bot] commented on PR #19571:
URL: https://github.com/apache/pulsar/pull/19571#issuecomment-1544990699

   The pr had no activity for 30 days, mark with Stale label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] github-actions[bot] commented on pull request #20065: [improve] [txn] respond to client when txn update status to committing or aborting



github-actions[bot] commented on PR #20065:
URL: https://github.com/apache/pulsar/pull/20065#issuecomment-1544990332

   The pr had no activity for 30 days, mark with Stale label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] github-actions[bot] commented on pull request #18937: [fix][admin] Fix the conversion from the response to string in client error handling



github-actions[bot] commented on PR #18937:
URL: https://github.com/apache/pulsar/pull/18937#issuecomment-1544991156

   The pr had no activity for 30 days, mark with Stale label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] github-actions[bot] commented on pull request #19815: [fix] [txn] fix consumer can receive aborted txn message when readType is replay



github-actions[bot] commented on PR #19815:
URL: https://github.com/apache/pulsar/pull/19815#issuecomment-1544990498

   The pr had no activity for 30 days, mark with Stale label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] github-actions[bot] commented on pull request #19775: [fix] [Perf] PerformanceProducer do not produce expected number of messages.



github-actions[bot] commented on PR #19775:
URL: https://github.com/apache/pulsar/pull/19775#issuecomment-1544990565

   The pr had no activity for 30 days, mark with Stale label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] github-actions[bot] commented on pull request #19867: [improve][client] add additional logging and stacktrace client encryption failure



github-actions[bot] commented on PR #19867:
URL: https://github.com/apache/pulsar/pull/19867#issuecomment-1544990456

   The pr had no activity for 30 days, mark with Stale label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] Technoboy- commented on a diff in pull request #19993: [improve][broker] Save createIfMissing in TopicLoadingContext



Technoboy- commented on code in PR #19993:
URL: https://github.com/apache/pulsar/pull/19993#discussion_r1191826984


##
pulsar-broker/src/test/java/org/apache/pulsar/broker/service/BrokerServiceTest.java:
##
@@ -1159,6 +1160,75 @@ public void testTopicLoadingOnDisableNamespaceBundle() 
throws Exception {
 }
 }
 
+@Test
+public void testConcurrentLoadTopicExceedLimitShouldNotBeAutoCreated() 
throws Exception {
+boolean needDeleteTopic = false;
+final String namespace = "prop/concurrentLoad";
+try {
+// set up broker disable auto create and set concurrent load to 1 
qps.
+cleanup();
+conf.setMaxConcurrentTopicLoadRequest(1);
+conf.setAllowAutoTopicCreation(false);
+setup();
+
+try {
+admin.namespaces().createNamespace(namespace);
+} catch (PulsarAdminException.ConflictException e) {
+// Ok.. (if test fails intermittently and namespace is already 
created)
+}
+
+// create 3 topic
+
+String topicName = "persistent://" + namespace + "/my-topic";
+
+for (int i = 0; i < 3; i++) {
+admin.topics().createNonPartitionedTopic(topicName + "_" + i);
+}
+
+needDeleteTopic = true;
+
+
+// try to load 10 topic
+

Review Comment:
   Could you remove the empty lines ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] nodece commented on pull request #20305: [fix][build] Fix publish image script



nodece commented on PR #20305:
URL: https://github.com/apache/pulsar/pull/20305#issuecomment-1544979220

   /pulsarbot rerun-failure-checks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] HattoriHenzo commented on issue #20274: Producer with JSON schema type with RecordSchemaBuilder doesn't throw an error if any extra parameter is passed in the message payload.



HattoriHenzo commented on issue #20274:
URL: https://github.com/apache/pulsar/issues/20274#issuecomment-1544964776

   @vpeddada Your point is good, but I am not sure this is not the role of the 
Broker to handle message validation. It can be a good feature to implement.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-site] Anonymitaet commented on pull request #571: Add client memory limits docs.



Anonymitaet commented on PR #571:
URL: https://github.com/apache/pulsar-site/pull/571#issuecomment-1544944609

   This PR adds doc for 
https://github.com/apache/pulsar-client-go/pull/991#issuecomment-1512391159


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-site] Anonymitaet commented on a diff in pull request #571: Add client memory limits docs.



Anonymitaet commented on code in PR #571:
URL: https://github.com/apache/pulsar-site/pull/571#discussion_r1191788445


##
docs/client-libraries-clients.md:
##
@@ -51,4 +51,36 @@ The following example creates a Python client using multiple 
advertised listener
 
   
 
+
+
+## Set memory limits
+
+You can uses memory limits param to control client overall memory usage,

Review Comment:
   ```suggestion
   You can use memory limits parameters to control the total client memory 
usage,
   ```



##
docs/client-libraries-clients.md:
##
@@ -51,4 +51,36 @@ The following example creates a Python client using multiple 
advertised listener
 
   
 
+
+
+## Set memory limits
+
+You can uses memory limits param to control client overall memory usage,
+the producers and consumers under this client will compete for the memory 
assigned. See [PIP 74: Pulsar client memory 
limits](https://github.com/apache/pulsar/wiki/PIP-74%3A-Pulsar-client-memory-limits)

Review Comment:
   ```suggestion
   the producers and consumers under this client will compete for the memory 
assigned. For implementation details, see [PIP 74: Pulsar client memory 
limits](https://github.com/apache/pulsar/wiki/PIP-74%3A-Pulsar-client-memory-limits).
   ```



##
docs/client-libraries-clients.md:
##
@@ -51,4 +51,36 @@ The following example creates a Python client using multiple 
advertised listener
 
   
 
+
+
+## Set memory limits
+
+You can uses memory limits param to control client overall memory usage,
+the producers and consumers under this client will compete for the memory 
assigned. See [PIP 74: Pulsar client memory 
limits](https://github.com/apache/pulsar/wiki/PIP-74%3A-Pulsar-client-memory-limits)

Review Comment:
   Do we have any considerations/limitations that need to remind users?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] JooHyukKim opened a new pull request, #20310: [fix][broker] Fix class name typo `PrecisPublishLimiter` to "Precise"



JooHyukKim opened a new pull request, #20310:
URL: https://github.com/apache/pulsar/pull/20310

   ### Motivation
   
   To fix a typo on a class name. Class name is not a good place for a typo.
   
   ### Modifications
   
   - Changed class name from `PrecisPublishLimiter` to `PrecisePublishLimiter`, 
ending with "e"
   - Renamed relevent local declarations like wise, from `precisPublishLimiter` 
to `precisePublishLimiter`
   
   ### Verifying this change
   
   This change is a trivial rework / code cleanup without any test coverage.
   
   ### Does this pull request potentially affect one of the following parts:
   
   
   
   *If the box was checked, please highlight the changes*
   
   - [ ] Dependencies (add or upgrade a dependency)
   - [ ] The public API
   - [ ] The schema
   - [ ] The default values of configurations
   - [ ] The threading model
   - [ ] The binary protocol
   - [ ] The REST endpoints
   - [ ] The admin CLI options
   - [ ] The metrics
   - [ ] Anything that affects deployment
   
   ### Documentation
   
   
   
   - [ ] `doc` 
   - [ ] `doc-required` 
   - [x] `doc-not-needed` 
   - [ ] `doc-complete` 
   
   ### Matching PR in forked repository
   
   PR in forked repository: https://github.com/JooHyukKim/pulsar/pull/3


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] michaeljmarshall opened a new pull request, #20309: WIP: PIP 97 Async HTTP Auth Filter



michaeljmarshall opened a new pull request, #20309:
URL: https://github.com/apache/pulsar/pull/20309

   This is a draft. I haven't been able to get tests passing yet.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] github-actions[bot] commented on pull request #20309: WIP: PIP 97 Async HTTP Auth Filter



github-actions[bot] commented on PR #20309:
URL: https://github.com/apache/pulsar/pull/20309#issuecomment-1544798151

   @michaeljmarshall Please add the following content to your PR description 
and select a checkbox:
   ```
   - [ ] `doc` 
   - [ ] `doc-required` 
   - [ ] `doc-not-needed` 
   - [ ] `doc-complete` 
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[pulsar] branch master updated (4b24c9e85ad -> 525dd2f3b54)

2023-05-11 Thread mmarshall

This is an automated email from the ASF dual-hosted git repository.

mmarshall pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git


from 4b24c9e85ad [fix][broker] Fix NPE cause by topic publish rate limiter. 
(#20302)
 add 525dd2f3b54 [fix][fn] Correct TLS cert config translation from broker 
to fn worker (#20297)

No new revisions were added by this update.

Summary of changes:
 .../org/apache/pulsar/broker/PulsarService.java|  3 ++-
 .../functions/worker/PulsarFunctionTlsTest.java| 23 +-
 2 files changed, 20 insertions(+), 6 deletions(-)

[GitHub] [pulsar] michaeljmarshall merged pull request #20297: [fix][fn] Correct TLS cert config translation from broker to fn worker



michaeljmarshall merged PR #20297:
URL: https://github.com/apache/pulsar/pull/20297


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-manager] bpereto commented on pull request #500: Support different environments to use different tokens.



bpereto commented on PR #500:
URL: https://github.com/apache/pulsar-manager/pull/500#issuecomment-1544736810

   works perfectly.
   @tisonkun can we merge this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] HattoriHenzo commented on issue #12207: How to turn on debug mode in pulsar cluster



HattoriHenzo commented on issue #12207:
URL: https://github.com/apache/pulsar/issues/12207#issuecomment-1544556936

   @gaozhangmin Hi, I think the question is about how to use break point into 
your IDE to debug your Pulsar Docker. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] michaeljmarshall commented on pull request #20297: [fix][fn] Correct TLS cert config translation from broker to fn worker



michaeljmarshall commented on PR #20297:
URL: https://github.com/apache/pulsar/pull/20297#issuecomment-1544418991

   /pulsarbot rerun-failure-checks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] haibingtown commented on issue #11371: [Pulsar Client Tool] Add support for creating a replicated subscription in pulsar-client consume



haibingtown commented on issue #11371:
URL: https://github.com/apache/pulsar/issues/11371#issuecomment-1544329855

   Maybe I can try to see if I can solve this, can you assign it to me?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[pulsar] branch branch-2.11 updated: [fix][broker] Fix NPE cause by topic publish rate limiter. (#20302)

2023-05-11 Thread technoboy

This is an automated email from the ASF dual-hosted git repository.

technoboy pushed a commit to branch branch-2.11
in repository https://gitbox.apache.org/repos/asf/pulsar.git


The following commit(s) were added to refs/heads/branch-2.11 by this push:
 new df96ff40e00 [fix][broker] Fix NPE cause by topic publish rate limiter. 
(#20302)
df96ff40e00 is described below

commit df96ff40e0011b114f2159e25007059fc78f6e70
Author: Shen Liu 
AuthorDate: Thu May 11 22:28:40 2023 +0800

[fix][broker] Fix NPE cause by topic publish rate limiter. (#20302)

Co-authored-by: druidliu 
---
 .../src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java   | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java
index c1fcee4a059..8a76f615189 100644
--- 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java
@@ -110,7 +110,7 @@ public abstract class AbstractTopic implements Topic, 
TopicPolicyListener

[GitHub] [pulsar] michaeljmarshall opened a new issue, #20308: Flaky-test: LeaderElectionImplTest.validateDeadLock



michaeljmarshall opened a new issue, #20308:
URL: https://github.com/apache/pulsar/issues/20308

   ### Search before asking
   
   - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) 
and found nothing similar.
   
   
   ### Example failure
   
   
https://github.com/apache/pulsar/actions/runs/4949638662/jobs/8852414583?pr=20297#step:10:1573
   
   ### Exception stacktrace
   
   
   ```
 Error:  Tests run: 9, Failures: 1, Errors: 0, Skipped: 5, Time elapsed: 
19.697 s <<< FAILURE! - in 
org.apache.pulsar.metadata.coordination.impl.LeaderElectionImplTest
 Error:  
org.apache.pulsar.metadata.coordination.impl.LeaderElectionImplTest.validateDeadLock[Etcd,
 
org.apache.pulsar.metadata.BaseMetadataStoreTest$$Lambda$379/0x0008013a5888@6dc9da2d](10)
  Time elapsed: 0.007 s  <<< FAILURE!
 java.lang.IllegalStateException: Mapped port can only be obtained after 
the container is started
at 
org.testcontainers.shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:174)
at 
org.testcontainers.containers.ContainerState.getMappedPort(ContainerState.java:149)
at 
io.etcd.jetcd.launcher.EtcdContainer.clientEndpoint(EtcdContainer.java:201)
at 
java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at 
java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
at 
java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at 
java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at 
java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
at 
java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at 
java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
at 
io.etcd.jetcd.launcher.EtcdClusterFactory$2.getClientEndpoints(EtcdClusterFactory.java:112)
at 
org.apache.pulsar.metadata.BaseMetadataStoreTest.getEtcdClusterConnectString(BaseMetadataStoreTest.java:90)
at 
org.apache.pulsar.metadata.BaseMetadataStoreTest.lambda$implementations$3(BaseMetadataStoreTest.java:81)
at 
org.apache.pulsar.metadata.coordination.impl.LeaderElectionImplTest.validateDeadLock(LeaderElectionImplTest.java:41)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at 
org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)
at 
org.testng.internal.invokers.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:47)
at 
org.testng.internal.invokers.InvokeMethodRunnable.call(InvokeMethodRunnable.java:76)
at 
org.testng.internal.invokers.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
   ```
   
   
   ### Are you willing to submit a PR?
   
   - [ ] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] lifepuzzlefun opened a new issue, #20307: PIP-270 Add config to set metadata size threshold for compression.



lifepuzzlefun opened a new issue, #20307:
URL: https://github.com/apache/pulsar/issues/20307

   # Background knowledge
   
   current we support to set compression on ManagedLedger and ManagedCursor 
metadata
   by `managedLedgerInfoCompressionType` and `managedCursorInfoCompressionType`
   
   # Motivation
   
   sometimes the metadata is too small to compress. so a size based config is 
needed.
   
   # Goals
   
   ## In Scope
   add config to set size threshold for compression on metadata.
   
   `managedLedgerInfoCompressionSizeThreshold`
   `managedCursorInfoCompressionSizeThreshold`
   
   default value is 0.
   so this won't change the behaviour if set `managedLedgerInfoCompressionType`
   
   if the setting is set, only compress the metadata if the persistent size if 
above the setting.
   
   
   # High Level Design
   add config in `ServerConfiguration`
   when init `ManagedLedgerConfig` the setting is passed.
   
   when persistent metadata the config will affect the compression logic.
   
   ### Configuration
   new configuration: 
   
   `managedLedgerInfoCompressionSizeThreshold`
   `managedCursorInfoCompressionSizeThreshold`
   
   # Backward & Forward Compatability
   
   ## Revert
   
   the compression logic is transparent. if remove the config nothing changed.
   
   
   
   # Links
   
   pr that introduce metadata compression: 
   https://github.com/apache/pulsar/pull/11490
   https://github.com/apache/pulsar/pull/14542
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] michaeljmarshall commented on a diff in pull request #20297: [fix][fn] Correct TLS cert config translation from broker to fn worker



michaeljmarshall commented on code in PR #20297:
URL: https://github.com/apache/pulsar/pull/20297#discussion_r1191356879


##
pulsar-broker/src/test/java/org/apache/pulsar/functions/worker/PulsarFunctionTlsTest.java:
##
@@ -233,6 +240,11 @@ public void testFunctionsCreation() throws Exception {
 functionConfig, jarFilePathUrl
 );
 
+// Function creation is not strongly consistent, so this test can 
fail with a get that is too eager and
+// does not have retries.
+final PulsarAdmin admin = pulsarAdmins[i];
+Awaitility.await().untilAsserted(() -> 
admin.functions().getFunction(testTenant, "my-ns", functionName));

Review Comment:
   I didn't realize it worked that way. The test started passing after I added 
this block, so I incorrectly assumed it was right. Thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] lifepuzzlefun commented on pull request #19993: [improve][broker] Save createIfMissing in TopicLoadingContext



lifepuzzlefun commented on PR #19993:
URL: https://github.com/apache/pulsar/pull/19993#issuecomment-1544204134

   unit test added : - )


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-site] shibd opened a new pull request, #571: Add client memory limits docs.



shibd opened a new pull request, #571:
URL: https://github.com/apache/pulsar-site/pull/571

   ### Preview
   
   https://github.com/apache/pulsar-site/assets/33416836/d55b2ff1-53a8-499e-bce0-394e3f4f6b36;>
   
   https://github.com/apache/pulsar-site/assets/33416836/e71a9eff-886e-4716-b44f-ee5606bf4456;>
   
   
   ### Documentation
   
   
   
   - [x] `doc` 
   - [ ] `doc-required` 
   - [ ] `doc-not-needed` 
   - [ ] `doc-complete` 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[pulsar] branch branch-2.10 updated (d8bf38ebc3e -> f3edfc578fa)

2023-05-11 Thread yubiao

This is an automated email from the ASF dual-hosted git repository.

yubiao pushed a change to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/pulsar.git


from d8bf38ebc3e [fix][monitor] topic with double quote breaks the 
prometheus format (#20230)
 new 14dd76f2e3d Revert "[improve] [broker] Skip split boundle if only one 
broker (#20190)"
 new f3edfc578fa [improve] [broker] Skip split boundle if only one broker 
(#20190)

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../pulsar/broker/stats/PrometheusMetricsTest.java | 52 +-
 1 file changed, 50 insertions(+), 2 deletions(-)

[pulsar] 01/02: Revert "[improve] [broker] Skip split boundle if only one broker (#20190)"

2023-05-11 Thread yubiao

This is an automated email from the ASF dual-hosted git repository.

yubiao pushed a commit to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/pulsar.git

commit 14dd76f2e3d25b0a7e515a5557593243acc9db35
Author: fengyubiao 
AuthorDate: Thu May 11 22:17:06 2023 +0800

Revert "[improve] [broker] Skip split boundle if only one broker (#20190)"

This reverts commit 7da787378089e0142a5235a15d0915925c177edd.
---
 .../loadbalance/impl/ModularLoadManagerImpl.java   |  2 +-
 .../pulsar/broker/stats/PrometheusMetricsTest.java | 10 +---
 .../pulsar/client/api/BrokerServiceLookupTest.java | 58 --
 3 files changed, 2 insertions(+), 68 deletions(-)

diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/ModularLoadManagerImpl.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/ModularLoadManagerImpl.java
index 293ff2760f0..d81f6949f43 100644
--- 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/ModularLoadManagerImpl.java
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/ModularLoadManagerImpl.java
@@ -700,7 +700,7 @@ public class ModularLoadManagerImpl implements 
ModularLoadManager {
 public void checkNamespaceBundleSplit() {
 
 if (!conf.isLoadBalancerAutoBundleSplitEnabled() || 
pulsar.getLeaderElectionService() == null
-|| !pulsar.getLeaderElectionService().isLeader() || 
loadData.getBrokerData().size() <= 1) {
+|| !pulsar.getLeaderElectionService().isLeader()) {
 return;
 }
 final boolean unloadSplitBundles = 
pulsar.getConfiguration().isLoadBalancerAutoUnloadSplitBundlesEnabled();
diff --git 
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/PrometheusMetricsTest.java
 
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/PrometheusMetricsTest.java
index 1f21706f7dc..473adc26daa 100644
--- 
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/PrometheusMetricsTest.java
+++ 
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/PrometheusMetricsTest.java
@@ -36,7 +36,6 @@ import java.nio.charset.StandardCharsets;
 import java.text.NumberFormat;
 import java.util.Arrays;
 import java.util.Collection;
-import java.util.Collections;
 import java.util.Date;
 import java.util.HashMap;
 import java.util.List;
@@ -75,7 +74,6 @@ import org.apache.pulsar.client.api.PulsarClient;
 import org.apache.pulsar.client.api.PulsarClientException;
 import org.apache.pulsar.client.api.SubscriptionType;
 import org.apache.pulsar.compaction.Compactor;
-import org.apache.zookeeper.CreateMode;
 import org.awaitility.Awaitility;
 import org.testng.Assert;
 import org.testng.annotations.AfterMethod;
@@ -550,10 +548,6 @@ public class PrometheusMetricsTest extends BrokerTestBase {
 c1.acknowledge(c1.receive());
 }
 
-// Mock another broker to make split task work.
-String mockedBroker = "/loadbalance/brokers/127.0.0.1:0";
-mockZooKeeper.create(mockedBroker, new byte[]{0}, 
Collections.emptyList(), CreateMode.EPHEMERAL);
-
 pulsar.getBrokerService().updateRates();
 Awaitility.await().untilAsserted(() -> 
assertTrue(pulsar.getBrokerService().getBundleStats().size() > 0));
 ModularLoadManagerWrapper loadManager = 
(ModularLoadManagerWrapper)pulsar.getLoadManager().get();
@@ -577,9 +571,7 @@ public class PrometheusMetricsTest extends BrokerTestBase {
 assertTrue(metrics.containsKey("pulsar_lb_bandwidth_in_usage"));
 assertTrue(metrics.containsKey("pulsar_lb_bandwidth_out_usage"));
 
-assertTrue(metrics.containsKey("pulsar_lb_bundles_split_total"));
-
-mockZooKeeper.delete(mockedBroker, 0);
+assertTrue(metrics.containsKey("pulsar_lb_bundles_split_count"));
 }
 
 @Test
diff --git 
a/pulsar-broker/src/test/java/org/apache/pulsar/client/api/BrokerServiceLookupTest.java
 
b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/BrokerServiceLookupTest.java
index 396fda82b59..e3bb6a92dc0 100644
--- 
a/pulsar-broker/src/test/java/org/apache/pulsar/client/api/BrokerServiceLookupTest.java
+++ 
b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/BrokerServiceLookupTest.java
@@ -66,7 +66,6 @@ import javax.net.ssl.KeyManagerFactory;
 import javax.net.ssl.SSLContext;
 import javax.net.ssl.TrustManager;
 import lombok.Cleanup;
-import org.apache.pulsar.broker.BrokerTestUtil;
 import org.apache.pulsar.broker.PulsarService;
 import org.apache.pulsar.broker.ServiceConfiguration;
 import org.apache.pulsar.broker.authentication.AuthenticationDataSource;
@@ -79,7 +78,6 @@ import 
org.apache.pulsar.broker.loadbalance.impl.SimpleResourceUnit;
 import org.apache.pulsar.broker.namespace.NamespaceService;
 import org.apache.pulsar.broker.service.BrokerService;
 import org.apache.pulsar.common.naming.NamespaceBundle;
-import org.apache.pulsar.common.naming.NamespaceName;
 import

[pulsar] 02/02: [improve] [broker] Skip split boundle if only one broker (#20190)

2023-05-11 Thread yubiao

This is an automated email from the ASF dual-hosted git repository.

yubiao pushed a commit to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/pulsar.git

commit f3edfc578fab3d74298615429efeb2aaa72a4d32
Author: fengyubiao 
AuthorDate: Sat Apr 29 01:54:05 2023 +0800

[improve] [broker] Skip split boundle if only one broker (#20190)

Co-authored-by: Zixuan Liu 
(cherry picked from commit d135c4a115038dc61f8fe2d230cb1f0c02239f92)
---
 .../loadbalance/impl/ModularLoadManagerImpl.java   |  2 +-
 .../pulsar/broker/stats/PrometheusMetricsTest.java | 56 +
 .../pulsar/client/api/BrokerServiceLookupTest.java | 58 ++
 3 files changed, 115 insertions(+), 1 deletion(-)

diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/ModularLoadManagerImpl.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/ModularLoadManagerImpl.java
index d81f6949f43..293ff2760f0 100644
--- 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/ModularLoadManagerImpl.java
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/ModularLoadManagerImpl.java
@@ -700,7 +700,7 @@ public class ModularLoadManagerImpl implements 
ModularLoadManager {
 public void checkNamespaceBundleSplit() {
 
 if (!conf.isLoadBalancerAutoBundleSplitEnabled() || 
pulsar.getLeaderElectionService() == null
-|| !pulsar.getLeaderElectionService().isLeader()) {
+|| !pulsar.getLeaderElectionService().isLeader() || 
loadData.getBrokerData().size() <= 1) {
 return;
 }
 final boolean unloadSplitBundles = 
pulsar.getConfiguration().isLoadBalancerAutoUnloadSplitBundlesEnabled();
diff --git 
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/PrometheusMetricsTest.java
 
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/PrometheusMetricsTest.java
index 473adc26daa..103b8f94df1 100644
--- 
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/PrometheusMetricsTest.java
+++ 
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/PrometheusMetricsTest.java
@@ -36,6 +36,7 @@ import java.nio.charset.StandardCharsets;
 import java.text.NumberFormat;
 import java.util.Arrays;
 import java.util.Collection;
+import java.util.Collections;
 import java.util.Date;
 import java.util.HashMap;
 import java.util.List;
@@ -74,6 +75,7 @@ import org.apache.pulsar.client.api.PulsarClient;
 import org.apache.pulsar.client.api.PulsarClientException;
 import org.apache.pulsar.client.api.SubscriptionType;
 import org.apache.pulsar.compaction.Compactor;
+import org.apache.zookeeper.CreateMode;
 import org.awaitility.Awaitility;
 import org.testng.Assert;
 import org.testng.annotations.AfterMethod;
@@ -548,6 +550,58 @@ public class PrometheusMetricsTest extends BrokerTestBase {
 c1.acknowledge(c1.receive());
 }
 
+// Mock another broker to make split task work.
+String json =
+"{"
++ "\"webServiceUrl\": \"http://127.0.0.1:0\",;
++ "\"webServiceUrlTls\": \"https://127.0.0.1:0\",;
++ "\"pulsarServiceUrl\": \"pulsar://127.0.0.1:0\","
++ "\"pulsarServiceUrlTls\": \"pulsar+ssl://127.0.0.1:0\","
++ "\"persistentTopicsEnabled\": true,"
++ "\"nonPersistentTopicsEnabled\": true,"
++ "\"cpu\": {"
++ "\"usage\": 0.0,"
++ "\"limit\": 1000.0"
++ "},"
++ "\"memory\": {"
++ "\"usage\": 124.1398696899414,"
++ "\"limit\": 1024.0"
++ "},"
++ "\"directMemory\": {"
++ "\"usage\": 4.0,"
++ "\"limit\": 1024.0"
++ "},"
++ "\"bandwidthIn\": {"
++ "\"usage\": -1.0,"
++ "\"limit\": -1.0"
++ "},"
++ "\"bandwidthOut\": {"
++ "\"usage\": -1.0,"
++ "\"limit\": -1.0"
++ "},"
++ "\"msgThroughputIn\": 0.0,"
++ "\"msgThroughputOut\": 0.0,"
++ "\"msgRateIn\": 0.0,"
++ "\"msgRateOut\": 0.0,"
++ "\"lastUpdate\": 1683812593521,"
++ "\"lastStats\": {},"
++ "\"numTopics\": 0,"
++ "\"numBundles\": 0,"
++ "\"numConsumers\": 0,"
++ "\"numProducers\": 0,"
++ "\"bundles\": [],"
++ "\"lastBundleGains\": [],"
++ "\"lastBundleLosses\": [],"
++ "\"brokerVersionString\": \"2.10.4\","
++ "\"protocols\": {},"
++ "\"advertisedListeners\": {},"
++ "\"maxResourceUsage\": 0.00390625,"
++ "\"loadReportType\":

[GitHub] [pulsar] tisonkun closed issue #20301: [Bug] [broker] Broker get NPE caused by topic publisher throttling



tisonkun closed issue #20301: [Bug] [broker] Broker get NPE caused by topic 
publisher throttling
URL: https://github.com/apache/pulsar/issues/20301


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[pulsar] branch master updated: [fix][broker] Fix NPE cause by topic publish rate limiter. (#20302)

2023-05-11 Thread tison

This is an automated email from the ASF dual-hosted git repository.

tison pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new 4b24c9e85ad [fix][broker] Fix NPE cause by topic publish rate limiter. 
(#20302)
4b24c9e85ad is described below

commit 4b24c9e85ad534c2087063e7873eded4f9ab7a21
Author: Shen Liu 
AuthorDate: Thu May 11 22:28:40 2023 +0800

[fix][broker] Fix NPE cause by topic publish rate limiter. (#20302)

Co-authored-by: druidliu 
---
 .../src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java   | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java
index 6245ce19eeb..8269dc0c3d1 100644
--- 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java
@@ -115,7 +115,7 @@ public abstract class AbstractTopic implements Topic, 
TopicPolicyListener

[GitHub] [pulsar] tisonkun merged pull request #20302: [fix][broker] Fix NPE cause by topic publish rate limiter.



tisonkun merged PR #20302:
URL: https://github.com/apache/pulsar/pull/20302


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] tisonkun commented on pull request #20302: [fix][broker] Fix NPE cause by topic publish rate limiter.



tisonkun commented on PR #20302:
URL: https://github.com/apache/pulsar/pull/20302#issuecomment-1544089210

   Merging...
   
   Thanks for your contribution @dragonls!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] codecov-commenter commented on pull request #20302: [fix][broker] Fix NPE cause by topic publish rate limiter.



codecov-commenter commented on PR #20302:
URL: https://github.com/apache/pulsar/pull/20302#issuecomment-1544040844

   ## 
[Codecov](https://app.codecov.io/gh/apache/pulsar/pull/20302?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)
 Report
   > Merging 
[#20302](https://app.codecov.io/gh/apache/pulsar/pull/20302?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)
 (bcd3a41) into 
[master](https://app.codecov.io/gh/apache/pulsar/commit/b07f2a9b85a0f22b89fed5b51ee8625b00b7262e?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)
 (b07f2a9) will **increase** coverage by `42.70%`.
   > The diff coverage is `100.00%`.
   
   [![Impacted file tree 
graph](https://app.codecov.io/gh/apache/pulsar/pull/20302/graphs/tree.svg?width=650=150=pr=acYqCpsK9J_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)](https://app.codecov.io/gh/apache/pulsar/pull/20302?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)
   
   ```diff
   @@  Coverage Diff  @@
   ## master   #20302   +/-   ##
   =
   + Coverage 30.23%   72.94%   +42.70% 
   - Complexity  29531942+31647 
   =
 Files  1689 1868  +179 
 Lines129020   138602 +9582 
 Branches  1407015240 +1170 
   =
   + Hits  39013   101106+62093 
   + Misses8414829442-54706 
   - Partials   5859 8054 +2195 
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | inttests | `24.12% <100.00%> (-0.05%)` | :arrow_down: |
   | systests | `24.99% <100.00%> (+0.03%)` | :arrow_up: |
   | unittests | `72.22% <100.00%> (?)` | |
   
   Flags with carried forward coverage won't be shown. [Click 
here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment)
 to find out more.
   
   | [Impacted 
Files](https://app.codecov.io/gh/apache/pulsar/pull/20302?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)
 | Coverage Δ | |
   |---|---|---|
   | 
[...rg/apache/pulsar/broker/service/AbstractTopic.java](https://app.codecov.io/gh/apache/pulsar/pull/20302?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9zZXJ2aWNlL0Fic3RyYWN0VG9waWMuamF2YQ==)
 | `88.59% <100.00%> (+31.05%)` | :arrow_up: |
   
   ... and [1489 files with indirect coverage 
changes](https://app.codecov.io/gh/apache/pulsar/pull/20302/indirect-changes?src=pr=tree-more_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-site] shibd commented on a diff in pull request #565: Add CPP client table view doc.



shibd commented on code in PR #565:
URL: https://github.com/apache/pulsar-site/pull/565#discussion_r1191214011


##
docs/client-libraries-tableviews.md:
##
@@ -4,36 +4,106 @@ title: Work with TableView
 sidebar_label: "Work with TableView"
 ---
 
+mdx-code-block
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+
 After setting up your clients, you can explore more to start working with 
[TableView](concepts-clients.md#tableview).
 
 ## Configure TableView
 
-The following is an example of how to configure a TableView.
 
-```java
-TableView tv = client.newTableViewBuilder(Schema.STRING)
-  .topic("my-tableview")
-  .create()
-```
+mdx-code-block
+
+
+
+  The following is an example of how to configure a TableView.
+  
+  ```java
+TableView tv = client.newTableViewBuilder(Schema.STRING)
+.topic("my-tableview")
+.create()
+  ```
+
+You can use the available parameters in the `loadConf` configuration or the 
API 
[`TableViewBuilder`](/api/client/@pulsar:version_number@/org/apache/pulsar/client/api/TableViewBuilder.html)
 to customize your TableView.

Review Comment:
   oh, sorry. this link is wrong.
   
   https://github.com/apache/pulsar-site/assets/33416836/40f9f32d-8c4d-4b16-abe3-3cd78c681fcc;>
   
   @tisonkun Can you help take a look?
   
   The original thread is here
   https://github.com/apache/pulsar-site/pull/565#discussion_r1189257265



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[pulsar] branch branch-3.0 updated: [fix][monitor] topic with double quote breaks the prometheus format (#20230)

This is an automated email from the ASF dual-hosted git repository.

nicoloboschi pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git


The following commit(s) were added to refs/heads/branch-3.0 by this push:
 new dae05da63bc [fix][monitor] topic with double quote breaks the 
prometheus format (#20230)
dae05da63bc is described below

commit dae05da63bc999dc4fd0adf540d573098971cee1
Author: Nicolò Boschi 
AuthorDate: Wed May 10 13:26:16 2023 +0200

[fix][monitor] topic with double quote breaks the prometheus format (#20230)

(cherry picked from commit ea56197d18d8d1f5b06c26246d920cb736d4e6a6)
(cherry picked from commit c5b18542b8cf0806616a3848dbe14ef2394e57df)
---
 .../stats/prometheus/PrometheusMetricStreams.java  |  6 -
 .../pulsar/broker/stats/PrometheusMetricsTest.java | 26 ++
 2 files changed, 31 insertions(+), 1 deletion(-)

diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricStreams.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricStreams.java
index df107b6d7d9..93cbad4e195 100644
--- 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricStreams.java
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricStreams.java
@@ -41,7 +41,11 @@ public class PrometheusMetricStreams {
 SimpleTextOutputStream stream = initGaugeType(metricName);
 stream.write(metricName).write('{');
 for (int i = 0; i < labelsAndValuesArray.length; i += 2) {
-
stream.write(labelsAndValuesArray[i]).write("=\"").write(labelsAndValuesArray[i 
+ 1]).write('\"');
+String labelValue = labelsAndValuesArray[i + 1];
+if (labelValue != null) {
+labelValue = labelValue.replace("\"", "\\\"");
+}
+
stream.write(labelsAndValuesArray[i]).write("=\"").write(labelValue).write('\"');
 if (i + 2 != labelsAndValuesArray.length) {
 stream.write(',');
 }
diff --git 
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/PrometheusMetricsTest.java
 
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/PrometheusMetricsTest.java
index 7a23e56fe0b..a7a28afd8ac 100644
--- 
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/PrometheusMetricsTest.java
+++ 
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/PrometheusMetricsTest.java
@@ -1953,4 +1953,30 @@ public class PrometheusMetricsTest extends 
BrokerTestBase {
 }
 }
 
+@Test
+public void testEscapeLabelValue() throws Exception {
+String ns1 = "prop/ns-abc1";
+admin.namespaces().createNamespace(ns1);
+String topic = "persistent://" + ns1 + "/\"mytopic";
+admin.topics().createNonPartitionedTopic(topic);
+
+@Cleanup
+final Consumer consumer = pulsarClient.newConsumer()
+.subscriptionName("sub")
+.topic(topic)
+.subscribe();
+@Cleanup
+ByteArrayOutputStream statsOut = new ByteArrayOutputStream();
+PrometheusMetricsGenerator.generate(pulsar, true, false,
+false, statsOut);
+String metricsStr = statsOut.toString();
+final List subCountLines = metricsStr.lines()
+.filter(line -> line.startsWith("pulsar_subscriptions_count"))
+.collect(Collectors.toList());
+System.out.println(subCountLines);
+assertEquals(subCountLines.size(), 1);
+assertEquals(subCountLines.get(0),
+
"pulsar_subscriptions_count{cluster=\"test\",namespace=\"prop/ns-abc1\",topic=\"persistent://prop/ns-abc1/\\\"mytopic\"}
 1");
+}
+
 }

[GitHub] [pulsar] HattoriHenzo added a comment to the discussion: Docker image not build with maven



GitHub user HattoriHenzo added a comment to the discussion: Docker image not 
build with maven

It seems to be on Windows. I used this 
[link](https://stackoverflow.com/questions/27820268/trying-to-run-docker-resulted-in-exit-code-127)
 to resolve the issue by changing the end of file to **LF** in the file 
**pulsar\docker\pulsar\scripts\install-pulsar-client.sh**.
On VS Code you can use this 
[link](https://stackoverflow.com/questions/39525417/visual-studio-code-how-to-show-line-endings).

GitHub link: 
https://github.com/apache/pulsar/discussions/20296#discussioncomment-5873123


This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] nodece commented on pull request #20305: [fix][build] Fix publish image script



nodece commented on PR #20305:
URL: https://github.com/apache/pulsar/pull/20305#issuecomment-1543991388

   /pulsarbot rerun-failure-checks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] nodece opened a new pull request, #20305: [fix][build] Fix publish image script



nodece opened a new pull request, #20305:
URL: https://github.com/apache/pulsar/pull/20305

   ### Motivation
   
   In the old version, we have multiple images:
   - `pulsar:latest`
   - `pulsar:VERSION`
   - `pulsar-all:latest`
   - `pulsar-all:VERSION`
   - `apachepulsar/pulsar:latest`
   - `apachepulsar/pulsar:VERSION`
   - `apachepulsar/pulsar-all:latest`
   - `apachepulsar/pulsar-all:VERSION`
   
   After https://github.com/apache/pulsar/pull/19432 is merged, we only have 
the following images:
   - `apachepulsar/pulsar:latest`
   - `apachepulsar/pulsar:VERSION`
   - `apachepulsar/pulsar-all:latest`
   - `apachepulsar/pulsar-all:VERSION`
   
   
   The `pulsar:latest` and `pulsar-all:latest` images are used in the 
`docker/publish.sh` script, #19432 breaks the publish script.
   
   ### Modifications
   
   - Use `apachepulsar/pulsar:latest` instead of `pulsar:latest`
   - Use `apachepulsar/pulsar-all:latest` instead of `pulsar-all:latest`
   
   ### Documentation
   
   
   
   - [ ] `doc` 
   - [ ] `doc-required` 
   - [x] `doc-not-needed` 
   - [ ] `doc-complete` 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[pulsar] branch branch-2.11 updated: [fix][monitor] topic with double quote breaks the prometheus format (#20230)

This is an automated email from the ASF dual-hosted git repository.

nicoloboschi pushed a commit to branch branch-2.11
in repository https://gitbox.apache.org/repos/asf/pulsar.git


The following commit(s) were added to refs/heads/branch-2.11 by this push:
 new c5b18542b8c [fix][monitor] topic with double quote breaks the 
prometheus format (#20230)
c5b18542b8c is described below

commit c5b18542b8cf0806616a3848dbe14ef2394e57df
Author: Nicolò Boschi 
AuthorDate: Wed May 10 13:26:16 2023 +0200

[fix][monitor] topic with double quote breaks the prometheus format (#20230)

(cherry picked from commit ea56197d18d8d1f5b06c26246d920cb736d4e6a6)
---
 .../stats/prometheus/PrometheusMetricStreams.java  |  6 -
 .../pulsar/broker/stats/PrometheusMetricsTest.java | 27 ++
 2 files changed, 32 insertions(+), 1 deletion(-)

diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricStreams.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricStreams.java
index c7e09b5df60..2e32e216a02 100644
--- 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricStreams.java
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricStreams.java
@@ -41,7 +41,11 @@ public class PrometheusMetricStreams {
 SimpleTextOutputStream stream = initGaugeType(metricName);
 stream.write(metricName).write('{');
 for (int i = 0; i < labelsAndValuesArray.length; i += 2) {
-
stream.write(labelsAndValuesArray[i]).write("=\"").write(labelsAndValuesArray[i 
+ 1]).write('\"');
+String labelValue = labelsAndValuesArray[i + 1];
+if (labelValue != null) {
+labelValue = labelValue.replace("\"", "\\\"");
+}
+
stream.write(labelsAndValuesArray[i]).write("=\"").write(labelValue).write('\"');
 if (i + 2 != labelsAndValuesArray.length) {
 stream.write(',');
 }
diff --git 
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/PrometheusMetricsTest.java
 
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/PrometheusMetricsTest.java
index 615aaac293d..e1aa5edb1d9 100644
--- 
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/PrometheusMetricsTest.java
+++ 
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/PrometheusMetricsTest.java
@@ -51,6 +51,7 @@ import java.util.concurrent.TimeUnit;
 import java.util.concurrent.atomic.AtomicReference;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
+import java.util.stream.Collectors;
 import javax.crypto.SecretKey;
 import javax.naming.AuthenticationException;
 import lombok.Cleanup;
@@ -1733,4 +1734,30 @@ public class PrometheusMetricsTest extends 
BrokerTestBase {
 }
 }
 
+@Test
+public void testEscapeLabelValue() throws Exception {
+String ns1 = "prop/ns-abc1";
+admin.namespaces().createNamespace(ns1);
+String topic = "persistent://" + ns1 + "/\"mytopic";
+admin.topics().createNonPartitionedTopic(topic);
+
+@Cleanup
+final Consumer consumer = pulsarClient.newConsumer()
+.subscriptionName("sub")
+.topic(topic)
+.subscribe();
+@Cleanup
+ByteArrayOutputStream statsOut = new ByteArrayOutputStream();
+PrometheusMetricsGenerator.generate(pulsar, true, false,
+false, statsOut);
+String metricsStr = statsOut.toString();
+final List subCountLines = metricsStr.lines()
+.filter(line -> line.startsWith("pulsar_subscriptions_count"))
+.collect(Collectors.toList());
+System.out.println(subCountLines);
+assertEquals(subCountLines.size(), 1);
+assertEquals(subCountLines.get(0),
+
"pulsar_subscriptions_count{cluster=\"test\",namespace=\"prop/ns-abc1\",topic=\"persistent://prop/ns-abc1/\\\"mytopic\"}
 1");
+}
+
 }

[GitHub] [pulsar] eolivelli commented on a diff in pull request #20304: [fix][broker] Allow Access to System Topic Metadata for Reader Creation Post-Namespace Deletion



eolivelli commented on code in PR #20304:
URL: https://github.com/apache/pulsar/pull/20304#discussion_r1191112174


##
pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/PersistentTopicsBase.java:
##
@@ -4409,7 +4409,8 @@ public CompletableFuture 
getPartitionedTopicMetadata(
 // serve/redirect request else fail partitioned-metadata-request so, 
client fails while creating
 // producer/consumer
 authorizationFuture.thenCompose(__ ->
-checkLocalOrGetPeerReplicationCluster(pulsar, 
topicName.getNamespaceObject()))
+checkLocalOrGetPeerReplicationCluster(pulsar, 
topicName.getNamespaceObject(),
+SystemTopicNames.isSystemTopic(topicName)))

Review Comment:
   please add a comment here, about why system topics should be allowed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[pulsar] branch branch-2.10 updated: [fix][monitor] topic with double quote breaks the prometheus format (#20230)

This is an automated email from the ASF dual-hosted git repository.

nicoloboschi pushed a commit to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/pulsar.git


The following commit(s) were added to refs/heads/branch-2.10 by this push:
 new d8bf38ebc3e [fix][monitor] topic with double quote breaks the 
prometheus format (#20230)
d8bf38ebc3e is described below

commit d8bf38ebc3e0e44cf42d5aa821657596b8126d50
Author: Nicolò Boschi 
AuthorDate: Wed May 10 13:26:16 2023 +0200

[fix][monitor] topic with double quote breaks the prometheus format (#20230)

(cherry picked from commit ea56197d18d8d1f5b06c26246d920cb736d4e6a6)
(cherry picked from commit f94c8cb1fc137f72ce23e53552749c60c36b03a9)
---
 .../stats/prometheus/PrometheusMetricStreams.java  |  6 -
 .../pulsar/broker/stats/PrometheusMetricsTest.java | 27 ++
 2 files changed, 32 insertions(+), 1 deletion(-)

diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricStreams.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricStreams.java
index c7e09b5df60..2e32e216a02 100644
--- 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricStreams.java
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricStreams.java
@@ -41,7 +41,11 @@ public class PrometheusMetricStreams {
 SimpleTextOutputStream stream = initGaugeType(metricName);
 stream.write(metricName).write('{');
 for (int i = 0; i < labelsAndValuesArray.length; i += 2) {
-
stream.write(labelsAndValuesArray[i]).write("=\"").write(labelsAndValuesArray[i 
+ 1]).write('\"');
+String labelValue = labelsAndValuesArray[i + 1];
+if (labelValue != null) {
+labelValue = labelValue.replace("\"", "\\\"");
+}
+
stream.write(labelsAndValuesArray[i]).write("=\"").write(labelValue).write('\"');
 if (i + 2 != labelsAndValuesArray.length) {
 stream.write(',');
 }
diff --git 
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/PrometheusMetricsTest.java
 
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/PrometheusMetricsTest.java
index 8aec2b3eed0..1f21706f7dc 100644
--- 
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/PrometheusMetricsTest.java
+++ 
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/PrometheusMetricsTest.java
@@ -51,6 +51,7 @@ import java.util.concurrent.TimeUnit;
 import java.util.concurrent.atomic.AtomicReference;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
+import java.util.stream.Collectors;
 import javax.crypto.SecretKey;
 import javax.naming.AuthenticationException;
 import lombok.Cleanup;
@@ -1684,4 +1685,30 @@ public class PrometheusMetricsTest extends 
BrokerTestBase {
 }
 }
 
+@Test
+public void testEscapeLabelValue() throws Exception {
+String ns1 = "prop/ns-abc1";
+admin.namespaces().createNamespace(ns1);
+String topic = "persistent://" + ns1 + "/\"mytopic";
+admin.topics().createNonPartitionedTopic(topic);
+
+@Cleanup
+final Consumer consumer = pulsarClient.newConsumer()
+.subscriptionName("sub")
+.topic(topic)
+.subscribe();
+@Cleanup
+ByteArrayOutputStream statsOut = new ByteArrayOutputStream();
+PrometheusMetricsGenerator.generate(pulsar, true, false,
+false, statsOut);
+String metricsStr = statsOut.toString();
+final List subCountLines = 
Arrays.stream(metricsStr.split("\n"))
+.filter(line -> line.startsWith("pulsar_subscriptions_count") 
&& line.contains("topic="))
+.collect(Collectors.toList());
+System.out.println(subCountLines);
+assertEquals(subCountLines.size(), 1);
+assertEquals(subCountLines.get(0),
+
"pulsar_subscriptions_count{cluster=\"test\",namespace=\"prop/ns-abc1\",topic=\"persistent://prop/ns-abc1/\\\"mytopic\"}
 1");
+}
+
 }

[GitHub] [pulsar] liangyepianzhou opened a new pull request, #20304: [fix][broker] Allow Access to System Topic Metadata for Reader Creation Post-Namespace Deletion



liangyepianzhou opened a new pull request, #20304:
URL: https://github.com/apache/pulsar/pull/20304

   ## Motivation
   After initiating the snapshot segment function, deletion of topics 
necessitates the activation of readers. Furthermore, these readers should be 
opened and deleted as they are used, which implies that we should not pre-store 
readers. However, after initiating the deletion of namespaces currently, it is 
not allowed to obtain the metadata of partition topics, making it impossible to 
create readers. This results in the inability to delete namespaces.
   
   ## Modification
   Allow the acquisition of system topic metadata after initiating namespace 
deletion, thus creating readers to clean up topic data.
   
   
   ### Verifying this change
   
   - [ ] Make sure that the change passes the CI checks.
   
   *(Please pick either of the following options)*
   
   This change is a trivial rework / code cleanup without any test coverage.
   
   *(or)*
   
   This change is already covered by existing tests, such as *(please describe 
tests)*.
   
   *(or)*
   
   This change added tests and can be verified as follows:
   
   *(example:)*
 - *Added integration tests for end-to-end deployment with large payloads 
(10MB)*
 - *Extended integration test for recovery after broker failure*
   
   ### Does this pull request potentially affect one of the following parts:
   
   
   
   *If the box was checked, please highlight the changes*
   
   - [ ] Dependencies (add or upgrade a dependency)
   - [ ] The public API
   - [ ] The schema
   - [ ] The default values of configurations
   - [ ] The threading model
   - [ ] The binary protocol
   - [ ] The REST endpoints
   - [ ] The admin CLI options
   - [ ] The metrics
   - [ ] Anything that affects deployment
   
   ### Documentation
   
   
   
   - [ ] `doc` 
   - [ ] `doc-required` 
   - [x] `doc-not-needed` 
   - [ ] `doc-complete` 
   
   ### Matching PR in forked repository
   
   PR in forked repository: 
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] aloyszhang commented on a diff in pull request #20292: [feat][broker] introduce precise broker publish rate limiting



aloyszhang commented on code in PR #20292:
URL: https://github.com/apache/pulsar/pull/20292#discussion_r1191044794


##
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java:
##
@@ -1116,6 +1116,12 @@ public boolean isBrokerPublishRateExceeded() {
 return  getBrokerPublishRateLimiter().isPublishRateExceeded();
 }
 
+public boolean isBrokerPublishRateExceeded(int numberMessages, int bytes) {
+// whether broker publish rate exceed
+return preciseTopicPublishRateLimitingEnable

Review Comment:
   It's better to use a distinguish configuration instead of 
`preciseTopicPublishRateLimitingEnable`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] aloyszhang commented on a diff in pull request #20292: [feat][broker] introduce precise broker publish rate limiting



aloyszhang commented on code in PR #20292:
URL: https://github.com/apache/pulsar/pull/20292#discussion_r1191044794


##
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java:
##
@@ -1116,6 +1116,12 @@ public boolean isBrokerPublishRateExceeded() {
 return  getBrokerPublishRateLimiter().isPublishRateExceeded();
 }
 
+public boolean isBrokerPublishRateExceeded(int numberMessages, int bytes) {
+// whether broker publish rate exceed
+return preciseTopicPublishRateLimitingEnable

Review Comment:
   It's better two use a distinguish configuration instead of 
`preciseTopicPublishRateLimitingEnable`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] JooHyukKim commented on issue #10124: Don't require webServicePort to allow for TLS only listeners



JooHyukKim commented on issue #10124:
URL: https://github.com/apache/pulsar/issues/10124#issuecomment-1543837951

   > the issued file now always check `isPresent` before `get`
   
   I may have provided wrong link, but you caught my intention right on "how" 
this issue was resolved  @tisonkun Thank you 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] AnonHxy closed pull request #20286: [fix][test] Flaky-test: SimpleProducerConsumerTestStreamingDispatcherTest. rest



AnonHxy closed pull request #20286: [fix][test] Flaky-test: 
SimpleProducerConsumerTestStreamingDispatcherTest. rest
URL: https://github.com/apache/pulsar/pull/20286


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] Technoboy- commented on a diff in pull request #20299: [improve][ws] Use allowTopicOperationAsync for authz checks



Technoboy- commented on code in PR #20299:
URL: https://github.com/apache/pulsar/pull/20299#discussion_r1191000611


##
pulsar-websocket/src/main/java/org/apache/pulsar/websocket/ProducerHandler.java:
##
@@ -242,7 +244,19 @@ public long getMsgPublishedCounter() {
 
 @Override
 protected Boolean isAuthorized(String authRole, AuthenticationDataSource 
authenticationData) throws Exception {
-return service.getAuthorizationService().canProduce(topic, authRole, 
authenticationData);
+try {
+return service.getAuthorizationService()
+.allowTopicOperationAsync(topic, TopicOperation.PRODUCE, 
authRole, authenticationData)
+
.get(service.getConfig().getMetadataStoreOperationTimeoutSeconds(), SECONDS);
+} catch (InterruptedException e) {

Review Comment:
   `InterruptedException` > `ExecutionException | InterruptedException | 
TimeoutException` ?



##
pulsar-websocket/src/main/java/org/apache/pulsar/websocket/ConsumerHandler.java:
##
@@ -467,8 +470,21 @@ protected ConsumerBuilder 
getConsumerConfiguration(PulsarClient client)
 
 @Override
 protected Boolean isAuthorized(String authRole, AuthenticationDataSource 
authenticationData) throws Exception {
-return service.getAuthorizationService().canConsume(topic, authRole, 
authenticationData,
-this.subscription);
+try {
+AuthenticationDataSubscription subscription = new 
AuthenticationDataSubscription(authenticationData,
+this.subscription);
+return service.getAuthorizationService()
+.allowTopicOperationAsync(topic, TopicOperation.CONSUME, 
authRole, subscription)
+
.get(service.getConfig().getMetadataStoreOperationTimeoutSeconds(), SECONDS);
+} catch (InterruptedException e) {

Review Comment:
   `InterruptedException` > `ExecutionException | InterruptedException | 
TimeoutException`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] Technoboy- commented on a diff in pull request #20299: [improve][ws] Use allowTopicOperationAsync for authz checks



Technoboy- commented on code in PR #20299:
URL: https://github.com/apache/pulsar/pull/20299#discussion_r1191000219


##
pulsar-websocket/src/main/java/org/apache/pulsar/websocket/ConsumerHandler.java:
##
@@ -467,8 +470,21 @@ protected ConsumerBuilder 
getConsumerConfiguration(PulsarClient client)
 
 @Override
 protected Boolean isAuthorized(String authRole, AuthenticationDataSource 
authenticationData) throws Exception {
-return service.getAuthorizationService().canConsume(topic, authRole, 
authenticationData,
-this.subscription);
+try {
+AuthenticationDataSubscription subscription = new 
AuthenticationDataSubscription(authenticationData,
+this.subscription);
+return service.getAuthorizationService()
+.allowTopicOperationAsync(topic, TopicOperation.CONSUME, 
authRole, subscription)
+
.get(service.getConfig().getMetadataStoreOperationTimeoutSeconds(), SECONDS);
+} catch (InterruptedException e) {

Review Comment:
   InterruptedException > `ExecutionException | InterruptedException | 
TimeoutException`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] lifepuzzlefun commented on pull request #20292: [feat][broker] introduce precise broker publish rate limiting



lifepuzzlefun commented on PR #20292:
URL: https://github.com/apache/pulsar/pull/20292#issuecomment-1543735608

   @lordcheng10 nice patch~ can you add a unit test to simulate this, maybe 
just check rate is below the setting is enough.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-client-go] RobertIndie opened a new pull request, #1017: [fix] Fix flaky test in `negative_acks_tracker_test.go`

RobertIndie opened a new pull request, #1017:
URL: https://github.com/apache/pulsar-client-go/pull/1017

Fixes #1016

### Motivation

The test for negative ack trackers needs to wait sometime to close the
message channel. We use this interval time to assert that the
negativeAckTracker can trigger the redelivering with the correct time interval.
But we can't assert this interval very precisely due to the concurrent
execution. Currently, we reserve only one millisecond for this concurrent
execution:

https://github.com/apache/pulsar-client-go/blob/3367cc0cf877faadce0671c412ec827af7d27a16/pulsar/negative_acks_tracker_test.go#L45-L49

Here, 100 milliseconds is for the client tick delay. But we only wait for
another 1 millisecond. This seems too short and introduces the flaky.

This PR waits for more 200 milliseconds. This will not affect the
correctness of the test. Because the minimum interval NackDelay we used in
these tests is 300 milliseconds. Waiting more 200 milliseconds will not affect
the test result.

### Modifications

* Wait for more 200 milliseconds to reduce the flaky

### Does this pull request potentially affect one of the following parts:

*If `yes` was chosen, please highlight the changes*

- Dependencies (does it add or upgrade a dependency): (yes / no)
- The public API: (yes / no)
- The schema: (yes / no / don't know)
- The default values of configurations: (yes / no)
- The wire protocol: (yes / no)

### Documentation

- Does this pull request introduce a new feature? (yes / no)
- If yes, how is the feature documented? (not applicable / docs / GoDocs /
not documented)
- If a feature is not applicable for documentation, explain why?
- If a feature is not documented yet in this PR, please create a followup
issue for adding the documentation

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[pulsar] branch branch-2.9 updated: [fix][monitor] topic with double quote breaks the prometheus format (#20230)

This is an automated email from the ASF dual-hosted git repository.

nicoloboschi pushed a commit to branch branch-2.9
in repository https://gitbox.apache.org/repos/asf/pulsar.git


The following commit(s) were added to refs/heads/branch-2.9 by this push:
 new f94c8cb1fc1 [fix][monitor] topic with double quote breaks the 
prometheus format (#20230)
f94c8cb1fc1 is described below

commit f94c8cb1fc137f72ce23e53552749c60c36b03a9
Author: Nicolò Boschi 
AuthorDate: Wed May 10 13:26:16 2023 +0200

[fix][monitor] topic with double quote breaks the prometheus format (#20230)

(cherry picked from commit ea56197d18d8d1f5b06c26246d920cb736d4e6a6)
---
 .../stats/prometheus/PrometheusMetricStreams.java  |  6 -
 .../pulsar/broker/stats/PrometheusMetricsTest.java | 27 ++
 2 files changed, 32 insertions(+), 1 deletion(-)

diff --git 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricStreams.java
 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricStreams.java
index c7e09b5df60..2e32e216a02 100644
--- 
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricStreams.java
+++ 
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricStreams.java
@@ -41,7 +41,11 @@ public class PrometheusMetricStreams {
 SimpleTextOutputStream stream = initGaugeType(metricName);
 stream.write(metricName).write('{');
 for (int i = 0; i < labelsAndValuesArray.length; i += 2) {
-
stream.write(labelsAndValuesArray[i]).write("=\"").write(labelsAndValuesArray[i 
+ 1]).write('\"');
+String labelValue = labelsAndValuesArray[i + 1];
+if (labelValue != null) {
+labelValue = labelValue.replace("\"", "\\\"");
+}
+
stream.write(labelsAndValuesArray[i]).write("=\"").write(labelValue).write('\"');
 if (i + 2 != labelsAndValuesArray.length) {
 stream.write(',');
 }
diff --git 
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/PrometheusMetricsTest.java
 
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/PrometheusMetricsTest.java
index 84a4e250a6a..e8c55c5236e 100644
--- 
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/PrometheusMetricsTest.java
+++ 
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/PrometheusMetricsTest.java
@@ -53,6 +53,7 @@ import java.util.concurrent.TimeUnit;
 import java.util.concurrent.atomic.AtomicReference;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
+import java.util.stream.Collectors;
 import javax.crypto.SecretKey;
 import javax.naming.AuthenticationException;
 import lombok.Cleanup;
@@ -1517,4 +1518,30 @@ public class PrometheusMetricsTest extends 
BrokerTestBase {
 }
 }
 
+@Test
+public void testEscapeLabelValue() throws Exception {
+String ns1 = "prop/ns-abc1";
+admin.namespaces().createNamespace(ns1);
+String topic = "persistent://" + ns1 + "/\"mytopic";
+admin.topics().createNonPartitionedTopic(topic);
+
+@Cleanup
+final Consumer consumer = pulsarClient.newConsumer()
+.subscriptionName("sub")
+.topic(topic)
+.subscribe();
+@Cleanup
+ByteArrayOutputStream statsOut = new ByteArrayOutputStream();
+PrometheusMetricsGenerator.generate(pulsar, true, false,
+false, statsOut);
+String metricsStr = statsOut.toString();
+final List subCountLines = 
Arrays.stream(metricsStr.split("\n"))
+.filter(line -> line.startsWith("pulsar_subscriptions_count") 
&& line.contains("topic="))
+.collect(Collectors.toList());
+System.out.println(subCountLines);
+assertEquals(subCountLines.size(), 1);
+assertEquals(subCountLines.get(0),
+
"pulsar_subscriptions_count{cluster=\"test\",namespace=\"prop/ns-abc1\",topic=\"persistent://prop/ns-abc1/\\\"mytopic\"}
 1");
+}
+
 }

[GitHub] [pulsar] lordcheng10 commented on pull request #20292: [feat][broker] introduce precise broker publish rate limiting



lordcheng10 commented on PR #20292:
URL: https://github.com/apache/pulsar/pull/20292#issuecomment-1543721453

   @aloyszhang PTAL,thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] hpvd edited a comment on the discussion: Use Automatic Fuzzing to find bugs (e.g. as part of CI / via github action)



GitHub user hpvd edited a comment on the discussion: Use Automatic Fuzzing to 
find bugs (e.g. as part of CI / via github action)

@tisonkun since this was moved from type Issue to Discussion, would be great to 
remove the automatically set `stale` label and the enhancement label.

GitHub link: 
https://github.com/apache/pulsar/discussions/20252#discussioncomment-5871243


This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] poorbarcode opened a new pull request, #20303: [fix] [meta]Switch to the metadata store thread after zk operation



poorbarcode opened a new pull request, #20303:
URL: https://github.com/apache/pulsar/pull/20303

   ### Motivation
   
   #5358 Switched the thread to the metadata store thread.
   #13043 Add the feature `batch operation`, but after the batch is done, the 
thread is not switched back.
   
   Now, if we call `put /a`, it will use the `event thread` of the ZK client 
after the operation; but if we call `put /a/b/c`(the node `/a` is not exists), 
it will use the `metadata store thread`  after the operation.
   
   ### Modifications
   
   Switch to the metadata store thread after the batch operation.
   
   
   ### Documentation
   
   
   
   - [ ] `doc` 
   - [ ] `doc-required` 
   - [x] `doc-not-needed` 
   - [ ] `doc-complete` 
   
   ### Matching PR in forked repository
   
   PR in forked repository: 
   - x
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] hpvd added a comment to the discussion: Support alias for the Schema and Message



GitHub user hpvd added a comment to the discussion: Support alias for the 
Schema and Message

@tisonkun since this was moved from type Issue to Discussion, would be great to 
remove the automatically set `stale` label.

GitHub link: 
https://github.com/apache/pulsar/discussions/20248#discussioncomment-5871234


This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org

[GitHub] [pulsar-client-go] RobertIndie commented on a diff in pull request #878: [Issue 877]Fix send stuck



RobertIndie commented on code in PR #878:
URL: https://github.com/apache/pulsar-client-go/pull/878#discussion_r1190912395


##
pulsar/producer_partition.go:
##
@@ -1035,9 +1035,13 @@ func (p *partitionProducer) Send(ctx context.Context, 
msg *ProducerMessage) (Mes
}, true)
 
// wait for send request to finish
-   <-doneCh
-
-   return msgID, err
+   select {
+   case <-ctx.Done():
+   isDone.Store(true)
+   return msgID, ctx.Err()
+   case <-doneCh:
+   return msgID, err

Review Comment:
   We shouldn't return msgID and err here. They are not assigned here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-client-go] RobertIndie opened a new issue, #1016: [Flaky Test] TestNacksTracker



RobertIndie opened a new issue, #1016:
URL: https://github.com/apache/pulsar-client-go/issues/1016

   
https://github.com/apache/pulsar-client-go/actions/runs/4946174480/jobs/8843893910?pr=1015#step:5:8741
   
   ```
   === RUN   TestNacksTracker
   negative_acks_tracker_test.go:102: 
Error Trace:
/pulsar/pulsar-client-go/pulsar/negative_acks_tracker_test.go:102
Error:  Not equal: 
expected: 2
actual  : 0
Test:   TestNacksTracker
   --- FAIL: TestNacksTracker (0.40s)
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[pulsar-client-go] branch master updated: [fix][Flaky Test] Fix flaky test TestMaxPendingChunkMessages() (#1003)

2023-05-11 Thread zike

This is an automated email from the ASF dual-hosted git repository.

zike pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar-client-go.git


The following commit(s) were added to refs/heads/master by this push:
 new 3367cc0  [fix][Flaky Test] Fix flaky test 
TestMaxPendingChunkMessages() (#1003)
3367cc0 is described below

commit 3367cc0cf877faadce0671c412ec827af7d27a16
Author: Jiaqi Shen <18863662...@163.com>
AuthorDate: Thu May 11 17:25:12 2023 +0800

[fix][Flaky Test] Fix flaky test TestMaxPendingChunkMessages() (#1003)

### Motivation

Old TestMaxPendingChunkMessages() uses the concurrent message publish to 
make the consumer discard unavailable chunk. And it's flaky.

So the `sendSingeChunk()` is introduced to manual create scenarios where 
old chunks should be discarded.

### Modifications

- Fix `TestMaxPendingChunkMessages()`
---
 pulsar/message_chunking_test.go | 135 
 1 file changed, 67 insertions(+), 68 deletions(-)

diff --git a/pulsar/message_chunking_test.go b/pulsar/message_chunking_test.go
index 0e1214d..ee3ab17 100644
--- a/pulsar/message_chunking_test.go
+++ b/pulsar/message_chunking_test.go
@@ -24,10 +24,13 @@ import (
"math/rand"
"net/http"
"strings"
-   "sync"
"testing"
"time"
 
+   "github.com/apache/pulsar-client-go/pulsar/internal"
+
+   "google.golang.org/protobuf/proto"
+
"github.com/stretchr/testify/assert"
 )
 
@@ -148,8 +151,6 @@ func TestLargeMessage(t *testing.T) {
 }
 
 func TestMaxPendingChunkMessages(t *testing.T) {
-   rand.Seed(time.Now().Unix())
-
client, err := NewClient(ClientOptions{
URL: lookupURL,
})
@@ -157,83 +158,52 @@ func TestMaxPendingChunkMessages(t *testing.T) {
defer client.Close()
 
topic := newTopicName()
+   producer, err := client.CreateProducer(ProducerOptions{
+   Topic:   topic,
+   DisableBatching: true,
+   EnableChunking:  true,
+   ChunkMaxMessageSize: 10,
+   })
+   assert.NoError(t, err)
+   assert.NotNil(t, producer)
 
-   totalProducers := 5
-   producers := make([]Producer, 0, 20)
-   defer func() {
-   for _, p := range producers {
-   p.Close()
-   }
-   }()
-
-   clients := make([]Client, 0, 20)
-   defer func() {
-   for _, c := range clients {
-   c.Close()
-   }
-   }()
-
-   for j := 0; j < totalProducers; j++ {
-   pc, err := NewClient(ClientOptions{
-   URL: lookupURL,
-   })
-   assert.Nil(t, err)
-   clients = append(clients, pc)
-   producer, err := pc.CreateProducer(ProducerOptions{
-   Topic:   topic,
-   DisableBatching: true,
-   EnableChunking:  true,
-   ChunkMaxMessageSize: 10,
-   })
-   assert.NoError(t, err)
-   assert.NotNil(t, producer)
-   producers = append(producers, producer)
-   }
-
-   consumer, err := client.Subscribe(ConsumerOptions{
+   c, err := client.Subscribe(ConsumerOptions{
Topic:topic,
Type: Exclusive,
SubscriptionName: "chunk-subscriber",
MaxPendingChunkedMessage: 1,
})
assert.NoError(t, err)
-   assert.NotNil(t, consumer)
-   defer consumer.Close()
+   assert.NotNil(t, c)
+   defer c.Close()
+   pc := c.(*consumer).consumers[0]
 
-   totalMsgs := 40
-   wg := sync.WaitGroup{}
-   wg.Add(totalMsgs * totalProducers)
-   for i := 0; i < totalMsgs; i++ {
-   for j := 0; j < totalProducers; j++ {
-   p := producers[j]
-   go func() {
-   ID, err := p.Send(context.Background(), 
{
-   Payload: createTestMessagePayload(50),
-   })
-   assert.NoError(t, err)
-   assert.NotNil(t, ID)
-   wg.Done()
-   }()
-   }
-   }
-   wg.Wait()
+   sendSingleChunk(producer, "0", 0, 2)
+   // MaxPendingChunkedMessage is 1, the chunked message with uuid 0 will 
be discarded
+   sendSingleChunk(producer, "1", 0, 2)
 
-   received := 0
-   for i := 0; i < totalMsgs*totalProducers; i++ {
-   ctx, cancel := context.WithTimeout(context.Background(), 
time.Second*10)
-   msg, err := consumer.Receive(ctx)
-   cancel()
-   if msg == nil || (err != nil && errors.Is(err,

[GitHub] [pulsar-client-go] RobertIndie merged pull request #1003: [Flaky Test]: Fix flaky test TestMaxPendingChunkMessages()



RobertIndie merged PR #1003:
URL: https://github.com/apache/pulsar-client-go/pull/1003


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-client-go] RobertIndie closed issue #983: Flaky Test: TestMaxPendingChunkMessages



RobertIndie closed issue #983: Flaky Test: TestMaxPendingChunkMessages
URL: https://github.com/apache/pulsar-client-go/issues/983


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] dragonls opened a new pull request, #20302: [fix][broker] Fix NPE cause by topic publish rate limiter.



dragonls opened a new pull request, #20302:
URL: https://github.com/apache/pulsar/pull/20302

   Fixes #20301
   
   ### Motivation
   
   `topicPublishRateLimiter` in 
`org.apache.pulsar.broker.service.AbstractTopic` can only be set in 
`updatePublishDispatcher`. So the `topicPublishRateLimiter` in topic can 
shortly be `null` just when the topic object is created and before 
`initialize`. At this time, if the broker service starts to do 
`checkTopicPublishThrottlingRate` 
(`org.apache.pulsar.broker.service.BrokerService#setupTopicPublishRateLimiterMonitor`),
 which will cause NPE.
   
   Error log sample:
   ```
   2023-05-11T14:18:10,426+0800 
[pulsar-topic-publish-rate-limiter-monitor-39-1] ERROR 
org.apache.bookkeeper.common.util.SafeRunnable - Unexpected throwable caught
   java.lang.NullPointerException: null
   at 
org.apache.pulsar.broker.service.AbstractTopic.checkTopicPublishThrottlingRate(AbstractTopic.java:843)
 ~[org.apache.pulsar-pulsar-broker-2.10.4.jar:2.10.4]
   at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
   at 
org.apache.pulsar.broker.service.BrokerService.lambda$forEachTopic$80(BrokerService.java:1874)
 ~[org.apache.pulsar-pulsar-broker-2.10.4.jar:2.10.4]
   at 
org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:544)
 ~[org.apache.pulsar-pulsar-common-2.10.4.jar:2.10.4]
   at 
org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:272)
 ~[org.apache.pulsar-pulsar-common-2.10.4.jar:2.10.4]
   at 
org.apache.pulsar.broker.service.BrokerService.forEachTopic(BrokerService.java:1872)
 ~[org.apache.pulsar-pulsar-broker-2.10.4.jar:2.10.4]
   at 
org.apache.pulsar.broker.service.BrokerService.checkTopicPublishThrottlingRate(BrokerService.java:1849)
 ~[org.apache.pulsar-pulsar-broker-2.10.4.jar:2.10.4]
   at 
org.apache.pulsar.broker.service.BrokerService.lambda$setupTopicPublishRateLimiterMonitor$7(BrokerService.java:620)
 ~[org.apache.pulsar-pulsar-broker-2.10.4.jar:2.10.4]
   at 
org.apache.bookkeeper.mledger.util.SafeRun$1.safeRun(SafeRun.java:32) 
~[org.apache.pulsar-managed-ledger-2.10.4.jar:2.10.4]
   at 
org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) 
~[org.apache.bookkeeper-bookkeeper-common-4.14.7.gemini.0.jar:4.14.7.gemini.0]
   at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
   at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) 
~[?:?]
   at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
 ~[?:?]
   at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) 
~[?:?]
   at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) 
~[?:?]
   at 
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
 ~[io.netty-netty-common-4.1.87.Final.jar:4.1.87.Final]
   at java.lang.Thread.run(Thread.java:829) ~[?:?]
   ```
   
   ### Modifications
   
   Set the default value `PublishRateLimiter.DISABLED_RATE_LIMITER` to 
`topicPublishRateLimiter` in `AbstractTopic`.
   
   ### Verifying this change
   
   - [ ] Make sure that the change passes the CI checks.
   
   This change is hard to be covered by tests.
   
   ### Does this pull request potentially affect one of the following parts:
   
   
   
   *If the box was checked, please highlight the changes*
   
   - [ ] Dependencies (add or upgrade a dependency)
   - [ ] The public API
   - [ ] The schema
   - [ ] The default values of configurations
   - [ ] The threading model
   - [ ] The binary protocol
   - [ ] The REST endpoints
   - [ ] The admin CLI options
   - [ ] The metrics
   - [ ] Anything that affects deployment
   
   ### Documentation
   
   
   
   - [ ] `doc` 
   - [ ] `doc-required` 
   - [x] `doc-not-needed` 
   - [ ] `doc-complete` 
   
   ### Matching PR in forked repository
   
   PR in forked repository: [](https://github.com/dragonls/pulsar/pull/9)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-client-go] RobertIndie opened a new pull request, #1015: [fix] Fix event time not being set when batching is disabled



RobertIndie opened a new pull request, #1015:
URL: https://github.com/apache/pulsar-client-go/pull/1015

   
   
   Fixes #1013
   
   ### Motivation
   
   The event time is not set when batching is disabled. The event time will be 
lost.
   This is a regression bug in 0.10.0.
   
   ### Modifications
   
   * Set the event time when sending single message
   
   
   ### Does this pull request potentially affect one of the following parts:
   
   *If `yes` was chosen, please highlight the changes*
   
 - Dependencies (does it add or upgrade a dependency): (yes / no)
 - The public API: (yes / no)
 - The schema: (yes / no / don't know)
 - The default values of configurations: (yes / no)
 - The wire protocol: (yes / no)
   
   ### Documentation
   
 - Does this pull request introduce a new feature? (yes / no)
 - If yes, how is the feature documented? (not applicable / docs / GoDocs / 
not documented)
 - If a feature is not applicable for documentation, explain why?
 - If a feature is not documented yet in this PR, please create a followup 
issue for adding the documentation
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] dragonls opened a new issue, #20301: [Bug] [broker] Broker get NPE when default topic publisher throttling is enabled (`maxPublishRatePerTopicInBytes`>0)



dragonls opened a new issue, #20301:
URL: https://github.com/apache/pulsar/issues/20301

   ### Search before asking
   
   - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) 
and found nothing similar.
   
   
   ### Version
   
   2.10.4
   
   ### Minimal reproduce step
   
   1. Set up a pulsar cluster with configuration 
`maxPublishRatePerTopicInBytes>0`
   2. When topic unload, there is a chance to trigger this bug and show NPE 
error log.
   
   
   
   ### What did you expect to see?
   
   No error log and all topics go well.
   
   ### What did you see instead?
   
   `java.lang.NullPointerException` and the topic become unwritable.
   
   ### Anything else?
   
   broker log:
   ```
   2023-05-11T14:18:10,366+0800 
[pulsar-topic-publish-rate-limiter-monitor-39-1] ERROR 
org.apache.bookkeeper.common.util.SafeRunnable - Unexpected throwable caught 
   java.lang.NullPointerException: null
   at 
org.apache.pulsar.broker.service.AbstractTopic.checkTopicPublishThrottlingRate(AbstractTopic.java:843)
 ~[org.apache.pulsar-pulsar-broker-2.10.4.jar:2.10.4]
   at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
   at 
org.apache.pulsar.broker.service.BrokerService.lambda$forEachTopic$80(BrokerService.java:1874)
 ~[org.apache.pulsar-pulsar-broker-2.10.4.jar:2.10.4]
   at 
org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:544)
 ~[org.apache.pulsar-pulsar-common-2.10.4.jar:2.10.4]
   at 
org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:272)
 ~[org.apache.pulsar-pulsar-common-2.10.4.jar:2.10.4]
   at 
org.apache.pulsar.broker.service.BrokerService.forEachTopic(BrokerService.java:1872)
 ~[org.apache.pulsar-pulsar-broker-2.10.4.jar:2.10.4]
   at 
org.apache.pulsar.broker.service.BrokerService.checkTopicPublishThrottlingRate(BrokerService.java:1849)
 ~[org.apache.pulsar-pulsar-broker-2.10.4.jar:2.10.4]
   at 
org.apache.pulsar.broker.service.BrokerService.lambda$setupTopicPublishRateLimiterMonitor$7(BrokerService.java:620)
 ~[org.apache.pulsar-pulsar-broker-2.10.4.jar:2.10.4]
   at 
org.apache.bookkeeper.mledger.util.SafeRun$1.safeRun(SafeRun.java:32) 
~[org.apache.pulsar-managed-ledger-2.10.4.jar:2.10.4]
   at 
org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) 
~[org.apache.bookkeeper-bookkeeper-common-4.14.7.gemini.0.jar:4.14.7.gemini.0]
   at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
   at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) 
~[?:?]
   at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
 ~[?:?]
   at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) 
~[?:?]
   at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) 
~[?:?]
   at 
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
 ~[io.netty-netty-common-4.1.87.Final.jar:4.1.87.Final]
   at java.lang.Thread.run(Thread.java:829) ~[?:?]
   ```
   
   ### Are you willing to submit a PR?
   
   - [X] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-client-go] RobertIndie commented on pull request #1011: [feature] Message payload processor PIP-96



RobertIndie commented on PR #1011:
URL: 
https://github.com/apache/pulsar-client-go/pull/1011#issuecomment-1543565860

   > I am working on fixing this test as well, but I wanted to bring to your 
attention that it is also failing on the upstream master branch(when testing 
locally).I haven't made any changes to the code related to this test.
   
   @ksankeerth 
   I can't reproduce the issue on the master branch. Did you turn on 
`acknowledgmentAtBatchIndexLevelEnabled` for the broker?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[pulsar-site] branch main updated: Add PIP-195 to 3.0 release note (#562)

2023-05-11 Thread zhaocong

This is an automated email from the ASF dual-hosted git repository.

zhaocong pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/pulsar-site.git


The following commit(s) were added to refs/heads/main by this push:
 new dff67f42e6b Add PIP-195 to 3.0 release note (#562)
dff67f42e6b is described below

commit dff67f42e6b0ec5f1657b847bc55cdcb73a3ab2a
Author: Cong Zhao 
AuthorDate: Thu May 11 15:43:26 2023 +0800

Add PIP-195 to 3.0 release note (#562)
---
 release-notes/versioned/pulsar-3.0.0.md | 48 +
 1 file changed, 25 insertions(+), 23 deletions(-)

diff --git a/release-notes/versioned/pulsar-3.0.0.md 
b/release-notes/versioned/pulsar-3.0.0.md
index b02ba2a8739..6078c4245ee 100644
--- a/release-notes/versioned/pulsar-3.0.0.md
+++ b/release-notes/versioned/pulsar-3.0.0.md
@@ -35,6 +35,7 @@ sidebar_label: Apache Pulsar 3.0.0
 - PIP-188: Cluster migration or Blue-Green cluster deployment support in 
Pulsar [#16551](https://github.com/apache/pulsar/pull/16551)
 - PIP-192: New Pulsar Broker Load Balancer 
[#16691](https://github.com/apache/pulsar/pull/16691)
 - PIP-193: Sink preprocessing Function 
[#16739](https://github.com/apache/pulsar/issues/16739)
+- PIP-195: New bucket based delayed message tracker 
[#16763](https://github.com/apache/pulsar/issues/16763)
 - PIP-214: Add broker level metrics statistics and expose to prometheus 
[#18056](https://github.com/apache/pulsar/pull/18056)
 - PIP-218: Consumer batchReceive() single partition every receive 
[#18182](https://github.com/apache/pulsar/pull/18182)
 - PIP-229: Add a common interface to get fields of the MessageIdData 
[#18950](https://github.com/apache/pulsar/pull/18950)
@@ -68,15 +69,34 @@ sidebar_label: Apache Pulsar 3.0.0
   - Implement load data store by @Demogorgon314 in 
[#18777](https://github.com/apache/pulsar/pull/18777)
   - Implement broker registry for new load manager by @Demogorgon314 in 
[#18810](https://github.com/apache/pulsar/pull/18810)
   - Added ServiceUnitStateChannelImpl by @heesung-sn in 
[#18489](https://github.com/apache/pulsar/pull/18489)
+- [Broker] PIP-195: New bucket based delayed message tracker 
[#16763](https://github.com/apache/pulsar/issues/16763)
+  - New bucket based delayed message tracker - interface -part 1 
by @coderzc in [#17344](https://github.com/apache/pulsar/pull/17344)
+  - Implement delayed message index bucket snapshot (create/load) - part2 by 
@coderzc in [#17611](https://github.com/apache/pulsar/pull/17611)
+  - Implement BookkeeperBucketSnapshotStorage - part3 by @coderzc in 
[#17677](https://github.com/apache/pulsar/pull/17677)
+  - Support internal cursor properties - part4 by @coderzc in 
[#17712](https://github.com/apache/pulsar/pull/17712)
+  - Implement BucketDelayedDeliveryTrackerFactory and load 
BucketDelayedDeliveryTracker - part6 by @coderzc in 
[#17756](https://github.com/apache/pulsar/pull/17756)
+  - Implement delayed message bucket snapshot recover - part5 by @coderzc in 
[#18420](https://github.com/apache/pulsar/pull/18420)
+  - Implement Filter out all delayed messages and skip them when reading 
messages from bookies - part7 by @coderzc in 
[#19035](https://github.com/apache/pulsar/pull/19035)
+  - Fix cursor skip read by @coderzc in 
[#19124](https://github.com/apache/pulsar/pull/19124)
+  - Implement delayed message index bucket snapshot(merge/delete) - part8 by 
@coderzc in [#19138](https://github.com/apache/pulsar/pull/19138)
+  - Make BucketDelayedDeliveryTracker can retry snapshot operation & improve 
logs by @coderzc in [#19577](https://github.com/apache/pulsar/pull/19577)
+  - Fix BucketDelayedDeliveryTracker merge issues by @coderzc in 
[#19615](https://github.com/apache/pulsar/pull/19615)
+  - Cut off snapshot segment according to maxIndexesPerBucketSnapshotSegment 
by @coderzc in [#19706](https://github.com/apache/pulsar/pull/19706)
+  - Add metrics for bucket delayed message tracker by @coderzc in 
[#19716](https://github.com/apache/pulsar/pull/19716)
+  - Don't clean up BucketDelayedDeliveryTracker when all consumer disconnect 
by @coderzc in [#19801](https://github.com/apache/pulsar/pull/19801)
+  - Add topicName and cursorName for ledger metadata of bucket snapshot by 
@coderzc in [#19802](https://github.com/apache/pulsar/pull/19802)
+  - Make bucket merge operation asynchronous by @coderzc in 
[#19873](https://github.com/apache/pulsar/pull/19873)
+  - Clear delayed message when unsubscribe &  Make clear operation 
asynchronous by @coderzc in 
[#19901](https://github.com/apache/pulsar/pull/19901)
+  - Merge multiple buckets at once by @coderzc in 
[#19927](https://github.com/apache/pulsar/pull/19927)
+  - Ensure previous delayed index be removed from 
snapshotSegmentLastIndexTable & Make load operate asynchronous by @coderzc in 
[#20086](https://github.com/apache/pulsar/pull/20086)
+  - Fix the issue of the `future` of clean-up delayed messages can't be 
completed by @coderzc in

[GitHub] [pulsar-site] coderzc merged pull request #562: Add PIP-195 to 3.0 release note



coderzc merged PR #562:
URL: https://github.com/apache/pulsar-site/pull/562


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-client-go] Gleiphir2769 commented on pull request #1003: [Flaky Test]: Fix flaky test TestMaxPendingChunkMessages()