[GitHub] [pulsar] lhotari opened a new pull request, #19391: [fix][broker] Replace sync method in NamespacesBase#internalDeleteNamespaceBundleAsync
lhotari opened a new pull request, #19391: URL: https://github.com/apache/pulsar/pull/19391 ### Motivation Calling synchronous methods in an async call chain could cause deadlocks and poor performance. ### Modifications Fix an issue in NamespacesBase#internalDeleteNamespaceBundleAsync. - replace `validateNamespaceBundleOwnership` with `validateNamespaceBundleOwnershipAsync` ### Documentation - [ ] `doc` - [ ] `doc-required` - [x] `doc-not-needed` - [ ] `doc-complete` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] nodece opened a new pull request, #19390: [fix][authentication] Check the original auth when using anonymous role
nodece opened a new pull request, #19390: URL: https://github.com/apache/pulsar/pull/19390 ### Motivation When `anonymousUserRole` is configured, and auth method is `none`, the broker only stores the `authRole`, ignores the original auth data and role forwarded by the proxy, see https://github.com/apache/pulsar/blob/b3432f4ed9c9c19eef4ed696253eb2c18ebbf59d/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java#L898-L904 We cannot ingore the original authentication data and original role forwarded by the proxy, the original authentication data from `CommandConnect#getOriginalAuthData`, and the original role from `CommandConnect#getOriginalPrincipal`. ### Modifications - Add `checkOriginalAuthDataForwardedByProxy` method to check the original authentication - When using anonymous role, we also check the original authentication ### Verifying this change - [x] Make sure that the change passes the CI checks. Added `ProxyAnonymousRoleTest` test. ### Documentation - [ ] `doc` - [ ] `doc-required` - [x] `doc-not-needed` - [ ] `doc-complete` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] lhotari merged pull request #19386: [improve][ci] Update to Gradle Enterprise Maven Extension 1.16.3
lhotari merged PR #19386: URL: https://github.com/apache/pulsar/pull/19386 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch master updated (add792baf54 -> d9a097d1853)
This is an automated email from the ASF dual-hosted git repository. lhotari pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git from add792baf54 [fix] [admin] set offload threshold should fail if ns policies is read-only (#19383) add d9a097d1853 [improve][ci] Update to Gradle Enterprise Maven Extension 1.16.3 (#19386) No new revisions were added by this update. Summary of changes: .mvn/ge-extensions.xml | 2 +- .../apache/pulsar/client/api/SimpleSchemaTest.java | 16 + ...mpleSchemaWithSchemaValidationEnforcedTest.java | 28 -- 3 files changed, 12 insertions(+), 34 deletions(-) delete mode 100644 pulsar-broker/src/test/java/org/apache/pulsar/client/api/SimpleSchemaWithSchemaValidationEnforcedTest.java
[GitHub] [pulsar] lhotari opened a new pull request, #19389: [fix][bug] Fix concurrency bug in PersistentTopicsBase#internalGetReplicatedSubscriptionStatus
lhotari opened a new pull request, #19389: URL: https://github.com/apache/pulsar/pull/19389 ### Motivation There's a concurrency bug in PersistentTopicsBase#internalGetReplicatedSubscriptionStatus. A `java.util.HashMap` is used to collect results of concurrent asynchronous method calls. ### Modifications - use `ConcurrentHashMap` instead of `java.util.HashMap` ### Documentation - [ ] `doc` - [ ] `doc-required` - [x] `doc-not-needed` - [ ] `doc-complete` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] lhotari opened a new pull request, #19388: [improve][broker] Reduce calls on metadata store / ZK event thread & Netty threads in PersistentTopic
lhotari opened a new pull request, #19388: URL: https://github.com/apache/pulsar/pull/19388 ### Motivation It is not recommended to run possible blocking operations on Zookeeper event thread, Metadata Store notification thread or Netty IO threads. Execution could happen unintentionally on these threads since `CompletableFuture` callbacks (`thenCompose`, `thenAccept`, `thenRun`, `whenComplete`, ...) run on the thread that completes the `CompletableFuture`. In PersistentTopic, there are a few locations where there are potential issues. It's better to run the callbacks on another thread pool when there's a heavy operation or any operation that uses synchronization or locks. ### Modifications - replace `thenAccept` -> `thenAcceptAsync` and `thenRun` -> `thenRunAsync` in a few call chains - use `brokerService.getTopicOrderedExecutor().chooseThread(topic)` as the executor. ### Documentation - [ ] `doc` - [ ] `doc-required` - [x] `doc-not-needed` - [ ] `doc-complete` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-site] Sherlock113 commented on pull request #398: [BLOG] Add the Pulsar 600th contributor blog
Sherlock113 commented on PR #398: URL: https://github.com/apache/pulsar-site/pull/398#issuecomment-1413258316 @tuhaihe @tisonkun @Anonymitaet PTAL. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-site] Sherlock113 opened a new pull request, #398: [BLOG] Add the Pulsar 600th contributor blog
Sherlock113 opened a new pull request, #398: URL: https://github.com/apache/pulsar-site/pull/398 ### Documentation - [ ] `doc` - [ ] `doc-required` - [x] `doc-not-needed` - [ ] `doc-complete` Submitted a blog to celebrate the 600th contributor to Pulsar's main GitHub repo. Previews:     -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-go] panszobe commented on issue #949: Consume Performance drops when set EnableBatchIndexAcknowledgment = true
panszobe commented on issue #949: URL: https://github.com/apache/pulsar-client-go/issues/949#issuecomment-1413252488 > It might take some time for that, I will start the feature catch up next week. Assign this issue to me first, if someone else is interested, please ping me in this issue. Ok, thanks. Looking forward to new PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] lhotari opened a new pull request, #19387: [fix][broker] Remove synchronous method call in async call chain in PersistentTopicsBase
lhotari opened a new pull request, #19387: URL: https://github.com/apache/pulsar/pull/19387 ### Motivation Calling synchronous methods in an async call chain could cause deadlocks and poor performance. ### Modifications Fix an issue in PerrsistentTopicsBase's internalSetReplicationClusters method - use async calls ### Documentation - [ ] `doc` - [ ] `doc-required` - [x] `doc-not-needed` - [ ] `doc-complete` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar-client-python] branch main updated: Update the release process for versioning (#91)
This is an automated email from the ASF dual-hosted git repository. mmerli pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/pulsar-client-python.git The following commit(s) were added to refs/heads/main by this push: new cbd31e7 Update the release process for versioning (#91) cbd31e7 is described below commit cbd31e75adc592ee16dd1698335f6f34b8fb4844 Author: Yunze Xu AuthorDate: Thu Feb 2 14:21:05 2023 +0800 Update the release process for versioning (#91) ### Motivation Adopt the same versioning rule with the Node.js client, see https://github.com/apache/pulsar-client-node/pull/287. Add an extra step to commit the version update directly before pushing the tag. For example, https://github.com/apache/pulsar-client-python/commit/fda50867a9c7bf927309527fade2f53eb3907bed --- RELEASE.md | 51 +++ 1 file changed, 31 insertions(+), 20 deletions(-) diff --git a/RELEASE.md b/RELEASE.md index 9ccd2c4..9902cee 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -23,23 +23,16 @@ This page contains instructions for Pulsar committers on how to perform a release for the Pulsar Python client. -## Preparation +## Versioning +Bump up the version number as follows. -> **Note** -> -> The term `major/minor releases` used throughout this document is defined as follows: -> - Major releases refer to feature releases, such as 3.0.0, 3.1.0, and so on. -> - Minor releases refer to bug-fix releases, such as 3.0.1, 3.0.2, and so on. -> -> This guide use `X.Y.Z` or `X.Y` to represent the actual versions like `3.0.0` or `3.0`. - -For major releases, you should create a new branch named `branch-X.Y` once all PRs with the X.Y.0 milestone are merged. If some PRs with the X.Y.0 milestone are still working in progress and might take much time to complete, you can move them to the next milestone if they are not important. In this case, you'd better notify the author in the PR. - -For minor releases, if there are no disagreements, you should cherry-pick all merged PRs with the `release/X.Y.Z` labels into `branch-X.Y`. After these PRs are cherry-picked, you should add the `cherry-picked/branch-X.Y` labels. - -Sometimes some PRs cannot be cherry-picked cleanly, you might need to create a separate PR and move the `release/X.Y.Z` label from the original PR to it. In this case, you can ask the author to help create the new PR. - -For PRs that are still open, you can choose to delay them to the next release or ping other committers to review so that they can be merged. +* Major version (e.g. 3.0.0 => 4.0.0) + * Changes that break backward compatibility +* Minor version (e.g. 3.0.0 => 3.1.0) + * Backward compatible new features +* Patch version (e.g. 3.0.0 => 3.0.1) + * Backward compatible bug fixes + * C++ Client upgrade (even though there are no new commits in the Python client) ## Requirements @@ -55,12 +48,30 @@ Example: https://github.com/apache/pulsar-client-python/pull/62 After all necessary PRs are cherry-picked to `branch-X.Y`, you should cut the release by pushing a tag. +For major and minor releases (`X.Y.0`), you need to create a new branch: + +```bash +git checkout -b branch-X.Y +sed -i 's/__version__.*/__version__=X.Y.0/' pulsar/__about__.py +git add pulsar/__about__.py +git commit -m "Bump version to X.Y.0" +git push origin branch-X.Y +# N starts with 1 +git tag vX.Y.0-candidate-N +git push origin vX.Y.0-candidate-N +``` + +For patch releases (`X.Y.Z`), you need to reuse the existing branch: + ```bash git checkout branch-X.Y +sed -i 's/__version__.*/__version__=X.Y.Z/' pulsar/__about__.py +git add pulsar/__about__.py +git commit -m "Bump version to X.Y.Z" git push origin branch-X.Y # N starts with 1 -git tag vX.Y.Y-candidate-N -git push origin vX.Y.Y-candidate-N +git tag vX.Y.Z-candidate-N +git push origin vX.Y.Z-candidate-N ``` Then, [create a new milestone](https://github.com/apache/pulsar-client-python/milestones/new) for the next major release. @@ -103,7 +114,7 @@ Send an email to d...@pulsar.apache.org to start the vote for the candidate: To: d...@pulsar.apache.org Subject: [VOTE] Pulsar Client Python Release X.Y.Z Candidate N -This is the third release candidate for Apache Pulsar Client Python, +This is the Nth release candidate for Apache Pulsar Client Python, version X.Y.Z. It fixes the following issues: @@ -115,7 +126,7 @@ stay open for at least 72 hours *** Python wheels: https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-python-X.Y.Z-candidate-N/ -The supported python versions are 3.7, 3.8, 3.9 and 3.10. The +The supported python versions are 3.7, 3.8, 3.9, 3.10 and 3.11. The supported platforms and architectures are: - Windows x86_64 (windows/) - glibc-based Linux x86_64 (linux-glibc-x86_64/)
[GitHub] [pulsar-client-python] merlimat merged pull request #91: Update the release process for versioning
merlimat merged PR #91: URL: https://github.com/apache/pulsar-client-python/pull/91 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-python] keenborder786 commented on a diff in pull request #90: [fix] exception bug as per #89
keenborder786 commented on code in PR #90:
URL:
https://github.com/apache/pulsar-client-python/pull/90#discussion_r1094085704
##
pulsar/schema/schema_avro.py:
##
@@ -82,15 +82,6 @@ def decode(self, data):
return self._record_cls(**d)
else:
return d
-
else:
-class AvroSchema(Schema):
-def __init__(self, _record_cls, _schema_definition=None):
-raise Exception("Avro library support was not found. Make sure to
install Pulsar client " +
+raise Exception("Avro library support was not found. Make sure to install
Pulsar client " +
Review Comment:
The thing is it resolves the issue but even I am not sure why. In order to
replicate the bug please follow the below steps:
1. Create module called schema_test.py with following code:
```python
## Create test schema
from pulsar.schema import
Record,String,JsonSchema,Boolean,Integer,Long,AvroSchema
import json
import requests
## Simple Schema Test
class account_schema(Record):
id_value = String()
id_set = Boolean()
txn_code_value = String()
txn_code_set = Boolean()
from_account_value = String()
from_account_set = Boolean()
amount_value = String()
amount_set = Boolean()
class source_schema(Record):
version = String()
connector = String()
name = String()
ts_ms = Long()
class final_account_payload_schema(Record):
before = account_schema()
after = account_schema()
source = source_schema()
op = String()
ts_ms = Long()
transaction = String()
new_col = String()
new_col_2 = String()
new_col_3 = String()
new_col_4 = String()
new_col_5 = String()
new_col_6 = String()
schema_test = {
"name":"test",
"type": "AVRO",
"schema": json.dumps(final_account_payload_schema.schema()),
"properties": {}
}
x =
requests.post('http://127.0.0.1:8080/admin/v2/schemas/public/default/dbank-data-test/schema'
, headers = {'Content-Type': 'application/json'} ,data =
json.dumps(schema_test))
print(x.content)
schema_get =
requests.get('http://127.0.0.1:8080/admin/v2/schemas/public/default/dbank-data-test/schema'
, headers = {'Content-Type': 'application/json'})
print( json.loads(schema_get.content.decode('utf-8').strip()))
```
2. Create module called producer.py with following code:
```python
from pulsar.schema import AvroSchema,JsonSchema
from schema_test import
final_account_payload_schema,account_schema,source_schema
import json
import requests
import pulsar
## Simple Test Producer
client = pulsar.Client('pulsar://localhost:6650')
producer = client.create_producer(topic = 'dbank-data-test' , producer_name
= 'python' ,schema = AvroSchema(final_account_payload_schema))
## The below code was not accessible for me
after_vals = account_schema(id_value = '21' , id_set = True , txn_code_value
='1231' , txn_code_set = True , from_account_value = '12321' , from_account_set
= True , amount_value = '123' , amount_set = True , new_col = '123' )
source_vals = source_schema(version = '213' , connector = '12321' , name =
'12312' , ts_ms = 123213214321)
test_val = final_account_payload_schema(before = None , after = after_vals ,
source = source_vals ,
op = 'c' , ts_ms = 123213124 , transaction = '78181')
producer.send(test_val)
client.close()
```
Tested On following OS and conda environment:
- Distributor ID:Ubuntu
Description: Ubuntu 22.04.1 LTS
Release: 22.04
Codename: jammy
```yaml
name: data-platform-dev
channels:
- conda-forge
- defaults
- pypi
dependencies:
- _libgcc_mutex=0.1
- _openmp_mutex=4.5
- binutils=2.39
- binutils_impl_linux-64=2.39
- binutils_linux-64=2.39
- bzip2=1.0.8
- c-ares=1.18.1
- c-compiler=1.5.2
- ca-certificates=2022.12.7
- cmake=3.25.2
- cxx-compiler=1.5.2
- expat=2.5.0
- gcc=11.3.0
- gcc_impl_linux-64=11.3.0
- gcc_linux-64=11.3.0
- gxx=11.3.0
- gxx_impl_linux-64=11.3.0
- gxx_linux-64=11.3.0
- kernel-headers_linux-64=2.6.32
- keyutils=1.6.1
- krb5=1.20.1
- ld_impl_linux-64=2.39
- libcurl=7.87.0
- libedit=3.1.20191231
- libev=4.33
- libffi=3.4.2
- libgcc-devel_linux-64=11.3.0
- libgcc-ng=12.2.0
- libgomp=12.2.0
- libnghttp2=1.51.0
- libnsl=2.0.0
- libsanitizer=11.3.0
- libsqlite=3.40.0
- libssh2=1.10.0
- libstdcxx-devel_linux-64=11.3.0
- libstdcxx-ng=12.2.0
- libuuid=2.32.1
- libuv=1.44.2
- libzlib=1.2.13
- make=4.3
- ncurses=6.3
- openssl=3.0.7
- pip=22.3.1
- pybind11=2.10.3
-
[GitHub] [pulsar-client-python] keenborder786 commented on a diff in pull request #90: [fix] exception bug as per #89
keenborder786 commented on code in PR #90:
URL:
https://github.com/apache/pulsar-client-python/pull/90#discussion_r1094085704
##
pulsar/schema/schema_avro.py:
##
@@ -82,15 +82,6 @@ def decode(self, data):
return self._record_cls(**d)
else:
return d
-
else:
-class AvroSchema(Schema):
-def __init__(self, _record_cls, _schema_definition=None):
-raise Exception("Avro library support was not found. Make sure to
install Pulsar client " +
+raise Exception("Avro library support was not found. Make sure to install
Pulsar client " +
Review Comment:
The thing is it does but I am not sure why. In order to replicate the bug
please follow the below steps:
1. Create module called schema_test.py with following code:
`python
## Create test schema
from pulsar.schema import
Record,String,JsonSchema,Boolean,Integer,Long,AvroSchema
import json
import requests
## Simple Schema Test
class account_schema(Record):
id_value = String()
id_set = Boolean()
txn_code_value = String()
txn_code_set = Boolean()
from_account_value = String()
from_account_set = Boolean()
amount_value = String()
amount_set = Boolean()
class source_schema(Record):
version = String()
connector = String()
name = String()
ts_ms = Long()
class final_account_payload_schema(Record):
before = account_schema()
after = account_schema()
source = source_schema()
op = String()
ts_ms = Long()
transaction = String()
new_col = String()
new_col_2 = String()
new_col_3 = String()
new_col_4 = String()
new_col_5 = String()
new_col_6 = String()
schema_test = {
"name":"test",
"type": "AVRO",
"schema": json.dumps(final_account_payload_schema.schema()),
"properties": {}
}
x =
requests.post('http://127.0.0.1:8080/admin/v2/schemas/public/default/dbank-data-test/schema'
, headers = {'Content-Type': 'application/json'} ,data =
json.dumps(schema_test))
print(x.content)
schema_get =
requests.get('http://127.0.0.1:8080/admin/v2/schemas/public/default/dbank-data-test/schema'
, headers = {'Content-Type': 'application/json'})
print( json.loads(schema_get.content.decode('utf-8').strip()))
```
2. Create module called producer.py with following code:
```
from pulsar.schema import AvroSchema,JsonSchema
from schema_test import
final_account_payload_schema,account_schema,source_schema
import json
import requests
import pulsar
## Simple Test Producer
client = pulsar.Client('pulsar://localhost:6650')
producer = client.create_producer(topic = 'dbank-data-test' , producer_name
= 'python' ,schema = AvroSchema(final_account_payload_schema))
## The below code was not accessible for me
after_vals = account_schema(id_value = '21' , id_set = True , txn_code_value
='1231' , txn_code_set = True , from_account_value = '12321' , from_account_set
= True , amount_value = '123' , amount_set = True , new_col = '123' )
source_vals = source_schema(version = '213' , connector = '12321' , name =
'12312' , ts_ms = 123213214321)
test_val = final_account_payload_schema(before = None , after = after_vals ,
source = source_vals ,
op = 'c' , ts_ms = 123213124 , transaction = '78181')
producer.send(test_val)
client.close()
```
Tested On following OS and conda environment:
- Distributor ID:Ubuntu
Description: Ubuntu 22.04.1 LTS
Release: 22.04
Codename: jammy
```yaml
name: data-platform-dev
channels:
- conda-forge
- defaults
- pypi
dependencies:
- _libgcc_mutex=0.1
- _openmp_mutex=4.5
- binutils=2.39
- binutils_impl_linux-64=2.39
- binutils_linux-64=2.39
- bzip2=1.0.8
- c-ares=1.18.1
- c-compiler=1.5.2
- ca-certificates=2022.12.7
- cmake=3.25.2
- cxx-compiler=1.5.2
- expat=2.5.0
- gcc=11.3.0
- gcc_impl_linux-64=11.3.0
- gcc_linux-64=11.3.0
- gxx=11.3.0
- gxx_impl_linux-64=11.3.0
- gxx_linux-64=11.3.0
- kernel-headers_linux-64=2.6.32
- keyutils=1.6.1
- krb5=1.20.1
- ld_impl_linux-64=2.39
- libcurl=7.87.0
- libedit=3.1.20191231
- libev=4.33
- libffi=3.4.2
- libgcc-devel_linux-64=11.3.0
- libgcc-ng=12.2.0
- libgomp=12.2.0
- libnghttp2=1.51.0
- libnsl=2.0.0
- libsanitizer=11.3.0
- libsqlite=3.40.0
- libssh2=1.10.0
- libstdcxx-devel_linux-64=11.3.0
- libstdcxx-ng=12.2.0
- libuuid=2.32.1
- libuv=1.44.2
- libzlib=1.2.13
- make=4.3
- ncurses=6.3
- openssl=3.0.7
- pip=22.3.1
- pybind11=2.10.3
- pybind11-global=2.10.3
-
[pulsar] branch master updated: [fix] [admin] set offload threshold should fail if ns policies is read-only (#19383)
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new add792baf54 [fix] [admin] set offload threshold should fail if ns
policies is read-only (#19383)
add792baf54 is described below
commit add792baf54918ab726a09b6a590aa6e814bdaa7
Author: fengyubiao
AuthorDate: Thu Feb 2 13:42:03 2023 +0800
[fix] [admin] set offload threshold should fail if ns policies is read-only
(#19383)
---
.../broker/resources/NamespaceResources.java | 2 +-
.../pulsar/broker/admin/impl/NamespacesBase.java | 2 +-
.../pulsar/broker/admin/AdminApiOffloadTest.java | 23 ++
3 files changed, 25 insertions(+), 2 deletions(-)
diff --git
a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/resources/NamespaceResources.java
b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/resources/NamespaceResources.java
index dd1c428380b..48f82596567 100644
---
a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/resources/NamespaceResources.java
+++
b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/resources/NamespaceResources.java
@@ -50,7 +50,7 @@ public class NamespaceResources extends
BaseResources {
private final PartitionedTopicResources partitionedTopicResources;
private final MetadataStore configurationStore;
-private static final String POLICIES_READONLY_FLAG_PATH =
"/admin/flags/policies-readonly";
+public static final String POLICIES_READONLY_FLAG_PATH =
"/admin/flags/policies-readonly";
private static final String NAMESPACE_BASE_PATH = "/namespace";
private static final String BUNDLE_DATA_BASE_PATH =
"/loadbalance/bundle-data";
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java
index 324c8404875..44e2f46174a 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java
@@ -2046,7 +2046,7 @@ public abstract class NamespacesBase extends
AdminResource {
CompletableFuture f = new CompletableFuture<>();
validateNamespacePolicyOperationAsync(namespaceName,
PolicyName.OFFLOAD, PolicyOperation.WRITE)
-.thenApply(v -> validatePoliciesReadOnlyAccessAsync())
+.thenCompose(v -> validatePoliciesReadOnlyAccessAsync())
.thenCompose(v -> updatePoliciesAsync(namespaceName,
policies -> {
if (policies.offload_policies == null) {
diff --git
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/AdminApiOffloadTest.java
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/AdminApiOffloadTest.java
index 604bc437f19..c3265897b87 100644
---
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/AdminApiOffloadTest.java
+++
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/AdminApiOffloadTest.java
@@ -47,14 +47,17 @@ import static org.testng.Assert.assertEquals;
import static org.testng.Assert.assertNotNull;
import static org.testng.Assert.assertNull;
import static org.testng.Assert.assertTrue;
+import static org.testng.Assert.fail;
import java.util.HashMap;
import java.util.Map;
+import java.util.Optional;
import java.util.Set;
import java.util.UUID;
import java.util.concurrent.CompletableFuture;
import org.apache.bookkeeper.mledger.LedgerOffloader;
import org.apache.bookkeeper.mledger.ManagedLedgerInfo;
import org.apache.pulsar.broker.auth.MockedPulsarServiceBaseTest;
+import org.apache.pulsar.broker.resources.NamespaceResources;
import org.apache.pulsar.broker.service.persistent.PersistentTopic;
import org.apache.pulsar.client.admin.LongRunningProcessStatus;
import org.apache.pulsar.client.admin.PulsarAdminException.ConflictException;
@@ -290,6 +293,26 @@ public class AdminApiOffloadTest extends
MockedPulsarServiceBaseTest {
assertEquals(admin.namespaces().getOffloadPolicies(myNamespace),
policies);
}
+@Test
+public void testSetNamespaceOffloadPoliciesFailByReadOnly() throws
Exception {
+boolean setNsPolicyReadOnlySuccess = false;
+try {
+
pulsar.getConfigurationMetadataStore().put(NamespaceResources.POLICIES_READONLY_FLAG_PATH,
"0".getBytes(),
+Optional.empty()).join();
+setNsPolicyReadOnlySuccess = true;
+admin.namespaces().setOffloadThresholdInSeconds(myNamespace, 300);
+fail("set offload threshold should fail when ns policies is
readonly");
+} catch (Exception ex){
+// ignore.
+} finally {
+// cleanup.
+if (setNsPolicyReadOnlySuccess) {
+
[GitHub] [pulsar] lhotari merged pull request #19383: [fix] [admin] set offload threshold should fail if ns policies is read-only
lhotari merged PR #19383: URL: https://github.com/apache/pulsar/pull/19383 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] gaoran10 commented on a diff in pull request #18865: [improve][broker] PIP-220 Added TransferShedder
gaoran10 commented on code in PR #18865: URL: https://github.com/apache/pulsar/pull/18865#discussion_r1094053463 ## pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java: ## @@ -2450,6 +2450,64 @@ The delayed message index bucket time step(in seconds) in per bucket snapshot se ) private long namespaceBundleUnloadingTimeoutMs = 6; +@FieldContext( +category = CATEGORY_LOAD_BALANCER, +doc = "Option to enable the debug mode for the load balancer logics. " ++ "The debug mode prints more logs to provide more information " ++ "such as load balance states and decisions. " ++ "(only used in load balancer extension logics)" +) +private boolean loadBalancerDebugModeEnabled = false; Review Comment: OK, but it's a little strange to add a separate param to control debug log information. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #19386: [improve][ci] Update to Gradle Enterprise Maven Extension 1.16.3
lhotari commented on PR #19386: URL: https://github.com/apache/pulsar/pull/19386#issuecomment-1413184461 /pulsarbot rerun-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] gaoran10 commented on a diff in pull request #18865: [improve][broker] PIP-220 Added TransferShedder
gaoran10 commented on code in PR #18865: URL: https://github.com/apache/pulsar/pull/18865#discussion_r1094053463 ## pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java: ## @@ -2450,6 +2450,64 @@ The delayed message index bucket time step(in seconds) in per bucket snapshot se ) private long namespaceBundleUnloadingTimeoutMs = 6; +@FieldContext( +category = CATEGORY_LOAD_BALANCER, +doc = "Option to enable the debug mode for the load balancer logics. " ++ "The debug mode prints more logs to provide more information " ++ "such as load balance states and decisions. " ++ "(only used in load balancer extension logics)" +) +private boolean loadBalancerDebugModeEnabled = false; Review Comment: OK, but it's a little strange to add a separate param to control debug log information, we also need to add extra judgment logic. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] tjiuming commented on a diff in pull request #19384: [fix] Close TransactionBuffer when create persistent topic timeout
tjiuming commented on code in PR #19384:
URL: https://github.com/apache/pulsar/pull/19384#discussion_r1094051770
##
pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/buffer/TopicTransactionBufferTest.java:
##
@@ -140,4 +144,38 @@ public void
testCheckDeduplicationFailedWhenCreatePersistentTopic() throws Excep
Assert.assertEquals(ttb.getState(), expectState);
}
+
+@Test
+public void testCloseTransactionBufferWhenTimeout() throws Exception {
+String topic = "persistent://" + NAMESPACE1 + "/test_" +
UUID.randomUUID();
+PulsarService pulsar = pulsarServiceList.get(0);
+BrokerService brokerService0 = pulsar.getBrokerService();
+BrokerService brokerService = Mockito.spy(brokerService0);
+AtomicReference reference = new AtomicReference<>();
+long topicLoadTimeout =
TimeUnit.SECONDS.toMillis(pulsar.getConfiguration().getTopicLoadTimeoutSeconds()
+ 1);
+
+Mockito
+.doAnswer(inv -> {
+Thread.sleep(topicLoadTimeout);
+PersistentTopic persistentTopic = (PersistentTopic)
inv.callRealMethod();
+reference.set(persistentTopic);
+return persistentTopic;
+})
+.when(brokerService)
+.newTopic(Mockito.eq(topic), Mockito.any(),
Mockito.eq(brokerService),
+Mockito.eq(PersistentTopic.class));
+
+CompletableFuture> f = brokerService.getTopic(topic,
true);
+
+Awaitility.waitAtMost(90, TimeUnit.SECONDS)
Review Comment:
because the default value of `topicLoadTimeoutSeconds` is 60.
but I've fixed this, PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar-client-cpp] tongsucn commented on a diff in pull request #129: [improve] Add configuration to limit times of client's lookup redirection.
tongsucn commented on code in PR #129:
URL: https://github.com/apache/pulsar-client-cpp/pull/129#discussion_r1094042915
##
include/pulsar/ClientConfiguration.h:
##
@@ -64,25 +64,25 @@ class PULSAR_PUBLIC ClientConfiguration {
*
* @param timeout the timeout after which the operation will be considered
as failed
*/
-ClientConfiguration& setOperationTimeoutSeconds(int timeout);
+ClientConfiguration& setOperationTimeoutSeconds(int32_t timeout);
Review Comment:
I'll change them later this week
##
include/pulsar/ClientConfiguration.h:
##
@@ -64,25 +64,25 @@ class PULSAR_PUBLIC ClientConfiguration {
*
* @param timeout the timeout after which the operation will be considered
as failed
*/
-ClientConfiguration& setOperationTimeoutSeconds(int timeout);
+ClientConfiguration& setOperationTimeoutSeconds(int32_t timeout);
Review Comment:
I'll fix them later this week
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar-client-go] BewareMyPower commented on issue #949: Consume Performance drops when set EnableBatchIndexAcknowledgment = true
BewareMyPower commented on issue #949: URL: https://github.com/apache/pulsar-client-go/issues/949#issuecomment-1413168934 It might take some time for that, I will start the feature catch up next week. Assign this issue to me first, if someone else is interested, please ping me in this issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-go] BewareMyPower commented on issue #949: Consume Performance drops when set EnableBatchIndexAcknowledgment = true
BewareMyPower commented on issue #949: URL: https://github.com/apache/pulsar-client-go/issues/949#issuecomment-1413162825 I think the root cause is the lack of the ACK grouping tracker feature in Golang client. Here is a similar issue and fix for C++ client: https://github.com/apache/pulsar/pull/6534 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] Demogorgon314 commented on a diff in pull request #19102: [improve][broker] PIP-192: Implement extensible load manager
Demogorgon314 commented on code in PR #19102:
URL: https://github.com/apache/pulsar/pull/19102#discussion_r1094032298
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/ExtensibleLoadManagerImpl.java:
##
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.BrokerFilterException;
+import
org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannel;
+import
org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLoadData;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.broker.loadbalance.extensions.data.TopBundlesLoadData;
+import org.apache.pulsar.broker.loadbalance.extensions.filter.BrokerFilter;
+import org.apache.pulsar.broker.loadbalance.extensions.store.LoadDataStore;
+import
org.apache.pulsar.broker.loadbalance.extensions.store.LoadDataStoreException;
+import
org.apache.pulsar.broker.loadbalance.extensions.store.LoadDataStoreFactory;
+import
org.apache.pulsar.broker.loadbalance.extensions.strategy.BrokerSelectionStrategy;
+import
org.apache.pulsar.broker.loadbalance.extensions.strategy.LeastResourceUsageWithWeight;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.ServiceUnitId;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+
+@Slf4j
+public class ExtensibleLoadManagerImpl implements ExtensibleLoadManager {
+
+public static final String BROKER_LOAD_DATA_STORE_TOPIC = TopicName.get(
+TopicDomain.non_persistent.value(),
+NamespaceName.SYSTEM_NAMESPACE,
+"loadbalancer-broker-load-data").toString();
+
+public static final String TOP_BUNDLES_LOAD_DATA_STORE_TOPIC =
TopicName.get(
+TopicDomain.non_persistent.value(),
+NamespaceName.SYSTEM_NAMESPACE,
+"loadbalancer-top-bundles-load-data").toString();
+
+private PulsarService pulsar;
+
+private ServiceConfiguration conf;
+
+@Getter
+private BrokerRegistry brokerRegistry;
+
+private ServiceUnitStateChannel serviceUnitStateChannel;
+
+private LoadDataStore brokerLoadDataStore;
+private LoadDataStore topBundlesLoadDataStore;
Review Comment:
Sorry, I forgot to close the load data store. Now it is used in `start` and
`close` method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on a diff in pull request #19102: [improve][broker] PIP-192: Implement extensible load manager
BewareMyPower commented on code in PR #19102:
URL: https://github.com/apache/pulsar/pull/19102#discussion_r1094021602
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/ExtensibleLoadManagerImpl.java:
##
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.BrokerFilterException;
+import
org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannel;
+import
org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLoadData;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.broker.loadbalance.extensions.data.TopBundlesLoadData;
+import org.apache.pulsar.broker.loadbalance.extensions.filter.BrokerFilter;
+import org.apache.pulsar.broker.loadbalance.extensions.store.LoadDataStore;
+import
org.apache.pulsar.broker.loadbalance.extensions.store.LoadDataStoreException;
+import
org.apache.pulsar.broker.loadbalance.extensions.store.LoadDataStoreFactory;
+import
org.apache.pulsar.broker.loadbalance.extensions.strategy.BrokerSelectionStrategy;
+import
org.apache.pulsar.broker.loadbalance.extensions.strategy.LeastResourceUsageWithWeight;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.ServiceUnitId;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+
+@Slf4j
+public class ExtensibleLoadManagerImpl implements ExtensibleLoadManager {
+
+public static final String BROKER_LOAD_DATA_STORE_TOPIC = TopicName.get(
+TopicDomain.non_persistent.value(),
+NamespaceName.SYSTEM_NAMESPACE,
+"loadbalancer-broker-load-data").toString();
+
+public static final String TOP_BUNDLES_LOAD_DATA_STORE_TOPIC =
TopicName.get(
+TopicDomain.non_persistent.value(),
+NamespaceName.SYSTEM_NAMESPACE,
+"loadbalancer-top-bundles-load-data").toString();
+
+private PulsarService pulsar;
+
+private ServiceConfiguration conf;
+
+@Getter
+private BrokerRegistry brokerRegistry;
+
+private ServiceUnitStateChannel serviceUnitStateChannel;
+
+private LoadDataStore brokerLoadDataStore;
+private LoadDataStore topBundlesLoadDataStore;
Review Comment:
Just a question, other code LGTM, so I approved it first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on a diff in pull request #18865: [improve][broker] PIP-220 Added TransferShedder
heesung-sn commented on code in PR #18865: URL: https://github.com/apache/pulsar/pull/18865#discussion_r1094001823 ## pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java: ## @@ -2450,6 +2450,64 @@ The delayed message index bucket time step(in seconds) in per bucket snapshot se ) private long namespaceBundleUnloadingTimeoutMs = 6; +@FieldContext( +category = CATEGORY_LOAD_BALANCER, +doc = "Option to enable the debug mode for the load balancer logics. " ++ "The debug mode prints more logs to provide more information " ++ "such as load balance states and decisions. " ++ "(only used in load balancer extension logics)" +) +private boolean loadBalancerDebugModeEnabled = false; Review Comment: That's another way of fine-configuring the log level, and I assume we can control the log level by the package as well. But with this dynamic config, we can more easily turn on/off the debugging logs too. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] gaoran10 commented on a diff in pull request #18865: [improve][broker] PIP-220 Added TransferShedder
gaoran10 commented on code in PR #18865:
URL: https://github.com/apache/pulsar/pull/18865#discussion_r1093997621
##
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java:
##
@@ -2450,6 +2450,64 @@ The delayed message index bucket time step(in seconds)
in per bucket snapshot se
)
private long namespaceBundleUnloadingTimeoutMs = 6;
+@FieldContext(
+category = CATEGORY_LOAD_BALANCER,
+doc = "Option to enable the debug mode for the load balancer
logics. "
++ "The debug mode prints more logs to provide more
information "
++ "such as load balance states and decisions. "
++ "(only used in load balancer extension logics)"
+)
+private boolean loadBalancerDebugModeEnabled = false;
Review Comment:
The Pulsar broker uses the log4j2, which supports custom log configurations
by modifying the config file `${PULSAR_HOME}/conf/log4j2.yaml`, which supports
log control at the class level.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] gaoran10 commented on a diff in pull request #18865: [improve][broker] PIP-220 Added TransferShedder
gaoran10 commented on code in PR #18865:
URL: https://github.com/apache/pulsar/pull/18865#discussion_r1093997621
##
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java:
##
@@ -2450,6 +2450,64 @@ The delayed message index bucket time step(in seconds)
in per bucket snapshot se
)
private long namespaceBundleUnloadingTimeoutMs = 6;
+@FieldContext(
+category = CATEGORY_LOAD_BALANCER,
+doc = "Option to enable the debug mode for the load balancer
logics. "
++ "The debug mode prints more logs to provide more
information "
++ "such as load balance states and decisions. "
++ "(only used in load balancer extension logics)"
+)
+private boolean loadBalancerDebugModeEnabled = false;
Review Comment:
The Pulsar broker uses the log4j2, which supports custom log configurations
by modifying the log config file `${PULSAR_HOME}/conf/log4j2.yaml`, which
supports log control at class level.
##
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java:
##
@@ -2450,6 +2450,64 @@ The delayed message index bucket time step(in seconds)
in per bucket snapshot se
)
private long namespaceBundleUnloadingTimeoutMs = 6;
+@FieldContext(
+category = CATEGORY_LOAD_BALANCER,
+doc = "Option to enable the debug mode for the load balancer
logics. "
++ "The debug mode prints more logs to provide more
information "
++ "such as load balance states and decisions. "
++ "(only used in load balancer extension logics)"
+)
+private boolean loadBalancerDebugModeEnabled = false;
Review Comment:
The Pulsar broker uses the log4j2, which supports custom log configurations
by modifying the log config file `${PULSAR_HOME}/conf/log4j2.yaml`, which
supports log control at class level.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on a diff in pull request #19102: [improve][broker] PIP-192: Implement extensible load manager
BewareMyPower commented on code in PR #19102:
URL: https://github.com/apache/pulsar/pull/19102#discussion_r1093994357
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/ExtensibleLoadManagerImpl.java:
##
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.BrokerFilterException;
+import
org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannel;
+import
org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLoadData;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.broker.loadbalance.extensions.data.TopBundlesLoadData;
+import org.apache.pulsar.broker.loadbalance.extensions.filter.BrokerFilter;
+import org.apache.pulsar.broker.loadbalance.extensions.store.LoadDataStore;
+import
org.apache.pulsar.broker.loadbalance.extensions.store.LoadDataStoreException;
+import
org.apache.pulsar.broker.loadbalance.extensions.store.LoadDataStoreFactory;
+import
org.apache.pulsar.broker.loadbalance.extensions.strategy.BrokerSelectionStrategy;
+import
org.apache.pulsar.broker.loadbalance.extensions.strategy.LeastResourceUsageWithWeight;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.ServiceUnitId;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+
+@Slf4j
+public class ExtensibleLoadManagerImpl implements ExtensibleLoadManager {
+
+public static final String BROKER_LOAD_DATA_STORE_TOPIC = TopicName.get(
+TopicDomain.non_persistent.value(),
+NamespaceName.SYSTEM_NAMESPACE,
+"loadbalancer-broker-load-data").toString();
+
+public static final String TOP_BUNDLES_LOAD_DATA_STORE_TOPIC =
TopicName.get(
+TopicDomain.non_persistent.value(),
+NamespaceName.SYSTEM_NAMESPACE,
+"loadbalancer-top-bundles-load-data").toString();
+
+private PulsarService pulsar;
+
+private ServiceConfiguration conf;
+
+@Getter
+private BrokerRegistry brokerRegistry;
+
+private ServiceUnitStateChannel serviceUnitStateChannel;
+
+private LoadDataStore brokerLoadDataStore;
+private LoadDataStore topBundlesLoadDataStore;
Review Comment:
These two fields are only used in `start()`, will them be used somewhere
else in future? If not, we should change them to local variables.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar-site] Anonymitaet merged pull request #397: [BLOG] Add the Pulsar Summit Europe CFP blog
Anonymitaet merged PR #397: URL: https://github.com/apache/pulsar-site/pull/397 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on a diff in pull request #18865: [improve][broker] PIP-220 Added TransferShedder
heesung-sn commented on code in PR #18865: URL: https://github.com/apache/pulsar/pull/18865#discussion_r1093991281 ## pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java: ## @@ -2450,6 +2450,64 @@ The delayed message index bucket time step(in seconds) in per bucket snapshot se ) private long namespaceBundleUnloadingTimeoutMs = 6; +@FieldContext( +category = CATEGORY_LOAD_BALANCER, +doc = "Option to enable the debug mode for the load balancer logics. " ++ "The debug mode prints more logs to provide more information " ++ "such as load balance states and decisions. " ++ "(only used in load balancer extension logics)" +) +private boolean loadBalancerDebugModeEnabled = false; Review Comment: I think we can mark all these configs as dynamic. Updating. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-site] Anonymitaet commented on a diff in pull request #396: docs: contributing runbook
Anonymitaet commented on code in PR #396: URL: https://github.com/apache/pulsar-site/pull/396#discussion_r1093989812 ## CONTRIBUTING.md: ## @@ -0,0 +1,119 @@ +# Contribute to Apache Pulsar Site + +## How-tos + +### How to fix search index mismatches? + +First of all, you should get the permission to access `apache_pulsar` crawler on [Algolia Crawler console](https://crawler.algolia.com/). You can email d...@pulsar.apache.org to ask for permission. + +The most common fix for search index mismatches is to re-index the pages. You can do so by clicking "Restart crawling" button on the [crawler page](https://crawler.algolia.com/admin/crawlers/7a3458ba-2373-47d5-9520-90cc9cc10736/overview). Typically, it takes about 1 or 2 hours to complete. + +### How to update reference pages? + +The source of reference pages consists of: + +* Generated from configuration classes +* Generated from command-line interfaces +* Manually held under `static/referces` folder + +Read [update reference docs](https://pulsar.apache.org/contribute/document-contribution/#update-reference-docs) guide for details. Review Comment: Suggest consolidating this part into [update reference docs - guide](https://pulsar.apache.org/contribute/document-contribution/#update-reference-docs) to keep a single source of truth. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] Demogorgon314 commented on a diff in pull request #18865: [improve][broker] PIP-220 Added TransferShedder
Demogorgon314 commented on code in PR #18865: URL: https://github.com/apache/pulsar/pull/18865#discussion_r1093988251 ## pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java: ## @@ -2450,6 +2450,64 @@ The delayed message index bucket time step(in seconds) in per bucket snapshot se ) private long namespaceBundleUnloadingTimeoutMs = 6; +@FieldContext( +category = CATEGORY_LOAD_BALANCER, +doc = "Option to enable the debug mode for the load balancer logics. " ++ "The debug mode prints more logs to provide more information " ++ "such as load balance states and decisions. " ++ "(only used in load balancer extension logics)" +) +private boolean loadBalancerDebugModeEnabled = false; Review Comment: Should we mark this configuration as dynamic? Then we can change this configuration in real time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] onobc commented on pull request #18358: [fix][client] Set authentication when using loadConf in client and admin client
onobc commented on PR #18358: URL: https://github.com/apache/pulsar/pull/18358#issuecomment-1413118657 > @onobc there's a test failing : `testAuthTokenClientConfig`. Can you check ? Thx for heads up @cbornet - I will check as soon as we get power/internet back on in area in the next 24-48hrs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] Demogorgon314 commented on a diff in pull request #19102: [improve][broker] PIP-192: Implement extensible load manager
Demogorgon314 commented on code in PR #19102: URL: https://github.com/apache/pulsar/pull/19102#discussion_r1093986856 ## pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java: ## @@ -0,0 +1,222 @@ +/* Review Comment: The PR https://github.com/apache/pulsar/pull/18810 is merged. We can continue the review now : ) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-python] BewareMyPower opened a new pull request, #91: Update the release process for versioning
BewareMyPower opened a new pull request, #91: URL: https://github.com/apache/pulsar-client-python/pull/91 ### Motivation Adopt the same versioning rule with the Node.js client, see https://github.com/apache/pulsar-client-node/pull/287. Add an extra step to commit the version update directly before pushing the tag. For example, https://github.com/apache/pulsar-client-python/commit/fda50867a9c7bf927309527fade2f53eb3907bed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] Demogorgon314 commented on a diff in pull request #19102: [improve][broker] PIP-192: Implement extensible load manager
Demogorgon314 commented on code in PR #19102:
URL: https://github.com/apache/pulsar/pull/19102#discussion_r1093985988
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/ExtensibleLoadManagerImpl.java:
##
@@ -0,0 +1,232 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ThreadLocalRandom;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarServerException;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.loadbalance.BrokerFilterException;
+import
org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannel;
+import
org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl;
+import org.apache.pulsar.broker.loadbalance.extensions.data.BrokerLookupData;
+import org.apache.pulsar.broker.loadbalance.extensions.filter.BrokerFilter;
+import
org.apache.pulsar.broker.loadbalance.extensions.strategy.BrokerSelectionStrategy;
+import org.apache.pulsar.common.naming.ServiceUnitId;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap;
+
+@Slf4j
+public class ExtensibleLoadManagerImpl implements ExtensibleLoadManager {
+
+private PulsarService pulsar;
+
+private ServiceConfiguration conf;
+
+@Getter
+private BrokerRegistry brokerRegistry;
+
+private ServiceUnitStateChannel serviceUnitStateChannel;
+
+@Getter
+private LoadManagerContext context;
+
+@Getter
+private final BrokerSelectionStrategy brokerSelectionStrategy;
+
+@Getter
+private final List brokerFilterPipeline;
+
+private boolean started = false;
+
+private final ConcurrentOpenHashMap>>
+lookupRequests = ConcurrentOpenHashMap.>>newBuilder()
+.build();
+
+/**
+ * Life cycle: Constructor -> initialize -> start -> close.
+ */
+public ExtensibleLoadManagerImpl() {
+this.brokerFilterPipeline = new ArrayList<>();
+// TODO: Make brokerSelectionStrategy configurable.
+this.brokerSelectionStrategy = (brokers, bundle, context) -> {
+if (brokers.isEmpty()) {
+return Optional.empty();
+}
+return
Optional.of(brokers.get(ThreadLocalRandom.current().nextInt(brokers.size(;
+};
+}
+
+public static boolean isLoadManagerExtensionEnabled(ServiceConfiguration
conf) {
+return
ExtensibleLoadManagerImpl.class.getName().equals(conf.getLoadManagerClassName());
+}
+
+@Override
+public synchronized void start() throws PulsarServerException {
+if (this.started) {
+return;
+}
+this.brokerRegistry = new BrokerRegistryImpl(pulsar);
+this.serviceUnitStateChannel = new ServiceUnitStateChannelImpl(pulsar);
+this.brokerRegistry.start();
+this.serviceUnitStateChannel.start();
+
+// TODO: Start the load data store.
+
+this.context = LoadManagerContextImpl.builder()
+.configuration(conf)
+.brokerRegistry(brokerRegistry)
+.brokerLoadDataStore(null)
+.topBundleLoadDataStore(null).build();
+// TODO: Start load data reporter.
+
+// TODO: Start unload scheduler and bundle split scheduler
+this.started = true;
+}
+
+@Override
+public synchronized void initialize(PulsarService pulsar) {
+this.pulsar = pulsar;
+this.conf = pulsar.getConfiguration();
Review Comment:
I have rechecked the `start`, `initialize`, and `close` methods, the `start`
and `initialize` methods always call in the same thread, so we don't need to
add the synchronized modifier.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail:
[GitHub] [pulsar] freeznet commented on a diff in pull request #18951: [improve][fn] support reading config options from file in Function Python Runner
freeznet commented on code in PR #18951:
URL: https://github.com/apache/pulsar/pull/18951#discussion_r1093982208
##
pulsar-functions/instance/src/main/python/python_instance_main.py:
##
@@ -49,47 +49,160 @@
to_run = True
Log = log.Log
+
def atexit_function(signo, _frame):
global to_run
Log.info("Interrupted by %d, shutting down" % signo)
to_run = False
+
+def merge_arguments(args, config_file):
+ """
+ This function is used to merge arguments passed in via the command line
+ and those passed in via the configuration file during initialization.
+
+ :param args: arguments passed in via the command line
+ :param config_file: configuration file name (path)
+
+ During the merge process, the arguments passed in via the command line have
higher priority,
+ so only optional arguments need to be merged.
+ """
+ if config_file is None:
+return
+ config = util.read_config(config_file)
+ if not config:
+return
+ default_config = config["DEFAULT"]
+ if not default_config:
+return
+ if not args.client_auth_plugin and default_config.get("client_auth_plugin",
None):
Review Comment:
any chance to simplify this function by
[`vars()`](https://docs.python.org/3/library/functions.html#vars)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on a diff in pull request #18865: [improve][broker] PIP-220 Added TransferShedder
heesung-sn commented on code in PR #18865: URL: https://github.com/apache/pulsar/pull/18865#discussion_r1093985502 ## pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java: ## @@ -2450,6 +2450,64 @@ The delayed message index bucket time step(in seconds) in per bucket snapshot se ) private long namespaceBundleUnloadingTimeoutMs = 6; +@FieldContext( +category = CATEGORY_LOAD_BALANCER, +doc = "Option to enable the debug mode for the load balancer logics. " ++ "The debug mode prints more logs to provide more information " ++ "such as load balance states and decisions. " ++ "(only used in load balancer extension logics)" +) +private boolean loadBalancerDebugModeEnabled = false; Review Comment: Some systems do support fine-grained debugging logs for certain features. I am curious if the pulsar follows this practice, but I think this could be useful to see debug logs from load balancer logic only. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-go] panszobe opened a new issue, #949: Consume Performance drops when set EnableBatchIndexAcknowledgment = true
panszobe opened a new issue, #949:
URL: https://github.com/apache/pulsar-client-go/issues/949
According to PR: #938
use master version(v0.9.1-0.20230117072740-d9b18d0690c1) to consume messages
while EnableBatchIndexAcknowledgment set
true，but **consume performance drops to 2/3 of previous**。
The test situation is as follows:
Topic has 5 partitions, producer production rate is 20MB/s , 30 rows/s.
and consumers consume situations:
| SDK Version | Enable Batch Index Ack | Consumer Instances | Consume Rate |
| - | - | - | - |
| v0.9.1-0.20230117072740-d9b18d0690c1 | Yes | 3 | 10 rows/s |
| v0.9.1-0.20230117072740-d9b18d0690c1 | No | 3 | 30+ rows/s |
Analyze the problem by pprof，we found that
internal.(*connection).internalSendRequest and
pulsar.(*partitionConsumer).internalAck are much more resource intensive when
set EnableBatchIndexAcknowledgment as true.
Review the code:
```
func (pc *partitionConsumer) ackID(msgID MessageID, withResponse bool) error
{
if state := pc.getConsumerState(); state == consumerClosed || state ==
consumerClosing {
pc.log.WithField("state", state).Error("Failed to ack by
closing or closed consumer")
return errors.New("consumer state is closed")
}
if cmid, ok := toChunkedMessageID(msgID); ok {
return pc.unAckChunksTracker.ack(cmid)
}
trackingID, ok := toTrackingMessageID(msgID)
if !ok {
return errors.New("failed to convert trackingMessageID")
}
ackReq := new(ackRequest)
ackReq.doneCh = make(chan struct{})
ackReq.ackType = individualAck
if !trackingID.Undefined() && trackingID.ack() {
pc.metrics.AcksCounter.Inc()
pc.metrics.ProcessingTime.Observe(float64(time.Now().UnixNano()-trackingID.receivedTime.UnixNano())
/ 1.0e9)
ackReq.msgID = trackingID
// send ack request to eventsCh
pc.eventsCh <- ackReq
if withResponse {
<-ackReq.doneCh
}
pc.options.interceptors.OnAcknowledge(pc.parentConsumer, msgID)
} else if pc.options.enableBatchIndexAck {
ackReq.msgID = trackingID
pc.eventsCh <- ackReq
}
if withResponse {
return ackReq.err
}
return nil
}
```
Maybe problem is that partitionConsumer will send ack request to *Pulsar
Server* by every MessageID, without waiting all msg of one batch be acked by
ackTracker, it leads to ack requests becoming much more than BatchIndexAck
disabled, performance drops bacause of much more processing requests. And
backlog is lasting increasing, could not catch up with the production rate.
So, enableBatchIndexAck should follow the previous processing method or
there is another way.
@BewareMyPower Thank you for developing this feature. Could you take a look
at this problem ? Thanks a lot!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] gaoran10 commented on a diff in pull request #18865: [improve][broker] PIP-220 Added TransferShedder
gaoran10 commented on code in PR #18865: URL: https://github.com/apache/pulsar/pull/18865#discussion_r1093982471 ## pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java: ## @@ -2450,6 +2450,64 @@ The delayed message index bucket time step(in seconds) in per bucket snapshot se ) private long namespaceBundleUnloadingTimeoutMs = 6; +@FieldContext( +category = CATEGORY_LOAD_BALANCER, +doc = "Option to enable the debug mode for the load balancer logics. " ++ "The debug mode prints more logs to provide more information " ++ "such as load balance states and decisions. " ++ "(only used in load balancer extension logics)" +) +private boolean loadBalancerDebugModeEnabled = false; Review Comment: Does this config is necessary? Maybe we can do the same thing via modify the broker log config file. ## pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java: ## @@ -2450,6 +2450,64 @@ The delayed message index bucket time step(in seconds) in per bucket snapshot se ) private long namespaceBundleUnloadingTimeoutMs = 6; +@FieldContext( +category = CATEGORY_LOAD_BALANCER, +doc = "Option to enable the debug mode for the load balancer logics. " ++ "The debug mode prints more logs to provide more information " ++ "such as load balance states and decisions. " ++ "(only used in load balancer extension logics)" +) +private boolean loadBalancerDebugModeEnabled = false; Review Comment: Does this config is necessary? Maybe we can do the same thing via modifying the broker log config file. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] gaoran10 commented on a diff in pull request #18865: [improve][broker] PIP-220 Added TransferShedder
gaoran10 commented on code in PR #18865: URL: https://github.com/apache/pulsar/pull/18865#discussion_r1093982471 ## pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java: ## @@ -2450,6 +2450,64 @@ The delayed message index bucket time step(in seconds) in per bucket snapshot se ) private long namespaceBundleUnloadingTimeoutMs = 6; +@FieldContext( +category = CATEGORY_LOAD_BALANCER, +doc = "Option to enable the debug mode for the load balancer logics. " ++ "The debug mode prints more logs to provide more information " ++ "such as load balance states and decisions. " ++ "(only used in load balancer extension logics)" +) +private boolean loadBalancerDebugModeEnabled = false; Review Comment: Does this config is necessary? Maybe we can do the same thing via modify log configurations. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on a diff in pull request #19300: [improve][cli] improve admin `set-backlog-quota` more clear
codelipenghui commented on code in PR #19300:
URL: https://github.com/apache/pulsar/pull/19300#discussion_r1093975003
##
pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdTopicPolicies.java:
##
@@ -977,8 +977,8 @@ private class SetBacklogQuota extends CliCommand {
private String policyStr;
@Parameter(names = {"-t", "--type"}, description = "Backlog quota type
to set. Valid options are: "
-+ "destination_storage and message_age. "
-+ "destination_storage limits backlog by size (in bytes). "
++ "destination_storage (default) and message_age. "
Review Comment:
The default is backlog size, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[pulsar-client-python] tag v3.1.0-candidate-2 created (now fda5086)
This is an automated email from the ASF dual-hosted git repository. xyz pushed a change to tag v3.1.0-candidate-2 in repository https://gitbox.apache.org/repos/asf/pulsar-client-python.git at fda5086 (commit) No new revisions were added by this update.
[pulsar-client-python] branch branch-3.1 updated: Bump version to 3.1.0
This is an automated email from the ASF dual-hosted git repository. xyz pushed a commit to branch branch-3.1 in repository https://gitbox.apache.org/repos/asf/pulsar-client-python.git The following commit(s) were added to refs/heads/branch-3.1 by this push: new fda5086 Bump version to 3.1.0 fda5086 is described below commit fda50867a9c7bf927309527fade2f53eb3907bed Author: Yunze Xu AuthorDate: Thu Feb 2 11:23:49 2023 +0800 Bump version to 3.1.0 --- pulsar/__about__.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pulsar/__about__.py b/pulsar/__about__.py index a66d247..a91aa25 100644 --- a/pulsar/__about__.py +++ b/pulsar/__about__.py @@ -16,4 +16,4 @@ # specific language governing permissions and limitations # under the License. # -__version__='3.1.0a1' +__version__='3.1.0'
[GitHub] [pulsar] nodece commented on a diff in pull request #19359: [fix][authentication] Make AuthenticationProviderBasic refreshes data and role
nodece commented on code in PR #19359:
URL: https://github.com/apache/pulsar/pull/19359#discussion_r1093971890
##
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/authentication/AuthenticationProviderBasic.java:
##
@@ -177,4 +196,55 @@ public String getPassword() {
return password;
}
}
+
+private static class BasicAuthenticationState implements
AuthenticationState {
+
+private final AuthenticationProviderBasic authenticationProvider;
+private final SocketAddress remoteAddress;
+private final SSLSession sslSession;
+private String role;
+private AuthenticationDataSource authenticationDataSource;
+
+public BasicAuthenticationState(SocketAddress remoteAddress,
SSLSession sslSession,
+AuthenticationProviderBasic
authenticationProvider) {
+this.remoteAddress = remoteAddress;
+this.sslSession = sslSession;
+this.authenticationProvider = authenticationProvider;
+}
+
+@Override
+public String getAuthRole() throws AuthenticationException {
+if (authenticationDataSource == null) {
+throw new AuthenticationException("Must authenticate before
calling getAuthRole");
+}
+
+return role;
+}
+
+@Override
+public AuthData authenticate(AuthData authData) throws
AuthenticationException {
+AuthenticationDataSource dataSource =
+new AuthenticationDataCommand(new
String(authData.getBytes(), StandardCharsets.UTF_8),
+remoteAddress, sslSession);
+role = authenticationProvider.authenticationBasic(dataSource);
+authenticationDataSource = dataSource;
+
+return null;
+}
+
+@Override
+public AuthenticationDataSource getAuthDataSource() {
+return authenticationDataSource;
+}
+
+@Override
+public boolean isComplete() {
+return authenticationDataSource != null;
+}
+
+@Override
+public boolean isExpired() {
+return false;
+}
Review Comment:
Good explanation!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] nodece closed pull request #19359: [fix][authentication] Make AuthenticationProviderBasic refreshes data and role
nodece closed pull request #19359: [fix][authentication] Make AuthenticationProviderBasic refreshes data and role URL: https://github.com/apache/pulsar/pull/19359 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on a diff in pull request #19153: [improve] PIP-241: add TopicEventListener / topic events for the BrokerService
BewareMyPower commented on code in PR #19153:
URL: https://github.com/apache/pulsar/pull/19153#discussion_r1093955392
##
pulsar-broker/src/test/java/org/apache/pulsar/broker/TopicEventsListenerTest.java:
##
@@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker;
+
+import com.google.common.collect.Sets;
+
+import java.util.Queue;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.TimeUnit;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.ArrayUtils;
+import org.apache.pulsar.broker.service.BrokerTestBase;
+import org.apache.pulsar.client.admin.PulsarAdmin;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.Consumer;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.PulsarClientException;
+import org.apache.pulsar.common.policies.data.InactiveTopicDeleteMode;
+import org.apache.pulsar.common.policies.data.InactiveTopicPolicies;
+import org.apache.pulsar.common.policies.data.RetentionPolicies;
+import org.awaitility.Awaitility;
+import org.testng.Assert;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.AfterMethod;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.DataProvider;
+import org.testng.annotations.Test;
+
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertTrue;
+
+@Slf4j
+public class TopicEventsListenerTest extends BrokerTestBase {
+
+final static Queue events = new ConcurrentLinkedQueue<>();
+volatile String topicNameToWatch;
+String namespace;
+
+@DataProvider(name = "topicType")
+public static Object[][] topicType() {
+return new Object[][] {
+{"persistent", "partitioned", Boolean.TRUE},
+{"persistent", "non-partitioned", Boolean.TRUE},
+{"non-persistent", "partitioned", Boolean.TRUE},
+{"non-persistent", "non-partitioned", Boolean.TRUE},
+{"persistent", "partitioned", Boolean.FALSE},
+{"persistent", "non-partitioned", Boolean.FALSE},
+{"non-persistent", "partitioned", Boolean.FALSE},
+{"non-persistent", "non-partitioned", Boolean.FALSE}
+};
+}
+
+@DataProvider(name = "topicTypeNoDelete")
+public static Object[][] topicTypeNoDelete() {
+return new Object[][] {
+{"persistent", "partitioned"},
+{"persistent", "non-partitioned"},
+{"non-persistent", "partitioned"},
+{"non-persistent", "non-partitioned"}
+};
+}
+
+@BeforeClass
+@Override
+protected void setup() throws Exception {
+super.baseSetup();
+pulsar.getConfiguration().setForceDeleteNamespaceAllowed(true);
+
+pulsar.getBrokerService().addTopicEventListener((topic, event, stage,
t) -> {
+log.info("got event {}__{} for topic {}", event, stage, topic);
+if (topic.equals(topicNameToWatch)) {
+if (log.isDebugEnabled()) {
+log.debug("got event {}__{} for topic {} with detailed
stack",
+event, stage, topic, new Exception("tracing event
source"));
+}
+events.add(event.toString() + "__" + stage.toString());
+}
+});
+}
+
+@AfterClass(alwaysRun = true)
+@Override
+protected void cleanup() throws Exception {
+super.internalCleanup();
+}
+
+@BeforeMethod
+protected void setupTest() throws Exception {
+namespace = "prop/" + UUID.randomUUID();
+admin.namespaces().createNamespace(namespace, Sets.newHashSet("test"));
+
assertTrue(admin.namespaces().getNamespaces("prop").contains(namespace));
+admin.namespaces().setRetention(namespace, new RetentionPolicies(3,
10));
+try (PulsarAdmin admin2 = createPulsarAdmin()) {
+Awaitility.await().untilAsserted(() ->
+assertEquals(admin2.namespaces().getRetention(namespace),
[GitHub] [pulsar] Technoboy- closed pull request #19300: [improve][cli] improve admin `set-backlog-quota` more clear
Technoboy- closed pull request #19300: [improve][cli] improve admin `set-backlog-quota` more clear URL: https://github.com/apache/pulsar/pull/19300 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #18985: PIP-232: Introduce thread monitor to check if thread is blocked for long time.
github-actions[bot] commented on issue #18985: URL: https://github.com/apache/pulsar/issues/18985#issuecomment-1413043656 The issue had no activity for 30 days, mark with Stale label. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-site] sijie commented on pull request #397: [BLOG] Add the Pulsar Summit Europe CFP blog
sijie commented on PR #397: URL: https://github.com/apache/pulsar-site/pull/397#issuecomment-1413041667 @tisonkun I believe the Pulsar PMC has approved all the Pulsar Summit events for 2023. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-site] Sherlock113 commented on pull request #397: [BLOG] Add the Pulsar Summit Europe CFP blog
Sherlock113 commented on PR #397: URL: https://github.com/apache/pulsar-site/pull/397#issuecomment-1413038919 > The content looks good. > > May I ask if there's any background about this event? I can approve if I checked the resolution in PMC. The summit website is already updated for Europe Summit https://pulsar-summit.org/. You can find related links there. As for the CFP itself, all the background info can be found here https://sessionize.com/pulsar-virtual-summit-europe-2023/. These links are provided in the blog as well. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-site] Sherlock113 commented on pull request #397: [BLOG] Add the Pulsar Summit Europe CFP blog
Sherlock113 commented on PR #397: URL: https://github.com/apache/pulsar-site/pull/397#issuecomment-1413034869 @tisonkun @urfreespace @Anonymitaet PTAL. We hope to publish it asap. Thank you very much! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-site] Sherlock113 opened a new pull request, #397: Add the Pulsar Summit Europe CFP blog
Sherlock113 opened a new pull request, #397: URL: https://github.com/apache/pulsar-site/pull/397 ### Documentation - [ ] `doc` - [ ] `doc-required` - [x] `doc-not-needed` - [ ] `doc-complete` Submitted a blog about Pulsar Summit Europe CFP. Previews:     -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] tisonkun commented on pull request #17151: [fix][broker] Added check for invisible characters for subscription name
tisonkun commented on PR #17151: URL: https://github.com/apache/pulsar/pull/17151#issuecomment-1413030094 Closed as no consensus. I see @mattisonchao started a discussion about topic names on the mailing list: https://lists.apache.org/thread/5s4kop7qmxgpvsnh65s42r9mv7qc1pxt @lordcheng10 if you'd like to continue this work, you can follow this pattern to discuss and get more attention first. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] tisonkun closed pull request #17151: [fix][broker] Added check for invisible characters for subscription name
tisonkun closed pull request #17151: [fix][broker] Added check for invisible characters for subscription name URL: https://github.com/apache/pulsar/pull/17151 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar-site] branch main updated: [fix][doc] PMC -> PMC members
This is an automated email from the ASF dual-hosted git repository.
tison pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/pulsar-site.git
The following commit(s) were added to refs/heads/main by this push:
new 9dc279660ab [fix][doc] PMC -> PMC members
9dc279660ab is described below
commit 9dc279660ab06c80d25ea88e26722b50154bf48e
Author: tison
AuthorDate: Thu Feb 2 09:24:26 2023 +0800
[fix][doc] PMC -> PMC members
This refers to https://github.com/apache/pulsar/issues/17900
---
src/pages/community.tsx | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/pages/community.tsx b/src/pages/community.tsx
index af9eea30b5b..c85a254796f 100644
--- a/src/pages/community.tsx
+++ b/src/pages/community.tsx
@@ -541,7 +541,7 @@ export default function Community(): JSX.Element {
.
-PMC
+PMC members
[GitHub] [pulsar] BewareMyPower commented on a diff in pull request #19153: [improve] PIP-241: add TopicEventListener / topic events for the BrokerService
BewareMyPower commented on code in PR #19153:
URL: https://github.com/apache/pulsar/pull/19153#discussion_r1093916553
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/TopicEventsDispatcher.java:
##
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.service;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Objects;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CopyOnWriteArrayList;
+import lombok.extern.slf4j.Slf4j;
+
+/**
+ * Utility class to dispatch topic events.
+ */
+@Slf4j
+public class TopicEventsDispatcher {
+private final List topicEventListeners = new
CopyOnWriteArrayList<>();
+
+/**
+ * Adds listeners, ignores null listeners.
+ * @param listeners
+ */
+public void addTopicEventListener(TopicEventsListener... listeners) {
+Objects.requireNonNull(listeners);
+Arrays.stream(listeners)
+.filter(x -> x != null)
+.forEach(topicEventListeners::add);
+}
+
+/**
+ * Removes listeners.
+ * @param listeners
+ */
+public void removeTopicEventListener(TopicEventsListener... listeners) {
+Objects.requireNonNull(listeners);
+Arrays.stream(listeners)
+.filter(x -> x != null)
+.forEach(topicEventListeners::remove);
+}
+
+/**
+ * Dispatches notification to all currently added listeners.
+ * @param topic
+ * @param event
+ * @param stage
+ */
+public void notify(String topic,
+ TopicEventsListener.TopicEvent event,
+ TopicEventsListener.EventStage stage) {
+notify(topic, event, stage, null);
+}
+
+/**
+ * Dispatches notification to all currently added listeners.
+ * @param topic
+ * @param event
+ * @param stage
+ * @param t
+ */
+public void notify(String topic,
+ TopicEventsListener.TopicEvent event,
+ TopicEventsListener.EventStage stage,
+ Throwable t) {
+topicEventListeners
+.forEach(listener -> notify(listener, topic, event, stage, t));
+}
+
+/**
+ * Dispatches SUCCESS/FAILURE notification to all currently added
listeners on completion of the future.
+ * @param future
+ * @param topic
+ * @param event
+ * @param
+ * @return future of a new completion stage
+ */
+public CompletableFuture notifyOnCompletion(CompletableFuture
future,
+ String topic,
+
TopicEventsListener.TopicEvent event) {
+return future.whenComplete((r, ex) -> notify(topic,
+event,
+ex == null ? TopicEventsListener.EventStage.SUCCESS :
TopicEventsListener.EventStage.FAILURE,
+ex));
+}
Review Comment:
Good explanation. It makes sense to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar-client-python] merlimat commented on a diff in pull request #90: [fix] exception bug as per #89
merlimat commented on code in PR #90:
URL:
https://github.com/apache/pulsar-client-python/pull/90#discussion_r1093871144
##
pulsar/schema/schema_avro.py:
##
@@ -82,15 +82,6 @@ def decode(self, data):
return self._record_cls(**d)
else:
return d
-
else:
-class AvroSchema(Schema):
-def __init__(self, _record_cls, _schema_definition=None):
-raise Exception("Avro library support was not found. Make sure to
install Pulsar client " +
+raise Exception("Avro library support was not found. Make sure to install
Pulsar client " +
Review Comment:
@keenborder786 How would that solve the issue? There shouldn't be any
practical difference between the 2 version
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] chenjing1294 commented on issue #14790: Topic stats cannot work with transaction_coordinator_assign topic
chenjing1294 commented on issue #14790:
URL: https://github.com/apache/pulsar/issues/14790#issuecomment-1412934248
I still have this problem in apache-pulsar-2.11.0.
```
[GET]
http://node1:8080/admin/v2/persistent/pulsar/system/transaction_coordinator_assign/internalStats
```
```json
{
"reason": "Can not create transaction system topic
persistent://pulsar/system/transaction_coordinator_assign"
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] merlimat commented on issue #12944: ARM based docker image
merlimat commented on issue #12944: URL: https://github.com/apache/pulsar/issues/12944#issuecomment-1412928644 As of now, you can build ARM64 image, though we don't yet have multi-arch image. It needs some work on the maven-docker plugin to do so -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-python] keenborder786 commented on a diff in pull request #90: [fix] exception bug as per #89
keenborder786 commented on code in PR #90:
URL:
https://github.com/apache/pulsar-client-python/pull/90#discussion_r1093848818
##
pulsar/schema/schema_avro.py:
##
@@ -82,15 +82,6 @@ def decode(self, data):
return self._record_cls(**d)
else:
return d
-
else:
-class AvroSchema(Schema):
-def __init__(self, _record_cls, _schema_definition=None):
-raise Exception("Avro library support was not found. Make sure to
install Pulsar client " +
+raise Exception("Avro library support was not found. Make sure to install
Pulsar client " +
Review Comment:
As you will see that I am still keeping the class but rather than using the
boolean variable HAS_AVRO and then using if else to raise the exception, I am
creating the AvroSchema class in the main try-except block. This not only
resolves my issue as well mentioned in #89, but also deals with the point that
you raised above. Plus, the HAS_AVRO variable and using if-else seems like
extra work in the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar-client-python] erichare commented on a diff in pull request #90: [fix] exception bug as per #89
erichare commented on code in PR #90:
URL:
https://github.com/apache/pulsar-client-python/pull/90#discussion_r1093847507
##
pulsar/schema/schema_avro.py:
##
@@ -82,15 +82,6 @@ def decode(self, data):
return self._record_cls(**d)
else:
return d
-
else:
-class AvroSchema(Schema):
-def __init__(self, _record_cls, _schema_definition=None):
-raise Exception("Avro library support was not found. Make sure to
install Pulsar client " +
+raise Exception("Avro library support was not found. Make sure to install
Pulsar client " +
Review Comment:
This looks much better to me @keenborder786 . does it fix the original bug
you reported on your end? I wasn't able to re-produce it yet but i had a
slightly older version of the client, so I'm trying it out shortly. But as far
as the issue with the import, i think this should do the trick...
@merlimat does this look good to you?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar-client-python] keenborder786 commented on a diff in pull request #90: [fix] exception bug as per #89
keenborder786 commented on code in PR #90:
URL:
https://github.com/apache/pulsar-client-python/pull/90#discussion_r1093845654
##
pulsar/schema/schema_avro.py:
##
@@ -82,15 +82,6 @@ def decode(self, data):
return self._record_cls(**d)
else:
return d
-
else:
-class AvroSchema(Schema):
-def __init__(self, _record_cls, _schema_definition=None):
-raise Exception("Avro library support was not found. Make sure to
install Pulsar client " +
+raise Exception("Avro library support was not found. Make sure to install
Pulsar client " +
Review Comment:
@erichare , okay I understand the point that you have raised about
AvroSchema Class being used in other places and exceptions being raised when
fastavro has not been installed. Therefore I have remodified my PR. Can you
please review the latest commit? Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] cbornet commented on pull request #18358: [fix][client] Set authentication when using loadConf in client and admin client
cbornet commented on PR #18358: URL: https://github.com/apache/pulsar/pull/18358#issuecomment-1412890037 @onobc there's a test failing : `testAuthTokenClientConfig`. Can you check ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-python] erichare commented on a diff in pull request #90: [fix] exception bug as per #89
erichare commented on code in PR #90:
URL:
https://github.com/apache/pulsar-client-python/pull/90#discussion_r1093834455
##
pulsar/schema/schema_avro.py:
##
@@ -82,15 +82,6 @@ def decode(self, data):
return self._record_cls(**d)
else:
return d
-
else:
-class AvroSchema(Schema):
-def __init__(self, _record_cls, _schema_definition=None):
-raise Exception("Avro library support was not found. Make sure to
install Pulsar client " +
+raise Exception("Avro library support was not found. Make sure to install
Pulsar client " +
Review Comment:
@keenborder786 the build checks are failing because we're importing
everything from the schema module:
`from pulsar.schema import *`
Basically the issue is just that if `fastavro` isn't available at import
time, then the exception is raised... now someone actually intending to import
AvroSchema, of course may want to know about the missing dependency right away
(before initializing an instance of the class) - But i guess like the tests
they could be trying to import everything - not the best practice of course.
Still, I imagine we don't want to break that (i could see some users without
fastavro also importing the entire schema module)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar-client-python] erichare commented on a diff in pull request #90: [fix] exception bug as per #89
erichare commented on code in PR #90:
URL:
https://github.com/apache/pulsar-client-python/pull/90#discussion_r1093834455
##
pulsar/schema/schema_avro.py:
##
@@ -82,15 +82,6 @@ def decode(self, data):
return self._record_cls(**d)
else:
return d
-
else:
-class AvroSchema(Schema):
-def __init__(self, _record_cls, _schema_definition=None):
-raise Exception("Avro library support was not found. Make sure to
install Pulsar client " +
+raise Exception("Avro library support was not found. Make sure to install
Pulsar client " +
Review Comment:
@keenborder786 the build checks are failing because we're importing
everything from the schema:
`from pulsar.schema import *`
Basically the issue is just that if `fastavro` isn't available at import
time, then the exception is raised... now someone actually intending to import
AvroSchema, of course may want to know about the missing dependency right away
(before initializing an instance of the class) - But i guess like the tests
they could be trying to import everything - not the best practice of course.
Still, I imagine we don't want to break that (i could see some users without
fastavro also importing the entire schema module)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar-client-python] erichare commented on a diff in pull request #90: [fix] exception bug as per #89
erichare commented on code in PR #90:
URL:
https://github.com/apache/pulsar-client-python/pull/90#discussion_r1093821836
##
pulsar/schema/schema_avro.py:
##
@@ -82,15 +82,6 @@ def decode(self, data):
return self._record_cls(**d)
else:
return d
-
else:
-class AvroSchema(Schema):
-def __init__(self, _record_cls, _schema_definition=None):
-raise Exception("Avro library support was not found. Make sure to
install Pulsar client " +
+raise Exception("Avro library support was not found. Make sure to install
Pulsar client " +
Review Comment:
The bug is an interesting finding... but yeah, right now AvroSchema gets
imported in the schema init, and attempting to do so without fastavro installed
(and importable) will lead to this exception being raised, when really it
should only be raised when attempting to initialize a new AvroSchema instance.
So i don't think this should be merged in as is.
With that said, why you're experiencing the bug that you are @keenborder786
, no clue when it comes to that yet...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar-client-python] keenborder786 commented on a diff in pull request #90: [fix] exception bug as per #89
keenborder786 commented on code in PR #90:
URL:
https://github.com/apache/pulsar-client-python/pull/90#discussion_r1093821941
##
pulsar/schema/schema_avro.py:
##
@@ -82,15 +82,6 @@ def decode(self, data):
return self._record_cls(**d)
else:
return d
-
else:
-class AvroSchema(Schema):
-def __init__(self, _record_cls, _schema_definition=None):
-raise Exception("Avro library support was not found. Make sure to
install Pulsar client " +
+raise Exception("Avro library support was not found. Make sure to install
Pulsar client " +
Review Comment:
Nope, I tested it out with other Schemas and it works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar-client-python] erichare commented on a diff in pull request #90: [fix] exception bug as per #89
erichare commented on code in PR #90:
URL:
https://github.com/apache/pulsar-client-python/pull/90#discussion_r1093821836
##
pulsar/schema/schema_avro.py:
##
@@ -82,15 +82,6 @@ def decode(self, data):
return self._record_cls(**d)
else:
return d
-
else:
-class AvroSchema(Schema):
-def __init__(self, _record_cls, _schema_definition=None):
-raise Exception("Avro library support was not found. Make sure to
install Pulsar client " +
+raise Exception("Avro library support was not found. Make sure to install
Pulsar client " +
Review Comment:
The bug is an interesting finding... but yeah, right now AvroSchema gets
imported in various places, and attempting to do so without fastavro installed
(and importable) will lead to this exception being raised, when really it
should only be raised when attempting to initialize a new AvroSchema. So i
don't think this should be merged in as is.
With that said, why you're experiencing the bug that you are @keenborder786
, no clue when it comes to that yet...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on pull request #18865: [improve][broker] PIP-220 Added TransferShedder
heesung-sn commented on PR #18865: URL: https://github.com/apache/pulsar/pull/18865#issuecomment-1412858746 Please continue the review. I applied the changes from https://github.com/apache/pulsar/pull/19154. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] zhongduo commented on issue #12944: ARM based docker image
zhongduo commented on issue #12944: URL: https://github.com/apache/pulsar/issues/12944#issuecomment-1412854860 Any chance to support ARM64 docker in dockerhub? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] dlg99 commented on a diff in pull request #19153: [improve] PIP-241: add TopicEventListener / topic events for the BrokerService
dlg99 commented on code in PR #19153:
URL: https://github.com/apache/pulsar/pull/19153#discussion_r1093793450
##
pulsar-broker/src/test/java/org/apache/pulsar/broker/TopicEventsListenerTest.java:
##
@@ -0,0 +1,672 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker;
+
+import com.google.common.collect.Sets;
+import java.util.Queue;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.TimeUnit;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.service.BrokerTestBase;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.common.policies.data.InactiveTopicDeleteMode;
+import org.apache.pulsar.common.policies.data.InactiveTopicPolicies;
+import org.apache.pulsar.common.policies.data.RetentionPolicies;
+import org.awaitility.Awaitility;
+import org.testng.Assert;
+import org.testng.annotations.AfterMethod;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+import static org.testng.Assert.assertTrue;
+
+@Slf4j
+public class TopicEventsListenerTest extends BrokerTestBase {
+
+final static Queue events = new ConcurrentLinkedQueue<>();
+String topicNameToWatch;
+String namespace;
+
+@BeforeMethod
+@Override
+protected void setup() throws Exception {
+super.baseSetup();
+pulsar.getConfiguration().setForceDeleteNamespaceAllowed(true);
+
+pulsar.getBrokerService().addTopicEventListener((topic, event, stage,
t) -> {
+log.info("got event {}__{} for topic {}", event, stage, topic);
+if (topic.equals(topicNameToWatch)) {
+if (log.isDebugEnabled()) {
+log.debug("got event {}__{} for topic {} with detailed
stack",
+event, stage, topic, new Exception("tracing event
source"));
+}
+events.add(event.toString() + "__" + stage.toString());
+}
+});
+
+namespace = "prop/" + UUID.randomUUID();
+admin.namespaces().createNamespace(namespace, Sets.newHashSet("test"));
+
assertTrue(admin.namespaces().getNamespaces("prop").contains(namespace));
+admin.namespaces().setRetention(namespace, new RetentionPolicies(3,
10));
+
+events.clear();
+}
+
+@AfterMethod(alwaysRun = true)
+@Override
+protected void cleanup() throws Exception {
+deleteNamespaceWithRetry(namespace, true);
+
+super.internalCleanup();
+}
+
+@Test
+public void testEventsNonPersistentNonPartitionedTopic() throws Exception {
+topicNameToWatch = "non-persistent://" + namespace + "/NP-NP";
+admin.topics().createNonPartitionedTopic(topicNameToWatch);
+
+Awaitility.waitAtMost(10, TimeUnit.SECONDS).untilAsserted(() ->
+Assert.assertEquals(events.toArray(), new String[]{
+"LOAD__BEFORE",
+"LOAD__FAILURE",
+"LOAD__BEFORE",
+"CREATE__BEFORE",
+"CREATE__SUCCESS",
+"LOAD__SUCCESS"
+})
+);
+
+events.clear();
+admin.topics().delete(topicNameToWatch);
+
+Awaitility.waitAtMost(10, TimeUnit.SECONDS).untilAsserted(() ->
+Assert.assertEquals(events.toArray(), new String[]{
+"DELETE__BEFORE",
+"UNLOAD__BEFORE",
+"UNLOAD__SUCCESS",
+"DELETE__SUCCESS"
+})
+);
+}
+
+@Test
+public void testEventsNonPersistentNonPartitionedTopicWithUnload() throws
Exception {
+topicNameToWatch = "non-persistent://" + namespace + "/NP-NP";
+admin.topics().createNonPartitionedTopic(topicNameToWatch);
+
+Awaitility.waitAtMost(10, TimeUnit.SECONDS).untilAsserted(() ->
+Assert.assertEquals(events.toArray(), new String[]{
+"LOAD__BEFORE",
+"LOAD__FAILURE",
+"LOAD__BEFORE",
+"CREATE__BEFORE",
+"CREATE__SUCCESS",
+
[GitHub] [pulsar] clayburn opened a new pull request, #19386: [improve][ci] Update to Gradle Enterprise Maven Extension 1.16.3
clayburn opened a new pull request, #19386: URL: https://github.com/apache/pulsar/pull/19386 ### Motivation This change updates the Gradle Enterprise Maven Extension to 1.16.3 and removes a workaround that was necessary to support Gradle Enterprise Maven Extension 1.16.2. ### Modifications - Update the Gradle Enterprise Maven Extension to 1.16.3 - Revert the workaround needed to support Gradle Enterprise Maven Extension 1.16.2 compatibility with TestNG `@Factory` and `@DataProvider`used in combination. ### Verifying this change - [ ] Make sure that the change passes the CI checks. This change is a trivial rework / code cleanup without any test coverage. ### Does this pull request potentially affect one of the following parts: *If the box was checked, please highlight the changes* - [ ] Dependencies (add or upgrade a dependency) - [ ] The public API - [ ] The schema - [ ] The default values of configurations - [ ] The threading model - [ ] The binary protocol - [ ] The REST endpoints - [ ] The admin CLI options - [ ] The metrics - [ ] Anything that affects deployment ### Documentation - [ ] `doc` - [ ] `doc-required` - [x] `doc-not-needed` - [ ] `doc-complete` ### Matching PR in forked repository PR in forked repository: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on a diff in pull request #19374: [fix][broker] Fix delete namespace fail by a In-flight topic
mattisonchao commented on code in PR #19374:
URL: https://github.com/apache/pulsar/pull/19374#discussion_r1093780209
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java:
##
@@ -346,7 +346,7 @@ public CompletableFuture initialize() {
Policies policies = optPolicies.get();
-this.updateTopicPolicyByNamespacePolicy(policies);
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on a diff in pull request #19374: [fix][broker] Fix delete namespace fail by a In-flight topic
mattisonchao commented on code in PR #19374:
URL: https://github.com/apache/pulsar/pull/19374#discussion_r1093779898
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java:
##
@@ -346,7 +346,7 @@ public CompletableFuture initialize() {
Policies policies = optPolicies.get();
-this.updateTopicPolicyByNamespacePolicy(policies);
+updateTopicPolicyByNamespacePolicy(policies);
Review Comment:
```suggestion
this.updateTopicPolicyByNamespacePolicy(policies);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
svn commit: r59828 - in /dev/pulsar/pulsar-client-reactive-0.2.0-candidate-2: pulsar-client-reactive-0.2.0-src.tar.gz pulsar-client-reactive-0.2.0-src.tar.gz.asc pulsar-client-reactive-0.2.0-src.tar.g
Author: cbornet Date: Wed Feb 1 22:04:10 2023 New Revision: 59828 Log: Add files for pulsar-client-reactive 0.2.0-candidate-2 release Added: dev/pulsar/pulsar-client-reactive-0.2.0-candidate-2/pulsar-client-reactive-0.2.0-src.tar.gz (with props) dev/pulsar/pulsar-client-reactive-0.2.0-candidate-2/pulsar-client-reactive-0.2.0-src.tar.gz.asc dev/pulsar/pulsar-client-reactive-0.2.0-candidate-2/pulsar-client-reactive-0.2.0-src.tar.gz.sha512 Added: dev/pulsar/pulsar-client-reactive-0.2.0-candidate-2/pulsar-client-reactive-0.2.0-src.tar.gz == Binary file - no diff available. Propchange: dev/pulsar/pulsar-client-reactive-0.2.0-candidate-2/pulsar-client-reactive-0.2.0-src.tar.gz -- svn:mime-type = application/octet-stream Added: dev/pulsar/pulsar-client-reactive-0.2.0-candidate-2/pulsar-client-reactive-0.2.0-src.tar.gz.asc == --- dev/pulsar/pulsar-client-reactive-0.2.0-candidate-2/pulsar-client-reactive-0.2.0-src.tar.gz.asc (added) +++ dev/pulsar/pulsar-client-reactive-0.2.0-candidate-2/pulsar-client-reactive-0.2.0-src.tar.gz.asc Wed Feb 1 22:04:10 2023 @@ -0,0 +1,16 @@ +-BEGIN PGP SIGNATURE- + +iQIzBAABCgAdFiEEof+yNNmpbbtzoNn+Fo1l3cYE67IFAmPa4aQACgkQFo1l3cYE +67IAuQ/+PGRyTJZG+sFsLyfWOQOMbAsmcnX31m7CaD45pzW0GnturB2hfiYzbdx2 +7l3S+F+PtMfdXytXSU09W8ZdNXZJxmy/kYd5tum9u1rkz9ZKTnFEOoZI5GnuYDMy +6qD5aC4oCqMEpbHNq/wblnXd8hOs8Y/QiZ5+Tbyj66cltikV/EgxmVMpmv2g7VPp +rzzLFgZ3rYSY2yfqm6h/nDVmnJi+dP7QUUPAW0Zb03II/rndM3VSsq47xw41IEGj +fizbLG8Pb/KDvdSXgLm1ejgZtKbswPAfc8vIw+gKSv//k5MS84YKA+Iepw0OZMfg +U3dqtUtASdVpcjMFRYZsQDanQDHYkcBZNW1/+14RCM50FhpYvYfrm+RwOCv6xcP7 +66IOojSrZwSfvtY1UUAqsMpuOLH5e3edA0RwjorMmMhFrXud2hdpSJOuwDeYqv7D +/LQGMOR3lYL1ScGDI7AAhQ+rfPXQjFftFFQJo8ULBlSooTfPtoqX2VuB1yWa8Pud +BWXJsiYyoR/zxcmsC0sOohxGEFkQjFj2xLn4EVIWGHXHjlgr+dfdVnpXHAR5/mkL +81DiEunCOEtah2fD6LYbKR6Icc3e8Z1QlsoDFL/sRvroYE6IcGhZvzGZawwksbn8 +RIwW48N8MdK1d1tIDoVHQdNPcXUPzFTrna4l9gnaGAVnbDhIedk= +=TDeO +-END PGP SIGNATURE- Added: dev/pulsar/pulsar-client-reactive-0.2.0-candidate-2/pulsar-client-reactive-0.2.0-src.tar.gz.sha512 == --- dev/pulsar/pulsar-client-reactive-0.2.0-candidate-2/pulsar-client-reactive-0.2.0-src.tar.gz.sha512 (added) +++ dev/pulsar/pulsar-client-reactive-0.2.0-candidate-2/pulsar-client-reactive-0.2.0-src.tar.gz.sha512 Wed Feb 1 22:04:10 2023 @@ -0,0 +1 @@ +1498a63dd1021108baab61baae8f6f494530118a53c92bc316320878d5850de61313219255010885a9a04762a54077865a2724669f0294ec950db5b5460bedaa pulsar-client-reactive-0.2.0-src.tar.gz
svn commit: r59827 - /dev/pulsar/pulsar-client-reactive-0.2.0-candidate-2/
Author: cbornet Date: Wed Feb 1 22:01:55 2023 New Revision: 59827 Log: Add directory for pulsar-client-reactive 0.2.0-candidate-2 release Added: dev/pulsar/pulsar-client-reactive-0.2.0-candidate-2/
[GitHub] [pulsar] dlg99 commented on a diff in pull request #19153: [improve] PIP-241: add TopicEventListener / topic events for the BrokerService
dlg99 commented on code in PR #19153:
URL: https://github.com/apache/pulsar/pull/19153#discussion_r1093775837
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/TopicEventsDispatcher.java:
##
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.service;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Objects;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CopyOnWriteArrayList;
+import lombok.extern.slf4j.Slf4j;
+
+/**
+ * Utility class to dispatch topic events.
+ */
+@Slf4j
+public class TopicEventsDispatcher {
+private final List topicEventListeners = new
CopyOnWriteArrayList<>();
+
+/**
+ * Adds listeners, ignores null listeners.
+ * @param listeners
+ */
+public void addTopicEventListener(TopicEventsListener... listeners) {
+Objects.requireNonNull(listeners);
+Arrays.stream(listeners)
+.filter(x -> x != null)
+.forEach(topicEventListeners::add);
+}
+
+/**
+ * Removes listeners.
+ * @param listeners
+ */
+public void removeTopicEventListener(TopicEventsListener... listeners) {
+Objects.requireNonNull(listeners);
+Arrays.stream(listeners)
+.filter(x -> x != null)
+.forEach(topicEventListeners::remove);
+}
+
+/**
+ * Dispatches notification to all currently added listeners.
+ * @param topic
+ * @param event
+ * @param stage
+ */
+public void notify(String topic,
+ TopicEventsListener.TopicEvent event,
+ TopicEventsListener.EventStage stage) {
+notify(topic, event, stage, null);
+}
+
+/**
+ * Dispatches notification to all currently added listeners.
+ * @param topic
+ * @param event
+ * @param stage
+ * @param t
+ */
+public void notify(String topic,
+ TopicEventsListener.TopicEvent event,
+ TopicEventsListener.EventStage stage,
+ Throwable t) {
+topicEventListeners
+.forEach(listener -> notify(listener, topic, event, stage, t));
+}
+
+/**
+ * Dispatches SUCCESS/FAILURE notification to all currently added
listeners on completion of the future.
+ * @param future
+ * @param topic
+ * @param event
+ * @param
+ * @return future of a new completion stage
+ */
+public CompletableFuture notifyOnCompletion(CompletableFuture
future,
+ String topic,
+
TopicEventsListener.TopicEvent event) {
+return future.whenComplete((r, ex) -> notify(topic,
+event,
+ex == null ? TopicEventsListener.EventStage.SUCCESS :
TopicEventsListener.EventStage.FAILURE,
+ex));
+}
Review Comment:
see discussion with Michael above.
This narrows down to the order of event produced, example below.
`TopicEventsListener.TopicEvent...` is possible but it will create new new
`TopicEventsListener.TopicEvent[]` which I'd like to avoid.
```java
/*
produces
2nd completed
1st completed
3rd completed
4th completed
*/
@Test
public void cNotfChained() throws Exception {
CompletableFuture cf = new CompletableFuture<>();
cf.whenComplete((v, e) -> {
log.info("1st completed");
});
cf.whenComplete((v, e) -> {
log.info("2nd completed");
});
cf.complete(null);
cf = CompletableFuture.completedFuture(null);
cf.whenComplete((v, e) -> {
log.info("3rd completed");
});
cf.whenComplete((v, e) -> {
log.info("4th completed");
});
}
/*
produces
1st completed
2nd completed
3rd completed
4th completed
*/
@Test
public void cfChained() throws Exception {
CompletableFuture cf = new
[pulsar-client-reactive] annotated tag v0.2.0-candidate-2 updated (8c676db -> 1f3077e)
This is an automated email from the ASF dual-hosted git repository. cbornet pushed a change to annotated tag v0.2.0-candidate-2 in repository https://gitbox.apache.org/repos/asf/pulsar-client-reactive.git *** WARNING: tag v0.2.0-candidate-2 was modified! *** from 8c676db (commit) to 1f3077e (tag) tagging 8c676db0f70604e1abb0978f966c0916d9ea3aa9 (commit) replaces v0.2.0-candidate-1 by Christophe Bornet on Wed Feb 1 22:57:15 2023 +0100 - Log - Release v0.2.0-candidate-2 -BEGIN PGP SIGNATURE- iIkEABYKADEWIQSsJtR8kOYvBGFVou/hzYcUAI4EngUCY9rgPBMcY2Jvcm5ldEBh cGFjaGUub3JnAAoJEOHNhxQAjgSeBf0BAL5GogxAHs7jBr4+Sx8JPv+CNnL+4TW0 uWB0MVA0cWCcAP0RWeUOgGSSpX1zEto/4aENkt5kNYB28z9aETci29NHAQ== =KfXJ -END PGP SIGNATURE- --- No new revisions were added by this update. Summary of changes:
[GitHub] [pulsar] dlg99 commented on a diff in pull request #19374: [fix][broker] Fix delete namespace fail by a In-flight topic
dlg99 commented on code in PR #19374:
URL: https://github.com/apache/pulsar/pull/19374#discussion_r1093670810
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java:
##
@@ -346,7 +346,7 @@ public CompletableFuture initialize() {
Policies policies = optPolicies.get();
-this.updateTopicPolicyByNamespacePolicy(policies);
Review Comment:
nit: unnecessary change for this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[pulsar] branch master updated: [fix][broker] Execute per-topic entry filters with the same classloader (#19364)
This is an automated email from the ASF dual-hosted git repository.
nicoloboschi pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new a4c3034f52f [fix][broker] Execute per-topic entry filters with the
same classloader (#19364)
a4c3034f52f is described below
commit a4c3034f52f857ae0f4daf5d366ea9e578133bc2
Author: Nicolò Boschi
AuthorDate: Wed Feb 1 20:55:30 2023 +0100
[fix][broker] Execute per-topic entry filters with the same classloader
(#19364)
---
.../pulsar/broker/service/AbstractTopic.java | 35 ++-
.../pulsar/broker/service/BrokerService.java | 52 +---
.../pulsar/broker/service/EntryFilterSupport.java | 30 +--
.../org/apache/pulsar/broker/service/Topic.java| 5 +-
.../service/nonpersistent/NonPersistentTopic.java | 6 +-
.../broker/service/persistent/PersistentTopic.java | 6 +-
.../service/plugin/EntryFilterDefinition.java | 2 +
.../service/plugin/EntryFilterDefinitions.java | 28 ---
.../broker/service/plugin/EntryFilterProvider.java | 188 +--
.../service/plugin/EntryFilterWithClassLoader.java | 8 +
.../apache/pulsar/broker/admin/AdminApi2Test.java | 265 +++--
.../broker/service/AbstractBaseDispatcherTest.java | 16 +-
.../broker/service/plugin/FilterEntryTest.java | 133 +--
.../pulsar/broker/stats/ConsumerStatsTest.java | 4 +-
.../testcontext/MockEntryFilterProvider.java | 66 +
.../pulsar/common/policies/data/EntryFilters.java | 2 +-
16 files changed, 622 insertions(+), 224 deletions(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java
index c9f95ab524f..4e095cd66ba 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java
@@ -43,6 +43,7 @@ import lombok.Getter;
import org.apache.bookkeeper.mledger.util.StatsBuckets;
import org.apache.commons.collections4.CollectionUtils;
import org.apache.commons.collections4.MapUtils;
+import org.apache.commons.lang3.StringUtils;
import org.apache.commons.lang3.tuple.Pair;
import org.apache.pulsar.broker.PulsarService;
import org.apache.pulsar.broker.ServiceConfiguration;
@@ -53,7 +54,7 @@ import
org.apache.pulsar.broker.service.BrokerServiceException.ProducerBusyExcep
import
org.apache.pulsar.broker.service.BrokerServiceException.ProducerFencedException;
import
org.apache.pulsar.broker.service.BrokerServiceException.TopicMigratedException;
import
org.apache.pulsar.broker.service.BrokerServiceException.TopicTerminatedException;
-import org.apache.pulsar.broker.service.plugin.EntryFilterWithClassLoader;
+import org.apache.pulsar.broker.service.plugin.EntryFilter;
import org.apache.pulsar.broker.service.schema.BookkeeperSchemaStorage;
import org.apache.pulsar.broker.service.schema.SchemaRegistryService;
import
org.apache.pulsar.broker.service.schema.exceptions.IncompatibleSchemaException;
@@ -148,7 +149,7 @@ public abstract class AbstractTopic implements Topic,
TopicPolicyListener entryFilters;
+protected volatile Pair> entryFilters;
public AbstractTopic(String topic, BrokerService brokerService) {
this.topic = topic;
@@ -188,8 +189,8 @@ public abstract class AbstractTopic implements Topic,
TopicPolicyListener getEntryFilters() {
-return this.entryFilters;
+public List getEntryFilters() {
+return this.entryFilters.getRight();
}
public DispatchRateImpl getReplicatorDispatchRate() {
@@ -240,6 +241,8 @@ public abstract class AbstractTopic implements Topic,
TopicPolicyListener filters =
+
brokerService.getEntryFilterProvider().loadEntryFiltersForPolicy(entryFiltersPolicy);
+entryFilters = Pair.of(entryFilterNames, filters);
+} catch (Throwable e) {
+log.error("Failed to load entry filters on topic {}: {}", topic,
e.getMessage());
+throw new RuntimeException(e);
+}
+}
+
public long getMsgInCounter() {
return this.msgInCounter.longValue();
}
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
index d88f040f11b..f7020963fb7 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
@@ -116,7 +116,6 @@ import
org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleC
import org.apache.pulsar.broker.service.persistent.PersistentTopic;
import org.apache.pulsar.broker.service.persistent.SystemTopic;
import
[GitHub] [pulsar] nicoloboschi closed issue #19361: [Bug] Entry filters per topic/namespace load and unpack NAR files for each topic created
nicoloboschi closed issue #19361: [Bug] Entry filters per topic/namespace load and unpack NAR files for each topic created URL: https://github.com/apache/pulsar/issues/19361 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] nicoloboschi merged pull request #19364: [fix][broker] Execute per-topic entry filters with the same classloader
nicoloboschi merged PR #19364: URL: https://github.com/apache/pulsar/pull/19364 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on pull request #19383: [fix] [admin] set offload threshold should fail if ns policies is read-only
poorbarcode commented on PR #19383: URL: https://github.com/apache/pulsar/pull/19383#issuecomment-1412553051 /pulsarbot rerun-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-python] merlimat commented on issue #84: Deadlock in concurrent `send_async` calls w/ pulsar-client-3.0
merlimat commented on issue #84: URL: https://github.com/apache/pulsar-client-python/issues/84#issuecomment-1412485663 Thanks for the clear repro code and the above info. That made it really easy! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] merlimat commented on issue #12944: ARM based docker image
merlimat commented on issue #12944: URL: https://github.com/apache/pulsar/issues/12944#issuecomment-1412484186 Since Pulsar 2.11.0, there is no problem in running on ARM64 with default configs -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-python] zbentley commented on issue #84: Deadlock in concurrent `send_async` calls w/ pulsar-client-3.0
zbentley commented on issue #84: URL: https://github.com/apache/pulsar-client-python/issues/84#issuecomment-1412483703 I understand now. Thank you! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-python] merlimat commented on issue #84: Deadlock in concurrent `send_async` calls w/ pulsar-client-3.0
merlimat commented on issue #84: URL: https://github.com/apache/pulsar-client-python/issues/84#issuecomment-1412481804 The logger and the message listeners should be fine since they are both going to acquire the GIL when entering the Python callbacks. This is assuming that after this PR there are no other places that are going to lock on any Pulsar client internal locks while holding the GIL, as was the case with `send_async()`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] eolivelli commented on issue #12944: ARM based docker image
eolivelli commented on issue #12944: URL: https://github.com/apache/pulsar/issues/12944#issuecomment-1412457768 if you want to run pulsar standalone without docker you can simply add this to conf/standalone.conf, this will disable RocksDB `ledgerStorageClass=org.apache.bookkeeper.bookie.SortedLedgerStorage` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-python] zbentley commented on a diff in pull request #87: Fixed deadlock in producer.send_async
zbentley commented on code in PR #87:
URL:
https://github.com/apache/pulsar-client-python/pull/87#discussion_r1093518373
##
src/producer.cc:
##
@@ -34,6 +34,16 @@ MessageId Producer_send(Producer& producer, const Message&
message) {
return messageId;
}
+void Producer_sendAsync(Producer& producer, const Message& msg, SendCallback
callback) {
+Py_BEGIN_ALLOW_THREADS
+producer.sendAsync(msg, callback);
+Py_END_ALLOW_THREADS
+
+if (PyErr_CheckSignals() == -1) {
Review Comment:
Sure. Something I've seen before is issues around an error already being
"pre-raise" in a Python thread when some external code (the pulsar client C++
in this case) decides to invoke an unrelated Python callback. Does pybind
handle situations like [this
one](https://github.com/apache/pulsar/pull/16535/files#diff-b9b1b59dde469b15de56c46d20ae54d82e01a7a607ef4bf6f2d759807e5bfd79R119)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar-client-python] zbentley commented on issue #84: Deadlock in concurrent `send_async` calls w/ pulsar-client-3.0
zbentley commented on issue #84: URL: https://github.com/apache/pulsar-client-python/issues/84#issuecomment-1412425548 That makes sense @merlimat, thanks. What about "async" (I think it's more "c++ deciding to call Python" rather than the other way around) functionality like the Python logger, or the consumer's on-message callbacks? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] nicoloboschi commented on issue #19363: [Bug] pulsar-sink nested avro
nicoloboschi commented on issue #19363: URL: https://github.com/apache/pulsar/issues/19363#issuecomment-1412410250 @RonnelOvalles could you provide the error and reformat the code? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] tisonkun closed issue #18960: [Doc] Documentation site search navigates to the end of the page
tisonkun closed issue #18960: [Doc] Documentation site search navigates to the end of the page URL: https://github.com/apache/pulsar/issues/18960 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] tisonkun commented on issue #18960: [Doc] Documentation site search navigates to the end of the page
tisonkun commented on issue #18960: URL: https://github.com/apache/pulsar/issues/18960#issuecomment-1412389362 Thanks for your explanation @urfreespace! Closed as expected or at least won't fix by now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-site] tisonkun opened a new pull request, #396: docs: contributing runbook
tisonkun opened a new pull request, #396: URL: https://github.com/apache/pulsar-site/pull/396 Add a CONTRIBUTING file for inline guides. They're too details to be included in the Contribution Guide (or at least at the very first time). But I think when the content becomes stable and well-structured, we should publish them to the Contribution Guide and only leave the FAQs and links. Signed-off-by: tison ### Documentation - [x] `doc` - [ ] `doc-required` - [ ] `doc-not-needed` - [ ] `doc-complete` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-python] merlimat commented on a diff in pull request #90: [fix] exception bug as per #89
merlimat commented on code in PR #90:
URL:
https://github.com/apache/pulsar-client-python/pull/90#discussion_r1093481989
##
pulsar/schema/schema_avro.py:
##
@@ -82,15 +82,6 @@ def decode(self, data):
return self._record_cls(**d)
else:
return d
-
else:
-class AvroSchema(Schema):
-def __init__(self, _record_cls, _schema_definition=None):
-raise Exception("Avro library support was not found. Make sure to
install Pulsar client " +
+raise Exception("Avro library support was not found. Make sure to install
Pulsar client " +
Review Comment:
Wouldn't this fail even if one is not trying to use Avro schema?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] pgier commented on pull request #19279: [improve][cli] pulsar-perf: refactor to reduce code duplication
pgier commented on PR #19279: URL: https://github.com/apache/pulsar/pull/19279#issuecomment-1412376604 Rebased on latest master to re-run CI. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-python] merlimat commented on issue #84: Deadlock in concurrent `send_async` calls w/ pulsar-client-3.0
merlimat commented on issue #84: URL: https://github.com/apache/pulsar-client-python/issues/84#issuecomment-1412374679 @zbentley > can you expand on It's only the sendAsync that is impacted? In this Python wrapper we mostly expose sync methods, eg: `client.create_produce()`, etc.. The only async method is `producer.send_async()`. For sync methods, we internally use the C++ async version (to avoid blocking when checking for Python interruptions) and handle the GIL properly. We have a function for that purpose: https://github.com/apache/pulsar-client-python/blob/main/src/utils.h#L60-L87 > Unless something is handling grabbing the GIL when the callback is triggered, it seems like that PR may resolve the deadlock at the expense of making the python interactions non-thread-safe. You can see in the stack trace above that PyBind is already automatically acquiring the GIL when triggering the Python callback : ``` thread #2 frame #0: 0x7ff80a1943ea libsystem_kernel.dylib`__psynch_cvwait + 10 frame #1: 0x7ff80a1cea6f libsystem_pthread.dylib`_pthread_cond_wait + 1249 frame #2: 0x000105a3cb9f python3.7`take_gil + 255 frame #3: 0x000105a3cfb3 python3.7`PyEval_AcquireThread + 19 frame #4: 0x00010870be43 _pulsar.cpython-37m-darwin.so`pybind11::gil_scoped_acquire::gil_scoped_acquire() + 83 frame #5: 0x0001087714f3 _pulsar.cpython-37m-darwin.so`pybind11::detail::type_caster, void>::load(pybind11::handle, bool)::func_handle::func_handle(func_handle const&) + 35 frame #6: 0x0001087715f1 _pulsar.cpython-37m-darwin.so`std::__1::__function::__func, void>::load(pybind11::handle, bool)::func_wrapper, std::__1::allocator, void>::load(pybind11::handle, bool)::func_wrapper>, void (pulsar::Result, pulsar::MessageId const&)>::__clone() const + 49 frame #7: 0x0001088daccd _pulsar.cpython-37m-darwin.so`std::__1::__function::__func)::$_2, std::__1::allocator)::$_2>, void (pulsar::Result, pulsar::MessageId const&)>::__clone() const + 93 frame #8: 0x00010878f7de _pulsar.cpython-37m-darwin.so`pulsar::OpSendMsg::OpSendMsg(pulsar::OpSendMsg const&) + 126 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-python] merlimat commented on a diff in pull request #87: Fixed deadlock in producer.send_async
merlimat commented on code in PR #87:
URL:
https://github.com/apache/pulsar-client-python/pull/87#discussion_r1093472361
##
src/producer.cc:
##
@@ -34,6 +34,16 @@ MessageId Producer_send(Producer& producer, const Message&
message) {
return messageId;
}
+void Producer_sendAsync(Producer& producer, const Message& msg, SendCallback
callback) {
+Py_BEGIN_ALLOW_THREADS
+producer.sendAsync(msg, callback);
+Py_END_ALLOW_THREADS
+
+if (PyErr_CheckSignals() == -1) {
Review Comment:
This is in the main thread, irrespective of the callback.
`send_async()` is a potentially blocking operation (when
`block_if_queue_full=True`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
