[GitHub] [pulsar] poorbarcode commented on a diff in pull request #19514: [fix] [admin] Admin API can not work if uri too large

[via GitHub]

poorbarcode commented on code in PR #19514:
URL: https://github.com/apache/pulsar/pull/19514#discussion_r1133045643


##
pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/ProxyConfiguration.java:
##
@@ -661,6 +661,29 @@ public class ProxyConfiguration implements 
PulsarConfiguration {
 )
 private int httpOutputBufferSize = 32 * 1024;
 
+@FieldContext(
+minValue = 1,
+category = CATEGORY_HTTP,
+doc = """
+The maximum size in bytes of the request header.
+Larger headers will allow for more and/or larger cookies plus 
larger form content encoded in a URL.
+However, larger headers consume more memory and can make a 
server more vulnerable to denial of service
+attacks.
+  """
+)
+private int httpMaxRequestHeaderSize = 8 * 1024;
+
+@FieldContext(
+minValue = 1,
+category = CATEGORY_HTTP,
+doc = """
+ the size of the buffer used to write requests to Broker.
+ if "httpMaxRequestHeaderSize" is large than 
"httpClientRequestBufferSize", will set
+ "httpClientRequestBufferSize" to the value of 
"httpMaxRequestHeaderSize"
+  """
+)
+private int httpClientRequestBufferSize = 4096;

Review Comment:
   Proxy handles HTTP requests like this： `pulsar-admin.sh` -> `proxy web 
server` -> `(highlight) internal client in proxy` -> `pulsar web server`.  
   
   When the internal client forwards a request, it forwards the request header 
and the request body, and all the data passes through a buffer( we call it Buf 
), like this:
   - Receive a request
   - Put the request line and request headers input to the Buf.
   - (highlight)Flush the Buf ( If the data in the request 
line and request header exceeds the length of the buf, an error is reported )
   - Put the request body input to the Buf.
   - Flush the Buf if it is full.
   
   So we need a config to set the `buff size` of the Buf: 
`pulsar-proxy.conf.httpClientRequestBufferSize` -> `buf size of the internal 
client`.
   And if `pulsar-proxy.conf.httpMaxRequestHeaderSize` is large than 
`pulsar-proxy.conf.httpClientRequestBufferSize`, the error in Step 3 above 
occurs, so we should set the size of Buf to 
`max(pulsar-proxy.conf.httpClientRequestBufferSize, 
pulsar-proxy.conf.httpMaxRequestHeaderSize)`
   
   same as the comment: 
https://github.com/apache/pulsar/pull/19514#discussion_r1133045975



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] poorbarcode commented on a diff in pull request #19514: [fix] [admin] Admin API can not work if uri too large

[via GitHub]

poorbarcode commented on code in PR #19514:
URL: https://github.com/apache/pulsar/pull/19514#discussion_r1133045975


##
pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/WebServer.java:
##
@@ -214,6 +218,13 @@ public void addServlet(String basePath, ServletHolder 
servletHolder,
 handlers.add(context);
 }
 
+private static void popularServletParams(ServletHolder servletHolder, 
ProxyConfiguration config){
+if (config.getHttpClientRequestBufferSize() > 0 || 
config.getHttpMaxRequestHeaderSize() > 0){
+int v = Math.max(config.getHttpClientRequestBufferSize(), 
config.getHttpMaxRequestHeaderSize());
+servletHolder.setInitParameter(INIT_PARAM_REQUEST_BUFFER_SIZE, 
String.valueOf(v));
+}
+}

Review Comment:
   Proxy handles HTTP requests like this： `pulsar-admin.sh` -> `proxy web 
server` -> `(highlight) internal client in proxy` -> `pulsar web server`.  
   
   When the internal client forwards a request, it forwards the request header 
and the request body, and all the data passes through a buffer( we call it Buf 
), like this:
   - Receive a request
   - Put the request line and request headers input to the Buf.
   - (highlight)Flush the Buf ( If the data in the request 
line and request header exceeds the length of the buf, an error is reported )
   - Put the request body input to the Buf.
   - Flush the Buf if it is full.
   
   So we need a config to set the `buff size` of the Buf: 
`pulsar-proxy.conf.httpClientRequestBufferSize` -> `buf size of the internal 
client`.
   And if `pulsar-proxy.conf.httpMaxRequestHeaderSize` is large than 
`pulsar-proxy.conf.httpClientRequestBufferSize`, the error in Step 3 above 
occurs, so we should set the size of Buf to 
`max(pulsar-proxy.conf.httpClientRequestBufferSize, 
pulsar-proxy.conf.httpMaxRequestHeaderSize)`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] poorbarcode commented on a diff in pull request #19514: [fix] [admin] Admin API can not work if uri too large

[via GitHub]

poorbarcode commented on code in PR #19514:
URL: https://github.com/apache/pulsar/pull/19514#discussion_r1133045643


##
pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/ProxyConfiguration.java:
##
@@ -661,6 +661,29 @@ public class ProxyConfiguration implements 
PulsarConfiguration {
 )
 private int httpOutputBufferSize = 32 * 1024;
 
+@FieldContext(
+minValue = 1,
+category = CATEGORY_HTTP,
+doc = """
+The maximum size in bytes of the request header.
+Larger headers will allow for more and/or larger cookies plus 
larger form content encoded in a URL.
+However, larger headers consume more memory and can make a 
server more vulnerable to denial of service
+attacks.
+  """
+)
+private int httpMaxRequestHeaderSize = 8 * 1024;
+
+@FieldContext(
+minValue = 1,
+category = CATEGORY_HTTP,
+doc = """
+ the size of the buffer used to write requests to Broker.
+ if "httpMaxRequestHeaderSize" is large than 
"httpClientRequestBufferSize", will set
+ "httpClientRequestBufferSize" to the value of 
"httpMaxRequestHeaderSize"
+  """
+)
+private int httpClientRequestBufferSize = 4096;

Review Comment:
   Proxy handles HTTP requests like this： `pulsar-admin.sh` -> `proxy web 
server` -> `(highlight) internal client in proxy` -> `pulsar web server`.  
   
   When the internal client forwards a request, it forwards the request header 
and the request body, and all the data passes through a buffer( we call it Buf 
), like this:
   - Receive a request
   - Put the request line and request headers input to the Buf.
   - (highlight)Flush the Buf ( If the data in the request 
line and request header exceeds the length of the buf, an error is reported )
   - Put the request body input to the Buf.
   - Flush the Buf if it is full.
   
   So we need a config to set the `buff size` of the Buf.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[pulsar] branch master updated: [fix] [test] fix flaky test BucketDelayedDeliveryTrackerTest. testWithBkException (#19751)

[yubiao]

This is an automated email from the ASF dual-hosted git repository.

yubiao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new bfc620bf6f2 [fix] [test] fix flaky test 
BucketDelayedDeliveryTrackerTest. testWithBkException (#19751)
bfc620bf6f2 is described below

commit bfc620bf6f2586b099fe423d7b07cc88754ef312
Author: fengyubiao 
AuthorDate: Sat Mar 11 13:02:53 2023 +0800

[fix] [test] fix flaky test BucketDelayedDeliveryTrackerTest. 
testWithBkException (#19751)

This test sets the maximum number of buckets to 10, then creates multiple 
buckets by writing, then uses the merge mechanism to make the final number of 
buckets less than or equal to 10.

But `BucketDelayedDeliveryTracker` doesn't guarantee that every merger 
operation will work, these cases will make the operation fail:
- the last persistention of the bucket has not finished.
- all buckets are full.

So remove this validation
---
 .../broker/delayed/bucket/BucketDelayedDeliveryTrackerTest.java   | 4 
 1 file changed, 4 deletions(-)

diff --git 
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/delayed/bucket/BucketDelayedDeliveryTrackerTest.java
 
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/delayed/bucket/BucketDelayedDeliveryTrackerTest.java
index 08e1f78725b..0ba9e5f4ca2 100644
--- 
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/delayed/bucket/BucketDelayedDeliveryTrackerTest.java
+++ 
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/delayed/bucket/BucketDelayedDeliveryTrackerTest.java
@@ -308,10 +308,6 @@ public class BucketDelayedDeliveryTrackerTest extends 
AbstractDeliveryTrackerTes
 
 assertEquals(110, tracker.getNumberOfDelayedMessages());
 
-int size = tracker.getImmutableBuckets().asMapOfRanges().size();
-
-assertEquals(10, size);
-
 tracker.addMessage(111, 1011, 111 * 10);
 
 MutableLong delayedMessagesInSnapshot = new MutableLong();

[GitHub] [pulsar] poorbarcode closed issue #19720: Flaky-test: BucketDelayedDeliveryTrackerTest. testWithBkException

[via GitHub]

poorbarcode closed issue #19720: Flaky-test: BucketDelayedDeliveryTrackerTest. 
testWithBkException
URL: https://github.com/apache/pulsar/issues/19720


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] poorbarcode merged pull request #19751: [fix] [test] fix flaky test BucketDelayedDeliveryTrackerTest. testWithBkException

[via GitHub]

poorbarcode merged PR #19751:
URL: https://github.com/apache/pulsar/pull/19751


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] heesung-sn commented on a diff in pull request #19730: [improve][broker] PIP-192 Made only the leader consume TopBundlesLoadDataStore

[via GitHub]

heesung-sn commented on code in PR #19730:
URL: https://github.com/apache/pulsar/pull/19730#discussion_r1131556270


##
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/ExtensibleLoadManagerImpl.java:
##
@@ -398,6 +425,50 @@ private boolean isInternalTopic(String topic) {
 || topic.startsWith(TOP_BUNDLES_LOAD_DATA_STORE_TOPIC);
 }
 
+private void playLeader() {
+log.info("This broker:{} is the leader now.", 
pulsar.getLookupServiceAddress());
+serviceUnitStateChannel.scheduleOwnershipMonitor();
+this.pulsar.getLoadManagerExecutor().execute(() -> {
+try {
+loadStoreInitWaiter.await();
+} catch (InterruptedException e) {

Review Comment:
   In fact, I think we can retry when interrupted.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] 315157973 commented on pull request #19561: [improve][meta] Support set metadata size threshold for compression.

[via GitHub]

315157973 commented on PR #19561:
URL: https://github.com/apache/pulsar/pull/19561#issuecomment-1464814475

   @lifepuzzlefun  Please fix the unit test 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-client-go] Shoothzj closed pull request #959: add pulsar admin ability in pulsar-client-go

[via GitHub]

Shoothzj closed pull request #959: add pulsar admin ability in pulsar-client-go
URL: https://github.com/apache/pulsar-client-go/pull/959


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] yyj8 opened a new pull request, #19047: [improve][broker]PIP-214 Add broker level metrics statistics and expose to prometheus

[via GitHub]

yyj8 opened a new pull request, #19047:
URL: https://github.com/apache/pulsar/pull/19047

   For detailed improvement instructions, please refer to issues：
   https://github.com/apache/pulsar/issues/18056
   
   original PR：
   https://github.com/apache/pulsar/pull/18116
   
   ### Motivation
   Currently, pulsar does not statistics broker level metrics, and all relevant 
metrics are 0 by default.
   
   When the number of topic partitions reaches more than 10 or even 
millions, and the topic level metrics are exposed 
(exposeTopicLevelMetricsInPrometheus=true), if you want to query the metrics of 
the broker dimension, you need to summarize all topics, and the performance 
becomes very poor, or even the results cannot be queried from the promethus. 
Common broker level metrics include:
   ```
   pulsar_topics_count
   pulsar_subscriptions_count
   pulsar_producers_count
   pulsar_consumers_count
   pulsar_rate_in
   pulsar_rate_out
   pulsar_throughput_in
   pulsar_throughput_out
   pulsar_storage_size
   pulsar_storage_logical_size
   pulsar_storage_write_rate
   pulsar_storage_read_rate
   pulsar_msg_backlog
   ```
   
   ### Modifications
   We need to statistics the metrics of the broker dimension and expose them to 
prometheus to improve the performance of the monitoring of the broker 
dimension. Modify the original metrics name as follows:
   ```
   pulsar_broker_topics_count
   pulsar_broker_subscriptions_count
   pulsar_broker_producers_count
   pulsar_broker_consumers_count
   pulsar_broker_rate_in
   pulsar_broker_rate_out
   pulsar_broker_throughput_in
   pulsar_broker_throughput_out
   pulsar_broker_storage_size
   pulsar_broker_storage_logical_size
   pulsar_broker_storage_write_rate
   pulsar_broker_storage_read_rate
   pulsar_broker_msg_backlog
   ```
   
   ### Documentation
   - [ ] `doc` 
   - [ ] `doc-required` 
   - [ ] `doc-not-needed` 
   - [x] `doc-complete` 
   
   ### Matching PR in forked repository
   PR in forked repository: https://github.com/yyj8/pulsar/pull/2


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] yyj8 closed pull request #19789: [improve][broker]PIP-214 Add broker level metrics statistics and expose to prometheus

[via GitHub]

yyj8 closed pull request #19789: [improve][broker]PIP-214 Add broker level 
metrics statistics and expose to prometheus
URL: https://github.com/apache/pulsar/pull/19789


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] yyj8 opened a new pull request, #19789: [improve][broker]PIP-214 Add broker level metrics statistics and expose to prometheus

[via GitHub]

yyj8 opened a new pull request, #19789:
URL: https://github.com/apache/pulsar/pull/19789

   For detailed improvement instructions, please refer to issues：
   https://github.com/apache/pulsar/issues/18056
   
   original PR：
   https://github.com/apache/pulsar/pull/18116
   
   ### Motivation
   Currently, pulsar does not statistics broker level metrics, and all relevant 
metrics are 0 by default.
   
   When the number of topic partitions reaches more than 10 or even 
millions, and the topic level metrics are exposed 
(exposeTopicLevelMetricsInPrometheus=true), if you want to query the metrics of 
the broker dimension, you need to summarize all topics, and the performance 
becomes very poor, or even the results cannot be queried from the promethus. 
Common broker level metrics include:
   ```
   pulsar_topics_count
   pulsar_subscriptions_count
   pulsar_producers_count
   pulsar_consumers_count
   pulsar_rate_in
   pulsar_rate_out
   pulsar_throughput_in
   pulsar_throughput_out
   pulsar_storage_size
   pulsar_storage_logical_size
   pulsar_storage_write_rate
   pulsar_storage_read_rate
   pulsar_msg_backlog
   ```
   
   ### Modifications
   We need to statistics the metrics of the broker dimension and expose them to 
prometheus to improve the performance of the monitoring of the broker 
dimension. Modify the original metrics name as follows:
   ```
   pulsar_broker_topics_count
   pulsar_broker_subscriptions_count
   pulsar_broker_producers_count
   pulsar_broker_consumers_count
   pulsar_broker_rate_in
   pulsar_broker_rate_out
   pulsar_broker_throughput_in
   pulsar_broker_throughput_out
   pulsar_broker_storage_size
   pulsar_broker_storage_logical_size
   pulsar_broker_storage_write_rate
   pulsar_broker_storage_read_rate
   pulsar_broker_msg_backlog
   ```
   
   ### Documentation
   - [ ] `doc` 
   - [ ] `doc-required` 
   - [ ] `doc-not-needed` 
   - [x] `doc-complete` 
   
   ### Matching PR in forked repository
   PR in forked repository: https://github.com/yyj8/pulsar/pull/2


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[pulsar] branch branch-2.10 updated: [fix][client][branch-2.10]Return local thread for the `newThread` (#19779)

[xiangying]

This is an automated email from the ASF dual-hosted git repository.

xiangying pushed a commit to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/pulsar.git


The following commit(s) were added to refs/heads/branch-2.10 by this push:
 new 240c22cce69 [fix][client][branch-2.10]Return local thread for the 
`newThread` (#19779)
240c22cce69 is described below

commit 240c22cce69d99d69cbda84467787467b5bbbcb4
Author: Xiangying Meng <55571188+liangyepianz...@users.noreply.github.com>
AuthorDate: Sat Mar 11 10:10:29 2023 +0800

[fix][client][branch-2.10]Return local thread for the `newThread` (#19779)

### Motivation
The original fix is https://github.com/apache/pulsar/pull/18268.
But after the fix is cherry-picked, the fix is overridden when resolving
conflicts for cherry-pick https://github.com/apache/pulsar/pull/18211.
### Modifications
Return the local thread variables in the `newThread`.
---
 .../src/main/java/org/apache/pulsar/client/util/ExecutorProvider.java  | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git 
a/pulsar-client/src/main/java/org/apache/pulsar/client/util/ExecutorProvider.java
 
b/pulsar-client/src/main/java/org/apache/pulsar/client/util/ExecutorProvider.java
index 8997d714194..90e1a59ed4d 100644
--- 
a/pulsar-client/src/main/java/org/apache/pulsar/client/util/ExecutorProvider.java
+++ 
b/pulsar-client/src/main/java/org/apache/pulsar/client/util/ExecutorProvider.java
@@ -52,9 +52,10 @@ public class ExecutorProvider {
 
 @Override
 public Thread newThread(Runnable r) {
-thread = super.newThread(r);
+Thread thread = super.newThread(r);
 thread.setUncaughtExceptionHandler((t, e) ->
 log.error("Thread {} got uncaught Exception", t.getName(), 
e));
+this.thread = thread;
 return thread;
 }
 }

[GitHub] [pulsar] liangyepianzhou merged pull request #19779: [fix][client][branch-2.10]Return local thread for the `newThread`

[via GitHub]

liangyepianzhou merged PR #19779:
URL: https://github.com/apache/pulsar/pull/19779


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] github-actions[bot] commented on issue #19123: PIP-237: Make PulsarAdmin accessible in SinkContext and SourceContext

[via GitHub]

github-actions[bot] commented on issue #19123:
URL: https://github.com/apache/pulsar/issues/19123#issuecomment-1464780620

   The issue had no activity for 30 days, mark with Stale label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] github-actions[bot] commented on issue #19136: PIP-239: MAX RECONSUME TIMES per Message

[via GitHub]

github-actions[bot] commented on issue #19136:
URL: https://github.com/apache/pulsar/issues/19136#issuecomment-1464780550

   The issue had no activity for 30 days, mark with Stale label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] github-actions[bot] commented on issue #19363: [Bug] pulsar-sink nested avro

[via GitHub]

github-actions[bot] commented on issue #19363:
URL: https://github.com/apache/pulsar/issues/19363#issuecomment-1464780313

   The issue had no activity for 30 days, mark with Stale label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] github-actions[bot] commented on issue #19462: Support configure compression type for pulsar functions

[via GitHub]

github-actions[bot] commented on issue #19462:
URL: https://github.com/apache/pulsar/issues/19462#issuecomment-1464779963

   The issue had no activity for 30 days, mark with Stale label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] github-actions[bot] commented on issue #19460: [Bug][broker] Error deserializing message，Invalid unknonwn tag type: 7

[via GitHub]

github-actions[bot] commented on issue #19460:
URL: https://github.com/apache/pulsar/issues/19460#issuecomment-1464780010

   The issue had no activity for 30 days, mark with Stale label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] github-actions[bot] commented on pull request #19390: [fix][authentication] Store the original auth when using anonymous role

[via GitHub]

github-actions[bot] commented on PR #19390:
URL: https://github.com/apache/pulsar/pull/19390#issuecomment-1464780171

   The pr had no activity for 30 days, mark with Stale label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] github-actions[bot] commented on issue #19457: [Bug] Excessive logs regarding possible duplicate objects when using Producer in a concurrent way

[via GitHub]

github-actions[bot] commented on issue #19457:
URL: https://github.com/apache/pulsar/issues/19457#issuecomment-1464780056

   The issue had no activity for 30 days, mark with Stale label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] github-actions[bot] commented on issue #19463: Support configuring batch message behavior in Java client's DeadLetterPolicy / DLQ feature

[via GitHub]

github-actions[bot] commented on issue #19463:
URL: https://github.com/apache/pulsar/issues/19463#issuecomment-1464779908

   The issue had no activity for 30 days, mark with Stale label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] nlu90 closed pull request #19470: [improve][fn] Support configure compression type

[via GitHub]

nlu90 closed pull request #19470: [improve][fn] Support configure compression 
type
URL: https://github.com/apache/pulsar/pull/19470


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] tpiperatgod opened a new pull request, #18929: [improve][fn] Support processingGuarantees "EFFECTIVELY_ONCE" in python function

[via GitHub]

tpiperatgod opened a new pull request, #18929:
URL: https://github.com/apache/pulsar/pull/18929

   
   
   
   
   Fixes #18903 
   
   
   
   Master Issue: #xyz
   
   ### Motivation
   
   Make pulsar python functions support to set `processingGuarantees` to 
`EFFECTIVELY_ONCE`
   
   ### Modifications
   
   
   
   ### Verifying this change
   
   - [x] Make sure that the change passes the CI checks.
   
   *(Please pick either of the following options)*
   
   This change is a trivial rework / code cleanup without any test coverage.
   
   *(or)*
   
   This change is already covered by existing tests, such as *(please describe 
tests)*.
   
   *(or)*
   
   This change added tests and can be verified as follows:
   
   *(example:)*
 - *Added integration tests for end-to-end deployment with large payloads 
(10MB)*
 - *Extended integration test for recovery after broker failure*
   
   ### Does this pull request potentially affect one of the following parts:
   
   *If the box was checked, please highlight the changes*
   
   - [ ] Dependencies (add or upgrade a dependency)
   - [ ] The public API
   - [ ] The schema
   - [ ] The default values of configurations
   - [ ] The threading model
   - [ ] The binary protocol
   - [ ] The REST endpoints
   - [ ] The admin CLI options
   - [ ] Anything that affects deployment
   
   ### Documentation
   
   
   
   - [ ] `doc` 
   - [x] `doc-required` 
   - [ ] `doc-not-needed` 
   - [ ] `doc-complete` 
   
   ### Matching PR in forked repository
   
   PR in forked repository: https://github.com/tpiperatgod/pulsar/pull/9
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] nlu90 closed pull request #18929: [improve][fn] Support processingGuarantees "EFFECTIVELY_ONCE" in python function

[via GitHub]

nlu90 closed pull request #18929: [improve][fn] Support processingGuarantees 
"EFFECTIVELY_ONCE" in python function 
URL: https://github.com/apache/pulsar/pull/18929


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-client-node] JTBS commented on issue #272: Add how to use schema examples or docs

[via GitHub]

JTBS commented on issue #272:
URL: 
https://github.com/apache/pulsar-client-node/issues/272#issuecomment-1464493770

   Hi @shibd  
   I was able to use Schemas in Producers and Consumers in Java using Java 
clients.
   
   But with NodeJS - currently published pulsar-client in NPM, does not have 
these change
   So I built it locally to get latest changes from git repo.
   
   While this is good so far, I am not able to figure out what is right way to 
use Schema in NodeJS/Pulsar-Client?
   Below scenarios gives timeout when creating Producer.
   
   I really appreciate if you can share some details on Producer/Consumer to 
use Schema in NodeJs? 
   
   const txnSchema = {
 schemaType: "Json",
 name: "txn",  
 schema: "{ \"type\": \"record\",\"fields\": [{\"name\": \"Id\",\"type\": 
[\"null\",\"int\"]}]}",
 properties: {
   key1: 'value1',
   key2: 'value2'
 }
   }
   
   producer = await client.createProducer({
   topic: topicName,
   schema: this.schema
 });
   
   
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-client-go] LennyAndersen opened a new pull request, #990: returning errors back to the client.

[via GitHub]

LennyAndersen opened a new pull request, #990:
URL: https://github.com/apache/pulsar-client-go/pull/990

   
   ### Motivation
   When using the pulsar go client with oauth2, setting up the auth with the 
function NewAuthenticationOAuth2 doesn't return an error. In case of errors in 
the authentication flow nothing happens and the user/developer can only assume 
that the authentication went well.
   
   It turns out there is a pretty decent error handling in the entire flow, but 
for unknown reasons this error is discarded right before the connection is 
returned to the client.
   
   ### Modifications
   
   Instead of just discarding the error, it is now returned. Note that  the 
function now returns two parameters instead of just one and should be handled 
accordingly.
   Refer to the changelog of the file `pulsar/client.go` to see the actual 
changes.
   
   ### Verifying this change
   
   This change is a trivial rework / code cleanup without any test coverage.
   
   
   ### Documentation
 - Does this pull request introduce a new feature? no


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] heesung-sn commented on a diff in pull request #19730: [improve][broker] PIP-192 Made only the leader consume TopBundlesLoadDataStore

[via GitHub]

heesung-sn commented on code in PR #19730:
URL: https://github.com/apache/pulsar/pull/19730#discussion_r1132844370


##
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/store/TableViewLoadDataStoreImpl.java:
##
@@ -61,30 +70,58 @@ public CompletableFuture removeAsync(String key) {
 
 @Override
 public Optional get(String key) {
+validateTableViewStart();

Review Comment:
   
   > ExtensibleLoadManager.start() and playLeader() are complete
   
   I meant this `complete` as everything initialized correctly without an error.
   
   Yes. Currently, if `topBundlesLoadDataStore.startTableView();` fails, the 
leader broker still proceeds, and the TransferShedder will constantly make 
UnloadDecision(fail). 
   
   However, lazy-init(tableview.start ) in validateTableViewStart could also 
fail and lead to the same situation (TransferShedder will constantly make 
UnloadDecision(fail)). Besides, this lazy-init can lead to unknown 
behaviors(many consumers on this topBundlesLoad topic) when other brokers 
access the tableview.
   
   The bigger question is how Pulsar handles system state change failures. In 
this case, the state change is from playLeader or playFollower.
   
   I can think of the following options:
   
   Option 1: retry-forever
   Option 2: fail-fast(shutdown broker)
   Option 3: ignore (leader or followers will run in the invalid state, and 
there will be many error logs and metrics. Eventually, the operator needs to 
capture this and fix it, probably by broker restart)
   
   AFAIK, upon system state change failures, other systems immediately fail 
fast with meaningful logs and even with coredump. However, I don't think this 
`fail-fast` is the practice in the Pulsar project. 
   
   I can update the code for option 1 (retry-forever).
   
   Please let me know if you have more concerns.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] dlg99 opened a new pull request, #19788: [cherry-pick][branch-2.11] KCA: picking fixes from master

[via GitHub]

dlg99 opened a new pull request, #19788:
URL: https://github.com/apache/pulsar/pull/19788

   ### Motivation
   
   Cherry-picking from master, please do not squash when merging
   
   ```
   Author: Andrey Yegorov <8622884+dl...@users.noreply.github.com>
   Date:   Fri Mar 10 01:58:40 2023 -0800
   
   [fix][io] KCA: Option to use kafka connector's SourceConnector class to 
create task and task config (#19772)
   
   (cherry picked from commit 90b0f0a17579d22d413853ed4941d81debbe0cbe)
   
   Author: Andrey Yegorov <8622884+dl...@users.noreply.github.com>
   Date:   Thu Mar 9 01:09:34 2023 -0800
   
   [fix][io] KCA: 'desanitize' topic name for the pulsar's ctx calls 
(#19756)
   
   (cherry picked from commit d4930a31c052dd8fcd5982b649898967a24f8961)
   ```
   
   ### Modifications
   
   cherry-picks
   
   ### Verifying this change
   
   - [ ] Make sure that the change passes the CI checks.
   
   
   ### Does this pull request potentially affect one of the following parts:
   
   
   
   NO
   
   *If the box was checked, please highlight the changes*
   
   - [ ] Dependencies (add or upgrade a dependency)
   - [ ] The public API
   - [ ] The schema
   - [ ] The default values of configurations
   - [ ] The threading model
   - [ ] The binary protocol
   - [ ] The REST endpoints
   - [ ] The admin CLI options
   - [ ] The metrics
   - [ ] Anything that affects deployment
   
   ### Documentation
   
   
   
   - [ ] `doc` 
   - [ ] `doc-required` 
   - [x] `doc-not-needed` 
   - [ ] `doc-complete` 
   
   ### Matching PR in forked repository
   
   PR in forked repository: n/a
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] dlg99 opened a new pull request, #19787: [cherry-pick][branch-2.11] KCA: picking fixes from master

[via GitHub]

dlg99 opened a new pull request, #19787:
URL: https://github.com/apache/pulsar/pull/19787

   ### Motivation
   
   Cherry-picking from master, please do not squash when merging
   
   ```
   Author: Andrey Yegorov <8622884+dl...@users.noreply.github.com>
   Date:   Fri Mar 10 01:58:40 2023 -0800
   
   [fix][io] KCA: Option to use kafka connector's SourceConnector class to 
create task and task config (#19772)
   
   (cherry picked from commit 90b0f0a17579d22d413853ed4941d81debbe0cbe)
   
   Author: Andrey Yegorov <8622884+dl...@users.noreply.github.com>
   Date:   Thu Mar 9 01:09:34 2023 -0800
   
   [fix][io] KCA: 'desanitize' topic name for the pulsar's ctx calls 
(#19756)
   
   (cherry picked from commit d4930a31c052dd8fcd5982b649898967a24f8961)
   ```
   
   ### Modifications
   
   cherry-picks
   
   ### Verifying this change
   
   - [ ] Make sure that the change passes the CI checks.
   
   
   ### Does this pull request potentially affect one of the following parts:
   
   
   
   NO
   
   *If the box was checked, please highlight the changes*
   
   - [ ] Dependencies (add or upgrade a dependency)
   - [ ] The public API
   - [ ] The schema
   - [ ] The default values of configurations
   - [ ] The threading model
   - [ ] The binary protocol
   - [ ] The REST endpoints
   - [ ] The admin CLI options
   - [ ] The metrics
   - [ ] Anything that affects deployment
   
   ### Documentation
   
   
   
   - [ ] `doc` 
   - [ ] `doc-required` 
   - [x] `doc-not-needed` 
   - [ ] `doc-complete` 
   
   ### Matching PR in forked repository
   
   PR in forked repository: n/a
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] vineeth1995 commented on pull request #19605: [feat] [broker] PIP-188 support blue-green cluster migration [part-2]

[via GitHub]

vineeth1995 commented on PR #19605:
URL: https://github.com/apache/pulsar/pull/19605#issuecomment-1464252845

   /pulsarbot run-failure-checks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-site] DaveDuggins commented on pull request #462: [feat][doc] Add *Get Started* for API (Java + REST) and CLI (pulsar-admin)

[via GitHub]

DaveDuggins commented on PR #462:
URL: https://github.com/apache/pulsar-site/pull/462#issuecomment-1464216117

   This is an excellent proposal. Please coordinate with Asaf and Julien, as I
   believe they are planning something similar.
   
   On Thu, Mar 9, 2023 at 5:22 AM Anonymitaet ***@***.***> wrote:
   
   > Hi @hangc0276  you reviewed the content of
   > this PR before in
   > 
https://docs.google.com/document/d/129rxvfJUzfycDe1wbqJVBIcpWaOzgKwMV6yhcwVavr0/edit#,
   > can you take a quick look on this? Thank you!
   >
   > —
   > Reply to this email directly, view it on GitHub
   > ,
   > or unsubscribe
   > 

   > .
   > You are receiving this because you were mentioned.Message ID:
   > ***@***.***>
   >
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] nodece commented on pull request #19782: [fix][broker] Fix admin api status code compatibility

[via GitHub]

nodece commented on PR #19782:
URL: https://github.com/apache/pulsar/pull/19782#issuecomment-1464126611

   Could you add a test to cover this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-client-node] trevordowdle commented on issue #303: libc++abi: terminating with uncaught exception of type std::invalid_argument: The scheme part is missing:

[via GitHub]

trevordowdle commented on issue #303:
URL: 
https://github.com/apache/pulsar-client-node/issues/303#issuecomment-1464119531

   For some additional context, I recently upgraded an existing project to 
1.8.1 from 1.7.0 and we had some configurations that would loop over different 
pulsar environments to create client connections.  A number of these 
configurations were had blank url's.  
   
   Where previously this would fail gracefully, all of a sudden the application 
just died with the above error message.  There was no additional traceable 
information in order to locate the source of the error.  
   
   So I had to take the application apart little by little and finally found 
the source/cause of the error, but it took a good amount of time and almost 
dissuading me from upgrading despite having all the dependencies wrapped up in 
1.8.
   
   If anything can be done here, I think it would be beneficial, especially for 
those looking to upgrade from previous versions as the error is not easy to 
diagnose.  
   
   Thanks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] hpvd commented on issue #19776: [doc][feat] add *Observability* guide and *Structured Event Logging* chapters

[via GitHub]

hpvd commented on issue #19776:
URL: https://github.com/apache/pulsar/issues/19776#issuecomment-1464047564

   just as addition 2 links..
   
   this is the PIP:
   https://github.com/apache/pulsar/wiki/PIP-89:-Structured-document-logging
   
   this is the master issue:
   https://github.com/apache/pulsar/issues/11799


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] michaeljmarshall commented on pull request #19455: [improve][broker] Require authRole is proxyRole to set originalPrincipal

[via GitHub]

michaeljmarshall commented on PR #19455:
URL: https://github.com/apache/pulsar/pull/19455#issuecomment-1464036418

   > In a pulsar cluster there are not only super roles, but also non-super 
roles, so authorization is still required
   
   Sure, and that is why this PR is necessary for the older branches. If the 
proxy has a super user role that is not in the `proxyRoles` configured by the 
broker and the proxy is using mTLS to authenticate with the broker, all clients 
going through the proxy inherit the proxy's superuser role.
   
   > Before this change, if a client wanted to connect to the cluster using the 
super role, the role had to be configured in the proxy's superUserRoles, 
otherwise the proxy would not be able to authenticate it, after this change, if 
a client connected to the cluster using the super role, the role could not 
appear in superUserRoles of the proxy and proxyRoles of the broker, otherwise 
the connect also will fail, which seems to be an opposite behavior
   
   This analysis does not match my understanding of the PR. The proxy's 
`superUserRoles` has nothing to do with this change. This PR only changes the 
broker's logic to require an `originalPrincipal` to be supplied, the `role` 
must be one of the `proxyRoles` in the broker.conf. The core logic is here:
   
   
https://github.com/apache/pulsar/blob/d4be954dedcc7537b3d65b9a1d7b5662e6062fdf/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/authorization/AuthorizationService.java#L332-L335
   
   It sounds like the problem you're encountering is that your proxy's role is 
not in the `proxyRoles` list. The tests modified by this PR support my 
understanding of this change.
   
   > I think this is a security enhancement, not a vulnerability (I'm sure you 
agree)
   
   I disagree that this is only an enhancement. This change protects operators 
from dangerous misconfigurations.
   
   The only way this is not a vulnerability is if we agree that a proxy is 
supposed to connect with a role in the `proxyRoles`. It is a vulnerability if 
we decide that the `proxyRoles` list is irrelevant because it's not the user 
misconfiguring things but rather pulsar doing the wrong thing.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] poorbarcode commented on pull request #19751: [fix] [test] fix flaky test BucketDelayedDeliveryTrackerTest. testWithBkException

[via GitHub]

poorbarcode commented on PR #19751:
URL: https://github.com/apache/pulsar/pull/19751#issuecomment-1464018864

   /pulsarbot run-failure-checks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-site] cbornet commented on pull request #463: Add Pulsar Adapters 2.11.0 release

[via GitHub]

cbornet commented on PR #463:
URL: https://github.com/apache/pulsar-site/pull/463#issuecomment-1463979852

   AFAIK, the adapters use the Pulsar client so they have the same 
compatibility as the Pulsar client. Which I think is to be compatible with any 
2.x cluster. Others may know more about this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-client-cpp] BewareMyPower commented on a diff in pull request #196: [TableView-2] Implement all interfaces.

[via GitHub]

BewareMyPower commented on code in PR #196:
URL: https://github.com/apache/pulsar-client-cpp/pull/196#discussion_r1132534400


##
lib/TableViewImpl.cc:
##
@@ -0,0 +1,169 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+#include "TableViewImpl.h"
+
+#include "ClientImpl.h"
+#include "LogUtils.h"
+#include "ReaderImpl.h"
+#include "TimeUtils.h"
+
+namespace pulsar {
+
+DECLARE_LOG_OBJECT()
+
+TableViewImpl::TableViewImpl(ClientImplPtr client, const std::string& topic,
+ const TableViewConfiguration& conf)
+: client_(client), topic_(topic), conf_(conf) {}
+
+Future TableViewImpl::start() {
+Promise promise;
+ReaderConfiguration readerConfiguration;
+readerConfiguration.setSchema(conf_.schemaInfo);
+readerConfiguration.setReadCompacted(true);
+readerConfiguration.setInternalSubscriptionName(conf_.subscriptionName);
+
+TableViewImplPtr self = shared_from_this();
+ReaderCallback readerCallback = [self, promise](Result res, Reader reader) 
{
+if (res == ResultOk) {
+self->reader_ = reader.impl_;
+self->readAllExistingMessages(promise, 
TimeUtils::currentTimeMillis(), 0);
+} else {
+promise.setFailed(res);
+}
+};
+client_->createReaderAsync(topic_, MessageId::earliest(), 
readerConfiguration, readerCallback);
+return promise.getFuture();
+}
+
+bool TableViewImpl::retrieveValue(const std::string& key, std::string& value) {
+auto optValue = data_.remove(key);
+if (optValue) {
+value = optValue.value();
+return true;
+}
+return false;
+}
+
+bool TableViewImpl::getValue(const std::string& key, std::string& value) const 
{
+auto optValue = data_.find(key);
+if (optValue) {
+value = optValue.value();
+return true;
+}
+return false;
+}
+
+bool TableViewImpl::containsKey(const std::string& key) const { return 
data_.find(key) != boost::none; }
+
+std::unordered_map TableViewImpl::snapshot() { 
return data_.move(); }
+
+std::size_t TableViewImpl::size() const { return data_.size(); }
+
+void TableViewImpl::forEach(TableViewAction action) { data_.forEach(action); }
+
+void TableViewImpl::forEachAndListen(TableViewAction action) {
+data_.forEach(action);
+Lock lock(listenersMutex_);
+listeners_.emplace_back(action);
+}
+
+void TableViewImpl::closeAsync(ResultCallback callback) {
+if (reader_) {
+reader_->closeAsync([callback, this](Result result) {
+reader_.reset();
+callback(result);
+});
+} else {
+callback(ResultConsumerNotInitialized);
+}
+}
+
+void TableViewImpl::handleMessage(const Message& msg) {
+if (msg.hasPartitionKey()) {
+auto value = msg.getDataAsString();
+LOG_DEBUG("Applying message from " << topic_ << " key=" << 
msg.getPartitionKey()
+   << " value=" << value)
+
+if (msg.getLength() == 0) {
+data_.remove(msg.getPartitionKey());
+} else {
+data_.emplace(msg.getPartitionKey(), value);
+}
+
+Lock lock(listenersMutex_);
+for (const auto& listener : listeners_) {
+try {
+listener(msg.getPartitionKey(), value);
+} catch (const std::exception& exc) {
+LOG_ERROR("Table view listener raised an exception: " << 
exc.what());
+}
+}
+}
+}
+
+void TableViewImpl::readAllExistingMessages(Promise 
promise, long startTime,
+long messagesRead) {
+std::weak_ptr weakSelf{shared_from_this()};
+reader_->hasMessageAvailableAsync(
+[weakSelf, promise, startTime, messagesRead](Result result, bool 
hasMessage) {
+auto self = weakSelf.lock();
+if (!self || result != ResultOk) {
+promise.setFailed(result);
+return;
+}
+if (hasMessage) {
+Message msg;
+self->reader_->readNextAsync(
+[weakSelf, promise, startTime, messagesRead](Result res, 
const Message& msg) {
+auto self = 

[GitHub] [pulsar-site] 5exceptions commented on pull request #440: [fix][doc] Changed in the IBM login url

[via GitHub]

5exceptions commented on PR #440:
URL: https://github.com/apache/pulsar-site/pull/440#issuecomment-1463967709

   @Anonymitaet : Conflicts are resolved. Please review.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] gaoran10 commented on a diff in pull request #19730: [improve][broker] PIP-192 Made only the leader consume TopBundlesLoadDataStore

[via GitHub]

gaoran10 commented on code in PR #19730:
URL: https://github.com/apache/pulsar/pull/19730#discussion_r1132040771


##
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/store/TableViewLoadDataStoreImpl.java:
##
@@ -61,30 +70,58 @@ public CompletableFuture removeAsync(String key) {
 
 @Override
 public Optional get(String key) {
+validateTableViewStart();

Review Comment:
   According to this code block, even if it failed to start the 
`topBundlesLoadDataStore`,  the `playLeader` still can complete, if the 
`UnloadScheduler` tries to find bundles that need to unload, it will try to get 
topKBundlesData, at this moment it will encounter IllegalStateException("table 
view has not been started").
   
   ```
   private void playLeader() {
   this.pulsar.getLoadManagerExecutor().execute(() -> {
   serviceUnitStateChannel.scheduleOwnershipMonitor();
   waitForLoadStoreInit();
   try {
   topBundlesLoadDataStore.startTableView();
   } catch (Throwable e) {
   log.error("The new leader:{} failed to start 
topBundlesLoadDataStore tableview",
   pulsar.getLookupServiceAddress(), e);
   }
   ...
   log.info("This broker:{} is the leader now.", 
pulsar.getLookupServiceAddress());
   });
   }
   
   ```
   
   TransferShedder.java
   ```
   public UnloadDecision findBundlesForUnloading(LoadManagerContext context,
 Map 
recentlyUnloadedBundles,
 Map 
recentlyUnloadedBrokers) {
   try {
   ...
   // If topBundlesLoadDataStore is not initialized, 
IllegalStateException will be thrown.
   Optional bundlesLoadData = 
context.topBundleLoadDataStore().get(maxBroker);
   ...
   }  catch (Throwable e) {
   log.error("Failed to process unloading. ", e);
   decision.fail();
   }
   return decision;
   
   }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] gaoran10 commented on a diff in pull request #19730: [improve][broker] PIP-192 Made only the leader consume TopBundlesLoadDataStore

[via GitHub]

gaoran10 commented on code in PR #19730:
URL: https://github.com/apache/pulsar/pull/19730#discussion_r1132040771


##
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/store/TableViewLoadDataStoreImpl.java:
##
@@ -61,30 +70,58 @@ public CompletableFuture removeAsync(String key) {
 
 @Override
 public Optional get(String key) {
+validateTableViewStart();

Review Comment:
   According to this code block, even if it failed to start the 
`topBundlesLoadDataStore`,  the `playLeader` still can complete, if the 
`UnloadScheduler` tries to find bundles that need to unload, it will try to get 
topKBundlesData, at this moment it will encounter IllegalStateException("table 
view has not been started").
   
   ```
   private void playLeader() {
   this.pulsar.getLoadManagerExecutor().execute(() -> {
   serviceUnitStateChannel.scheduleOwnershipMonitor();
   waitForLoadStoreInit();
   try {
   topBundlesLoadDataStore.startTableView();
   } catch (Throwable e) {
   log.error("The new leader:{} failed to start 
topBundlesLoadDataStore tableview",
   pulsar.getLookupServiceAddress(), e);
   }
   ...
   log.info("This broker:{} is the leader now.", 
pulsar.getLookupServiceAddress());
   });
   }
   
   ```
   
   ```
   public UnloadDecision findBundlesForUnloading(LoadManagerContext context,
 Map 
recentlyUnloadedBundles,
 Map 
recentlyUnloadedBrokers) {
   try {
   ...
   // If topBundlesLoadDataStore is not initialized, 
IllegalStateException will be thrown.
   Optional bundlesLoadData = 
context.topBundleLoadDataStore().get(maxBroker);
   ...
   }  catch (Throwable e) {
   log.error("Failed to process unloading. ", e);
   decision.fail();
   }
   return decision;
   
   }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] gaoran10 commented on a diff in pull request #19730: [improve][broker] PIP-192 Made only the leader consume TopBundlesLoadDataStore

[via GitHub]

gaoran10 commented on code in PR #19730:
URL: https://github.com/apache/pulsar/pull/19730#discussion_r1132040771


##
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/store/TableViewLoadDataStoreImpl.java:
##
@@ -61,30 +70,58 @@ public CompletableFuture removeAsync(String key) {
 
 @Override
 public Optional get(String key) {
+validateTableViewStart();

Review Comment:
   According to this code block, even if it failed to start the 
`topBundlesLoadDataStore`,  the `playLeader` still can complete, if the 
`UnloadScheduler` tries to find bundles that need to unload, it will try to get 
topKBundlesData, at this moment it will encounter IllegalStateException("table 
view has not been started").
   
   ```
   private void playLeader() {
   this.pulsar.getLoadManagerExecutor().execute(() -> {
   serviceUnitStateChannel.scheduleOwnershipMonitor();
   waitForLoadStoreInit();
   try {
   topBundlesLoadDataStore.startTableView();
   } catch (Throwable e) {
   log.error("The new leader:{} failed to start 
topBundlesLoadDataStore tableview",
   pulsar.getLookupServiceAddress(), e);
   }
   ...
   log.info("This broker:{} is the leader now.", 
pulsar.getLookupServiceAddress());
   });
   }
   
   ```
   
   ```
   public UnloadDecision findBundlesForUnloading(LoadManagerContext context,
 Map 
recentlyUnloadedBundles,
 Map 
recentlyUnloadedBrokers) {
   try {
   ...
   Optional bundlesLoadData = 
context.topBundleLoadDataStore().get(maxBroker);
   ...
   }  catch (Throwable e) {
   log.error("Failed to process unloading. ", e);
   decision.fail();
   }
   return decision;
   
   }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] gaoran10 commented on a diff in pull request #19514: [fix] [admin] Admin API can not work if uri too large

[via GitHub]

gaoran10 commented on code in PR #19514:
URL: https://github.com/apache/pulsar/pull/19514#discussion_r1132499251


##
pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/ProxyConfiguration.java:
##
@@ -661,6 +661,29 @@ public class ProxyConfiguration implements 
PulsarConfiguration {
 )
 private int httpOutputBufferSize = 32 * 1024;
 
+@FieldContext(
+minValue = 1,
+category = CATEGORY_HTTP,
+doc = """
+The maximum size in bytes of the request header.
+Larger headers will allow for more and/or larger cookies plus 
larger form content encoded in a URL.
+However, larger headers consume more memory and can make a 
server more vulnerable to denial of service
+attacks.
+  """
+)
+private int httpMaxRequestHeaderSize = 8 * 1024;
+
+@FieldContext(
+minValue = 1,
+category = CATEGORY_HTTP,
+doc = """
+ the size of the buffer used to write requests to Broker.
+ if "httpMaxRequestHeaderSize" is large than 
"httpClientRequestBufferSize", will set
+ "httpClientRequestBufferSize" to the value of 
"httpMaxRequestHeaderSize"
+  """
+)
+private int httpClientRequestBufferSize = 4096;

Review Comment:
   Does this config is necessary?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] eolivelli opened a new issue, #19786: [transactions] Add admin API to abort forcibly a transaction

[via GitHub]

eolivelli opened a new issue, #19786:
URL: https://github.com/apache/pulsar/issues/19786

   ### Search before asking
   
   - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) 
and found nothing similar.
   
   
   ### Motivation
   
   If you have a pending stuck transaction the Consumers are stuck, they cannot 
read beyond the beginning of the transaction.
   
   You have to wait for the transaction timeout in order to see the system 
recover
   
   ### Solution
   
   In Kafka you have ad admin command "abortTransactions", we need something 
like that.
   
   
https://cwiki.apache.org/confluence/display/KAFKA/KIP-664%3A+Provide+tooling+to+detect+and+abort+hanging+transactions
   
   ### Alternatives
   
   _No response_
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [ ] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] jeantil opened a new issue, #19785: [Bug] pulsar admin client in pulsar-client-all logs a ClassNotFoundException at startup

[via GitHub]

jeantil opened a new issue, #19785:
URL: https://github.com/apache/pulsar/issues/19785

   ### Search before asking
   
   - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) 
and found nothing similar.
   
   
   ### Version
   
   2.11.0
   
   ### Minimal reproduce step
   
   add the pulsar-client-all-2.11.0.jar as a dependency, try to create an admin 
client 
   
   ### What did you expect to see?
   
   no warnings or a warning that reports a legitimate reason for conscrypt to 
be unavailable
   
   ### What did you see instead?
   
   ```
   java.lang.ClassNotFoundException: org.conscrypt.Conscrypt
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown 
Source)
at 
java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown 
Source)
at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Unknown Source)
at 
org.apache.pulsar.common.util.SecurityUtility.loadConscryptProvider(SecurityUtility.java:124)
at 
org.apache.pulsar.common.util.SecurityUtility.(SecurityUtility.java:81)
at 
org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.(AsyncHttpConnector.java:165)
at 
org.apache.pulsar.client.admin.internal.http.AsyncHttpConnectorProvider.getConnector(AsyncHttpConnectorProvider.java:52)
at 
org.apache.pulsar.client.admin.internal.PulsarAdminImpl.(PulsarAdminImpl.java:183)
at 
org.apache.pulsar.client.admin.internal.PulsarAdminBuilderImpl.build(PulsarAdminBuilderImpl.java:48)
   ...
   ```
   
   ### Anything else?
   
   I think the issue stems from shading the conscrypt jar in the 
pulsar-client-all jar. 
   The Conscrypt class is relocated under 
org.apache.pulsar.shade.io.netty.handler.ssl but the SecurityUtility class 
continues to look for `org.conscrypt.Conscrypt` which isn't found
   
   
![image](https://user-images.githubusercontent.com/22979/224327905-8b8b886f-c6ee-4760-8460-1dc552b1a58e.png)
   
   the following Junit5 test exhibits the issue
   ```
   package org.apache.james.queue.pulsar;
   
   import org.apache.james.backends.pulsar.Auth;
   import org.apache.pulsar.client.admin.PulsarAdmin;
   import org.apache.pulsar.client.api.Authentication;
   import org.apache.pulsar.client.api.PulsarClientException;
   import org.apache.pulsar.client.impl.auth.AuthenticationDisabled;
   import org.apache.pulsar.client.impl.auth.AuthenticationToken;
   import org.junit.jupiter.api.Test;
   
   import static org.assertj.core.api.AssertionsForClassTypes.assertThat;
   
   public class DemonstratePulsarIssue {
   @Test
   void should_not_log_about_conscrypt() {
   try {
   var adminclient= PulsarAdmin.builder()
   .serviceHttpUrl("https://some-pulsar-service.com;)
   .authentication( new AuthenticationToken("foo"))
   .build();
   adminclient.close();
   } catch (PulsarClientException e) {
   assertThat(e.toString()).isNull();
   }
   
   }
   }
   ```
   yields
   ```
   java.lang.ClassNotFoundException: org.conscrypt.Conscrypt
at 
java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
at 
java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:315)
at 
org.apache.pulsar.common.util.SecurityUtility.loadConscryptProvider(SecurityUtility.java:124)
at 
org.apache.pulsar.common.util.SecurityUtility.(SecurityUtility.java:81)
at 
org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.(AsyncHttpConnector.java:165)
at 
org.apache.pulsar.client.admin.internal.http.AsyncHttpConnectorProvider.getConnector(AsyncHttpConnectorProvider.java:52)
at 
org.apache.pulsar.client.admin.internal.PulsarAdminImpl.(PulsarAdminImpl.java:183)
at 
org.apache.pulsar.client.admin.internal.PulsarAdminBuilderImpl.build(PulsarAdminBuilderImpl.java:48)
at 
org.apache.james.queue.pulsar.DemonstratePulsarIssue.should_not_log_about_conscrypt(DemonstratePulsarIssue.java:20)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at 
org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
at 
org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at 

[GitHub] [pulsar] ThomasTaketurns opened a new issue, #19784: [Bug] Deadlock on Pulsar Java client side

[via GitHub]

ThomasTaketurns opened a new issue, #19784:
URL: https://github.com/apache/pulsar/issues/19784

   ### Search before asking
   
   - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) 
and found nothing similar.
   
   
   ### Version
   
   The broker version is 
https://pulsar.apache.org/release-notes/versioned/pulsar-2.10.1/
   And the client is : 
https://javadoc.io/doc/org.apache.pulsar/pulsar-client-api/2.10.3/index.html
   
   ### Minimal reproduce step
   
   We are currently facing what seems to be a deadlock issue on our testing 
environment.
   We are working with the Java SDK.
   In my thread dump I can see 24 threads stuck on this line :
   final Transaction txn = 
this.getClient().newTransaction().withTransactionTimeout(1, 
TimeUnit.MINUTES).build().get();
   Where getClient returns an instance of PulsarClient.
   at jdk.internal.misc.Unsafe.park(java.base@17.0.2/Native Method)
   - parking to wait for <0xad73c838> (a 
java.util.concurrent.CompletableFuture$Signaller)
   at 
java.util.concurrent.locks.LockSupport.park(java.base@17.0.2/LockSupport.java:211)
   at 
java.util.concurrent.CompletableFuture$Signaller.block(java.base@17.0.2/CompletableFuture.java:1864)
   at 
java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@17.0.2/ForkJoinPool.java:3463)
   at 
java.util.concurrent.ForkJoinPool.managedBlock(java.base@17.0.2/ForkJoinPool.java:3434)
   at 
java.util.concurrent.CompletableFuture.waitingGet(java.base@17.0.2/CompletableFuture.java:1898)
   at 
java.util.concurrent.CompletableFuture.get(java.base@17.0.2/CompletableFuture.java:2072)
   at 
[com.taketurns.es](http://com.taketurns.es/).pulsar.controller.EntityPulsarController.publishEvents(EntityPulsarController.java:75)
   Those threads are waiting for a CompletableFuture that never complete.
   I can also see that pulsar-timer-9-1 thread is blocked.
   java.lang.Thread.State: BLOCKED (on object monitor)
   at org.apache.pulsar.client.impl.ProducerImpl.run(ProducerImpl.java:1876)
   - waiting to lock <0xad6fd530> (a 
org.apache.pulsar.client.impl.ProducerImpl)
   at 
[org.apache.pulsar.shade.io](http://org.apache.pulsar.shade.io/).netty.util.HashedWheelTimer$HashedWheelTimeout.run(HashedWheelTimer.java:715)
   at 
[org.apache.pulsar.shade.io](http://org.apache.pulsar.shade.io/).netty.util.concurrent.ImmediateExecutor.execute(ImmediateExecutor.java:34)
   at 
[org.apache.pulsar.shade.io](http://org.apache.pulsar.shade.io/).netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:703)
   at 
[org.apache.pulsar.shade.io](http://org.apache.pulsar.shade.io/).netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:790)
   at 
[org.apache.pulsar.shade.io](http://org.apache.pulsar.shade.io/).netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:503)
   at 
[org.apache.pulsar.shade.io](http://org.apache.pulsar.shade.io/).netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
   at java.lang.Thread.run(java.base@17.0.2/Thread.java:833)
   Locked ownable synchronizers:
   - None
   And the thread holding the lock on <0xad6fd530> is the following.
   pulsar-client-io-1-1
   Stack Trace is:
   java.lang.Thread.State: WAITING (parking)
   at jdk.internal.misc.Unsafe.park(java.base@17.0.2/Native Method)
   - parking to wait for <0xad6fd400> (a 
java.util.concurrent.CompletableFuture$Signaller)
   at 
java.util.concurrent.locks.LockSupport.park(java.base@17.0.2/LockSupport.java:211)
   at 
java.util.concurrent.CompletableFuture$Signaller.block(java.base@17.0.2/CompletableFuture.java:1864)
   at 
java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@17.0.2/ForkJoinPool.java:3463)
   at 
java.util.concurrent.ForkJoinPool.managedBlock(java.base@17.0.2/ForkJoinPool.java:3434)
   at 
java.util.concurrent.CompletableFuture.waitingGet(java.base@17.0.2/CompletableFuture.java:1898)
   at 
java.util.concurrent.CompletableFuture.get(java.base@17.0.2/CompletableFuture.java:2072)
   at 
org.apache.pulsar.client.impl.schema.AbstractStructSchema.atSchemaVersion(AbstractStructSchema.java:100)
   at 
org.apache.pulsar.client.impl.MessageImpl.getReaderSchema(MessageImpl.java:400)
   at 
org.apache.pulsar.client.impl.ConsumerImpl.lambda$doReconsumeLater$11(ConsumerImpl.java:657)
   at 
org.apache.pulsar.client.impl.ConsumerImpl$$Lambda$4185/0x000801ee43e0.accept(Unknown
 Source)
   at 
java.util.concurrent.CompletableFuture$UniAccept.tryFire(java.base@17.0.2/CompletableFuture.java:718)
   at 
java.util.concurrent.CompletableFuture.postComplete(java.base@17.0.2/CompletableFuture.java:510)
   at 
java.util.concurrent.CompletableFuture.complete(java.base@17.0.2/CompletableFuture.java:2147)
   at 
org.apache.pulsar.client.impl.ProducerImpl.lambda$resendMessages$18(ProducerImpl.java:1795)
   - locked <0xad6fd530> (a org.apache.pulsar.client.impl.ProducerImpl)
   at 

[GitHub] [pulsar] tuteng commented on pull request #19455: [improve][broker] Require authRole is proxyRole to set originalPrincipal

[via GitHub]

tuteng commented on PR #19455:
URL: https://github.com/apache/pulsar/pull/19455#issuecomment-1463770662

   > > in many user environments it can be trusted and there is no such risk
   > 
   > Can you clarify what "it" is here? What is the purpose of authorization if 
the environment is trusted?
   
   `it` means is that the client application uses the proxy's super role to 
connect to the cluster
   In a pulsar cluster there are not only super roles, but also non-super 
roles, so authorization is still required, and if a client has been assigned a 
super role, it is of course trusted
   
   Before this change, if a client wanted to connect to the cluster using the 
super role, the role had to be configured in the proxy's superUserRoles, 
otherwise the proxy would not be able to authenticate it, after this change, if 
a client connected to the cluster using the super role, the role could not 
appear in superUserRoles of the proxy and proxyRoles of the broker, otherwise 
the connect also will fail, which seems to be an opposite behavior
   
   I think this is a security enhancement, not a vulnerability (I'm sure you 
agree), but now that this change has been introduced on most major branches.I 
understand it doesn't make sense to use the proxy's superRoles, but it works 
correctly, and if a user performs a minor version upgrade without understanding 
the context, such as upgrading a cluster from 2.10.3 to 2.10.4 (ideally there 
should be no breaking changes), that will result in clients not being able to 
successfully connect to the cluster, which will cause some failures, which have 
actually happened in some users' test environments
   
   I don't think it is a problem to introduce this change in a major release 
(e.g. 3.0) because it will give users enough time to understand it. Minor 
upgrades are frequent, so I don't think it is appropriate to introduce it on a 
minor release and not be configurable, which can very easily lead to 
unpredictable failures


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] yyj8 closed pull request #19047: [improve][broker]PIP-214 Add broker level metrics statistics and expose to prometheus

[via GitHub]

yyj8 closed pull request #19047: [improve][broker]PIP-214 Add broker level 
metrics statistics and expose to prometheus
URL: https://github.com/apache/pulsar/pull/19047


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-client-cpp] RobertIndie commented on pull request #207: [feat] Support pattern subscribe non-persistent topic.

[via GitHub]

RobertIndie commented on PR #207:
URL: 
https://github.com/apache/pulsar-client-cpp/pull/207#issuecomment-1463689936

   > I decide what subscriptionTopicsMode is based on the fact that the domain 
is included in the pattern.
   
   Great idea. Are there any other contexts that we need to use 
RegexSubscriptionMode except for this case in the Java client? Would it be 
possible also apply this solution to the Java client?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] yyj8 closed pull request #19046: [improve][doc]Add broker level metrics documentation

[via GitHub]

yyj8 closed pull request #19046: [improve][doc]Add broker level metrics 
documentation
URL: https://github.com/apache/pulsar/pull/19046


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] eolivelli opened a new pull request, #19783: [improve][offloaders] Automatically evict Offloaded Ledgers from memory

[via GitHub]

eolivelli opened a new pull request, #19783:
URL: https://github.com/apache/pulsar/pull/19783

   ### Motivation
   
   ManagedLedgerImpl retains eagerly a cache of all the BookKeeper ReadHandles.
   In case of Offloaded ReadHandler there is kind of a memory leak, as each 
BlobStoreBackedInputStreamImpl retains a DirectBuffer of 1MB, in the case of a 
topic with terabytes of data and thousands of ledger this leads to Direct OOM 
errors if something tries to open all the ledgers
   
   ### Modifications
   
   Add a new  background activity that evicts from memory all the Offloaded 
ReadHandles and release memory.
   
   The feature is controlled by the new configuration option 
managedLedgerInactiveOffloadedLedgerEvictionTimeSeconds=600
   
   Unfortunately this fix cannot fully prevent a OODM error because there is no 
global count and limit of the memory retained by such Handles, it allows to 
mitigate the problem by releasing automatically unused Ledger Handlers.
   The default value, 10 minutes, is very conservative, but it should work with 
real-world ledgers.
   
   The worst case scenario is a topic with tens of thousands of small ledgers 
with a consumer that reads from the topic from the beginning, in this case the 
broker will open the ReadHandlers probably more quickly than the eviction 
process pace.
   
   ### Verifying this change
   
   
   This change added tests.
   
   ### Does this pull request potentially affect one of the following parts:
   
   
   
   *If the box was checked, please highlight the changes*
   
   - [ ] Dependencies (add or upgrade a dependency)
   - [ ] The public API
   - [ ] The schema
   - [ ] The default values of configurations
   - [ ] The threading model
   - [ ] The binary protocol
   - [ ] The REST endpoints
   - [ ] The admin CLI options
   - [ ] The metrics
   - [ ] Anything that affects deployment
   
   ### Documentation
   
   
   
   - [ ] `doc` 
   - [ ] `doc-required` 
   - [x] `doc-not-needed` 
   - [ ] `doc-complete` 
   
   ### Matching PR in forked repository
   
   PR in forked repository: 
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-helm-chart] Shohou opened a new issue, #350: PULSAR_MEM variable doesn't set limits for bookie

[via GitHub]

Shohou opened a new issue, #350:
URL: https://github.com/apache/pulsar-helm-chart/issues/350

   **Describe the bug**
   I found that changing `PULSAR_MEM` variable for bookkeeper which can be 
found in `values.yaml` file doesn't change memory params of bookie at least for 
pulsar v2.9.4
   Some investigation shows that in version 2.6.x `bin/pulsar` script had this 
line
   ```
   OPTS="$OPTS $PULSAR_EXTRA_OPTS $PULSAR_MEM $PULSAR_GC"
   ```
   from version 2.7.x till 2.9.x it has this:
   ```
   if [ $COMMAND == "bookie" ]; then
 # Pass BOOKIE_EXTRA_OPTS option defined in bkenv.sh
 OPTS="$OPTS $BOOKIE_EXTRA_OPTS"
   else
 OPTS="$OPTS $PULSAR_EXTRA_OPTS $PULSAR_MEM $PULSAR_GC"
   fi
   ```
   and versions 2.10.x - 2.11.x has this:
   ```
   if [ $COMMAND == "bookie" ]; then
 # Pass BOOKIE_EXTRA_OPTS option defined in bkenv.sh
 OPTS="$OPTS $BOOKIE_MEM $BOOKIE_GC $BOOKIE_GC_LOG $BOOKIE_EXTRA_OPTS"
   else
 OPTS="$OPTS $PULSAR_MEM $PULSAR_GC $PULSAR_GC_LOG $PULSAR_EXTRA_OPTS"
   fi
   ```
   
   I think it could be changed to BOOKIE_MEM with some comment about earlier 
versions
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] mattisonchao closed pull request #19782: [fix][broker] Fix admin api status code compatibility

[via GitHub]

mattisonchao closed pull request #19782: [fix][broker] Fix admin api status 
code compatibility
URL: https://github.com/apache/pulsar/pull/19782


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] mattisonchao opened a new pull request, #19782: [fix][broker] Fix admin api status code compatibility issue

[via GitHub]

mattisonchao opened a new pull request, #19782:
URL: https://github.com/apache/pulsar/pull/19782

   ### Motivation
   
   https://github.com/apache/pulsar/pull/19166/files#r1129126330
   
   ### Modifications
   
   - Change the status code from 422 to 409.
   
   ### Verifying this change
   
   - [x] Make sure that the change passes the CI checks.
   
   ### Documentation
   
   
   
   - [ ] `doc` 
   - [ ] `doc-required` 
   - [x] `doc-not-needed` 
   - [ ] `doc-complete` 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] Denovo1998 commented on pull request #19753: [cleanup][broker] Remove duplicate code in the SchemaRegistryServiceImpl that checks for existing schema and new schema types

[via GitHub]

Denovo1998 commented on PR #19753:
URL: https://github.com/apache/pulsar/pull/19753#issuecomment-1463565822

   /pulsarbot run-failure-checks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] eolivelli commented on pull request #19772: [fix][io] KCA: Option to use kafka connector's SourceConnector class to create task and task config

[via GitHub]

eolivelli commented on PR #19772:
URL: https://github.com/apache/pulsar/pull/19772#issuecomment-1463559111

   @dlg99 I would like to cherry-pick this patch to branch-2.11 and to 
branch-2.10, but the patch doesn't apply.
   
   Could you please send a new PR ? or maybe there are dependencies that we are 
missing ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-dotpulsar] kandersen82 opened a new pull request, #142: Async lock leak

[via GitHub]

kandersen82 opened a new pull request, #142:
URL: https://github.com/apache/pulsar-dotpulsar/pull/142

   Fixes a memory leak in AsyncLock
   
   # Description
   
   Pending lock requests in AsyncLock had a registration that was never 
disposed.
   
   # Regression
   
   Issue was introduced in initial commit
   
   # Testing
   
   Manual testing with memory profiler


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[pulsar] branch master updated (9feb85b19ca -> 90b0f0a1757)

[eolivelli]

This is an automated email from the ASF dual-hosted git repository.

eolivelli pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git


from 9feb85b19ca [fix][client] Fix topic list watcher fail log (#19733)
 add 90b0f0a1757 [fix][io] KCA: Option to use kafka connector's 
SourceConnector class to create task and task config (#19772)

No new revisions were added by this update.

Summary of changes:
 .../kafka/connect/AbstractKafkaConnectSource.java  | 48 ---
 .../io/kafka/connect/KafkaConnectSourceTest.java   | 54 ++
 2 files changed, 77 insertions(+), 25 deletions(-)

[GitHub] [pulsar] eolivelli merged pull request #19772: [fix][io] KCA: Option to use kafka connector's SourceConnector class to create task and task config

[via GitHub]

eolivelli merged PR #19772:
URL: https://github.com/apache/pulsar/pull/19772


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-client-go] Gleiphir2769 opened a new pull request, #989: [perf]: More precise producer rate limiter

[via GitHub]

Gleiphir2769 opened a new pull request, #989:
URL: https://github.com/apache/pulsar-client-go/pull/989

   ### Motivation
   
   Currently, when the rate of the producer is set relatively small, the actual 
production rate is not accurate.
   
   For example, set the rate aas 1.
   
   ```
   $ ./pulsar-perf-old produce my-test-r1 -r 1
   
   INFO[17:20:12.524] Stats - Publish rate:2.9 msg/s -0.0 Mbps - 
   Latency ms: 50%   3.8 -95% 1004.0 - 99% 
1004.3 - 99.9% 1004.3 - max 1005.2 
   INFO[17:20:22.523] Stats - Publish rate:3.0 msg/s -0.0 Mbps - 
   Latency ms: 50%   3.8 -95% 1004.0 - 99% 
1004.2 - 99.9% 1004.2 - max 1004.9 
   INFO[17:20:32.523] Stats - Publish rate:3.0 msg/s -0.0 Mbps - 
   Latency ms: 50%   3.9 -95% 1004.0 - 99% 
1004.0 - 99.9% 1004.0 - max 1004.1 
   
   ```
   
   After modify the rate limiter, the result can be improved as that.
   
   ```
   $ ./pulsar-perf produce my-test-r1 -r 1
   
   INFO[17:22:45.971] Stats - Publish rate:1.0 msg/s -0.0 Mbps - 
   Latency ms: 50% 1004.4 -95% 1005.1 - 99% 
1005.1 - 99.9% 1005.1 - max 1006.2 
   INFO[17:22:55.971] Stats - Publish rate:1.0 msg/s -0.0 Mbps - 
   Latency ms: 50% 1004.2 -95% 1007.4 - 99% 
1007.4 - 99.9% 1007.4 - max 1008.0 
   INFO[17:23:05.971] Stats - Publish rate:1.0 msg/s -0.0 Mbps - 
   Latency ms: 50% 1003.4 -95% 1004.7 - 99% 
1004.7 - 99.9% 1004.7 - max 1008.1 
   INFO[17:23:15.971] Stats - Publish rate:1.0 msg/s -0.0 Mbps - 
   Latency ms: 50% 1003.5 -95% 1004.3 - 99% 
1004.3 - 99.9% 1004.3 - max 1004.6 
   
   ```
   
   ### Modifications
   
   - Use the "golang.org/x/time/rate" as producer limiter.
   
   ### Verifying this change
   
   - [x] Make sure that the change passes the CI checks.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] codecov-commenter commented on pull request #19561: [improve][meta] Support set metadata size threshold for compression.

[via GitHub]

codecov-commenter commented on PR #19561:
URL: https://github.com/apache/pulsar/pull/19561#issuecomment-1463476049

   ## 
[Codecov](https://codecov.io/gh/apache/pulsar/pull/19561?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)
 Report
   > Merging 
[#19561](https://codecov.io/gh/apache/pulsar/pull/19561?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)
 (4282e6d) into 
[master](https://codecov.io/gh/apache/pulsar/commit/e0b50c9ec5f12d0fb8275f235d8ac00e87a9099e?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)
 (e0b50c9) will **increase** coverage by `6.64%`.
   > The diff coverage is `50.00%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/pulsar/pull/19561/graphs/tree.svg?width=650=150=pr=acYqCpsK9J_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pulsar/pull/19561?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)
   
   ```diff
   @@ Coverage Diff  @@
   ## master   #19561  +/-   ##
   
   + Coverage 24.65%   31.30%   +6.64% 
   - Complexity  291  293   +2 
   
 Files  1591 1673  +82 
 Lines123073   131631+8558 
 Branches  1343415117+1683 
   
   + Hits  3034841209   +10861 
   + Misses8823384320-3913 
   - Partials   4492 6102+1610 
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | inttests | `25.53% <50.00%> (+0.88%)` | :arrow_up: |
   | systests | `25.34% <50.00%> (?)` | |
   
   Flags with carried forward coverage won't be shown. [Click 
here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment)
 to find out more.
   
   | [Impacted 
Files](https://codecov.io/gh/apache/pulsar/pull/19561?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation)
 | Coverage Δ | |
   |---|---|---|
   | 
[.../apache/bookkeeper/mledger/impl/MetaStoreImpl.java](https://codecov.io/gh/apache/pulsar/pull/19561?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-bWFuYWdlZC1sZWRnZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2Jvb2trZWVwZXIvbWxlZGdlci9pbXBsL01ldGFTdG9yZUltcGwuamF2YQ==)
 | `43.16% <18.51%> (+2.15%)` | :arrow_up: |
   | 
[.../bookkeeper/mledger/MetadataCompressionConfig.java](https://codecov.io/gh/apache/pulsar/pull/19561?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-bWFuYWdlZC1sZWRnZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2Jvb2trZWVwZXIvbWxlZGdlci9NZXRhZGF0YUNvbXByZXNzaW9uQ29uZmlnLmphdmE=)
 | `60.00% <60.00%> (ø)` | |
   | 
[...bookkeeper/mledger/ManagedLedgerFactoryConfig.java](https://codecov.io/gh/apache/pulsar/pull/19561?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-bWFuYWdlZC1sZWRnZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2Jvb2trZWVwZXIvbWxlZGdlci9NYW5hZ2VkTGVkZ2VyRmFjdG9yeUNvbmZpZy5qYXZh)
 | `90.00% <100.00%> (+2.50%)` | :arrow_up: |
   | 
[...kkeeper/mledger/impl/ManagedLedgerFactoryImpl.java](https://codecov.io/gh/apache/pulsar/pull/19561?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-bWFuYWdlZC1sZWRnZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2Jvb2trZWVwZXIvbWxlZGdlci9pbXBsL01hbmFnZWRMZWRnZXJGYWN0b3J5SW1wbC5qYXZh)
 | `40.87% <100.00%> (+0.11%)` | :arrow_up: |
   | 
[...org/apache/pulsar/broker/ServiceConfiguration.java](https://codecov.io/gh/apache/pulsar/pull/19561?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3B1bHNhci9icm9rZXIvU2VydmljZUNvbmZpZ3VyYXRpb24uamF2YQ==)
 | `98.00% <100.00%> (+0.12%)` | :arrow_up: |
   | 
[...ache/pulsar/broker/ManagedLedgerClientFactory.java](https://codecov.io/gh/apache/pulsar/pull/19561?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9NYW5hZ2VkTGVkZ2VyQ2xpZW50RmFjdG9yeS5qYXZh)
 | `46.15% <100.00%> (+2.91%)` | :arrow_up: |
   
   ... and [540 files with indirect coverage 

[pulsar-site] branch main updated: [fix][doc] Markdown link in reference-terminology.md (#464)

[tison]

This is an automated email from the ASF dual-hosted git repository.

tison pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/pulsar-site.git


The following commit(s) were added to refs/heads/main by this push:
 new 2072c3d7f3a [fix][doc] Markdown link in reference-terminology.md (#464)
2072c3d7f3a is described below

commit 2072c3d7f3acfc8f4882418d6ce79a1755dadb23
Author: 何晨 
AuthorDate: Fri Mar 10 16:46:03 2023 +0800

[fix][doc] Markdown link in reference-terminology.md (#464)
---
 docs/reference-terminology.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/reference-terminology.md b/docs/reference-terminology.md
index e1fc0771dee..b58c191f01a 100644
--- a/docs/reference-terminology.md
+++ b/docs/reference-terminology.md
@@ -153,7 +153,7 @@ handles all message transfers. Pulsar clusters typically 
consist of multiple bro
 
  Dispatcher
 
-An asynchronous TCP server used for all data transfers in and out of a Pulsar 
[broker](#broker)](#broker). The Pulsar
+An asynchronous TCP server used for all data transfers in and out of a Pulsar 
[broker](#broker). The Pulsar
 dispatcher uses a custom binary protocol for all communications.
 
 ### Storage

[GitHub] [pulsar-site] tisonkun commented on pull request #464: [fix][doc] Markdown link in reference-terminology.md

[via GitHub]

tisonkun commented on PR #464:
URL: https://github.com/apache/pulsar-site/pull/464#issuecomment-1463465095

   Merging...
   
   Thanks for your contribution!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-site] tisonkun merged pull request #464: [fix][doc] Markdown link in reference-terminology.md

[via GitHub]

tisonkun merged PR #464:
URL: https://github.com/apache/pulsar-site/pull/464


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-client-cpp] RobertIndie opened a new pull request, #210: [feat] Add consumer interceptor

[via GitHub]

RobertIndie opened a new pull request, #210:
URL: https://github.com/apache/pulsar-client-cpp/pull/210

   
   
   Master Issue: #150
   
   ### Motivation
   
   This is the consumer interceptor implementation of 
https://github.com/apache/pulsar-client-cpp/issues/150.
   
   There are some trigger events that are not implemented due to the current PR 
is big enough:
   * onNegativeAcksSend
   * onAckTimeoutSend
   * onPartitionsChange
   
   I will add them in another PR.
   
   ### Modifications
   
   * Add ProducerInterceptor interface
   * Add `intercept` and `getInterceptors` in `ProducerConfiguration`.
   * Triggering the interceptor when `beforeConsume`, `onAcknowledge`, 
`onAcknowledgeCumulative` and `close`.
   
   ### Verifying this change
   
   This change added tests.
   ### Documentation
   
   
   
   - [x] `doc-required` 
   (Your PR needs to update docs and you will update later)
   
   - [ ] `doc-not-needed` 
   (Please explain why)
   
   - [ ] `doc` 
   (Your PR contains doc changes)
   
   - [ ] `doc-complete`
   (Docs have been already added)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] jiangpengcheng commented on pull request #19470: [improve][fn] Support configure compression type

[via GitHub]

jiangpengcheng commented on PR #19470:
URL: https://github.com/apache/pulsar/pull/19470#issuecomment-1463456660

   /pulsarbot rerun-failure-checks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] jiangpengcheng commented on pull request #19470: [improve][fn] Support configure compression type

[via GitHub]

jiangpengcheng commented on PR #19470:
URL: https://github.com/apache/pulsar/pull/19470#issuecomment-1463456470

   > > the type is `String` and available values are: `None`, `LZ4`, `ZLIB`, 
`ZSTD` and `SNAPPY`, and default to `LZ4`
   > 
   > @jiangpengcheng thanks for the input. I've drafted 
[apache/pulsar-site#461](https://github.com/apache/pulsar-site/pull/461) to add 
the docs. Only one concern - the default config of compression type on client 
side, e.g., Java producers, is `None` (no compression), which is not aligned 
with functions. Is there any risk here?
   
   
   There is no risk, just for back compatibility since current pulsar functions 
are using `LZ4` as compression type.
   
   One thing I did wrong is that the `None` should be `NONE` instead
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] codelipenghui commented on a diff in pull request #19749: [improve][broker] PIP-192: Add metrics for unload operation

[via GitHub]

codelipenghui commented on code in PR #19749:
URL: https://github.com/apache/pulsar/pull/19749#discussion_r1132082205


##
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/models/UnloadCounter.java:
##
@@ -45,36 +48,57 @@ public class UnloadCounter {
 long unloadBrokerCount = 0;
 long unloadBundleCount = 0;
 
-final Map> 
breakdownCounters;
+@Getter
+@VisibleForTesting
+final Map> 
breakdownCounters;
 
+@Getter
+@VisibleForTesting
 double loadAvg;
+@Getter
+@VisibleForTesting
 double loadStd;
 
+private volatile long updatedAt = 0;
+
 public UnloadCounter() {
 breakdownCounters = Map.of(
 Success, Map.of(
-Overloaded, new MutableLong(),
-Underloaded, new MutableLong()),
+Overloaded, new AtomicLong(),
+Underloaded, new AtomicLong(),
+Admin, new AtomicLong()),
 Skip, Map.of(
-Balanced, new MutableLong(),
-NoBundles, new MutableLong(),
-CoolDown, new MutableLong(),
-OutDatedData, new MutableLong(),
-NoLoadData, new MutableLong(),
-NoBrokers, new MutableLong(),
-Unknown, new MutableLong()),
+Balanced, new AtomicLong(),
+NoBundles, new AtomicLong(),
+CoolDown, new AtomicLong(),
+OutDatedData, new AtomicLong(),
+NoLoadData, new AtomicLong(),
+NoBrokers, new AtomicLong(),
+Unknown, new AtomicLong()),
 Failure, Map.of(
-Unknown, new MutableLong())
+Unknown, new AtomicLong())
 );
 }
 
 public void update(UnloadDecision decision) {
-var unloads = decision.getUnloads();
-unloadBrokerCount += unloads.keySet().size();
-unloadBundleCount += unloads.values().size();
-
breakdownCounters.get(decision.getLabel()).get(decision.getReason()).increment();
-loadAvg = decision.loadAvg;
-loadStd = decision.loadStd;
+unloadBrokerCount++;
+unloadBundleCount++;
+
breakdownCounters.get(decision.getLabel()).get(decision.getReason()).incrementAndGet();
+updatedAt = System.currentTimeMillis();
+}
+
+public void update(UnloadDecision.Label label, UnloadDecision.Reason 
reason) {
+if (label == Success) {
+unloadBundleCount++;
+unloadBrokerCount++;

Review Comment:
   @Demogorgon314 @heesung-sn We'd better to not remove it in this PR. If it is 
worth to be removed, we can start a separate discussion and PR. Removing the 
metrics might break the Pulsar operators.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[pulsar] branch master updated: [fix][client] Fix topic list watcher fail log (#19733)

[houxiaoyu]

This is an automated email from the ASF dual-hosted git repository.

houxiaoyu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new 9feb85b19ca [fix][client] Fix topic list watcher fail log (#19733)
9feb85b19ca is described below

commit 9feb85b19ca160b4be3acbfe15f39edde07608f5
Author: houxiaoyu 
AuthorDate: Fri Mar 10 16:21:25 2023 +0800

[fix][client] Fix topic list watcher fail log (#19733)
---
 .../src/main/java/org/apache/pulsar/client/impl/TopicListWatcher.java | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git 
a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/TopicListWatcher.java
 
b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/TopicListWatcher.java
index 88ada6a344b..384d1b688b8 100644
--- 
a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/TopicListWatcher.java
+++ 
b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/TopicListWatcher.java
@@ -148,8 +148,8 @@ public class TopicListWatcher extends HandlerState 
implements ConnectionHandler.
 cnx.channel().close();
 return null;
 }
-log.warn("[{}][{}] Failed to subscribe to topic on 
{}", topic,
-getHandlerName(), 
cnx.channel().remoteAddress());
+log.warn("[{}][{}] Failed to create topic list watcher 
on {}",
+topic, getHandlerName(), 
cnx.channel().remoteAddress());
 
 if (e.getCause() instanceof PulsarClientException
 && 
PulsarClientException.isRetriableError(e.getCause())

[GitHub] [pulsar] AnonHxy merged pull request #19733: [fix][client] Fix topic list watcher fail log

[via GitHub]

AnonHxy merged PR #19733:
URL: https://github.com/apache/pulsar/pull/19733


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-client-go] shibd commented on issue #926: [feature request] Support KeyValue Schema.

[via GitHub]

shibd commented on issue #926:
URL: 
https://github.com/apache/pulsar-client-go/issues/926#issuecomment-1463435088

   @CrazyCollin Hi, You can refer to java related 
[impl](https://github.com/apache/pulsar/blob/82237d3684fe506bcb6426b3b23f413422e6e4fb/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/KeyValueSchemaImpl.java#L44).
 
   
   A point is: the schema of the `key` and `value` and the value of the `key` 
and `value` are joint, and the splicing rule is referenced: 
   
   
https://github.com/apache/pulsar/blob/82237d3684fe506bcb6426b3b23f413422e6e4fb/pulsar-client-api/src/main/java/org/apache/pulsar/common/schema/KeyValue.java#L100-L123


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-client-cpp] shibd commented on pull request #196: [TableView-2] Implement all interfaces.

[via GitHub]

shibd commented on PR #196:
URL: 
https://github.com/apache/pulsar-client-cpp/pull/196#issuecomment-1463426053

   @RobertIndie @BewareMyPower ping.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org