[GitHub] [pulsar] codelipenghui commented on issue #16753: Flaky-test: PulsarFunctionLocalRunTest.testE2EPulsarFunctionLocalRunMultipleInstances
codelipenghui commented on issue #16753: URL: https://github.com/apache/pulsar/issues/16753#issuecomment-1198909130 Another one https://github.com/apache/pulsar/runs/7572938477?check_suite_focus=true -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] zhanghaou commented on issue #16564: Confused with method getMemUsage() in LinuxBrokerHostUsageImpl.java
zhanghaou commented on issue #16564: URL: https://github.com/apache/pulsar/issues/16564#issuecomment-1198908900 > OpeJDK should be "container aware" now. See https://bugs.openjdk.java.net/browse/JDK-8203357. > > Would you please try the higher JDK version? Thanks for this tip, I will try. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui opened a new issue, #16863: Flaky-test: ManagedLedgerFactoryChangeLedgerPathTest.setUp
codelipenghui opened a new issue, #16863: URL: https://github.com/apache/pulsar/issues/16863 [example failure](https://github.com/apache/pulsar/runs/7572938224?check_suite_focus=true) ``` Error: Tests run: 6, Failures: 1, Errors: 0, Skipped: 5, Time elapsed: 15.043 s <<< FAILURE! - in org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryChangeLedgerPathTest Error: setUp(org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryChangeLedgerPathTest) Time elapsed: 15.009 s <<< FAILURE! org.apache.bookkeeper.bookie.BookieException$MetadataStoreException: Failed to initialize metadata bookie driver at org.apache.bookkeeper.bookie.BookieResources.createMetadataDriver(BookieResources.java:64) at org.apache.bookkeeper.test.ServerTester.(ServerTester.java:118) at org.apache.bookkeeper.test.BookKeeperClusterTestCase.startBookie(BookKeeperClusterTestCase.java:675) at org.apache.bookkeeper.test.BookKeeperClusterTestCase.startAndAddBookie(BookKeeperClusterTestCase.java:655) at org.apache.bookkeeper.test.BookKeeperClusterTestCase.startNewBookieAndReturnAddress(BookKeeperClusterTestCase.java:644) at org.apache.bookkeeper.test.BookKeeperClusterTestCase.startNewBookie(BookKeeperClusterTestCase.java:637) at org.apache.bookkeeper.test.BookKeeperClusterTestCase.startBKCluster(BookKeeperClusterTestCase.java:267) at org.apache.bookkeeper.test.BookKeeperClusterTestCase.setUp(BookKeeperClusterTestCase.java:166) at org.apache.bookkeeper.test.BookKeeperClusterTestCase.setUp(BookKeeperClusterTestCase.java:150) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132) at org.testng.internal.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:61) at org.testng.internal.ConfigInvoker.invokeConfigurationMethod(ConfigInvoker.java:366) at org.testng.internal.ConfigInvoker.invokeConfigurations(ConfigInvoker.java:320) at org.testng.TestRunner.invokeTestConfigurations(TestRunner.java:617) at org.testng.TestRunner.beforeRun(TestRunner.java:607) at org.testng.TestRunner.run(TestRunner.java:578) at org.testng.SuiteRunner.runTest(SuiteRunner.java:384) at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:378) at org.testng.SuiteRunner.privateRun(SuiteRunner.java:337) at org.testng.SuiteRunner.run(SuiteRunner.java:286) at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53) at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96) at org.testng.TestNG.runSuitesSequentially(TestNG.java:1218) at org.testng.TestNG.runSuitesLocally(TestNG.java:1140) at org.testng.TestNG.runSuites(TestNG.java:1069) at org.testng.TestNG.run(TestNG.java:1037) at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:135) at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeSingleClass(TestNGDirectoryTestSuite.java:112) at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeLazy(TestNGDirectoryTestSuite.java:123) at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:90) at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:146) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418) Caused by: org.apache.bookkeeper.meta.exceptions.MetadataException: Failed to create zookeeper client to localhost:40093 at org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase.initialize(ZKMetadataDriverBase.java:236) at org.apache.bookkeeper.meta.zk.ZKMetadataBookieDriver.initialize(ZKMetadataBookieDriver.java:55) at org.apache.bookkeeper.bookie.BookieResources.createMetadataDriver(BookieResources.java:61) ... 38 more Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) at org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase.waitForConnection(ZooKeeperWatcherBase.java:159) at
[GitHub] [pulsar] michaeljmarshall commented on pull request #16222: Update/fix Swagger Annotation for param: authoritative
michaeljmarshall commented on PR #16222: URL: https://github.com/apache/pulsar/pull/16222#issuecomment-1198908409 @BewareMyPower - done. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch branch-2.9 updated: Update/fix Swagger Annotation for param: authoritative (#16222)
This is an automated email from the ASF dual-hosted git repository.
mmarshall pushed a commit to branch branch-2.9
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-2.9 by this push:
new 085678aab22 Update/fix Swagger Annotation for param: authoritative
(#16222)
085678aab22 is described below
commit 085678aab22f4d3e8d3282e8a1331006dc16b341
Author: Michael Marshall
AuthorDate: Tue Jul 12 18:58:26 2022 -0500
Update/fix Swagger Annotation for param: authoritative (#16222)
* Update/fix Swagger Annotation for param: authoritative
* Fix Checkstyle
(cherry picked from commit b4ef4a3f4b752749277ae460d7e0739cf32672bc)
---
.../pulsar/broker/admin/v1/PersistentTopics.java | 10 +-
.../broker/admin/v2/NonPersistentTopics.java | 10 +-
.../pulsar/broker/admin/v2/PersistentTopics.java | 207 +++--
3 files changed, 114 insertions(+), 113 deletions(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v1/PersistentTopics.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v1/PersistentTopics.java
index fcb2ba83057..f3b5e6b56a3 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v1/PersistentTopics.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v1/PersistentTopics.java
@@ -194,7 +194,7 @@ public class PersistentTopics extends PersistentTopicsBase {
@PathParam("namespace") String namespace,
@ApiParam(value = "Specify topic name", required = true)
@PathParam("topic") @Encoded String encodedTopic,
-@ApiParam(value = "Is authentication required to perform this
operation")
+@ApiParam(value = "Whether leader broker redirected this call to
this broker. For internal use.")
@QueryParam("authoritative") @DefaultValue("false") boolean
authoritative) {
validateNamespaceName(tenant, cluster, namespace);
validateTopicName(tenant, cluster, namespace, encodedTopic);
@@ -229,7 +229,7 @@ public class PersistentTopics extends PersistentTopicsBase {
public void updatePartitionedTopic(@PathParam("property") String property,
@PathParam("cluster") String cluster,
@PathParam("namespace") String namespace, @PathParam("topic")
@Encoded String encodedTopic,
@QueryParam("updateLocalTopicOnly") @DefaultValue("false") boolean
updateLocalTopicOnly,
-@ApiParam(value = "Is authentication required to perform this
operation")
+@ApiParam(value = "Whether leader broker redirected this call to
this broker. For internal use.")
@QueryParam("authoritative") @DefaultValue("false") boolean
authoritative,
@QueryParam("force") @DefaultValue("false") boolean force,
int numPartitions) {
@@ -530,7 +530,7 @@ public class PersistentTopics extends PersistentTopicsBase {
@PathParam("topic") @Encoded String encodedTopic,
@ApiParam(value = "Subscription to be Expiry messages on")
@PathParam("subName") String encodedSubName,
-@ApiParam(value = "Is authentication required to perform this
operation")
+@ApiParam(value = "Whether leader broker redirected this call to
this broker. For internal use.")
@QueryParam("authoritative") @DefaultValue("false") boolean
authoritative,
@ApiParam(name = "messageId", value = "messageId to reset back to
(ledgerId:entryId)")
ResetCursorData resetCursorData) {
@@ -832,7 +832,7 @@ public class PersistentTopics extends PersistentTopicsBase {
@PathParam("namespace") String namespace,
@ApiParam(value = "Specify topic name", required = true)
@PathParam("topic") @Encoded String encodedTopic,
-@ApiParam(value = "Is authentication required to perform this
operation")
+@ApiParam(value = "Whether leader broker redirected this call to
this broker. For internal use.")
@QueryParam("authoritative") @DefaultValue("false") boolean
authoritative) {
try {
validateTopicName(tenant, cluster, namespace, encodedTopic);
@@ -867,7 +867,7 @@ public class PersistentTopics extends PersistentTopicsBase {
@PathParam("topic") @Encoded String encodedTopic,
@ApiParam(value = "Name of subscription", required = true)
@PathParam("subName") String encodedSubName,
-@ApiParam(value = "Is authentication required to perform this
operation")
+@ApiParam(value = "Whether leader broker redirected this call to
this broker. For internal use.")
@QueryParam("authoritative") @DefaultValue("false") boolean
authoritative,
@ApiParam(value = "Whether to enable replicated subscription",
required = true)
boolean enabled) {
diff --git
[pulsar] branch branch-2.8 updated: Update/fix Swagger Annotation for param: authoritative (#16222)
This is an automated email from the ASF dual-hosted git repository.
mmarshall pushed a commit to branch branch-2.8
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-2.8 by this push:
new b06b2ee056a Update/fix Swagger Annotation for param: authoritative
(#16222)
b06b2ee056a is described below
commit b06b2ee056a1ec4bf5826615587f100f73b4e66d
Author: Michael Marshall
AuthorDate: Tue Jul 12 18:58:26 2022 -0500
Update/fix Swagger Annotation for param: authoritative (#16222)
* Update/fix Swagger Annotation for param: authoritative
* Fix Checkstyle
(cherry picked from commit b4ef4a3f4b752749277ae460d7e0739cf32672bc)
---
.../pulsar/broker/admin/v1/PersistentTopics.java | 8 +-
.../broker/admin/v2/NonPersistentTopics.java | 10 +-
.../pulsar/broker/admin/v2/PersistentTopics.java | 203 +++--
3 files changed, 111 insertions(+), 110 deletions(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v1/PersistentTopics.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v1/PersistentTopics.java
index df4d553d148..d0a2bba279d 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v1/PersistentTopics.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v1/PersistentTopics.java
@@ -198,7 +198,7 @@ public class PersistentTopics extends PersistentTopicsBase {
public void updatePartitionedTopic(@PathParam("property") String property,
@PathParam("cluster") String cluster,
@PathParam("namespace") String namespace, @PathParam("topic")
@Encoded String encodedTopic,
@QueryParam("updateLocalTopicOnly") @DefaultValue("false") boolean
updateLocalTopicOnly,
-@ApiParam(value = "Is authentication required to perform this
operation")
+@ApiParam(value = "Whether leader broker redirected this call to
this broker. For internal use.")
@QueryParam("authoritative") @DefaultValue("false") boolean
authoritative,
int numPartitions) {
validateTopicName(property, cluster, namespace, encodedTopic);
@@ -497,7 +497,7 @@ public class PersistentTopics extends PersistentTopicsBase {
@PathParam("topic") @Encoded String encodedTopic,
@ApiParam(value = "Subscription to be Expiry messages on")
@PathParam("subName") String encodedSubName,
-@ApiParam(value = "Is authentication required to perform this
operation")
+@ApiParam(value = "Whether leader broker redirected this call to
this broker. For internal use.")
@QueryParam("authoritative") @DefaultValue("false") boolean
authoritative,
@ApiParam(name = "messageId", value = "messageId to reset back to
(ledgerId:entryId)")
ResetCursorData resetCursorData) {
@@ -798,7 +798,7 @@ public class PersistentTopics extends PersistentTopicsBase {
@PathParam("namespace") String namespace,
@ApiParam(value = "Specify topic name", required = true)
@PathParam("topic") @Encoded String encodedTopic,
-@ApiParam(value = "Is authentication required to perform this
operation")
+@ApiParam(value = "Whether leader broker redirected this call to
this broker. For internal use.")
@QueryParam("authoritative") @DefaultValue("false") boolean
authoritative) {
try {
validateTopicName(tenant, cluster, namespace, encodedTopic);
@@ -833,7 +833,7 @@ public class PersistentTopics extends PersistentTopicsBase {
@PathParam("topic") @Encoded String encodedTopic,
@ApiParam(value = "Name of subscription", required = true)
@PathParam("subName") String encodedSubName,
-@ApiParam(value = "Is authentication required to perform this
operation")
+@ApiParam(value = "Whether leader broker redirected this call to
this broker. For internal use.")
@QueryParam("authoritative") @DefaultValue("false") boolean
authoritative,
@ApiParam(value = "Whether to enable replicated subscription",
required = true)
boolean enabled) {
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v2/NonPersistentTopics.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v2/NonPersistentTopics.java
index 8a0f123d2ce..a8490a2cda8 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v2/NonPersistentTopics.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v2/NonPersistentTopics.java
@@ -95,7 +95,7 @@ public class NonPersistentTopics extends PersistentTopics {
@PathParam("namespace") String namespace,
@ApiParam(value = "Specify topic name", required = true)
@PathParam("topic") @Encoded String encodedTopic,
-@ApiParam(value = "Is authentication required
[pulsar] branch master updated (0f2060d0d28 -> 5d0eb9b71f9)
This is an automated email from the ASF dual-hosted git repository. technoboy pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git from 0f2060d0d28 [feature][broker] add config maxUnloadBundleNumPerShedding for UniformLoadShedder (#16409) add 5d0eb9b71f9 [fix][client] Fix load trust certificate (#16789) No new revisions were added by this update. Summary of changes: .../client/api/ClientAuthenticationTlsTest.java| 183 + .../apache/pulsar/common/util/SecurityUtility.java | 7 +- 2 files changed, 188 insertions(+), 2 deletions(-) create mode 100644 pulsar-broker/src/test/java/org/apache/pulsar/client/api/ClientAuthenticationTlsTest.java
[GitHub] [pulsar] Technoboy- merged pull request #16789: [fix][client] Fix load trust certificate
Technoboy- merged PR #16789: URL: https://github.com/apache/pulsar/pull/16789 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] michaeljmarshall commented on pull request #16222: Update/fix Swagger Annotation for param: authoritative
michaeljmarshall commented on PR #16222: URL: https://github.com/apache/pulsar/pull/16222#issuecomment-1198898251 @BewareMyPower - of course. I will look at cherry picking this now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on pull request #16222: Update/fix Swagger Annotation for param: authoritative
BewareMyPower commented on PR #16222: URL: https://github.com/apache/pulsar/pull/16222#issuecomment-1198897587 @michaeljmarshall Could you cherry-pick this PR to branch-2.8? It looks like to have too many conflicts to resolve. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch branch-2.8 updated (3b164f5ac0c -> f45985e7bff)
This is an automated email from the ASF dual-hosted git repository. xyz pushed a change to branch branch-2.8 in repository https://gitbox.apache.org/repos/asf/pulsar.git from 3b164f5ac0c Revert "[improve][broker] Avoid reconnection when a partitioned topic was created concurrently (#16043)" new 739c45e8075 [fix][Java Client] Fix thread safety issue of `LastCumulativeAck` (#16072) new 7a3ed537f38 Fix `messageQueue` release message issue. (#16155) new b607d09176f [fix][broker] Fix NPE when drop backlog for time limit. (#16235) new a97131eeabf [Branch-2.9][Cherry-pick] fix bug: fail to expose managed ledger client stats to prometheus if bookkeeperClientExposeStatsToPrometheus is true #16219 (#16343) new f45985e7bff Increase timeout in PersistentStreamingDispatcherBlockConsumerTest.testBlockBrokerDispatching (#12943) The 5 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../pulsar/broker/ManagedLedgerClientFactory.java | 5 +- .../pulsar/broker/service/BacklogQuotaManager.java | 4 + .../pulsar/broker/stats/PrometheusMetricsTest.java | 32 + .../client/api/DispatcherBlockConsumerTest.java| 2 +- .../PersistentAcknowledgmentsGroupingTracker.java | 141 +++-- .../pulsar/client/impl/LastCumulativeAckTest.java | 86 + .../pulsar/sql/presto/PulsarRecordCursor.java | 14 +- 7 files changed, 205 insertions(+), 79 deletions(-) create mode 100644 pulsar-client/src/test/java/org/apache/pulsar/client/impl/LastCumulativeAckTest.java
[pulsar] 05/05: Increase timeout in PersistentStreamingDispatcherBlockConsumerTest.testBlockBrokerDispatching (#12943)
This is an automated email from the ASF dual-hosted git repository.
xyz pushed a commit to branch branch-2.8
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit f45985e7bfffaee365af268bec6925953b7db5b0
Author: JiangHaiting
AuthorDate: Tue Nov 23 22:17:16 2021 +0800
Increase timeout in
PersistentStreamingDispatcherBlockConsumerTest.testBlockBrokerDispatching
(#12943)
(cherry picked from commit 5abf42cf24b43f7f1874eb0b385e093926ea0b21)
---
.../java/org/apache/pulsar/client/api/DispatcherBlockConsumerTest.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/pulsar-broker/src/test/java/org/apache/pulsar/client/api/DispatcherBlockConsumerTest.java
b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/DispatcherBlockConsumerTest.java
index cb1f44bf81e..4faf7834e94 100644
---
a/pulsar-broker/src/test/java/org/apache/pulsar/client/api/DispatcherBlockConsumerTest.java
+++
b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/DispatcherBlockConsumerTest.java
@@ -672,7 +672,7 @@ public class DispatcherBlockConsumerTest extends
ProducerConsumerBase {
*
*
*/
-@Test(timeOut = 1)
+@Test(timeOut = 6)
public void testBlockBrokerDispatching() {
log.info("-- Starting {} test --", methodName);
[pulsar] 04/05: [Branch-2.9][Cherry-pick] fix bug: fail to expose managed ledger client stats to prometheus if bookkeeperClientExposeStatsToPrometheus is true #16219 (#16343)
This is an automated email from the ASF dual-hosted git repository.
xyz pushed a commit to branch branch-2.8
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit a97131eeabf2febe167d775b4c8de70946f44c5c
Author: Qiang Huang
AuthorDate: Sat Jul 2 21:45:46 2022 +0800
[Branch-2.9][Cherry-pick] fix bug: fail to expose managed ledger client
stats to prometheus if bookkeeperClientExposeStatsToPrometheus is true #16219
(#16343)
(cherry picked from commit d56ae3797c18b1451b58c4b05f60ed23ae321c9f)
---
.../pulsar/broker/ManagedLedgerClientFactory.java | 5 ++--
.../pulsar/broker/stats/PrometheusMetricsTest.java | 32 ++
2 files changed, 35 insertions(+), 2 deletions(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/ManagedLedgerClientFactory.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/ManagedLedgerClientFactory.java
index 431cb729b22..dc0ac3c5b7b 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/ManagedLedgerClientFactory.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/ManagedLedgerClientFactory.java
@@ -82,7 +82,8 @@ public class ManagedLedgerClientFactory implements
ManagedLedgerStorage {
statsProvider.start(configuration);
StatsLogger statsLogger =
statsProvider.getStatsLogger("pulsar_managedLedger_client");
-this.defaultBkClient = bookkeeperProvider.create(conf, zkClient,
eventLoopGroup, Optional.empty(), null);
+this.defaultBkClient =
+bookkeeperProvider.create(conf, zkClient, eventLoopGroup,
Optional.empty(), null, statsLogger);
BookkeeperFactoryForCustomEnsemblePlacementPolicy bkFactory = (
EnsemblePlacementPolicyConfig ensemblePlacementPolicyConfig)
-> {
@@ -93,7 +94,7 @@ public class ManagedLedgerClientFactory implements
ManagedLedgerStorage {
try {
return bookkeeperProvider.create(conf, zkClient,
eventLoopGroup,
Optional.ofNullable(ensemblePlacementPolicyConfig.getPolicyClass()),
-ensemblePlacementPolicyConfig.getProperties());
+ensemblePlacementPolicyConfig.getProperties(),
statsLogger);
} catch (Exception e) {
log.error("Failed to initialize bk-client for policy
{}, properties {}",
ensemblePlacementPolicyConfig.getPolicyClass(),
diff --git
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/PrometheusMetricsTest.java
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/PrometheusMetricsTest.java
index 68dde0aaa26..c0383cd6e9f 100644
---
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/PrometheusMetricsTest.java
+++
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/PrometheusMetricsTest.java
@@ -821,6 +821,38 @@ public class PrometheusMetricsTest extends BrokerTestBase {
cm = (List)
metrics.get("pulsar_managedLedger_client_bookkeeper_ml_workers_task_execution_count");
assertEquals(cm.size(), 0);
+
+cm = (List) metrics.get(
+keyNameBySubstrings(metrics, "pulsar_managedLedger_client",
"bookkeeper_ml_scheduler_total_tasks"));
+assertEquals(cm.size(), 1);
+assertEquals(cm.get(0).tags.get("cluster"), "test");
+
+cm = (List) metrics.get(keyNameBySubstrings(metrics,
"pulsar_managedLedger_client",
+"bookkeeper_ml_scheduler_task_execution_sum"));
+assertEquals(cm.size(), 2);
+assertEquals(cm.get(0).tags.get("cluster"), "test");
+
+cm = (List) metrics.get(
+keyNameBySubstrings(metrics,
+"pulsar_managedLedger_client",
"bookkeeper_ml_scheduler_queue"));
+assertEquals(cm.size(), 1);
+assertEquals(cm.get(0).tags.get("cluster"), "test");
+}
+
+private static String keyNameBySubstrings(Multimap
metrics, String... substrings) {
+for (String key : metrics.keys()) {
+boolean found = true;
+for (String s : substrings) {
+if (!key.contains(s)) {
+found = false;
+break;
+}
+}
+if (found) {
+return key;
+}
+}
+return null;
}
@Test
[pulsar] 03/05: [fix][broker] Fix NPE when drop backlog for time limit. (#16235)
This is an automated email from the ASF dual-hosted git repository.
xyz pushed a commit to branch branch-2.8
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit b607d09176fd2c1586691de2bde5967ebaf4f539
Author: Jiwei Guo
AuthorDate: Tue Jun 28 14:39:00 2022 +0800
[fix][broker] Fix NPE when drop backlog for time limit. (#16235)
(cherry picked from commit d24d82780fd27a98c6cdbee28d756ee7d419495f)
---
.../java/org/apache/pulsar/broker/service/BacklogQuotaManager.java| 4
1 file changed, 4 insertions(+)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BacklogQuotaManager.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BacklogQuotaManager.java
index 65ffaabef96..6414904d73d 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BacklogQuotaManager.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BacklogQuotaManager.java
@@ -246,6 +246,10 @@ public class BacklogQuotaManager {
ManagedCursor slowestConsumer =
mLedger.getSlowestConsumer();
Position oldestPosition =
slowestConsumer.getMarkDeletedPosition();
ManagedLedgerInfo.LedgerInfo ledgerInfo =
mLedger.getLedgerInfo(oldestPosition.getLedgerId()).get();
+if (ledgerInfo == null) {
+
slowestConsumer.resetCursor(mLedger.getNextValidPosition((PositionImpl)
oldestPosition));
+continue;
+}
// Timestamp only > 0 if ledger has been closed
if (ledgerInfo.getTimestamp() > 0
&& currentMillis - ledgerInfo.getTimestamp() >
quota.getLimitTime()) {
[pulsar] 02/05: Fix `messageQueue` release message issue. (#16155)
This is an automated email from the ASF dual-hosted git repository.
xyz pushed a commit to branch branch-2.8
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 7a3ed537f383f87e96d263c95a3e2ffd32272601
Author: Jiwei Guo
AuthorDate: Tue Jun 21 17:13:00 2022 +0800
Fix `messageQueue` release message issue. (#16155)
(cherry picked from commit 141c44022a27be2fc07eab9827cfdb168e448953)
---
.../org/apache/pulsar/sql/presto/PulsarRecordCursor.java | 14 ++
1 file changed, 6 insertions(+), 8 deletions(-)
diff --git
a/pulsar-sql/presto-pulsar/src/main/java/org/apache/pulsar/sql/presto/PulsarRecordCursor.java
b/pulsar-sql/presto-pulsar/src/main/java/org/apache/pulsar/sql/presto/PulsarRecordCursor.java
index e646dcff4c9..9474376f77c 100644
---
a/pulsar-sql/presto-pulsar/src/main/java/org/apache/pulsar/sql/presto/PulsarRecordCursor.java
+++
b/pulsar-sql/presto-pulsar/src/main/java/org/apache/pulsar/sql/presto/PulsarRecordCursor.java
@@ -700,14 +700,6 @@ public class PulsarRecordCursor implements RecordCursor {
public void close() {
log.info("Closing cursor record");
-if (currentMessage != null) {
-currentMessage.release();
-}
-
-if (messageQueue != null) {
-messageQueue.drain(RawMessage::release);
-}
-
if (entryQueue != null) {
entryQueue.drain(Entry::release);
}
@@ -717,6 +709,12 @@ public class PulsarRecordCursor implements RecordCursor {
if (entryQueue != null) {
entryQueue.drain(Entry::release);
}
+if (messageQueue != null) {
+messageQueue.drain(RawMessage::release);
+}
+if (currentMessage != null) {
+currentMessage.release();
+}
});
}
[pulsar] 01/05: [fix][Java Client] Fix thread safety issue of `LastCumulativeAck` (#16072)
This is an automated email from the ASF dual-hosted git repository. xyz pushed a commit to branch branch-2.8 in repository https://gitbox.apache.org/repos/asf/pulsar.git commit 739c45e807524e003286db66563eb8e4421c89dd Author: Yunze Xu AuthorDate: Wed Jun 22 23:34:49 2022 +0800 [fix][Java Client] Fix thread safety issue of `LastCumulativeAck` (#16072) ### Motivation There were several issues caused by the thread safe issue of `LastCumulativeAck`, see: - https://github.com/apache/pulsar/pull/10586 - https://github.com/apache/pulsar/pull/12343 The root cause is that `LastCumulativeAck` could be accessed by different threads, especially in `flushAsync` method. But the fields are accessed directly and no thread safety can be guaranteed. In addition, the current `LastCumulativeAck` class was added in https://github.com/apache/pulsar/pull/8996 to hold two object references, but this modification is wrong. Before #8996, there are two CAS operations in `doCumulativeAck` method in case it's called concurretly. Though the composite CAS operation is not atomic. However, after #8996, only CAS operation was performed but it's compared with a `LastCumulativeAck` object, not the two fields (`messageId` and `bitSetRecyclable`). There is another issue that it uses a flag `cumulativeAckFlushRequired` to mark if `lastCumulativeAck` should flush. However, if `flushAsync` was called concurrently, both would send ACK commands to broker. ### Modifications To solve the thread safety issue, this PR move the `LastCumulativeAck` out of the `PersistentAcknowledgmentsGroupingTracker` to disable directly access to the internal fields. Then, the following synchronized methods were added to guarantee the thread safety: - `update`: Guarantee the safe write operations. It also recycles the `BitSetRecyclable` object before assigning new values and indicates itself can be flushed. - `flush`: If it can be flushed, return a thread local `LastCumulativeAck` instance that contains the message ID and the bit set. The bit set is deep copied to avoid the original reference being recycled in another `update` call. In addition, since the `messageId` field is volatile, the `getMessageId` method can always retrieve the latest reference. `LastCumulativeAckTest` is added to verify the sematics above. Based on the new design, we can only maintain a `LastCumulativeAck` field in `PersistentAcknowledgmentsGroupingTracker` and call the related methods in `doCumulativeAck` and `flushAsync`. It also fixes the problem that two concurrent `flushAsync` calls might send the same ACK command twice. (cherry picked from commit 936d6fdc780ea454e72e82b6c7a1885799158d02) --- .../PersistentAcknowledgmentsGroupingTracker.java | 141 +++-- .../pulsar/client/impl/LastCumulativeAckTest.java | 86 + 2 files changed, 159 insertions(+), 68 deletions(-) diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/PersistentAcknowledgmentsGroupingTracker.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/PersistentAcknowledgmentsGroupingTracker.java index 9ba965d37cc..9bd1c3d2a56 100644 --- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/PersistentAcknowledgmentsGroupingTracker.java +++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/PersistentAcknowledgmentsGroupingTracker.java @@ -21,6 +21,7 @@ package org.apache.pulsar.client.impl; import static org.apache.pulsar.common.util.Runnables.catchingAndLoggingThrowables; import io.netty.buffer.ByteBuf; import io.netty.channel.EventLoopGroup; +import io.netty.util.concurrent.FastThreadLocal; import java.util.ArrayList; import java.util.Collections; import java.util.HashSet; @@ -33,11 +34,8 @@ import java.util.concurrent.ConcurrentHashMap; import java.util.concurrent.ConcurrentSkipListSet; import java.util.concurrent.ScheduledFuture; import java.util.concurrent.TimeUnit; -import java.util.concurrent.atomic.AtomicReferenceFieldUpdater; import java.util.concurrent.locks.ReentrantReadWriteLock; - -import io.netty.util.Recycler; -import lombok.NonNull; +import lombok.Getter; import lombok.extern.slf4j.Slf4j; import org.apache.commons.lang3.tuple.Triple; import org.apache.pulsar.client.api.MessageId; @@ -69,17 +67,11 @@ public class PersistentAcknowledgmentsGroupingTracker implements Acknowledgments private volatile TimedCompletableFuture currentIndividualAckFuture; private volatile TimedCompletableFuture currentCumulativeAckFuture; -private volatile LastCumulativeAck lastCumulativeAck = -LastCumulativeAck.create((MessageIdImpl) MessageIdImpl.earliest, null); - -private volatile boolean cumulativeAckFlushRequired = false; +private final LastCumulativeAck
[GitHub] [pulsar] codelipenghui commented on pull request #16715: [feature][client] PIP-184: Topic specific consumer priorityLevel
codelipenghui commented on PR #16715: URL: https://github.com/apache/pulsar/pull/16715#issuecomment-119618 The PR can only merge after the proposal is closed https://lists.apache.org/thread/08olgc0bjr13ld3g0hz58p1sj9c4pk98 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on pull request #16219: [fix][broker]fix bug: fail to expose managed ledger client stats to prometheus if bookkeeperClientExposeStatsToPrometheus is true
BewareMyPower commented on PR #16219: URL: https://github.com/apache/pulsar/pull/16219#issuecomment-1198883513 cherry-pick https://github.com/apache/pulsar/pull/16343 to branch-2.8 instead. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar-client-go] branch master updated: [issue 814] consumer and producer reconnect failure metrics counter (#815)
This is an automated email from the ASF dual-hosted git repository.
mmarshall pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar-client-go.git
The following commit(s) were added to refs/heads/master by this push:
new bd19458 [issue 814] consumer and producer reconnect failure metrics
counter (#815)
bd19458 is described below
commit bd19458b32ff89206c135cc647336e690e99c32f
Author: ming
AuthorDate: Fri Jul 29 01:00:33 2022 -0400
[issue 814] consumer and producer reconnect failure metrics counter (#815)
* consumer and producer reconnect failure metrics counter
* increment on every reconnect failure
* producer consumer reconnect max retry counter
Implement #814
### Motivation
In a Pulsar cluster's kubernetes deployment or a deployment with Proxy/LB
in the front, we need metrics counter to track the re-connection failure
producers and consumers.
When brokers go offline but the proxy/LB is still functioning, TCP
connection can still be established but the topic look up failed.
pulsar_client_connections_establishment_errors counter is not incremented in
this case. Therefore new counters are required to track such failure cases.
### Modifications
Two new counter metrics `pulsar_client_producers_reconnect_failure` and
`pulsar_client_consumers_reconnect_failure` will be incremented at the
producer_partition and consumer_partition retry failure code block.
Two new counter metrics `pulsar_client_producers_reconnect_max_retry` and
`pulsar_client_consumers_reconnect_max_retry` will be incremented at the
producer_partition and consumer_partition when either the max retry or max back
off is reached.
The existing code logic already covers the case when the topic does not
exist. The counters will not be pegged if the topic does not exist. It simply
exists from the retry loop at once.
### Verifying this change
This has been verified in the Pulsar cluster deployment with Proxy. We do
not have such set up in CI because it's not possible to test with Pulsar
standalone mode.
### Does this pull request potentially affect one of the following parts:
*If `yes` was chosen, please highlight the changes*
- Dependencies (does it add or upgrade a dependency): ( no)
- The public API: ( no)
- The schema: (no)
- The default values of configurations: ( no)
- The wire protocol: (no)
### Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable / docs / GoDocs
/ not documented)
- If a feature is not applicable for documentation, explain why?
- If a feature is not documented yet in this PR, please create a followup
issue for adding the documentation
---
pulsar/consumer_partition.go| 4 ++
pulsar/internal/backoff.go | 5 ++
pulsar/internal/backoff_test.go | 2 +
pulsar/internal/metrics.go | 108 +++-
pulsar/producer_partition.go| 4 ++
5 files changed, 99 insertions(+), 24 deletions(-)
diff --git a/pulsar/consumer_partition.go b/pulsar/consumer_partition.go
index fac9d4b..50fa3c4 100644
--- a/pulsar/consumer_partition.go
+++ b/pulsar/consumer_partition.go
@@ -1155,6 +1155,10 @@ func (pc *partitionConsumer) reconnectToBroker() {
if maxRetry > 0 {
maxRetry--
}
+ pc.metrics.ConsumersReconnectFailure.Inc()
+ if maxRetry == 0 || backoff.IsMaxBackoffReached() {
+ pc.metrics.ConsumersReconnectMaxRetry.Inc()
+ }
}
}
diff --git a/pulsar/internal/backoff.go b/pulsar/internal/backoff.go
index f172bdc..ff9b0bc 100644
--- a/pulsar/internal/backoff.go
+++ b/pulsar/internal/backoff.go
@@ -51,3 +51,8 @@ func (b *Backoff) Next() time.Duration {
return b.backoff + time.Duration(jitter)
}
+
+// IsMaxBackReached evaluates if the max number of retries is reached
+func (b *Backoff) IsMaxBackoffReached() bool {
+ return b.backoff >= maxBackoff
+}
diff --git a/pulsar/internal/backoff_test.go b/pulsar/internal/backoff_test.go
index 46c9211..ad6e764 100644
--- a/pulsar/internal/backoff_test.go
+++ b/pulsar/internal/backoff_test.go
@@ -42,6 +42,7 @@ func TestBackoff_NextExponentialBackoff(t *testing.T) {
// the jitter introduces at most 20% difference so delay is
less than twice the previous value
assert.LessOrEqual(t, int64(float64(delay)*.8),
int64(2*float64(previousDelay)))
previousDelay = delay
+ assert.Equal(t, false, backoff.IsMaxBackoffReached())
}
}
@@ -54,6 +55,7 @@ func TestBackoff_NextMaxValue(t *testing.T) {
cappedDelay := backoff.Next()
assert.GreaterOrEqual(t, int64(cappedDelay), int64(maxBackoff))
+
[GitHub] [pulsar-client-go] michaeljmarshall merged pull request #815: [issue 814] consumer and producer reconnect failure metrics counter
michaeljmarshall merged PR #815: URL: https://github.com/apache/pulsar-client-go/pull/815 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] lordcheng10 closed pull request #12604: Optimize OverloadShedder: Sort bundle by throughput in ascending order
lordcheng10 closed pull request #12604: Optimize OverloadShedder: Sort bundle by throughput in ascending order URL: https://github.com/apache/pulsar/pull/12604 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] lordcheng10 closed pull request #12471: Optimize ThresholdShedder strategy: the low-load Broker cannot be fully utilized
lordcheng10 closed pull request #12471: Optimize ThresholdShedder strategy: the low-load Broker cannot be fully utilized URL: https://github.com/apache/pulsar/pull/12471 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] michaeljmarshall opened a new pull request, #16862: [Pulsar Admin CLI Tool] Use NoSplitter for subscription properties
michaeljmarshall opened a new pull request, #16862: URL: https://github.com/apache/pulsar/pull/16862 ### Motivation Subscription properties are a new feature coming in 2.11. The admin api will allow for properties with commas, but the admin cli tool splits parameters with commas into multiple properties. This PR switches the splitter for the `properties` field and makes it possible to specify any valid property with the cli tool. Note: there are other properties fields that the CLI tool allows you to set, but because those have already been released, I am not updating those here. ### Modifications * Use the `NoSplitter.class` as the `splitter` for the subscription `properties` parameter. * Add test coverage for all changes. ### Verifying this change This PR includes new test coverage. ### Does this pull request potentially affect one of the following parts: This PR updates a CLI tool method that has not yet been released, so it does not break any existing functionality. ### Documentation - [x] `doc-not-needed` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on pull request #16205: [fix][broker] Fix compaction subscription acknowledge Marker msg issue.
BewareMyPower commented on PR #16205: URL: https://github.com/apache/pulsar/pull/16205#issuecomment-1198874486 @Technoboy- Could you help cherry-pick this PR to branch-2.8? It's weird that the `createNonPartitionedTopic` call failed in my local env. ``` 12:37:49.693 [AsyncHttpClient-43-1] WARN org.apache.pulsar.client.admin.internal.BaseResource - [http://localhost:53870/admin/persistent/my-property/use/my-ns/testWriteMarker-caf36fc4-1e8c-45f8-a49f-674b1d145a19] Failed to perform http put request: javax.ws.rs.NotAllowedException: HTTP 405 Method Not Allowed org.apache.pulsar.client.admin.PulsarAdminException$NotAllowedException: HTTP 405 Method Not Allowed at org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:232) at org.apache.pulsar.client.admin.internal.BaseResource$1.failed(BaseResource.java:130) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch master updated: [feature][broker] add config maxUnloadBundleNumPerShedding for UniformLoadShedder (#16409)
This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 0f2060d0d28 [feature][broker] add config maxUnloadBundleNumPerShedding
for UniformLoadShedder (#16409)
0f2060d0d28 is described below
commit 0f2060d0d2815ae5691b7dcee44878bfb3a9db82
Author: LinChen <1572139...@qq.com>
AuthorDate: Fri Jul 29 12:23:15 2022 +0800
[feature][broker] add config maxUnloadBundleNumPerShedding for
UniformLoadShedder (#16409)
---
.../apache/pulsar/broker/ServiceConfiguration.java | 8 +
.../loadbalance/impl/UniformLoadShedder.java | 4 +++
.../loadbalance/impl/UniformLoadShedderTest.java | 40 ++
3 files changed, 52 insertions(+)
diff --git
a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
index 91925bc85e8..a6bd9dde24f 100644
---
a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
+++
b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
@@ -2184,6 +2184,14 @@ public class ServiceConfiguration implements
PulsarConfiguration {
)
private double
loadBalancerMsgThroughputMultiplierDifferenceShedderThreshold = 4;
+@FieldContext(
+dynamic = true,
+category = CATEGORY_LOAD_BALANCER,
+doc = "For each uniform balanced unload, the maximum number of
bundles that can be unloaded."
++ " The default value is -1, which means no limit"
+)
+private int maxUnloadBundleNumPerShedding = -1;
+
@FieldContext(
dynamic = true,
category = CATEGORY_LOAD_BALANCER,
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/UniformLoadShedder.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/UniformLoadShedder.java
index 486f5e21a81..c31f9e14196 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/UniformLoadShedder.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/UniformLoadShedder.java
@@ -140,6 +140,10 @@ public class UniformLoadShedder implements
LoadSheddingStrategy {
return Triple.of(bundle, bundleData, throughput);
}).filter(e ->
!recentlyUnloadedBundles.containsKey(e.getLeft()))
.sorted((e1, e2) -> Double.compare(e2.getRight(),
e1.getRight())).forEach((e) -> {
+if (conf.getMaxUnloadBundleNumPerShedding() != -1
+&& selectedBundlesCache.size() >=
conf.getMaxUnloadBundleNumPerShedding()) {
+return;
+}
String bundle = e.getLeft();
BundleData bundleData = e.getMiddle();
TimeAverageMessageData shortTermData =
bundleData.getShortTermData();
diff --git
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/impl/UniformLoadShedderTest.java
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/impl/UniformLoadShedderTest.java
index 7affb52c2e4..7498386c113 100644
---
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/impl/UniformLoadShedderTest.java
+++
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/impl/UniformLoadShedderTest.java
@@ -24,6 +24,7 @@ import org.apache.pulsar.broker.loadbalance.LoadData;
import org.apache.pulsar.policies.data.loadbalancer.*;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Test;
+import static org.testng.Assert.assertEquals;
import static org.testng.Assert.assertFalse;
@Test(groups = "broker")
@@ -41,6 +42,45 @@ public class UniformLoadShedderTest {
uniformLoadShedder = new UniformLoadShedder();
}
+@Test
+public void testMaxUnloadBundleNumPerShedding(){
+conf.setMaxUnloadBundleNumPerShedding(2);
+int numBundles = 20;
+LoadData loadData = new LoadData();
+
+LocalBrokerData broker1 = new LocalBrokerData();
+LocalBrokerData broker2 = new LocalBrokerData();
+
+String broker2Name = "broker2";
+
+double brokerThroughput = 0;
+
+for (int i = 1; i <= numBundles; ++i) {
+broker1.getBundles().add("bundle-" + i);
+
+BundleData bundle = new BundleData();
+
+TimeAverageMessageData timeAverageMessageData = new
TimeAverageMessageData();
+
+double throughput = 1 * 1024 * 1024;
+timeAverageMessageData.setMsgThroughputIn(throughput);
+timeAverageMessageData.setMsgThroughputOut(throughput);
+
[GitHub] [pulsar] codelipenghui merged pull request #16409: [feature][broker] add config maxUnloadBundleNumPerShedding for UniformLoadShedder
codelipenghui merged PR #16409: URL: https://github.com/apache/pulsar/pull/16409 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on pull request #16842: [Java Client] fix PatternTopicsChangedListener blocked when topic removed
BewareMyPower commented on PR #16842: URL: https://github.com/apache/pulsar/pull/16842#issuecomment-1198865003 Please rebase to master to have https://github.com/apache/pulsar/pull/16812 included. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on pull request #16647: [fix][broker]Ensure that the ledgerId updated by currentLedger is monotonically increasing
lordcheng10 commented on PR #16647: URL: https://github.com/apache/pulsar/pull/16647#issuecomment-1198862263 @codelipenghui @hangc0276 PTAL,thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on pull request #16836: [feat]: support check PR name and automatically add labels based on PR title
Anonymitaet commented on PR #16836: URL: https://github.com/apache/pulsar/pull/16836#issuecomment-1198860960 @mangoGoForward wechat invitation sent -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] tjiuming commented on a diff in pull request #16659: [schema][client][improve] Add decode InputStream for Schema
tjiuming commented on code in PR #16659:
URL: https://github.com/apache/pulsar/pull/16659#discussion_r932851305
##
pulsar-client/src/main/java/org/apache/pulsar/client/util/ReadonlyByteBufInputStream.java:
##
@@ -0,0 +1,53 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.client.util;
+
+import io.netty.buffer.ByteBuf;
+import java.io.IOException;
+import java.io.InputStream;
+
+/**
+ * Wrap ByteBuf to InputStream, this will not move ByteBuf readerIndex.
+ */
+public final class ReadonlyByteBufInputStream extends InputStream {
Review Comment:
yes, I just have a try and see if the CI checks passed, will delete them
later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[pulsar-site] branch main updated: Add qraft case study (#155)
This is an automated email from the ASF dual-hosted git repository.
mmarshall pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/pulsar-site.git
The following commit(s) were added to refs/heads/main by this push:
new 6ad7791a8e8 Add qraft case study (#155)
6ad7791a8e8 is described below
commit 6ad7791a8e8a4c5eddc7d53c06dfe07f78d47a83
Author: Sherlock Xu <65327072+sherlock...@users.noreply.github.com>
AuthorDate: Fri Jul 29 12:09:29 2022 +0800
Add qraft case study (#155)
---
site2/website-next/src/pages/case-studies.js | 6 ++
site2/website-next/static/img/qraft.png | Bin 0 -> 83280 bytes
2 files changed, 6 insertions(+)
diff --git a/site2/website-next/src/pages/case-studies.js
b/site2/website-next/src/pages/case-studies.js
index b1da33210c4..547b6d54b8d 100644
--- a/site2/website-next/src/pages/case-studies.js
+++ b/site2/website-next/src/pages/case-studies.js
@@ -140,6 +140,12 @@ const csObj =
image: require("@site/static/img/proxyclick.png").default,
description: "It gives us consistency with the messages in the queue.
It also allows us to replay messages, and it’s a very powerful tool for the
distributed systems that like us."
},
+ {
+company: "Qraft",
+link: 'https://streamnative.io/success-story/qraft/',
+image: require("@site/static/img/qraft.png").default,
+description: "We choose Pulsar for its ability to manage distributed
transactions within a microservice architecture and its feature flexibility.
Pulsar now plays an essential part in helping our AI-powered order execution
system to find the optimal strategy in real time."
+ },
{
company: "Softtech",
link: 'https://www.youtube.com/watch?v=meDBzeintyE',
diff --git a/site2/website-next/static/img/qraft.png
b/site2/website-next/static/img/qraft.png
new file mode 100644
index 000..667a365685d
Binary files /dev/null and b/site2/website-next/static/img/qraft.png differ
[GitHub] [pulsar] lordcheng10 commented on pull request #16409: add config maxUnloadBundleNumPerShedding for UniformLoadShedder
lordcheng10 commented on PR #16409: URL: https://github.com/apache/pulsar/pull/16409#issuecomment-1198858847 @codelipenghui @Technoboy- PTAL,thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] aloyszhang commented on a diff in pull request #16409: add config maxUnloadBundleNumPerShedding for UniformLoadShedder
aloyszhang commented on code in PR #16409:
URL: https://github.com/apache/pulsar/pull/16409#discussion_r932848471
##
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java:
##
@@ -2153,6 +2153,14 @@ public class ServiceConfiguration implements
PulsarConfiguration {
)
private double
loadBalancerMsgThroughputMultiplierDifferenceShedderThreshold = 4;
+@FieldContext(
+dynamic = true,
+category = CATEGORY_LOAD_BALANCER,
+doc = "For each uniform balanced unload, the maximum number of
bundles that can be unloaded."
++ " The default value is -1, which means no limit"
+)
+private int maxUnloadBundleNumPerShedding = -1;
Review Comment:
Got it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on pull request #16071: [Flakey-test] fix flaky-test RackAwareTest.testRackUpdate
BewareMyPower commented on PR #16071: URL: https://github.com/apache/pulsar/pull/16071#issuecomment-1198856887 Remove the release labels because `testRackUpdate` was first introduced in https://github.com/apache/pulsar/pull/14248, which was added since 2.10. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on a diff in pull request #16409: add config maxUnloadBundleNumPerShedding for UniformLoadShedder
lordcheng10 commented on code in PR #16409:
URL: https://github.com/apache/pulsar/pull/16409#discussion_r932844551
##
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java:
##
@@ -2153,6 +2153,14 @@ public class ServiceConfiguration implements
PulsarConfiguration {
)
private double
loadBalancerMsgThroughputMultiplierDifferenceShedderThreshold = 4;
+@FieldContext(
+dynamic = true,
+category = CATEGORY_LOAD_BALANCER,
+doc = "For each uniform balanced unload, the maximum number of
bundles that can be unloaded."
++ " The default value is -1, which means no limit"
+)
+private int maxUnloadBundleNumPerShedding = -1;
Review Comment:
0 means unload cannot be performed. @aloyszhang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] Nicklee007 commented on pull request #16837: [fix][client]Fix client memory limit currentUsage leak and semaphore release duplicated in ProducerImpl
Nicklee007 commented on PR #16837: URL: https://github.com/apache/pulsar/pull/16837#issuecomment-1198851455 @codelipenghui @Jason918 @Shoothzj Also find producer semaphore release duplicated in `batchMessageAndSend`, if some error cause in `batchMessageAndSend`, the `semaphoreRelease` will be invoked in `failPendingBatchMessages` . Fixed it and add some unit test to cover it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on pull request #16647: [fix][broker]Ensure that the ledgerId updated by currentLedger is monotonically increasing
lordcheng10 commented on PR #16647: URL: https://github.com/apache/pulsar/pull/16647#issuecomment-1198850308 > @lordcheng10 Looks like we have the lock to ensure concurrency problem, could you describe the detail? When there is a serious Full GC, there may be multiple ledgers to update the metadata information on zookeeper at the same time. Suppose we create ledger in the order of ledger1, ledger2, ledger3, ledger4, ledger5, and assume that a comparison occurs. In severe Full GC, ledger5 may grab the metadataMutex lock before ledger4, so ledger5 may trigger the zookeeper callback method operationComplete before ledger4, so that ledger4 may cover ledger5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on a diff in pull request #16202: [feature][broker] PIP 37: Support chunking with Shared subscription
BewareMyPower commented on code in PR #16202:
URL: https://github.com/apache/pulsar/pull/16202#discussion_r932841311
##
pulsar-broker/src/test/java/org/apache/pulsar/broker/service/SharedConsumerAssignorTest.java:
##
@@ -0,0 +1,221 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.service;
+
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertNull;
+import static org.testng.Assert.assertSame;
+import static org.testng.Assert.assertTrue;
+import com.google.common.collect.Sets;
+import io.netty.buffer.ByteBuf;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import lombok.RequiredArgsConstructor;
+import org.apache.bookkeeper.mledger.impl.EntryImpl;
+import org.apache.pulsar.common.allocator.PulsarByteBufAllocator;
+import org.apache.pulsar.common.api.proto.MessageMetadata;
+import org.apache.pulsar.common.protocol.Commands;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+public class SharedConsumerAssignorTest {
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] aloyszhang commented on a diff in pull request #16409: add config maxUnloadBundleNumPerShedding for UniformLoadShedder
aloyszhang commented on code in PR #16409:
URL: https://github.com/apache/pulsar/pull/16409#discussion_r932835884
##
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java:
##
@@ -2153,6 +2153,14 @@ public class ServiceConfiguration implements
PulsarConfiguration {
)
private double
loadBalancerMsgThroughputMultiplierDifferenceShedderThreshold = 4;
+@FieldContext(
+dynamic = true,
+category = CATEGORY_LOAD_BALANCER,
+doc = "For each uniform balanced unload, the maximum number of
bundles that can be unloaded."
++ " The default value is -1, which means no limit"
+)
+private int maxUnloadBundleNumPerShedding = -1;
Review Comment:
It seems maxUnloadBundleNumPerShedding=0 also means no limit, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] coderzc commented on pull request #16832: [feature][broker] Support cgroup v2 by using `jdk.internal.platform.Metrics` in Pulsar Loadbalancer
coderzc commented on PR #16832: URL: https://github.com/apache/pulsar/pull/16832#issuecomment-1198847499 /pulsarbot run-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] keyboardbobo commented on issue #16859: [broker] Cursor status has always been SwitchingLedger and pendingMarkDeleteOps has accumulated tens of thousands of requests
keyboardbobo commented on issue #16859: URL: https://github.com/apache/pulsar/issues/16859#issuecomment-1198846624 My version is 2.9.2, the dump file is very large, and it is difficult to upload it on the intranet due to security reasons. The values of the ManagedCursorMXBean mbean are as follows: > persistLedgeSucceed：cellsBusy=0,base=63308,cells:null > persistLedgeFailed：cellsBusy=0,base=1,cells:null > persistZookeeperSucceed：cellsBusy=0,base=1,cells:null > persistZookeeperFailed：cellsBusy=0,base=0,cells:null > writeCursorLedgerSize：cellsBusy=0,base=3437666,cells:null > writeCursorLedgerLogicalSize：cellsBusy=0,base=1718833,cells:null > readCursorLedgerSize：cellsBusy=0,base=9,cells:null -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui merged pull request #16391: [improve][broker] Add dynamic configuration for UniformLoadShedder
codelipenghui merged PR #16391: URL: https://github.com/apache/pulsar/pull/16391 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch master updated: [improve][broker] Add dynamic configuration to UniformLoadShedder (#16391)
This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 4d64e2e6668 [improve][broker] Add dynamic configuration to
UniformLoadShedder (#16391)
4d64e2e6668 is described below
commit 4d64e2e66689381ebbb94fbfc03eb4e1dfba0405
Author: LinChen <1572139...@qq.com>
AuthorDate: Fri Jul 29 11:35:15 2022 +0800
[improve][broker] Add dynamic configuration to UniformLoadShedder (#16391)
---
.../apache/pulsar/broker/ServiceConfiguration.java | 21 +
.../broker/loadbalance/impl/UniformLoadShedder.java | 14 +-
2 files changed, 26 insertions(+), 9 deletions(-)
diff --git
a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
index 178e047a32d..91925bc85e8 100644
---
a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
+++
b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
@@ -2144,6 +2144,27 @@ public class ServiceConfiguration implements
PulsarConfiguration {
)
private int loadBalancerAverageResourceUsageDifferenceThresholdPercentage
= 10;
+@FieldContext(
+dynamic = true,
+category = CATEGORY_LOAD_BALANCER,
+doc = "In the UniformLoadShedder strategy, the minimum message
that triggers unload."
+)
+private int minUnloadMessage = 1000;
+
+@FieldContext(
+dynamic = true,
+category = CATEGORY_LOAD_BALANCER,
+doc = "In the UniformLoadShedder strategy, the minimum throughput
that triggers unload."
+)
+private int minUnloadMessageThroughput = 1 * 1024 * 1024;
+
+@FieldContext(
+dynamic = true,
+category = CATEGORY_LOAD_BALANCER,
+doc = "In the UniformLoadShedder strategy, the maximum unload
ratio."
+)
+private double maxUnloadPercentage = 0.2;
+
@FieldContext(
dynamic = true,
category = CATEGORY_LOAD_BALANCER,
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/UniformLoadShedder.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/UniformLoadShedder.java
index 3d400b07cae..486f5e21a81 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/UniformLoadShedder.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/UniformLoadShedder.java
@@ -44,11 +44,6 @@ import
org.apache.pulsar.policies.data.loadbalancer.TimeAverageMessageData;
*/
@Slf4j
public class UniformLoadShedder implements LoadSheddingStrategy {
-
-private static final int MB = 1024 * 1024;
-private static final double MAX_UNLOAD_PERCENTAGE = 0.2;
-private static final int MIN_UNLOAD_MESSAGE = 1000;
-private static final int MIN_UNLOAD_THROUGHPUT = 1 * MB;
private final Multimap selectedBundlesCache =
ArrayListMultimap.create();
private static final double EPS = 1e-6;
@@ -122,14 +117,15 @@ public class UniformLoadShedder implements
LoadSheddingStrategy {
underloadedBroker.getValue(), minMsgRate.getValue(),
minThroughputRate.getValue());
}
MutableInt msgRateRequiredFromUnloadedBundles = new MutableInt(
-(int) ((maxMsgRate.getValue() - minMsgRate.getValue()) *
MAX_UNLOAD_PERCENTAGE));
+(int) ((maxMsgRate.getValue() - minMsgRate.getValue()) *
conf.getMaxUnloadPercentage()));
MutableInt msgThroughputRequiredFromUnloadedBundles = new
MutableInt(
-(int) ((maxThroughputRate.getValue() -
minThroughputRate.getValue()) * MAX_UNLOAD_PERCENTAGE));
+(int) ((maxThroughputRate.getValue() -
minThroughputRate.getValue())
+* conf.getMaxUnloadPercentage()));
LocalBrokerData overloadedBrokerData =
brokersData.get(overloadedBroker.getValue()).getLocalData();
if (overloadedBrokerData.getBundles().size() > 1
-&& (msgRateRequiredFromUnloadedBundles.getValue() >=
MIN_UNLOAD_MESSAGE
-|| msgThroughputRequiredFromUnloadedBundles.getValue() >=
MIN_UNLOAD_THROUGHPUT)) {
+&& (msgRateRequiredFromUnloadedBundles.getValue() >=
conf.getMinUnloadMessage()
+|| msgThroughputRequiredFromUnloadedBundles.getValue() >=
conf.getMinUnloadMessageThroughput())) {
// Sort bundles by throughput, then pick the bundle which can
help to reduce load uniformly with
// under-loaded broker
loadBundleData.entrySet().stream()
[GitHub] [pulsar] mangoGoForward commented on pull request #16836: [feat]: support check PR name and automatically add labels based on PR title
mangoGoForward commented on PR #16836: URL: https://github.com/apache/pulsar/pull/16836#issuecomment-1198845083 > @mangoGoForward How can I contact you personally? QQ: 1253319389 please @Anonymitaet -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on pull request #16409: add config maxUnloadBundleNumPerShedding for UniformLoadShedder
lordcheng10 commented on PR #16409: URL: https://github.com/apache/pulsar/pull/16409#issuecomment-1198844016 /pulsarbot run-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on pull request #16836: [feat]: support check PR name and automatically add labels based on PR title
Anonymitaet commented on PR #16836: URL: https://github.com/apache/pulsar/pull/16836#issuecomment-1198843960 @mangoGoForward How can I contact you personally? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on pull request #16409: add config maxUnloadBundleNumPerShedding for UniformLoadShedder
lordcheng10 commented on PR #16409: URL: https://github.com/apache/pulsar/pull/16409#issuecomment-1198843685 /pulsarbot run-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on pull request #16409: add config maxUnloadBundleNumPerShedding for UniformLoadShedder
lordcheng10 commented on PR #16409: URL: https://github.com/apache/pulsar/pull/16409#issuecomment-1198843530 /pulsarbot run-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on pull request #16391: [improve][broker] Add dynamic configuration for UniformLoadShedder
lordcheng10 commented on PR #16391: URL: https://github.com/apache/pulsar/pull/16391#issuecomment-1198843136 /pulsarbot run-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on issue #16861: [branch-2.8] testMaxNumPartitionsPerPartitionedTopicTopicCreation is broken
BewareMyPower commented on issue #16861: URL: https://github.com/apache/pulsar/issues/16861#issuecomment-1198841386 Revert it first. It's because in branch-2.8, `checkMaxTopicsPerNamespace` is a synchronous method that cannot be invoked in the callback. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch branch-2.8 updated: Revert "[improve][broker] Avoid reconnection when a partitioned topic was created concurrently (#16043)"
This is an automated email from the ASF dual-hosted git repository.
xyz pushed a commit to branch branch-2.8
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-2.8 by this push:
new 3b164f5ac0c Revert "[improve][broker] Avoid reconnection when a
partitioned topic was created concurrently (#16043)"
3b164f5ac0c is described below
commit 3b164f5ac0c900d6c162596892ec7e702e48f937
Author: Yunze Xu
AuthorDate: Fri Jul 29 11:25:00 2022 +0800
Revert "[improve][broker] Avoid reconnection when a partitioned topic was
created concurrently (#16043)"
This reverts commit b01d5d2da269c5f072167e8f21d569e25a35306f.
---
.../pulsar/broker/service/BrokerService.java | 48 +-
1 file changed, 10 insertions(+), 38 deletions(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
index d943734e0a8..bdaded637b6 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
@@ -2412,44 +2412,16 @@ public class BrokerService implements Closeable,
ZooKeeperCacheListener {
return fetchPartitionedTopicMetadataAsync(topicName)
.thenCompose(metadata -> {
-CompletableFuture
future = new CompletableFuture<>();
-
-// There are a couple of potentially blocking
calls, which we cannot make from the
-// MetadataStore callback thread.
-pulsar.getExecutor().execute(() -> {
-// If topic is already exist, creating
partitioned topic is not allowed.
-
-if (metadata.partitions == 0
-&& !topicExists
-&& !topicName.isPartitioned()
-&&
pulsar.getBrokerService().isAllowAutoTopicCreation(topicName)
-&&
pulsar.getBrokerService().isDefaultTopicTypePartitioned(topicName)) {
-
-
pulsar.getBrokerService().createDefaultPartitionedTopicAsync(topicName)
-.thenAccept(md ->
future.complete(md))
-.exceptionally(ex -> {
-if (ex.getCause()
-instanceof
MetadataStoreException.AlreadyExistsException) {
-// The partitioned
topic might be created concurrently
-
fetchPartitionedTopicMetadataAsync(topicName)
-
.whenComplete((metadata2, ex2) -> {
-if (ex2 ==
null) {
-
future.complete(metadata2);
-} else {
-
future.completeExceptionally(ex2);
-}
-});
-} else {
-
future.completeExceptionally(ex);
-}
-return null;
-});
-} else {
-future.complete(metadata);
-}
-});
-
-return future;
+// If topic is already exist, creating
partitioned topic is not allowed.
+if (metadata.partitions == 0
+&& !topicExists
+&& !topicName.isPartitioned()
+&&
pulsar.getBrokerService().isAllowAutoTopicCreation(topicName)
+&&
pulsar.getBrokerService().isDefaultTopicTypePartitioned(topicName)) {
+return
pulsar.getBrokerService().createDefaultPartitionedTopicAsync(topicName);
+} else {
+return
CompletableFuture.completedFuture(metadata);
+
[GitHub] [pulsar] lordcheng10 commented on pull request #16409: add config maxUnloadBundleNumPerShedding for UniformLoadShedder
lordcheng10 commented on PR #16409: URL: https://github.com/apache/pulsar/pull/16409#issuecomment-1198840597 /pulsarbot run-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch master updated: [fix][broker] ManagedCursor: mark delete no callback when create meta-ledger fail (#16841)
This is an automated email from the ASF dual-hosted git repository.
technoboy pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 5faac761372 [fix][broker] ManagedCursor: mark delete no callback when
create meta-ledger fail (#16841)
5faac761372 is described below
commit 5faac761372c5b0848924d6607a8da77f209679c
Author: fengyubiao
AuthorDate: Fri Jul 29 11:21:11 2022 +0800
[fix][broker] ManagedCursor: mark delete no callback when create
meta-ledger fail (#16841)
---
.../java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java| 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java
index e7ec241291d..83ee356e5c3 100644
---
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java
+++
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java
@@ -1868,8 +1868,10 @@ public class ManagedCursorImpl implements ManagedCursor {
return;
case NoLedger:
-// We need to create a new ledger to write into
+pendingMarkDeleteOps.add(mdEntry);
+// We need to create a new ledger to write into.
startCreatingNewMetadataLedger();
+break;
// fall through
case SwitchingLedger:
pendingMarkDeleteOps.add(mdEntry);
[GitHub] [pulsar] Technoboy- merged pull request #16841: [fix][broker] ManagedCursor: mark delete no callback when create meta-ledger fail
Technoboy- merged PR #16841: URL: https://github.com/apache/pulsar/pull/16841 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] Technoboy- closed issue #16711: Flaky-test: ManagedCursorTest.markDeleteWithZKErrors
Technoboy- closed issue #16711: Flaky-test: ManagedCursorTest.markDeleteWithZKErrors URL: https://github.com/apache/pulsar/issues/16711 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on pull request #16391: [improve][broker] Add dynamic configuration for UniformLoadShedder
lordcheng10 commented on PR #16391: URL: https://github.com/apache/pulsar/pull/16391#issuecomment-1198839211 /pulsarbot run-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch master updated: [improve][broker] refactor ManagedLedger cacheEvictionTask implement (#14488)
This is an automated email from the ASF dual-hosted git repository.
technoboy pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 3619edc624a [improve][broker] refactor ManagedLedger cacheEvictionTask
implement (#14488)
3619edc624a is described below
commit 3619edc624ac223637f3e2b3fff2674ad9c5e0b6
Author: AloysZhang
AuthorDate: Fri Jul 29 11:18:12 2022 +0800
[improve][broker] refactor ManagedLedger cacheEvictionTask implement
(#14488)
---
conf/broker.conf | 9 ---
deployment/terraform-ansible/templates/broker.conf | 7 +++--
.../mledger/ManagedLedgerFactoryConfig.java| 4 +--
.../mledger/impl/ManagedLedgerFactoryImpl.java | 31 +-
.../mledger/impl/EntryCacheManagerTest.java| 4 +--
.../bookkeeper/mledger/impl/ManagedLedgerTest.java | 2 +-
.../apache/pulsar/broker/ServiceConfiguration.java | 22 +--
.../pulsar/broker/ManagedLedgerClientFactory.java | 2 +-
.../pulsar/client/api/ConsumerRedeliveryTest.java | 2 +-
.../pulsar/client/api/PartitionCreationTest.java | 2 +-
10 files changed, 46 insertions(+), 39 deletions(-)
diff --git a/conf/broker.conf b/conf/broker.conf
index 7f07ae54449..41809f87116 100644
--- a/conf/broker.conf
+++ b/conf/broker.conf
@@ -1014,8 +1014,8 @@ managedLedgerCacheCopyEntries=false
# Threshold to which bring down the cache level when eviction is triggered
managedLedgerCacheEvictionWatermark=0.9
-# Configure the cache eviction frequency for the managed ledger cache
(evictions/sec)
-managedLedgerCacheEvictionFrequency=100.0
+# Configure the cache eviction interval in milliseconds for the managed ledger
cache
+managedLedgerCacheEvictionIntervalMs=10
# All entries that have stayed in cache for more than the configured time,
will be evicted
managedLedgerCacheEvictionTimeThresholdMillis=1000
@@ -1572,4 +1572,7 @@ managedLedgerMaxUnackedRangesToPersistInZooKeeper=-1
# If enabled, the maximum "acknowledgment holes" will not be limited and
"acknowledgment holes" are stored in
# multiple entries.
-persistentUnackedRangesWithMultipleEntriesEnabled=false
\ No newline at end of file
+persistentUnackedRangesWithMultipleEntriesEnabled=false
+
+# Deprecated - Use managedLedgerCacheEvictionIntervalMs instead
+managedLedgerCacheEvictionFrequency=0
\ No newline at end of file
diff --git a/deployment/terraform-ansible/templates/broker.conf
b/deployment/terraform-ansible/templates/broker.conf
index fb4456f0b1d..1ed98cd4e02 100644
--- a/deployment/terraform-ansible/templates/broker.conf
+++ b/deployment/terraform-ansible/templates/broker.conf
@@ -772,8 +772,8 @@ managedLedgerCacheCopyEntries=false
# Threshold to which bring down the cache level when eviction is triggered
managedLedgerCacheEvictionWatermark=0.9
-# Configure the cache eviction frequency for the managed ledger cache
(evictions/sec)
-managedLedgerCacheEvictionFrequency=100.0
+# Configure the cache eviction interval in milliseconds for the managed ledger
cache
+managedLedgerCacheEvictionIntervalMs=10
# All entries that have stayed in cache for more than the configured time,
will be evicted
managedLedgerCacheEvictionTimeThresholdMillis=1000
@@ -1137,6 +1137,9 @@ brokerServicePurgeInactiveFrequencyInSeconds=60
# Deprecated: use managedLedgerMaxUnackedRangesToPersistInMetadataStore
managedLedgerMaxUnackedRangesToPersistInZooKeeper=-1
+# Deprecated - Use managedLedgerCacheEvictionIntervalMs instead
+managedLedgerCacheEvictionFrequency=0
+
### --- Transaction config variables --- ###
# Enable transaction coordinator in broker
diff --git
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/ManagedLedgerFactoryConfig.java
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/ManagedLedgerFactoryConfig.java
index 25fcb377e3e..78314be45c3 100644
---
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/ManagedLedgerFactoryConfig.java
+++
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/ManagedLedgerFactoryConfig.java
@@ -43,9 +43,9 @@ public class ManagedLedgerFactoryConfig {
private int numManagedLedgerSchedulerThreads =
Runtime.getRuntime().availableProcessors();
/**
- * Frequency of cache eviction triggering. Default is 100 times per second.
+ * Interval of cache eviction triggering. Default is 10 ms times.
*/
-private double cacheEvictionFrequency = 100;
+private long cacheEvictionIntervalMs = 10;
/**
* All entries that have stayed in cache for more than the configured
time, will be evicted.
diff --git
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerFactoryImpl.java
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerFactoryImpl.java
index 8e3271a0393..629e96ba3e3 100644
---
[GitHub] [pulsar] Technoboy- merged pull request #14488: [improve][broker] refactor ManagedLedger cacheEvictionTask implement
Technoboy- merged PR #14488: URL: https://github.com/apache/pulsar/pull/14488 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on a diff in pull request #16408: [fix][broker] Support loadBalancerSheddingIntervalMinutes dynamic configuration
lordcheng10 commented on code in PR #16408:
URL: https://github.com/apache/pulsar/pull/16408#discussion_r932832121
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/LoadSheddingTask.java:
##
@@ -39,6 +49,13 @@ public void run() {
loadManager.get().doLoadShedding();
} catch (Exception e) {
LOG.warn("Error during the load shedding", e);
+} finally {
+if (!loadManagerExecutor.isShutdown()) {
+loadManagerExecutor.schedule(
+new LoadSheddingTask(loadManager, loadManagerExecutor,
config),
Review Comment:
fixed, PTAL,thanks! @Jason918
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on pull request #16391: Add dynamic configuration for UniformLoadShedder
lordcheng10 commented on PR #16391: URL: https://github.com/apache/pulsar/pull/16391#issuecomment-1198834806 @Technoboy- @codelipenghui PTAL,thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] horizonzy commented on pull request #16590: [PIP-186] Introduce two phase deletion protocol based on system topic
horizonzy commented on PR #16590: URL: https://github.com/apache/pulsar/pull/16590#issuecomment-1198831319 See #16569 firstly, the pip has changed, the code is not match the new pip, after the pip passing, I will change this code. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] mangoGoForward commented on pull request #16836: [feat]: support check PR name and automatically add labels based on PR title
mangoGoForward commented on PR #16836: URL: https://github.com/apache/pulsar/pull/16836#issuecomment-1198827288 > @mangoGoForward I have some questions on this PR. Can we have a talk through Zoom/Tencent/...? Want to align requirements firstly OK, Zoom or Tencent all is ok. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on issue #16861: [branch-2.8] testMaxNumPartitionsPerPartitionedTopicTopicCreation is broken
BewareMyPower commented on issue #16861: URL: https://github.com/apache/pulsar/issues/16861#issuecomment-1198825462 It was introduced by the b01d5d2d -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on pull request #16836: [feat]: support check PR name and automatically add labels based on PR title
Anonymitaet commented on PR #16836: URL: https://github.com/apache/pulsar/pull/16836#issuecomment-1198825035 @mangoGoForward I have some questions on this PR. Can we have a talk through Zoom/Tencent/...? Want to align requirements firstly -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch master updated: Forget to update memory usage when invalid message (#16835)
This is an automated email from the ASF dual-hosted git repository.
shoothzj pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 57b008a4114 Forget to update memory usage when invalid message (#16835)
57b008a4114 is described below
commit 57b008a411463bce4c26350177dae4346f7b84d2
Author: ZhangJian He
AuthorDate: Fri Jul 29 10:43:50 2022 +0800
Forget to update memory usage when invalid message (#16835)
### Modifications
release memory usage when invalid message.
Only need to release memory usage here, no need to release semaphore. Both
add testcases.
coauthored by @pengxiangrui127.
### Verifying this change
- add unit tests for this change
### Documentation
Check the box below or label this PR directly.
Need to update docs?
- [x] `doc-not-needed`
bug fix, no need doc
---
.../client/impl/ProducerMemoryLimitTest.java | 27 ++
.../pulsar/client/impl/ProducerSemaphoreTest.java | 26 +
.../client/impl/BatchMessageContainerImpl.java | 2 ++
3 files changed, 55 insertions(+)
diff --git
a/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/ProducerMemoryLimitTest.java
b/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/ProducerMemoryLimitTest.java
index 264ec306413..0856dfc88b2 100644
---
a/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/ProducerMemoryLimitTest.java
+++
b/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/ProducerMemoryLimitTest.java
@@ -23,6 +23,8 @@ import org.apache.pulsar.client.api.ProducerConsumerBase;
import org.apache.pulsar.client.api.PulsarClient;
import org.apache.pulsar.client.api.PulsarClientException;
import org.apache.pulsar.client.api.SizeUnit;
+import org.mockito.MockedStatic;
+import org.mockito.Mockito;
import org.testng.Assert;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeMethod;
@@ -47,6 +49,31 @@ public class ProducerMemoryLimitTest extends
ProducerConsumerBase {
super.internalCleanup();
}
+@Test(timeOut = 10_000)
+public void testProducerInvalidMessageMemoryRelease() throws Exception {
+initClientWithMemoryLimit();
+@Cleanup
+ProducerImpl producer = (ProducerImpl)
pulsarClient.newProducer()
+.topic("testProducerMemoryLimit")
+.sendTimeout(5, TimeUnit.SECONDS)
+.batchingMaxPublishDelay(100, TimeUnit.MILLISECONDS)
+.batchingMaxBytes(10240)
+.enableBatching(true)
+.create();
+this.stopBroker();
+try {
+try (MockedStatic mockedStatic =
Mockito.mockStatic(ClientCnx.class)) {
+mockedStatic.when(ClientCnx::getMaxMessageSize).thenReturn(8);
+producer.send("memory-test".getBytes(StandardCharsets.UTF_8));
+}
+throw new IllegalStateException("can not reach here");
+} catch (PulsarClientException.InvalidMessageException ex) {
+PulsarClientImpl clientImpl = (PulsarClientImpl) this.pulsarClient;
+final MemoryLimitController memoryLimitController =
clientImpl.getMemoryLimitController();
+Assert.assertEquals(memoryLimitController.currentUsage(), 0);
+}
+}
+
@Test(timeOut = 10_000)
public void testProducerTimeoutMemoryRelease() throws Exception {
initClientWithMemoryLimit();
diff --git
a/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/ProducerSemaphoreTest.java
b/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/ProducerSemaphoreTest.java
index 78fc659a205..cc7b601e42a 100644
---
a/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/ProducerSemaphoreTest.java
+++
b/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/ProducerSemaphoreTest.java
@@ -25,12 +25,15 @@ import org.apache.pulsar.client.api.PulsarClientException;
import org.apache.pulsar.client.api.Schema;
import org.apache.pulsar.common.api.proto.MessageMetadata;
import org.apache.pulsar.common.util.FutureUtil;
+import org.mockito.MockedStatic;
+import org.mockito.Mockito;
import org.testng.Assert;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Test;
import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.CompletableFuture;
@@ -52,6 +55,29 @@ public class ProducerSemaphoreTest extends
ProducerConsumerBase {
super.internalCleanup();
}
+@Test(timeOut = 10_000)
+public void testProducerSemaphoreInvalidMessage() throws Exception {
+final int pendingQueueSize = 100;
+
+@Cleanup
+ProducerImpl producer = (ProducerImpl)
pulsarClient.newProducer()
+
[GitHub] [pulsar] Shoothzj merged pull request #16835: Forget to update memory usage when invalid message
Shoothzj merged PR #16835: URL: https://github.com/apache/pulsar/pull/16835 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] Nicklee007 commented on a diff in pull request #16837: [fix][client]Fix client memory limit currentUsage leak in ProducerImpl
Nicklee007 commented on code in PR #16837:
URL: https://github.com/apache/pulsar/pull/16837#discussion_r932820240
##
pulsar-broker/src/test/java/org/apache/pulsar/client/impl/ProducerMemoryLimitTest.java:
##
@@ -69,6 +69,35 @@ public void testProducerTimeoutMemoryRelease() throws
Exception {
}
+@Test(timeOut = 10_000)
+public void testProducerBatchSendTimeoutMemoryRelease() throws Exception {
+initClientWithMemoryLimit();
+@Cleanup
+ProducerImpl producer = (ProducerImpl)
pulsarClient.newProducer()
+.topic("testProducerMemoryLimit")
+.sendTimeout(5, TimeUnit.SECONDS)
+.maxPendingMessages(0)
+.enableBatching(true)
+.batchingMaxPublishDelay(100, TimeUnit.MILLISECONDS)
+.batchingMaxBytes(12)
+.create();
+this.stopBroker();
+try {
+
producer.newMessage().value("memory-test".getBytes(StandardCharsets.UTF_8)).sendAsync();
+try {
+
producer.newMessage().value("memory-test".getBytes(StandardCharsets.UTF_8)).sendAsync().get();
Review Comment:
@Jason918 Through some test, find the memory limit release operator add in
`batchMessageAndSend` will cause duplicate release with add in
`failPendingBatchMessages`. So remove the fixed in `batchMessageAndSend`.
Please take another look, Thks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[pulsar] branch master updated: [feature][blog] Add 2.9.3 release blog (#16772)
This is an automated email from the ASF dual-hosted git repository. mattisonchao pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/master by this push: new 17a9d2d4546 [feature][blog] Add 2.9.3 release blog (#16772) 17a9d2d4546 is described below commit 17a9d2d45462eee682f1eba29bb2584f211eb1fb Author: momo-jun <60642177+momo-...@users.noreply.github.com> AuthorDate: Fri Jul 29 10:35:19 2022 +0800 [feature][blog] Add 2.9.3 release blog (#16772) --- .../website/blog/2022-07-27-Apache-Pulsar-2-9-3.md | 82 ++ 1 file changed, 82 insertions(+) diff --git a/site2/website/blog/2022-07-27-Apache-Pulsar-2-9-3.md b/site2/website/blog/2022-07-27-Apache-Pulsar-2-9-3.md new file mode 100644 index 000..196883a739a --- /dev/null +++ b/site2/website/blog/2022-07-27-Apache-Pulsar-2-9-3.md @@ -0,0 +1,82 @@ +--- +title: "What’s New in Apache Pulsar 2.9.3" +date: 2022-07-27 +author: "mattisonchao, momo-jun" +--- + +The Apache Pulsar community releases version 2.9.3! 50 contributors provided improvements and bug fixes that delivered 200+ commits. Thanks for all your contributions. + +The highlight of the 2.9.3 release is introducing 30+ transaction fixes and improvements. Earlier-adoption users of Pulsar transactions have documented long-term use in their production environments and reported valuable findings in real applications. This provides the Pulsar community with the opportunity to make a difference. + +This blog walks through the most noteworthy changes. For the complete list including all feature enhancements and bug fixes, check out the [Pulsar 2.9.3 Release Notes](https://pulsar.apache.org/release-notes/versioned/pulsar-2.9.3/). + + +### Enabled cursor data compression to reduce persistent cursor data size. [14542](https://github.com/apache/pulsar/pull/14542) + + Issue +The cursor data is managed by the ZooKeeper/Etcd metadata store. When the data size increases, it may take too much time to pull the data, and brokers may end up writing large chunks of data to the ZooKeeper/Etcd metadata store. + + Resolution +Provide the ability to enable compression mechanisms to reduce cursor data size and the pulling time. + + +### Reduced the memory occupied by `metadataPositions` and avoid OOM. [15137](https://github.com/apache/pulsar/pull/15137) + + Issue +The map `metadataPositions` in MLPendingAckStore is used to clear useless data in PendingAck, where the key is the position that is persistent in PendingAck and the value is the max position acked by an operation. It judges whether the max subscription cursor position is smaller than the subscription cursor’s `markDeletePosition`. If the max position is smaller, then the log cursor will mark to delete the position. It causes two main issues: +* In normal cases, this map stores all transaction ack operations. This is a waste of memory and CPU. +* If a transaction that has not been committed for a long time acks a message in a later position, the map will not be cleaned up, which finally leads to OOM (out-of-memory). + + Resolution +Regularly store a small amount of data according to certain rules. For more detailed implementation, refer to [PIP-153](https://github.com/apache/pulsar/issues/15073). + + +### Checked `lowWaterMark` before appending transaction entries to Transaction Buffer. [15424](https://github.com/apache/pulsar/pull/15424) + + Issue +When a client sends messages using a previously committed transaction, these messages are visible to consumers unexpectedly. + + Resolution +Add a map to store the `lowWaterMark` of Transaction Coordinator in Trasanction Buffer, and check `lowWaterMark` before appending transaction entries to Trasanction Buffer. So when sending messages using an invalid transaction, clients will receive `NotAllowedException`. + + +### Fixed the consumption performance regression. [PR-15162](https://github.com/apache/pulsar/pull/15162) + + Issue +This performance regression was introduced in 2.10.0, 2.9.1, and 2.8.3. You may find a significant performance drop with message listeners while using Java Client. The root cause is each message will introduce the thread switching from the external thread pool to the internal thread poll and then to the external thread pool. + + Resolution +Avoid the thread switching for each message to improve consumption throughput. + + +### Fixed a deadlock issue of topic creation. [PR-15570](https://github.com/apache/pulsar/pull/15570) + + Issue +This deadlock issue occurred during topic creation by trying to re-acquire the same `StampedLock` from the same thread when removing it. This will cause the topic to stop service for a long time, and ultimately with a failure in the deduplication or geo-replication check. The workaround is restarting the broker. + + +### Optimized the memory usage of brokers. +
[GitHub] [pulsar] mattisonchao merged pull request #16772: [feature][blog] Add 2.9.3 release blog
mattisonchao merged PR #16772: URL: https://github.com/apache/pulsar/pull/16772 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] BewareMyPower opened a new issue, #16861: [branch-2.8] testMaxNumPartitionsPerPartitionedTopicTopicCreation is broken
BewareMyPower opened a new issue, #16861: URL: https://github.com/apache/pulsar/issues/16861 **Describe the bug** `BrokerServiceAutoTopicCreationTest#testMaxNumPartitionsPerPartitionedTopicTopicCreation` is broken in branch-2.8, see https://github.com/apache/pulsar/runs/7569323751?check_suite_focus=true. It has been broken for a long time but skipped by another failed test that was fixed recently. **Expected behavior** The test should pass. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #16860: Flaky-test: BookKeeperPackagesStorageTest.setUp
codelipenghui commented on issue #16860: URL: https://github.com/apache/pulsar/issues/16860#issuecomment-1198815024 Looks similar with https://github.com/apache/pulsar/issues/15773, but the stack is not 100% same -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui opened a new issue, #16860: Flaky-test: BookKeeperPackagesStorageTest.setUp
codelipenghui opened a new issue, #16860: URL: https://github.com/apache/pulsar/issues/16860 [example failure](https://github.com/apache/pulsar/runs/7571561624?check_suite_focus=true) ``` Error: Tests run: 31, Failures: 1, Errors: 0, Skipped: 28, Time elapsed: 45.57 s <<< FAILURE! - in org.apache.pulsar.packages.management.storage.bookkeeper.BookKeeperPackagesStorageTest Error: setUp(org.apache.pulsar.packages.management.storage.bookkeeper.BookKeeperPackagesStorageTest) Time elapsed: 24.472 s <<< FAILURE! org.apache.bookkeeper.bookie.BookieException$MetadataStoreException: Failed to initialize metadata bookie driver at org.apache.bookkeeper.bookie.BookieResources.createMetadataDriver(BookieResources.java:64) at org.apache.pulsar.packages.management.storage.bookkeeper.bookkeeper.test.BookKeeperClusterTestCase$ServerTester.(BookKeeperClusterTestCase.java:849) at org.apache.pulsar.packages.management.storage.bookkeeper.bookkeeper.test.BookKeeperClusterTestCase.startBookie(BookKeeperClusterTestCase.java:674) at org.apache.pulsar.packages.management.storage.bookkeeper.bookkeeper.test.BookKeeperClusterTestCase.startAndAddBookie(BookKeeperClusterTestCase.java:654) at org.apache.pulsar.packages.management.storage.bookkeeper.bookkeeper.test.BookKeeperClusterTestCase.startNewBookieAndReturnAddress(BookKeeperClusterTestCase.java:643) at org.apache.pulsar.packages.management.storage.bookkeeper.bookkeeper.test.BookKeeperClusterTestCase.startNewBookie(BookKeeperClusterTestCase.java:636) at org.apache.pulsar.packages.management.storage.bookkeeper.bookkeeper.test.BookKeeperClusterTestCase.startBKCluster(BookKeeperClusterTestCase.java:267) at org.apache.pulsar.packages.management.storage.bookkeeper.bookkeeper.test.BookKeeperClusterTestCase.setUp(BookKeeperClusterTestCase.java:181) at org.apache.pulsar.packages.management.storage.bookkeeper.bookkeeper.test.BookKeeperClusterTestCase.setUp(BookKeeperClusterTestCase.java:166) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132) at org.testng.internal.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:61) at org.testng.internal.ConfigInvoker.invokeConfigurationMethod(ConfigInvoker.java:366) at org.testng.internal.ConfigInvoker.invokeConfigurations(ConfigInvoker.java:320) at org.testng.internal.TestInvoker.runConfigMethods(TestInvoker.java:701) at org.testng.internal.TestInvoker.invokeMethod(TestInvoker.java:527) at org.testng.internal.TestInvoker.invokeTestMethod(TestInvoker.java:174) at org.testng.internal.MethodRunner.runInSequence(MethodRunner.java:46) at org.testng.internal.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:822) at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:147) at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146) at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128) at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) at org.testng.TestRunner.privateRun(TestRunner.java:764) at org.testng.TestRunner.run(TestRunner.java:585) at org.testng.SuiteRunner.runTest(SuiteRunner.java:384) at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:378) at org.testng.SuiteRunner.privateRun(SuiteRunner.java:337) at org.testng.SuiteRunner.run(SuiteRunner.java:286) at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53) at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96) at org.testng.TestNG.runSuitesSequentially(TestNG.java:1218) at org.testng.TestNG.runSuitesLocally(TestNG.java:1140) at org.testng.TestNG.runSuites(TestNG.java:1069) at org.testng.TestNG.run(TestNG.java:1037) at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:135) at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeSingleClass(TestNGDirectoryTestSuite.java:112) at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeLazy(TestNGDirectoryTestSuite.java:123) at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:90) at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:146) at
[GitHub] [pulsar] github-actions[bot] commented on issue #14732: IPv6 Support for broker and proxy with bindAddress
github-actions[bot] commented on issue #14732: URL: https://github.com/apache/pulsar/issues/14732#issuecomment-1198812407 The issue had no activity for 30 days, mark with Stale label. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #16017: [Discuss][Doc]: Remove active-standby replication mode on Geo Replication doc
github-actions[bot] commented on issue #16017: URL: https://github.com/apache/pulsar/issues/16017#issuecomment-1198811497 The issue had no activity for 30 days, mark with Stale label. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #16262: After version 2.10, it is recommended to fix more than 20 exceptions during startup
github-actions[bot] commented on issue #16262: URL: https://github.com/apache/pulsar/issues/16262#issuecomment-1198811227 The issue had no activity for 30 days, mark with Stale label. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #16210: Problems with Reader mode
github-actions[bot] commented on issue #16210: URL: https://github.com/apache/pulsar/issues/16210#issuecomment-1198811292 The issue had no activity for 30 days, mark with Stale label. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #16244: 2.8->2.10 version，start pulsar show exception：org.apache.pulsar.broker.web.RestException: Cluster already exists
github-actions[bot] commented on issue #16244: URL: https://github.com/apache/pulsar/issues/16244#issuecomment-1198811268 The issue had no activity for 30 days, mark with Stale label. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #16253: 2.10 version run：./pulsar standalone -nss -nfw result exception
github-actions[bot] commented on issue #16253: URL: https://github.com/apache/pulsar/issues/16253#issuecomment-1198811246 The issue had no activity for 30 days, mark with Stale label. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] pengxiangrui127 commented on pull request #16835: Forget to update memory usage when invalid message
pengxiangrui127 commented on PR #16835: URL: https://github.com/apache/pulsar/pull/16835#issuecomment-1198808959 来信已收到，我会尽快查阅，祝你生活愉快。 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on pull request #16835: Forget to update memory usage when invalid message
mattisonchao commented on PR #16835: URL: https://github.com/apache/pulsar/pull/16835#issuecomment-1198808785 @gaozhangmin Please take the final look. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on pull request #16202: [feature][broker] PIP 37: Support chunking with Shared subscription
codelipenghui commented on PR #16202: URL: https://github.com/apache/pulsar/pull/16202#issuecomment-1198808403 @BewareMyPower Please help resolve the conflicts. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on pull request #16841: [fix][broker] ManagedCursor: mark delete no callback when create meta-ledger fail
poorbarcode commented on PR #16841: URL: https://github.com/apache/pulsar/pull/16841#issuecomment-1198807469 Hi @keyboardbobo > If you have any ideas, please remind me Thank you for the information，I have open a issue #16859, We can discuss there. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on pull request #16834: [fix][proxy] Fix client service url
github-actions[bot] commented on PR #16834: URL: https://github.com/apache/pulsar/pull/16834#issuecomment-1198807020 @nodece Please provide a correct documentation label for your PR. Instructions see [Pulsar Documentation Label Guide](https://docs.google.com/document/d/1Qw7LHQdXWBW9t2-r-A7QdFDBwmZh6ytB4guwMoXHqc0). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on issue #16859: [broker] Cursor status has always been SwitchingLedger and pendingMarkDeleteOps has accumulated tens of thousands of requests
poorbarcode commented on issue #16859: URL: https://github.com/apache/pulsar/issues/16859#issuecomment-1198806994 Hi @keyboardbobo Could you please provide the version used? If possible, you can upload the dump file (which may cause data leakage of the company). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] poorbarcode opened a new issue, #16859: [broker] Cursor status has always been SwitchingLedger and pendingMarkDeleteOps has accumulated tens of thousands of requests
poorbarcode opened a new issue, #16859: URL: https://github.com/apache/pulsar/issues/16859 **Describe the bug** The pendingMarkDeleteOps has accumulated tens of thousands of requests, and the cursor status has always been CreatingLedger, and the retention cannot be executed, resulting in the situation that the disk space cannot be reclaimed. **Screenshots**  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] HQebupt commented on pull request #16814: [improve][broker] part 1:make some methods async in Namespaces
HQebupt commented on PR #16814: URL: https://github.com/apache/pulsar/pull/16814#issuecomment-1198804767 /pulsarbot run-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on pull request #16841: [fix][broker] ManagedCursor: mark delete no callback when create meta-ledger fail
poorbarcode commented on PR #16841: URL: https://github.com/apache/pulsar/pull/16841#issuecomment-1198803358 /pulsarbot rerun-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on pull request #16409: add config maxUnloadBundleNumPerShedding for UniformLoadShedder
lordcheng10 commented on PR #16409: URL: https://github.com/apache/pulsar/pull/16409#issuecomment-1198803127 /pulsarbot run-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on pull request #16391: Add dynamic configuration for UniformLoadShedder
lordcheng10 commented on PR #16391: URL: https://github.com/apache/pulsar/pull/16391#issuecomment-1198803005 /pulsarbot run-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on a diff in pull request #16130: [improve][pulsar-client-tools][ISSUE-16082] - Updated set retention time description and added test cases including default ti
Anonymitaet commented on code in PR #16130:
URL: https://github.com/apache/pulsar/pull/16130#discussion_r932802359
##
pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdTopicPolicies.java:
##
@@ -466,11 +466,16 @@ private class SetRetention extends CliCommand {
private List params;
@Parameter(names = { "--time",
-"-t" }, description = "Retention time in minutes (or minutes,
hours,days,weeks eg: 100m, 3h, 2d, 5w). "
-+ "0 means no retention and -1 means infinite time retention",
required = true)
+"-t" }, description = "Retention time with optional time unit
suffix "
++ "eg: 100m, 3h, 2d, 5w. "
Review Comment:
```suggestion
+ "For example, 100m, 3h, 2d, 5w. "
```
##
pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdTopicPolicies.java:
##
@@ -466,11 +466,16 @@ private class SetRetention extends CliCommand {
private List params;
@Parameter(names = { "--time",
-"-t" }, description = "Retention time in minutes (or minutes,
hours,days,weeks eg: 100m, 3h, 2d, 5w). "
-+ "0 means no retention and -1 means infinite time retention",
required = true)
+"-t" }, description = "Retention time with optional time unit
suffix "
++ "eg: 100m, 3h, 2d, 5w. "
++ "If no time unit is specified the default is seconds, eg: "
Review Comment:
```suggestion
+ "If the time unit is not specified, the default unit is
second. For example, "
```
##
pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdTopicPolicies.java:
##
@@ -466,11 +466,16 @@ private class SetRetention extends CliCommand {
private List params;
@Parameter(names = { "--time",
-"-t" }, description = "Retention time in minutes (or minutes,
hours,days,weeks eg: 100m, 3h, 2d, 5w). "
-+ "0 means no retention and -1 means infinite time retention",
required = true)
+"-t" }, description = "Retention time with optional time unit
suffix "
++ "eg: 100m, 3h, 2d, 5w. "
++ "If no time unit is specified the default is seconds, eg: "
++ "-t 120 will set retention to 2 minutes. "
Review Comment:
```suggestion
+ "-t 120 sets retention to 2 minutes. "
```
Write in the simple present tense as much as possible if you are covering
facts that were, are, and forever shall be true.
https://docs.google.com/document/d/1lc5j4RtuLIzlEYCBo97AC8-U_3Erzs_lxpkDuseU0n4/edit?pli=1#bookmark=id.e8uqh1awkcnp
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] Shoothzj commented on pull request #16835: Forget to update memory usage when invalid message
Shoothzj commented on PR #16835: URL: https://github.com/apache/pulsar/pull/16835#issuecomment-1198800087 /pulsarbot run-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] keyboardbobo commented on pull request #16841: [fix][broker] ManagedCursor: mark delete no callback when create meta-ledger fail
keyboardbobo commented on PR #16841: URL: https://github.com/apache/pulsar/pull/16841#issuecomment-1198800066  @poorbarcode -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on pull request #16853: [feature][doc] Show language support for pulsar-admin functions flags
Anonymitaet commented on PR #16853: URL: https://github.com/apache/pulsar/pull/16853#issuecomment-1198799686 @urfreespace could you please review this PR from the technical perspective? Thank you! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on pull request #16834: [fix][proxy] Fix client service url
github-actions[bot] commented on PR #16834: URL: https://github.com/apache/pulsar/pull/16834#issuecomment-1198799532 @nodece Please provide a correct documentation label for your PR. Instructions see [Pulsar Documentation Label Guide](https://docs.google.com/document/d/1Qw7LHQdXWBW9t2-r-A7QdFDBwmZh6ytB4guwMoXHqc0). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch master updated: [improve][doc] Quote package name to adapt zsh (#16851)
This is an automated email from the ASF dual-hosted git repository.
liuyu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new ae7b6385825 [improve][doc] Quote package name to adapt zsh (#16851)
ae7b6385825 is described below
commit ae7b6385825f54726ec00f708bb06e3baabefc69
Author: tison
AuthorDate: Fri Jul 29 09:47:24 2022 +0800
[improve][doc] Quote package name to adapt zsh (#16851)
---
site2/docs/client-libraries-python.md | 75 +--
1 file changed, 9 insertions(+), 66 deletions(-)
diff --git a/site2/docs/client-libraries-python.md
b/site2/docs/client-libraries-python.md
index 66c690404db..11aa73d590e 100644
--- a/site2/docs/client-libraries-python.md
+++ b/site2/docs/client-libraries-python.md
@@ -34,16 +34,14 @@ $ pip install pulsar-client==@pulsar:version_number@
If you install the client libraries on Linux to support services like Pulsar
functions or Avro serialization, you can install optional components alongside
the `pulsar-client` library.
```shell
-
# avro serialization
-$ pip install pulsar-client[avro]=='@pulsar:version_number@'
+$ pip install 'pulsar-client[avro]==@pulsar:version_number@'
# functions runtime
-$ pip install pulsar-client[functions]=='@pulsar:version_number@'
+$ pip install 'pulsar-client[functions]==@pulsar:version_number@'
# all optional components
-$ pip install pulsar-client[all]=='@pulsar:version_number@'
-
+$ pip install 'pulsar-client[all]==@pulsar:version_number@'
```
Installation via PyPi is available for the following Python versions:
@@ -61,11 +59,9 @@ To install the `pulsar-client` library by building from
source, follow [instruct
To install the built Python bindings:
```shell
-
$ git clone https://github.com/apache/pulsar
$ cd pulsar/pulsar-client-cpp/python
$ sudo python setup.py install
-
```
## API Reference
@@ -81,7 +77,6 @@ You can find a variety of Python code examples for the
`pulsar-client` library.
The following example creates a Python producer for the `my-topic` topic and
sends 10 messages on that topic:
```python
-
import pulsar
client = pulsar.Client('pulsar://localhost:6650')
@@ -92,7 +87,6 @@ for i in range(10):
producer.send(('Hello-%d' % i).encode('utf-8'))
client.close()
-
```
### Consumer example
@@ -100,7 +94,6 @@ client.close()
The following example creates a consumer with the `my-subscription`
subscription name on the `my-topic` topic, receives incoming messages, prints
the content and ID of messages that arrive, and acknowledges each message to
the Pulsar broker.
```python
-
import pulsar
client = pulsar.Client('pulsar://localhost:6650')
@@ -118,13 +111,11 @@ while True:
consumer.negative_acknowledge(msg)
client.close()
-
```
This example shows how to configure negative acknowledgement.
```python
-
from pulsar import Client, schema
client = Client('pulsar://localhost:6650')
consumer =
client.subscribe('negative_acks','test',schema=schema.StringSchema())
@@ -147,7 +138,6 @@ try:
except:
print("no more msg")
pass
-
```
### Reader interface example
@@ -155,7 +145,6 @@ except:
You can use the Pulsar Python API to use the Pulsar [reader
interface](concepts-clients.md#reader-interface). Here's an example:
```python
-
# MessageId taken from a previously fetched message
msg_id = msg.message_id()
@@ -165,7 +154,6 @@ while True:
msg = reader.read_next()
print("Received message '{}' id='{}'".format(msg.data(), msg.message_id()))
# No acknowledgment
-
```
### Multi-topic subscriptions
@@ -175,7 +163,6 @@ In addition to subscribing a consumer to a single Pulsar
topic, you can also sub
The following is an example:
```python
-
import re
consumer = client.subscribe(re.compile('persistent://public/default/topic-*'),
'my-subscription')
while True:
@@ -188,7 +175,6 @@ while True:
# Message failed to be processed
consumer.negative_acknowledge(msg)
client.close()
-
```
### Create a Python client with multiple advertised listeners
@@ -197,11 +183,9 @@ To ensure clients in both internal and external networks
can connect to a Pulsar
The following example creates a Python client using multiple advertised
listeners:
```python
-
import pulsar
client = pulsar.Client('pulsar://localhost:6650', listener_name='external')
-
```
## Schema
@@ -254,19 +238,16 @@ When adding a field, you can use these parameters in the
constructor.
# Simple definition
```python
-
class Example(Record):
a = String()
b = Integer()
c = Array(String())
i = Map(String())
-
```
# Using enums
```python
-
from enum import Enum
class Color(Enum):
@@ -277,13 +258,11 @@ class Color(Enum):
class Example(Record):
name = String()
color = Color
-
```
# Complex types
```python
-
class MySubRecord(Record):
[GitHub] [pulsar] Anonymitaet merged pull request #16851: [improve][doc] Quote package name to adapt zsh
Anonymitaet merged PR #16851: URL: https://github.com/apache/pulsar/pull/16851 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] Technoboy- commented on pull request #16715: [feature][client] PIP-184: Topic specific consumer priorityLevel
Technoboy- commented on PR #16715: URL: https://github.com/apache/pulsar/pull/16715#issuecomment-1198796789 There seems a failure test related to this change, could you help check it ? https://github.com/apache/pulsar/runs/7571444716?check_suite_focus=true -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] Technoboy- opened a new issue, #16858: Flaky-test: LedgerOffloaderMetricsTest.cleanup
Technoboy- opened a new issue, #16858: URL: https://github.com/apache/pulsar/issues/16858 LedgerOffloaderMetricsTest.cleanup is flaky. It fails sporadically. ### Failure CI https://github.com/apache/pulsar/runs/7562071273?check_suite_focus=true ### StackTrace ``` Error: Tests run: 6, Failures: 2, Errors: 0, Skipped: 3, Time elapsed: 10.064 s <<< FAILURE! - in org.apache.pulsar.broker.stats.LedgerOffloaderMetricsTest Error: cleanup(org.apache.pulsar.broker.stats.LedgerOffloaderMetricsTest) Time elapsed: 3.515 s <<< FAILURE! org.mockito.exceptions.misusing.WrongTypeOfReturnValue: BrokerService cannot be returned by getBrokerServiceUrl() getBrokerServiceUrl() should return String *** If you're unsure why you're getting above error read on. Due to the nature of the syntax above problem might occur because: 1. This exception *might* occur in wrongly written multi-threaded tests. Please refer to Mockito FAQ on limitations of concurrency testing. 2. A spy is stubbed using when(spy.foo()).then() syntax. It is safer to stub spies - - with doReturn|Throw() family of methods. More in javadocs for Mockito.spy() method. at org.apache.pulsar.broker.PulsarService.getBrokerServiceUrl(PulsarService.java:227) at org.apache.pulsar.broker.auth.MockedPulsarServiceBaseTest.stopBroker(MockedPulsarServiceBaseTest.java:296) at org.apache.pulsar.broker.auth.MockedPulsarServiceBaseTest.internalCleanup(MockedPulsarServiceBaseTest.java:239) at org.apache.pulsar.broker.stats.LedgerOffloaderMetricsTest.cleanup(LedgerOffloaderMetricsTest.java:57) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch master updated: Fix RetentionPolicies comment typo (#16823)
This is an automated email from the ASF dual-hosted git repository. technoboy pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/master by this push: new f27d8253b17 Fix RetentionPolicies comment typo (#16823) f27d8253b17 is described below commit f27d8253b17d4859a29921378faf819dbbe5ea5d Author: Xiaoyu Hou AuthorDate: Fri Jul 29 09:20:31 2022 +0800 Fix RetentionPolicies comment typo (#16823) --- .../java/org/apache/pulsar/common/policies/data/RetentionPolicies.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pulsar-client-admin-api/src/main/java/org/apache/pulsar/common/policies/data/RetentionPolicies.java b/pulsar-client-admin-api/src/main/java/org/apache/pulsar/common/policies/data/RetentionPolicies.java index 4049708e16a..2010ce0bf1e 100644 --- a/pulsar-client-admin-api/src/main/java/org/apache/pulsar/common/policies/data/RetentionPolicies.java +++ b/pulsar-client-admin-api/src/main/java/org/apache/pulsar/common/policies/data/RetentionPolicies.java @@ -22,7 +22,7 @@ package org.apache.pulsar.common.policies.data; * Definition of the retention policy. * * When you set a retention policy you must set **both** a *size limit* and a *time limit*. - * In the case where you don't want to limit by either time or set, the value must be set to `-1`. + * In the case where you don't want to limit by either time or size, the value must be set to `-1`. * Retention policy will be effectively disabled and it won't prevent the deletion of acknowledged * messages when either size or time limit is set to `0`. * Infinite retention can be achieved by setting both time and size limits to `-1`.
[GitHub] [pulsar] Technoboy- merged pull request #16823: [fix][broker]Fix RetentionPolicies comment typo
Technoboy- merged PR #16823: URL: https://github.com/apache/pulsar/pull/16823 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
