[GitHub] [pulsar] lordcheng10 commented on pull request #17151: [fix][broker] Added check for invisible characters for subscription name
lordcheng10 commented on PR #17151: URL: https://github.com/apache/pulsar/pull/17151#issuecomment-1250597999 > I think the change makes sense. Is there a reason we don't do this validation for subscriptions created via auto subscription creation in the Pulsar Protocol? Fixed. Add this validation when automatically creating a subscription. @michaeljmarshall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] gaoran10 commented on a diff in pull request #17704: [fix][metrics]wrong metrics text generated when label_cluster specified
gaoran10 commented on code in PR #17704:
URL: https://github.com/apache/pulsar/pull/17704#discussion_r973878222
##
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricsGeneratorUtils.java:
##
@@ -67,14 +68,21 @@ public static void
generateSystemMetrics(SimpleTextOutputStream stream, String c
stream.write(sample.name);
if (!sample.labelNames.contains("cluster")) {
Review Comment:
Maybe we can add this `stream.write("{");` before the check `if
(!sample.labelNames.contains("cluster"))`. The symbol `{` is necessary and it's
easier to understand.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] gaoran10 commented on a diff in pull request #17704: [fix][metrics]wrong metrics text generated when label_cluster specified
gaoran10 commented on code in PR #17704:
URL: https://github.com/apache/pulsar/pull/17704#discussion_r973878222
##
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricsGeneratorUtils.java:
##
@@ -67,14 +68,21 @@ public static void
generateSystemMetrics(SimpleTextOutputStream stream, String c
stream.write(sample.name);
if (!sample.labelNames.contains("cluster")) {
Review Comment:
Maybe we can add this `stream.write("{");` before the check `if
(!sample.labelNames.contains("cluster")) {`. The symbol `{` is necessary and
it's easier to understand.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] gaoran10 commented on a diff in pull request #17704: [fix][metrics]wrong metrics text generated when label_cluster specified
gaoran10 commented on code in PR #17704:
URL: https://github.com/apache/pulsar/pull/17704#discussion_r973878222
##
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricsGeneratorUtils.java:
##
@@ -67,14 +68,21 @@ public static void
generateSystemMetrics(SimpleTextOutputStream stream, String c
stream.write(sample.name);
if (!sample.labelNames.contains("cluster")) {
Review Comment:
Maybe we can add this `stream.write("{");` before the check `if
(!sample.labelNames.contains("cluster")) {`. The char `{` is necessary and it's
easier to understand.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] gaoran10 commented on a diff in pull request #17704: [fix][metrics]wrong metrics text generated when label_cluster specified
gaoran10 commented on code in PR #17704:
URL: https://github.com/apache/pulsar/pull/17704#discussion_r973878222
##
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricsGeneratorUtils.java:
##
@@ -67,14 +68,21 @@ public static void
generateSystemMetrics(SimpleTextOutputStream stream, String c
stream.write(sample.name);
if (!sample.labelNames.contains("cluster")) {
Review Comment:
Maybe we can add this `stream.write("{");` before this check `if
(!sample.labelNames.contains("cluster")) {`. The char `{` is necessary and it's
easier to understand.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] gaoran10 commented on a diff in pull request #17704: [fix][metrics]wrong metrics text generated when label_cluster specified
gaoran10 commented on code in PR #17704:
URL: https://github.com/apache/pulsar/pull/17704#discussion_r973878222
##
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricsGeneratorUtils.java:
##
@@ -67,14 +68,21 @@ public static void
generateSystemMetrics(SimpleTextOutputStream stream, String c
stream.write(sample.name);
if (!sample.labelNames.contains("cluster")) {
Review Comment:
Maybe we can add this `stream.write("{");` before this check. The char `{`
is necessary and it's easier to understand.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] coderzc commented on pull request #17712: [feat][broker][PIP-195] Support internal cursor properties - part4
coderzc commented on PR #17712: URL: https://github.com/apache/pulsar/pull/17712#issuecomment-1250587002 @lhotari @eolivelli @Jason918 @codelipenghui @mattisonchao PTAL. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] coderzc commented on pull request #17164: [improve][broker] Make cursor properties support modify single value concurrently.
coderzc commented on PR #17164: URL: https://github.com/apache/pulsar/pull/17164#issuecomment-1250586880 @lhotari @eolivelli @Jason918 @codelipenghui @mattisonchao PTAL. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] 01/03: [fix][broker] Fix namespace backlog quota check with retention. (#17706)
This is an automated email from the ASF dual-hosted git repository.
technoboy pushed a commit to branch branch-2.11
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 8b2694397ca8a26663c726402e4fa1ee01a076a0
Author: JiangHaiting
AuthorDate: Mon Sep 19 10:29:51 2022 +0800
[fix][broker] Fix namespace backlog quota check with retention. (#17706)
---
.../apache/pulsar/broker/admin/AdminResource.java | 10 +--
.../apache/pulsar/broker/admin/AdminApiTest.java | 32 ++
2 files changed, 39 insertions(+), 3 deletions(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/AdminResource.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/AdminResource.java
index 432611a69c0..c9078c63de7 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/AdminResource.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/AdminResource.java
@@ -355,17 +355,21 @@ public abstract class AdminResource extends
PulsarWebResource {
}
protected boolean checkBacklogQuota(BacklogQuota quota, RetentionPolicies
retention) {
-if (retention == null || retention.getRetentionSizeInMB() <= 0 ||
retention.getRetentionTimeInMinutes() <= 0) {
+if (retention == null
+|| (retention.getRetentionSizeInMB() <= 0 &&
retention.getRetentionTimeInMinutes() <= 0)) {
return true;
}
if (quota == null) {
quota =
pulsar().getBrokerService().getBacklogQuotaManager().getDefaultQuota();
}
-if (quota.getLimitSize() >= (retention.getRetentionSizeInMB() * 1024 *
1024)) {
+
+if (retention.getRetentionSizeInMB() > 0
+&& quota.getLimitSize() >= (retention.getRetentionSizeInMB() *
1024 * 1024)) {
return false;
}
// time based quota is in second
-if (quota.getLimitTime() >= (retention.getRetentionTimeInMinutes() *
60)) {
+if (retention.getRetentionTimeInMinutes() > 0
+&& quota.getLimitTime() >=
retention.getRetentionTimeInMinutes() * 60) {
return false;
}
return true;
diff --git
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/AdminApiTest.java
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/AdminApiTest.java
index 2ec0d76551a..7993750583f 100644
---
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/AdminApiTest.java
+++
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/AdminApiTest.java
@@ -3397,4 +3397,36 @@ public class AdminApiTest extends
MockedPulsarServiceBaseTest {
long value2 = partitionedStats.getEarliestMsgPublishTimeInBacklogs();
Assert.assertNotEquals(value2, 0);
}
+
+@Test
+public void testRetentionAndBacklogQuotaCheck() throws
PulsarAdminException {
+String namespace = "prop-xyz/ns1";
+//test size check.
+admin.namespaces().setRetention(namespace, new RetentionPolicies(-1,
10));
+admin.namespaces().setBacklogQuota(namespace,
BacklogQuota.builder().limitSize(9 * 1024 * 1024).build());
+
Assert.expectThrows(PulsarAdminException.PreconditionFailedException.class, ()
-> {
+admin.namespaces().setBacklogQuota(namespace,
BacklogQuota.builder().limitSize(100 * 1024 * 1024).build());
+});
+
+//test time check
+admin.namespaces().setRetention(namespace, new RetentionPolicies(10,
-1));
+admin.namespaces().setBacklogQuota(namespace,
BacklogQuota.builder().limitTime(9 * 60).build());
+
Assert.expectThrows(PulsarAdminException.PreconditionFailedException.class, ()
-> {
+admin.namespaces().setBacklogQuota(namespace,
BacklogQuota.builder().limitTime(11 * 60).build());
+});
+
+// test both size and time.
+admin.namespaces().setRetention(namespace, new RetentionPolicies(10,
10));
+admin.namespaces().setBacklogQuota(namespace,
BacklogQuota.builder().limitSize(9 * 1024 * 1024).build());
+admin.namespaces().setBacklogQuota(namespace,
BacklogQuota.builder().limitTime(9 * 60).build());
+admin.namespaces().setBacklogQuota(namespace,
BacklogQuota.builder().limitSize(9 * 1024 * 1024).
+limitTime(9 * 60).build());
+
Assert.expectThrows(PulsarAdminException.PreconditionFailedException.class, ()
-> {
+admin.namespaces().setBacklogQuota(namespace,
BacklogQuota.builder().limitSize(100 * 1024 * 1024).build());
+});
+
Assert.expectThrows(PulsarAdminException.PreconditionFailedException.class, ()
-> {
+admin.namespaces().setBacklogQuota(namespace,
BacklogQuota.builder().limitTime(100 * 60).build());
+});
+
+}
}
[pulsar] 03/03: [fix][common] Fix parsing partitionedKey with Base64 encode issue. (#17687)
This is an automated email from the ASF dual-hosted git repository.
technoboy pushed a commit to branch branch-2.11
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit e802b586eb6c7cf8e2a6842aff2a5def5f70749a
Author: Jiwei Guo
AuthorDate: Fri Sep 16 23:56:03 2022 +0800
[fix][common] Fix parsing partitionedKey with Base64 encode issue. (#17687)
* Fix parsing partitionedKey with Base64 encode issue.
* release the buf
* fix checkstyle issue.
---
.../apache/pulsar/common/protocol/Commands.java| 4 +++
.../pulsar/common/compression/CommandsTest.java| 41 +++---
2 files changed, 40 insertions(+), 5 deletions(-)
diff --git
a/pulsar-common/src/main/java/org/apache/pulsar/common/protocol/Commands.java
b/pulsar-common/src/main/java/org/apache/pulsar/common/protocol/Commands.java
index 62243063741..61ce70a9a0a 100644
---
a/pulsar-common/src/main/java/org/apache/pulsar/common/protocol/Commands.java
+++
b/pulsar-common/src/main/java/org/apache/pulsar/common/protocol/Commands.java
@@ -30,6 +30,7 @@ import io.netty.util.concurrent.FastThreadLocal;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
+import java.util.Base64;
import java.util.Collections;
import java.util.List;
import java.util.Map;
@@ -1888,6 +1889,9 @@ public class Commands {
if (metadata.hasOrderingKey()) {
return metadata.getOrderingKey();
} else if (metadata.hasPartitionKey()) {
+if (metadata.isPartitionKeyB64Encoded()) {
+return
Base64.getDecoder().decode(metadata.getPartitionKey());
+}
return
metadata.getPartitionKey().getBytes(StandardCharsets.UTF_8);
}
} catch (Throwable t) {
diff --git
a/pulsar-common/src/test/java/org/apache/pulsar/common/compression/CommandsTest.java
b/pulsar-common/src/test/java/org/apache/pulsar/common/compression/CommandsTest.java
index 24d34ac547f..207c6202426 100644
---
a/pulsar-common/src/test/java/org/apache/pulsar/common/compression/CommandsTest.java
+++
b/pulsar-common/src/test/java/org/apache/pulsar/common/compression/CommandsTest.java
@@ -18,22 +18,23 @@
*/
package org.apache.pulsar.common.compression;
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static
org.apache.pulsar.common.protocol.Commands.serializeMetadataAndPayload;
import static org.testng.Assert.assertEquals;
import static org.testng.Assert.assertTrue;
-
import com.scurrilous.circe.checksum.Crc32cIntChecksum;
-
import io.netty.buffer.ByteBuf;
import io.netty.buffer.ByteBufUtil;
import io.netty.buffer.Unpooled;
-
import java.io.IOException;
-
+import java.util.Base64;
+import io.netty.util.ReferenceCountUtil;
import org.apache.pulsar.common.allocator.PulsarByteBufAllocator;
import org.apache.pulsar.common.api.proto.MessageMetadata;
import org.apache.pulsar.common.protocol.ByteBufPair;
import org.apache.pulsar.common.protocol.Commands;
import org.apache.pulsar.common.protocol.Commands.ChecksumType;
+import org.testng.Assert;
import org.testng.annotations.Test;
public class CommandsTest {
@@ -93,5 +94,35 @@ public class CommandsTest {
return computedChecksum;
}
-
+@Test
+public void testPeekStickyKey() {
+String message = "msg-1";
+String partitionedKey = "key1";
+MessageMetadata messageMetadata2 = new MessageMetadata()
+.setSequenceId(1)
+.setProducerName("testProducer")
+.setPartitionKey(partitionedKey)
+.setPartitionKeyB64Encoded(false)
+.setPublishTime(System.currentTimeMillis());
+ByteBuf byteBuf =
serializeMetadataAndPayload(Commands.ChecksumType.Crc32c, messageMetadata2,
+Unpooled.copiedBuffer(message.getBytes(UTF_8)));
+byte[] bytes = Commands.peekStickyKey(byteBuf, "topic-1", "sub-1");
+String key = new String(bytes);
+Assert.assertEquals(partitionedKey, key);
+ReferenceCountUtil.safeRelease(byteBuf);
+// test 64 encoded
+String partitionedKey2 =
Base64.getEncoder().encodeToString("key2".getBytes(UTF_8));
+MessageMetadata messageMetadata = new MessageMetadata()
+.setSequenceId(1)
+.setProducerName("testProducer")
+.setPartitionKey(partitionedKey2)
+.setPartitionKeyB64Encoded(true)
+.setPublishTime(System.currentTimeMillis());
+ByteBuf byteBuf2 =
serializeMetadataAndPayload(Commands.ChecksumType.Crc32c, messageMetadata,
+Unpooled.copiedBuffer(message.getBytes(UTF_8)));
+byte[] bytes2 = Commands.peekStickyKey(byteBuf2, "topic-2", "sub-2");
+String key2 = Base64.getEncoder().encodeToString(bytes2);;
+Assert.assertEquals(partitionedKey2, key2);
+ReferenceCountUtil.safeRelease(byteBuf2);
+}
}
[pulsar] 02/03: [fix][metadata] Set revalidateAfterReconnection true for certain failures (#17664)
This is an automated email from the ASF dual-hosted git repository.
technoboy pushed a commit to branch branch-2.11
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 30d7fce0802b20c354481a923a3b900586885b24
Author: Michael Marshall
AuthorDate: Sun Sep 18 06:27:59 2022 -0700
[fix][metadata] Set revalidateAfterReconnection true for certain failures
(#17664)
---
.../org/apache/pulsar/metadata/coordination/impl/ResourceLockImpl.java | 1 +
1 file changed, 1 insertion(+)
diff --git
a/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/coordination/impl/ResourceLockImpl.java
b/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/coordination/impl/ResourceLockImpl.java
index bc3f47d41dc..22eaccc278b 100644
---
a/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/coordination/impl/ResourceLockImpl.java
+++
b/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/coordination/impl/ResourceLockImpl.java
@@ -196,6 +196,7 @@ public class ResourceLockImpl implements ResourceLock
{
// We failed to revalidate the lock due to
connectivity issue
// Continue assuming we hold the lock, until we
can revalidate it, either
// on Reconnected or SessionReestablished events.
+revalidateAfterReconnection = true;
log.warn("Failed to revalidate the lock at {}.
Retrying later on reconnection {}", path,
ex.getCause().getMessage());
}
[pulsar] branch branch-2.11 updated (43ada027134 -> e802b586eb6)
This is an automated email from the ASF dual-hosted git repository. technoboy pushed a change to branch branch-2.11 in repository https://gitbox.apache.org/repos/asf/pulsar.git from 43ada027134 [fix][cli] Fix mbeans to json (#17676) new 8b2694397ca [fix][broker] Fix namespace backlog quota check with retention. (#17706) new 30d7fce0802 [fix][metadata] Set revalidateAfterReconnection true for certain failures (#17664) new e802b586eb6 [fix][common] Fix parsing partitionedKey with Base64 encode issue. (#17687) The 3 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../apache/pulsar/broker/admin/AdminResource.java | 10 -- .../apache/pulsar/broker/admin/AdminApiTest.java | 32 + .../apache/pulsar/common/protocol/Commands.java| 4 +++ .../pulsar/common/compression/CommandsTest.java| 41 +++--- .../coordination/impl/ResourceLockImpl.java| 1 + 5 files changed, 80 insertions(+), 8 deletions(-)
[GitHub] [pulsar] coderzc opened a new pull request, #17712: [feat][broker][PIP-195] Support internal cursor properties - part4
coderzc opened a new pull request, #17712: URL: https://github.com/apache/pulsar/pull/17712 Master Issue: #16763 ### Motivation Now, the cursor properties support modifying a single value concurrently and overall update, but `setCursorProperties` will all key-value is removed before update properties, this will cause internal property loss, so use a special prefix to prevent internal properties. More context see: https://github.com/apache/pulsar/pull/17164#issuecomment-1236872277, https://github.com/apache/pulsar/pull/17164#issuecomment-1241523618, https://github.com/apache/pulsar/pull/17164#issuecomment-1241536546 ### Modifications Use a special prefix to prevent internal properties. ### Verifying this change - [x] Make sure that the change passes the CI checks. *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Documentation - [ ] `doc-required` (Your PR needs to update docs and you will update later) - [x] `doc-not-needed` (Please explain why) - [ ] `doc` (Your PR contains doc changes) - [ ] `doc-complete` (Docs have been already added) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] Jason918 opened a new pull request, #17711: [feat][broker]PIP-180 Shadow Topic - Part V - Support shadow topic creation.
Jason918 opened a new pull request, #17711: URL: https://github.com/apache/pulsar/pull/17711 Master Issue: https://github.com/apache/pulsar/issues/16153 ### Motivation The main purpose of this part is to support shadow topic creation. Note: 1. A topic must defined as shadow topic at creation, or else it will create its own ledgers. So this PR uses topic property(key="PULSAR.SHADOW_SOURCE", value="$SourceTopic") to identify the topic is topic. 2. shadow topic can't change to non-shadow or change its shadow source in its whole life time. You must delete it first and recreate. 3. Deleting an shadow topic is just like deleting normal topic. ### Modifications Key changes: 1. Read topic properties from storage before loading a topic, see `checkOwnershipAndCreatePersistentTopic` 2. Add admin interface and cli for `createShadowTopic` and `getShadowSource` 3. Add `ShadowManagedLedgerImpl`, but it's not implemented yet. It's quite complicated and it will be implemented in the following PRs. ### Verifying this change - [ ] Make sure that the change passes the CI checks. This change added tests and can be verified as follows: - ShadowTopicTest ### Does this pull request potentially affect one of the following parts: *If the box was checked, please highlight the changes* - [ ] Dependencies (add or upgrade a dependency) - [ ] The public API - [ ] The schema - [ ] The default values of configurations - [ ] The binary protocol - [ ] The REST endpoints - [x] **The admin CLI options** - [ ] Anything that affects deployment ### Documentation - [ ] `doc-required` (Your PR needs to update docs and you will update later) - [ ] `doc-not-needed` (Please explain why) - [x] `doc` Java doc added in `pulsar-client-admin-api/src/main/java/org/apache/pulsar/client/admin/Topics.java` - [ ] `doc-complete` (Docs have been already added) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on pull request #17151: [fix][broker] Added check for invisible characters for subscription name
lordcheng10 commented on PR #17151: URL: https://github.com/apache/pulsar/pull/17151#issuecomment-1250538841 > > I think the change makes sense. Is there a reason we don't do this validation for subscriptions created via auto subscription creation in the Pulsar Protocol? > > We should add the validation. It's better to keep consistent with the REST API and Pulsar Protocol. OK, I will add this validation for subscriptions created via auto subscription creation in the Pulsar Protocol -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on a diff in pull request #17700: [fix][metadata] Cleanup state when lock revalidation gets `LockBusyException`
mattisonchao commented on code in PR #17700:
URL: https://github.com/apache/pulsar/pull/17700#discussion_r973835719
##
pulsar-metadata/src/test/java/org/apache/pulsar/metadata/LockManagerTest.java:
##
@@ -293,4 +296,55 @@ public void revalidateLockOnDifferentSession(String
provider, Supplier u
assertEquals(new
String(store1.get(path2).join().get().getValue()), "\"value-1\"");
});
}
+
+@Test(dataProvider = "impl")
+public void testCleanUpStateWhenRevalidationGotLockBusy(String provider,
Supplier urlSupplier)
+throws Exception {
+
+if (provider.equals("Memory") || provider.equals("RocksDB")) {
+// Local memory provider doesn't really have the concept of
multiple sessions
+return;
+}
+
+@Cleanup
+MetadataStoreExtended store1 =
MetadataStoreExtended.create(urlSupplier.get(),
+MetadataStoreConfig.builder().build());
+@Cleanup
+MetadataStoreExtended store2 =
MetadataStoreExtended.create(urlSupplier.get(),
+MetadataStoreConfig.builder().build());
+
+@Cleanup
+CoordinationService cs1 = new CoordinationServiceImpl(store1);
+@Cleanup
+LockManager lm1 = cs1.getLockManager(String.class);
+
+@Cleanup
+CoordinationService cs2 = new CoordinationServiceImpl(store2);
+@Cleanup
+LockManager lm2 = cs2.getLockManager(String.class);
+
+String path1 = newKey();
+
+ResourceLock lock1 = lm1.acquireLock(path1, "value-1").join();
+AtomicReference> lock2 = new AtomicReference<>();
+// lock 2 will steal the distributed lock first.
Review Comment:
When I am writing this test, I found an interesting thing. We can allow the
new lock to steal the existing lock that may hold by others(same value).
I'm not sure if it's a big problem. You can use this test to verify this
behaviour.
Plus, steal lock behaviour may cause an infinity loop when they use the same
value or a different value in the same session. the details please see
`ResourceLockImpl#doRevalidate`
https://github.com/apache/pulsar/blob/69f3f7471fa6faf24d4776d65e0509538c105d37/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/coordination/impl/ResourceLockImpl.java#L239-L310
Case:
- lock 1 holds the lock with value `value-1`.
- lock 2 tries to acquire the lock with value `value-1` got the
`LockBusyException` then invoked `revalidate` to steal the lock. (delete and
re-create it)
- lock 1 receives the delete notification **after lock 2 already acquires
the lock**. then lock1 tries to invoke `revalidate` to steal the lock again.
- Under this assumption, we're going to fall into an infinite loop.
This is a theoretical assumption because the situation is more difficult to
simulate. Please take a look, I'm not sure if I missing something.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar-site] urfreespace merged pull request #208: Add versioning for docs
urfreespace merged PR #208: URL: https://github.com/apache/pulsar-site/pull/208 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch branch-2.11 updated: [fix][cli] Fix mbeans to json (#17676)
This is an automated email from the ASF dual-hosted git repository.
zixuan pushed a commit to branch branch-2.11
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-2.11 by this push:
new 43ada027134 [fix][cli] Fix mbeans to json (#17676)
43ada027134 is described below
commit 43ada027134b8477469765580ef6c8e98596f8dc
Author: Zixuan Liu
AuthorDate: Fri Sep 16 23:56:29 2022 +0800
[fix][cli] Fix mbeans to json (#17676)
Signed-off-by: Zixuan Liu
(cherry picked from commit ae0f86b5dcbe1847b294dc28f23cc4afa73e4dc5)
---
bin/pulsar | 7 ++-
pom.xml| 3 +-
.../apache/pulsar/broker/admin/BrokerStatTest.java | 50 ++
3 files changed, 57 insertions(+), 3 deletions(-)
diff --git a/bin/pulsar b/bin/pulsar
index 0658762c96b..608bdb51ced 100755
--- a/bin/pulsar
+++ b/bin/pulsar
@@ -289,8 +289,11 @@ if [[ -z "$IS_JAVA_8" ]]; then
# https://github.com/netty/netty/issues/12265
OPTS="$OPTS --add-opens java.base/java.nio=ALL-UNNAMED --add-opens
java.base/jdk.internal.misc=ALL-UNNAMED"
# netty.DnsResolverUtil
- # JvmDefaultGCMetricsLogger
- OPTS="$OPTS --add-opens java.base/sun.net=ALL-UNNAMED --add-opens
java.management/sun.management=ALL-UNNAMED"
+ OPTS="$OPTS --add-opens java.base/sun.net=ALL-UNNAMED"
+ # JvmDefaultGCMetricsLogger & MBeanStatsGenerator
+ OPTS="$OPTS --add-opens java.management/sun.management=ALL-UNNAMED"
+ # MBeanStatsGenerator
+ OPTS="$OPTS --add-opens
jdk.management/com.sun.management.internal=ALL-UNNAMED"
fi
OPTS="-cp $PULSAR_CLASSPATH $OPTS"
diff --git a/pom.xml b/pom.xml
index 8f2af596d25..5879a81348f 100644
--- a/pom.xml
+++ b/pom.xml
@@ -98,7 +98,8 @@ flexible messaging model and an intuitive client
API.
--add-opens java.base/java.io=ALL-UNNAMED
--add-opens java.base/java.util=ALL-UNNAMED
--add-opens java.base/sun.net=ALL-UNNAMED
- --add-opens java.management/sun.management=ALL-UNNAMED
+ --add-opens java.management/sun.management=ALL-UNNAMED
+ --add-opens jdk.management/com.sun.management.internal=ALL-UNNAMED
true
4
diff --git
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/BrokerStatTest.java
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/BrokerStatTest.java
new file mode 100644
index 000..58c934c544d
--- /dev/null
+++
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/BrokerStatTest.java
@@ -0,0 +1,50 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.admin;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.auth.MockedPulsarServiceBaseTest;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.common.util.ObjectMapperFactory;
+import org.testng.annotations.AfterMethod;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+@Slf4j
+@Test(groups = "broker-admin")
+public class BrokerStatTest extends MockedPulsarServiceBaseTest {
+@BeforeMethod
+@Override
+public void setup() throws Exception {
+super.internalSetup();
+}
+
+@AfterMethod(alwaysRun = true)
+@Override
+public void cleanup() throws Exception {
+super.internalCleanup();
+}
+
+@Test
+public void testGetMBeans() throws PulsarAdminException,
JsonProcessingException {
+String data = admin.brokerStats().getMBeans();
+ObjectMapperFactory.create().readTree(data);
+}
+}
[GitHub] [pulsar] mattisonchao commented on a diff in pull request #17700: [fix][metadata] Cleanup state when lock revalidation gets `LockBusyException`
mattisonchao commented on code in PR #17700:
URL: https://github.com/apache/pulsar/pull/17700#discussion_r973835719
##
pulsar-metadata/src/test/java/org/apache/pulsar/metadata/LockManagerTest.java:
##
@@ -293,4 +296,55 @@ public void revalidateLockOnDifferentSession(String
provider, Supplier u
assertEquals(new
String(store1.get(path2).join().get().getValue()), "\"value-1\"");
});
}
+
+@Test(dataProvider = "impl")
+public void testCleanUpStateWhenRevalidationGotLockBusy(String provider,
Supplier urlSupplier)
+throws Exception {
+
+if (provider.equals("Memory") || provider.equals("RocksDB")) {
+// Local memory provider doesn't really have the concept of
multiple sessions
+return;
+}
+
+@Cleanup
+MetadataStoreExtended store1 =
MetadataStoreExtended.create(urlSupplier.get(),
+MetadataStoreConfig.builder().build());
+@Cleanup
+MetadataStoreExtended store2 =
MetadataStoreExtended.create(urlSupplier.get(),
+MetadataStoreConfig.builder().build());
+
+@Cleanup
+CoordinationService cs1 = new CoordinationServiceImpl(store1);
+@Cleanup
+LockManager lm1 = cs1.getLockManager(String.class);
+
+@Cleanup
+CoordinationService cs2 = new CoordinationServiceImpl(store2);
+@Cleanup
+LockManager lm2 = cs2.getLockManager(String.class);
+
+String path1 = newKey();
+
+ResourceLock lock1 = lm1.acquireLock(path1, "value-1").join();
+AtomicReference> lock2 = new AtomicReference<>();
+// lock 2 will steal the distributed lock first.
Review Comment:
When I am writing this test, I found an interesting thing. We can allow the
new lock to steal the existing lock that may hold by others(same value).
I'm not sure if it's a big problem. You can use this test to verify this
behaviour.
Plus, steal lock behaviour may cause an infinity loop when they use the same
value or a different value in the same session. the details please see
`ResourceLockImpl#doRevalidate`
https://github.com/apache/pulsar/blob/69f3f7471fa6faf24d4776d65e0509538c105d37/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/coordination/impl/ResourceLockImpl.java#L239-L310
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on a diff in pull request #17700: [fix][metadata] Cleanup state when lock revalidation gets `LockBusyException`
mattisonchao commented on code in PR #17700:
URL: https://github.com/apache/pulsar/pull/17700#discussion_r973835719
##
pulsar-metadata/src/test/java/org/apache/pulsar/metadata/LockManagerTest.java:
##
@@ -293,4 +296,55 @@ public void revalidateLockOnDifferentSession(String
provider, Supplier u
assertEquals(new
String(store1.get(path2).join().get().getValue()), "\"value-1\"");
});
}
+
+@Test(dataProvider = "impl")
+public void testCleanUpStateWhenRevalidationGotLockBusy(String provider,
Supplier urlSupplier)
+throws Exception {
+
+if (provider.equals("Memory") || provider.equals("RocksDB")) {
+// Local memory provider doesn't really have the concept of
multiple sessions
+return;
+}
+
+@Cleanup
+MetadataStoreExtended store1 =
MetadataStoreExtended.create(urlSupplier.get(),
+MetadataStoreConfig.builder().build());
+@Cleanup
+MetadataStoreExtended store2 =
MetadataStoreExtended.create(urlSupplier.get(),
+MetadataStoreConfig.builder().build());
+
+@Cleanup
+CoordinationService cs1 = new CoordinationServiceImpl(store1);
+@Cleanup
+LockManager lm1 = cs1.getLockManager(String.class);
+
+@Cleanup
+CoordinationService cs2 = new CoordinationServiceImpl(store2);
+@Cleanup
+LockManager lm2 = cs2.getLockManager(String.class);
+
+String path1 = newKey();
+
+ResourceLock lock1 = lm1.acquireLock(path1, "value-1").join();
+AtomicReference> lock2 = new AtomicReference<>();
+// lock 2 will steal the distributed lock first.
Review Comment:
When I am writing this test, I found an interesting thing. We can allow the
new lock to steal the existing lock that may hold by others(same value).
I'm not sure if it's a big problem. You can use this test to verify this
behaviour.
Plus, steal lock behaviour may cause an infinity loop when they use the same
value or a different value in the same session. the details please see
`ResourceLockImpl#doRevalidate`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on a diff in pull request #17700: [fix][metadata] Cleanup state when lock revalidation gets `LockBusyException`
mattisonchao commented on code in PR #17700:
URL: https://github.com/apache/pulsar/pull/17700#discussion_r973835719
##
pulsar-metadata/src/test/java/org/apache/pulsar/metadata/LockManagerTest.java:
##
@@ -293,4 +296,55 @@ public void revalidateLockOnDifferentSession(String
provider, Supplier u
assertEquals(new
String(store1.get(path2).join().get().getValue()), "\"value-1\"");
});
}
+
+@Test(dataProvider = "impl")
+public void testCleanUpStateWhenRevalidationGotLockBusy(String provider,
Supplier urlSupplier)
+throws Exception {
+
+if (provider.equals("Memory") || provider.equals("RocksDB")) {
+// Local memory provider doesn't really have the concept of
multiple sessions
+return;
+}
+
+@Cleanup
+MetadataStoreExtended store1 =
MetadataStoreExtended.create(urlSupplier.get(),
+MetadataStoreConfig.builder().build());
+@Cleanup
+MetadataStoreExtended store2 =
MetadataStoreExtended.create(urlSupplier.get(),
+MetadataStoreConfig.builder().build());
+
+@Cleanup
+CoordinationService cs1 = new CoordinationServiceImpl(store1);
+@Cleanup
+LockManager lm1 = cs1.getLockManager(String.class);
+
+@Cleanup
+CoordinationService cs2 = new CoordinationServiceImpl(store2);
+@Cleanup
+LockManager lm2 = cs2.getLockManager(String.class);
+
+String path1 = newKey();
+
+ResourceLock lock1 = lm1.acquireLock(path1, "value-1").join();
+AtomicReference> lock2 = new AtomicReference<>();
+// lock 2 will steal the distributed lock first.
Review Comment:
When I am writing this test, I found an interesting thing. We can allow the
new lock to steal the existing lock that may hold by others(same value).
I'm not sure if it's a big problem. You can use this test to verify this
behaviour.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] nodece commented on pull request #17382: [improve][CI] Generate unit test code coverage reports and upload to Codecov
nodece commented on PR #17382: URL: https://github.com/apache/pulsar/pull/17382#issuecomment-1250525352 Hi @yaalsn, the CI looks broken, we should rerun all jobs. > Error: Failed to execute goal on project pulsar-all-docker-image: Could not resolve dependencies for project org.apache.pulsar:pulsar-all-docker-image:pom:2.11.0-SNAPSHOT: The following artifacts could not be resolved: org.apache.pulsar:pulsar-io-distribution:pom:2.11.0-SNAPSHOT, org.apache.pulsar:pulsar-offloader-distribution:tar.gz:bin:2.11.0-SNAPSHOT: org.apache.pulsar:pulsar-io-distribution:pom:2.11.0-SNAPSHOT was not found in https://repository.apache.org/snapshots during a previous attempt. This failure was cached in the local repository and resolution is not reattempted until the update interval of apache.snapshots has elapsed or updates are forced -> [Help 1] -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on pull request #17318: [fix][client] Messages with inconsistent consumer epochs are not filtered when using batch receive and trigger timeout.
codelipenghui commented on PR #17318: URL: https://github.com/apache/pulsar/pull/17318#issuecomment-1250524811 @congbobo184 Do you remember why the consumer epoch check happened after polling the messages from the receiver queue? @shibd @congbobo184 Can we move it to the beginning of message receiving https://github.com/apache/pulsar/blob/69f3f7471fa6faf24d4776d65e0509538c105d37/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java#L1271-L1285 It looks like if the message with invalid epoch, we don't need to add it to the receiver queue. And we will also introduce client-side stats issues, because users haven't seen that message, but from stats they see the received messages are increased. https://github.com/apache/pulsar/blob/69f3f7471fa6faf24d4776d65e0509538c105d37/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java#L1628-L1644 The messages with invalid epoch should be the skipped messages like here https://github.com/apache/pulsar/blob/69f3f7471fa6faf24d4776d65e0509538c105d37/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java#L1559 It also makes sense to add skipped messages in `ConsumerStats` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on a diff in pull request #17700: [fix][metadata] Cleanup state when lock revalidation gets `LockBusyException`
mattisonchao commented on code in PR #17700:
URL: https://github.com/apache/pulsar/pull/17700#discussion_r973835719
##
pulsar-metadata/src/test/java/org/apache/pulsar/metadata/LockManagerTest.java:
##
@@ -293,4 +296,55 @@ public void revalidateLockOnDifferentSession(String
provider, Supplier u
assertEquals(new
String(store1.get(path2).join().get().getValue()), "\"value-1\"");
});
}
+
+@Test(dataProvider = "impl")
+public void testCleanUpStateWhenRevalidationGotLockBusy(String provider,
Supplier urlSupplier)
+throws Exception {
+
+if (provider.equals("Memory") || provider.equals("RocksDB")) {
+// Local memory provider doesn't really have the concept of
multiple sessions
+return;
+}
+
+@Cleanup
+MetadataStoreExtended store1 =
MetadataStoreExtended.create(urlSupplier.get(),
+MetadataStoreConfig.builder().build());
+@Cleanup
+MetadataStoreExtended store2 =
MetadataStoreExtended.create(urlSupplier.get(),
+MetadataStoreConfig.builder().build());
+
+@Cleanup
+CoordinationService cs1 = new CoordinationServiceImpl(store1);
+@Cleanup
+LockManager lm1 = cs1.getLockManager(String.class);
+
+@Cleanup
+CoordinationService cs2 = new CoordinationServiceImpl(store2);
+@Cleanup
+LockManager lm2 = cs2.getLockManager(String.class);
+
+String path1 = newKey();
+
+ResourceLock lock1 = lm1.acquireLock(path1, "value-1").join();
+AtomicReference> lock2 = new AtomicReference<>();
+// lock 2 will steal the distributed lock first.
Review Comment:
When I writing this test, I found an interesting thing. we can allow the
new lock to steal the existing lock that may hold by others(same value).
I'm not sure if it's a big problem, I can paste my analysis later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[pulsar-site] branch main updated: fix rest api page crash (#211)
This is an automated email from the ASF dual-hosted git repository.
urfree pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/pulsar-site.git
The following commit(s) were added to refs/heads/main by this push:
new 07a20f724be fix rest api page crash (#211)
07a20f724be is described below
commit 07a20f724be7408aab12d1194951f7c543a390f7
Author: Li Li
AuthorDate: Mon Sep 19 11:10:16 2022 +0800
fix rest api page crash (#211)
---
site2/website-next/package.json | 2 +-
site2/website-next/src/components/CLI.js | 2 +-
site2/website-next/src/components/RestApi.js | 2 +-
site2/website-next/src/utils/index.js| 9 +
4 files changed, 8 insertions(+), 7 deletions(-)
diff --git a/site2/website-next/package.json b/site2/website-next/package.json
index 0f7fb9942cd..544cf0c01ca 100644
--- a/site2/website-next/package.json
+++ b/site2/website-next/package.json
@@ -9,7 +9,7 @@
"build": "NODE_OPTIONS=--max_old_space_size=16000 docusaurus build",
"build:all:split": "BUILD_ALL_VERSION=1 ./scripts/split-version-build.sh",
"sync:docs": "./scripts/sync-docs.sh",
-"preview": "npm run build && sh scripts/docker.sh",
+"preview": "docusaurus build --locale en && docker-compose up",
"swizzle": "docusaurus swizzle",
"deploy": "docusaurus deploy",
"clear": "docusaurus clear",
diff --git a/site2/website-next/src/components/CLI.js
b/site2/website-next/src/components/CLI.js
index 1042dd9feb0..5d568c92525 100644
--- a/site2/website-next/src/components/CLI.js
+++ b/site2/website-next/src/components/CLI.js
@@ -1,7 +1,7 @@
const React = require("react");
import Layout from "@theme/Layout";
const versionList = require("../../versions.json");
-import { setVersion, getVersion, getApiVersion } from "../utils/index.js";
+import { getVersion } from "../utils/index.js";
function parseVersion() {}
diff --git a/site2/website-next/src/components/RestApi.js
b/site2/website-next/src/components/RestApi.js
index 581c884c53f..ad648344f87 100644
--- a/site2/website-next/src/components/RestApi.js
+++ b/site2/website-next/src/components/RestApi.js
@@ -1,6 +1,6 @@
const React = require("react");
import Layout from "@theme/Layout";
-import { setVersion, getVersion, getApiVersion } from "../utils/index.js";
+import { getVersion, getApiVersion } from "../utils/index.js";
function parseVersion(pathName) {
let swagger = "swagger";
diff --git a/site2/website-next/src/utils/index.js
b/site2/website-next/src/utils/index.js
index 4ef5b937d03..7232243e5da 100644
--- a/site2/website-next/src/utils/index.js
+++ b/site2/website-next/src/utils/index.js
@@ -54,10 +54,10 @@ export function setVersion(version) {
}
export function getVersion() {
- if (!getCache()) {
-return latestStableVersion;
+ if (/version=(\d+\.?\x?)+/.test(location.href)) {
+return location.href.match(/version=(\d+\.?\x?)+/)[0];
}
- return getCache().getItem("version") || latestStableVersion;
+ return "master";
}
export function getApiVersion(anchor) {
@@ -66,9 +66,10 @@ export function getApiVersion(anchor) {
let _restApiVs = {};
let _vsGroups = {};
for (let [key, val] of Object.entries(restApiVersions)) {
-if (key == 'master' || compareVersions.compare(key, "2.8.0", "<")) {
+if (key == "master" || compareVersions.compare(key, "2.8.0", "<")) {
_restApiVs[key] = val;
} else {
+ _restApiVs[key] = val;
let [one, two] = key.split(".");
let _tKey = one + "." + two + ".x";
_vsGroups[_tKey] = [...(_vsGroups[_tKey] || []), key];
[GitHub] [pulsar-site] urfreespace merged pull request #211: fix rest api page crash
urfreespace merged PR #211: URL: https://github.com/apache/pulsar-site/pull/211 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar-site] branch fix-rest-api-page-crash updated (8a5be86a423 -> c46c49930d0)
This is an automated email from the ASF dual-hosted git repository. urfree pushed a change to branch fix-rest-api-page-crash in repository https://gitbox.apache.org/repos/asf/pulsar-site.git from 8a5be86a423 fix rest api page crash add c46c49930d0 update No new revisions were added by this update. Summary of changes: site2/website-next/src/utils/index.js | 5 - 1 file changed, 5 deletions(-)
[GitHub] [pulsar] mattisonchao commented on pull request #17700: [fix][metadata] Cleanup state when lock revalidation gets `LockBusyException`
mattisonchao commented on PR #17700: URL: https://github.com/apache/pulsar/pull/17700#issuecomment-1250517841 @eolivelli @Jason918 Already add the test. Please retake a look. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar-site] 01/01: fix rest api page crash
This is an automated email from the ASF dual-hosted git repository.
urfree pushed a commit to branch fix-rest-api-page-crash
in repository https://gitbox.apache.org/repos/asf/pulsar-site.git
commit 8a5be86a4231fbbf0b7c9910f2d0b10d975bc315
Author: Li Li
AuthorDate: Mon Sep 19 11:08:38 2022 +0800
fix rest api page crash
Signed-off-by: Li Li
---
site2/website-next/package.json | 2 +-
site2/website-next/src/components/CLI.js | 2 +-
site2/website-next/src/components/RestApi.js | 2 +-
site2/website-next/src/utils/index.js| 14 ++
4 files changed, 13 insertions(+), 7 deletions(-)
diff --git a/site2/website-next/package.json b/site2/website-next/package.json
index 0f7fb9942cd..544cf0c01ca 100644
--- a/site2/website-next/package.json
+++ b/site2/website-next/package.json
@@ -9,7 +9,7 @@
"build": "NODE_OPTIONS=--max_old_space_size=16000 docusaurus build",
"build:all:split": "BUILD_ALL_VERSION=1 ./scripts/split-version-build.sh",
"sync:docs": "./scripts/sync-docs.sh",
-"preview": "npm run build && sh scripts/docker.sh",
+"preview": "docusaurus build --locale en && docker-compose up",
"swizzle": "docusaurus swizzle",
"deploy": "docusaurus deploy",
"clear": "docusaurus clear",
diff --git a/site2/website-next/src/components/CLI.js
b/site2/website-next/src/components/CLI.js
index 1042dd9feb0..5d568c92525 100644
--- a/site2/website-next/src/components/CLI.js
+++ b/site2/website-next/src/components/CLI.js
@@ -1,7 +1,7 @@
const React = require("react");
import Layout from "@theme/Layout";
const versionList = require("../../versions.json");
-import { setVersion, getVersion, getApiVersion } from "../utils/index.js";
+import { getVersion } from "../utils/index.js";
function parseVersion() {}
diff --git a/site2/website-next/src/components/RestApi.js
b/site2/website-next/src/components/RestApi.js
index 581c884c53f..ad648344f87 100644
--- a/site2/website-next/src/components/RestApi.js
+++ b/site2/website-next/src/components/RestApi.js
@@ -1,6 +1,6 @@
const React = require("react");
import Layout from "@theme/Layout";
-import { setVersion, getVersion, getApiVersion } from "../utils/index.js";
+import { getVersion, getApiVersion } from "../utils/index.js";
function parseVersion(pathName) {
let swagger = "swagger";
diff --git a/site2/website-next/src/utils/index.js
b/site2/website-next/src/utils/index.js
index 4ef5b937d03..9d9c9b35fe1 100644
--- a/site2/website-next/src/utils/index.js
+++ b/site2/website-next/src/utils/index.js
@@ -54,10 +54,15 @@ export function setVersion(version) {
}
export function getVersion() {
- if (!getCache()) {
-return latestStableVersion;
+ // if (!getCache()) {
+ // return latestStableVersion;
+ // }
+ // return getCache().getItem("version") || latestStableVersion;
+ //version=2.10.1
+ if (/version=(\d+\.?\x?)+/.test(location.href)) {
+return location.href.match(/version=(\d+\.?\x?)+/)[0];
}
- return getCache().getItem("version") || latestStableVersion;
+ return "master";
}
export function getApiVersion(anchor) {
@@ -66,9 +71,10 @@ export function getApiVersion(anchor) {
let _restApiVs = {};
let _vsGroups = {};
for (let [key, val] of Object.entries(restApiVersions)) {
-if (key == 'master' || compareVersions.compare(key, "2.8.0", "<")) {
+if (key == "master" || compareVersions.compare(key, "2.8.0", "<")) {
_restApiVs[key] = val;
} else {
+ _restApiVs[key] = val;
let [one, two] = key.split(".");
let _tKey = one + "." + two + ".x";
_vsGroups[_tKey] = [...(_vsGroups[_tKey] || []), key];
[pulsar-site] branch fix-rest-api-page-crash created (now 8a5be86a423)
This is an automated email from the ASF dual-hosted git repository. urfree pushed a change to branch fix-rest-api-page-crash in repository https://gitbox.apache.org/repos/asf/pulsar-site.git at 8a5be86a423 fix rest api page crash This branch includes the following new commits: new 8a5be86a423 fix rest api page crash The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[GitHub] [pulsar] mattisonchao commented on a diff in pull request #17700: [fix][metadata] Cleanup state when lock revalidation gets `LockBusyException`
mattisonchao commented on code in PR #17700:
URL: https://github.com/apache/pulsar/pull/17700#discussion_r973831241
##
pulsar-metadata/src/main/java/org/apache/pulsar/metadata/coordination/impl/ResourceLockImpl.java:
##
@@ -184,38 +185,21 @@ synchronized void lockWasInvalidated() {
}
log.info("Lock on resource {} was invalidated", path);
-revalidate(value)
-.thenRun(() -> log.info("Successfully revalidated the lock on
{}", path))
-.exceptionally(ex -> {
-synchronized (ResourceLockImpl.this) {
-if (ex.getCause() instanceof BadVersionException) {
-log.warn("Failed to revalidate the lock at {}.
Marked as expired", path);
-state = State.Released;
-expiredFuture.complete(null);
-} else {
-// We failed to revalidate the lock due to
connectivity issue
-// Continue assuming we hold the lock, until we
can revalidate it, either
-// on Reconnected or SessionReestablished events.
-revalidateAfterReconnection = true;
-log.warn("Failed to revalidate the lock at {}.
Retrying later on reconnection {}", path,
-ex.getCause().getMessage());
-}
-}
-return null;
-});
+revalidate(value, true)
+.thenRun(() -> log.info("Successfully revalidated the lock on
{}", path));
}
synchronized CompletableFuture revalidateIfNeededAfterReconnection()
{
if (revalidateAfterReconnection) {
revalidateAfterReconnection = false;
log.warn("Revalidate lock at {} after reconnection", path);
-return revalidate(value);
+return revalidate(value, true);
} else {
return CompletableFuture.completedFuture(null);
}
}
-synchronized CompletableFuture revalidate(T newValue) {
+synchronized CompletableFuture revalidate(T newValue, boolean
retryWhenConnectionLost) {
Review Comment:
Sure, and fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on pull request #17710: ack message list
github-actions[bot] commented on PR #17710: URL: https://github.com/apache/pulsar/pull/17710#issuecomment-1250498835 @shibd Please provide a correct documentation label for your PR. Instructions see [Pulsar Documentation Label Guide](https://docs.google.com/document/d/1Qw7LHQdXWBW9t2-r-A7QdFDBwmZh6ytB4guwMoXHqc0). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] shibd opened a new pull request, #17710: ack message list
shibd opened a new pull request, #17710: URL: https://github.com/apache/pulsar/pull/17710 Fixes # Master Issue: # ### Motivation ### Modifications ### Verifying this change - [ ] Make sure that the change passes the CI checks. *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Does this pull request potentially affect one of the following parts: *If the box was checked, please highlight the changes* - [ ] Dependencies (add or upgrade a dependency) - [ ] The public API - [ ] The schema - [ ] The default values of configurations - [ ] The binary protocol - [ ] The REST endpoints - [ ] The admin CLI options - [ ] Anything that affects deployment ### Documentation - [ ] `doc-required` (Your PR needs to update docs and you will update later) - [ ] `doc-not-needed` (Please explain why) - [ ] `doc` (Your PR contains doc changes) - [ ] `doc-complete` (Docs have been already added) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on a diff in pull request #17700: [fix][metadata] Cleanup state when lock revalidation gets `LockBusyException`
mattisonchao commented on code in PR #17700:
URL: https://github.com/apache/pulsar/pull/17700#discussion_r973831241
##
pulsar-metadata/src/main/java/org/apache/pulsar/metadata/coordination/impl/ResourceLockImpl.java:
##
@@ -184,38 +185,21 @@ synchronized void lockWasInvalidated() {
}
log.info("Lock on resource {} was invalidated", path);
-revalidate(value)
-.thenRun(() -> log.info("Successfully revalidated the lock on
{}", path))
-.exceptionally(ex -> {
-synchronized (ResourceLockImpl.this) {
-if (ex.getCause() instanceof BadVersionException) {
-log.warn("Failed to revalidate the lock at {}.
Marked as expired", path);
-state = State.Released;
-expiredFuture.complete(null);
-} else {
-// We failed to revalidate the lock due to
connectivity issue
-// Continue assuming we hold the lock, until we
can revalidate it, either
-// on Reconnected or SessionReestablished events.
-revalidateAfterReconnection = true;
-log.warn("Failed to revalidate the lock at {}.
Retrying later on reconnection {}", path,
-ex.getCause().getMessage());
-}
-}
-return null;
-});
+revalidate(value, true)
+.thenRun(() -> log.info("Successfully revalidated the lock on
{}", path));
}
synchronized CompletableFuture revalidateIfNeededAfterReconnection()
{
if (revalidateAfterReconnection) {
revalidateAfterReconnection = false;
log.warn("Revalidate lock at {} after reconnection", path);
-return revalidate(value);
+return revalidate(value, true);
} else {
return CompletableFuture.completedFuture(null);
}
}
-synchronized CompletableFuture revalidate(T newValue) {
+synchronized CompletableFuture revalidate(T newValue, boolean
retryWhenConnectionLost) {
Review Comment:
Ok @Jason918
##
pulsar-metadata/src/main/java/org/apache/pulsar/metadata/coordination/impl/ResourceLockImpl.java:
##
@@ -184,38 +185,21 @@ synchronized void lockWasInvalidated() {
}
log.info("Lock on resource {} was invalidated", path);
-revalidate(value)
-.thenRun(() -> log.info("Successfully revalidated the lock on
{}", path))
-.exceptionally(ex -> {
-synchronized (ResourceLockImpl.this) {
-if (ex.getCause() instanceof BadVersionException) {
-log.warn("Failed to revalidate the lock at {}.
Marked as expired", path);
-state = State.Released;
-expiredFuture.complete(null);
-} else {
-// We failed to revalidate the lock due to
connectivity issue
-// Continue assuming we hold the lock, until we
can revalidate it, either
-// on Reconnected or SessionReestablished events.
-revalidateAfterReconnection = true;
-log.warn("Failed to revalidate the lock at {}.
Retrying later on reconnection {}", path,
-ex.getCause().getMessage());
-}
-}
-return null;
-});
+revalidate(value, true)
+.thenRun(() -> log.info("Successfully revalidated the lock on
{}", path));
}
synchronized CompletableFuture revalidateIfNeededAfterReconnection()
{
if (revalidateAfterReconnection) {
revalidateAfterReconnection = false;
log.warn("Revalidate lock at {} after reconnection", path);
-return revalidate(value);
+return revalidate(value, true);
} else {
return CompletableFuture.completedFuture(null);
}
}
-synchronized CompletableFuture revalidate(T newValue) {
+synchronized CompletableFuture revalidate(T newValue, boolean
retryWhenConnectionLost) {
Review Comment:
Ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on a diff in pull request #17700: [fix][metadata] Cleanup state when lock revalidation gets `LockBusyException`
mattisonchao commented on code in PR #17700:
URL: https://github.com/apache/pulsar/pull/17700#discussion_r973831103
##
pulsar-metadata/src/main/java/org/apache/pulsar/metadata/coordination/impl/ResourceLockImpl.java:
##
@@ -233,6 +217,30 @@ synchronized CompletableFuture revalidate(T
newValue) {
});
revalidateFuture = newFuture;
}
+revalidateFuture.exceptionally(ex -> {
+synchronized (ResourceLockImpl.this) {
+if (!retryWhenConnectionLost) {
Review Comment:
Sure, and fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[pulsar] branch master updated: [fix][metadata] Handle session events in separate thread (#17638)
This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 69f3f7471fa [fix][metadata] Handle session events in separate thread
(#17638)
69f3f7471fa is described below
commit 69f3f7471fa6faf24d4776d65e0509538c105d37
Author: Lari Hotari
AuthorDate: Mon Sep 19 05:40:18 2022 +0300
[fix][metadata] Handle session events in separate thread (#17638)
---
.../pulsar/metadata/impl/AbstractMetadataStore.java | 21 +
1 file changed, 13 insertions(+), 8 deletions(-)
diff --git
a/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/AbstractMetadataStore.java
b/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/AbstractMetadataStore.java
index 4b9ff914fcf..f7e90dc8e60 100644
---
a/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/AbstractMetadataStore.java
+++
b/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/AbstractMetadataStore.java
@@ -409,14 +409,19 @@ public abstract class AbstractMetadataStore implements
MetadataStoreExtended, Co
protected void receivedSessionEvent(SessionEvent event) {
isConnected = event.isConnected();
-
-sessionListeners.forEach(l -> {
-try {
-l.accept(event);
-} catch (Throwable t) {
-log.warn("Error in processing session event", t);
-}
-});
+try {
+executor.execute(() -> {
+sessionListeners.forEach(l -> {
+try {
+l.accept(event);
+} catch (Throwable t) {
+log.warn("Error in processing session event " + event,
t);
+}
+});
+});
+} catch (RejectedExecutionException e) {
+log.warn("Error in processing session event " + event, e);
+}
}
@Override
[GitHub] [pulsar] codelipenghui merged pull request #17638: [fix][metadata] Handle session events in a separate thread
codelipenghui merged PR #17638: URL: https://github.com/apache/pulsar/pull/17638 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #17225: Flaky-test: PersistentSubscriptionMessageDispatchStreamingDispatcherThrottlingTest.testMultiLevelDispatch
codelipenghui commented on issue #17225: URL: https://github.com/apache/pulsar/issues/17225#issuecomment-1250486145 Another failure: https://github.com/apache/pulsar/actions/runs/3050809344/jobs/4918810532 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] Jason918 commented on a diff in pull request #17700: [fix][metadata] Cleanup state when lock revalidation gets `LockBusyException`
Jason918 commented on code in PR #17700:
URL: https://github.com/apache/pulsar/pull/17700#discussion_r973827309
##
pulsar-metadata/src/main/java/org/apache/pulsar/metadata/coordination/impl/ResourceLockImpl.java:
##
@@ -184,38 +185,21 @@ synchronized void lockWasInvalidated() {
}
log.info("Lock on resource {} was invalidated", path);
-revalidate(value)
-.thenRun(() -> log.info("Successfully revalidated the lock on
{}", path))
-.exceptionally(ex -> {
-synchronized (ResourceLockImpl.this) {
-if (ex.getCause() instanceof BadVersionException) {
-log.warn("Failed to revalidate the lock at {}.
Marked as expired", path);
-state = State.Released;
-expiredFuture.complete(null);
-} else {
-// We failed to revalidate the lock due to
connectivity issue
-// Continue assuming we hold the lock, until we
can revalidate it, either
-// on Reconnected or SessionReestablished events.
-revalidateAfterReconnection = true;
-log.warn("Failed to revalidate the lock at {}.
Retrying later on reconnection {}", path,
-ex.getCause().getMessage());
-}
-}
-return null;
-});
+revalidate(value, true)
+.thenRun(() -> log.info("Successfully revalidated the lock on
{}", path));
}
synchronized CompletableFuture revalidateIfNeededAfterReconnection()
{
if (revalidateAfterReconnection) {
revalidateAfterReconnection = false;
log.warn("Revalidate lock at {} after reconnection", path);
-return revalidate(value);
+return revalidate(value, true);
} else {
return CompletableFuture.completedFuture(null);
}
}
-synchronized CompletableFuture revalidate(T newValue) {
+synchronized CompletableFuture revalidate(T newValue, boolean
retryWhenConnectionLost) {
Review Comment:
About the variable name, maybe just reuse `revalidateAfterReconnection`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] Jason918 commented on a diff in pull request #17700: [fix][metadata] Cleanup state when lock revalidation gets `LockBusyException`
Jason918 commented on code in PR #17700:
URL: https://github.com/apache/pulsar/pull/17700#discussion_r973826990
##
pulsar-metadata/src/main/java/org/apache/pulsar/metadata/coordination/impl/ResourceLockImpl.java:
##
@@ -233,6 +217,30 @@ synchronized CompletableFuture revalidate(T
newValue) {
});
revalidateFuture = newFuture;
}
+revalidateFuture.exceptionally(ex -> {
+synchronized (ResourceLockImpl.this) {
+if (!retryWhenConnectionLost) {
Review Comment:
This can just merge into the `if` at L229.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[pulsar] branch master updated: [fix][broker] Fix namespace backlog quota check with retention. (#17706)
This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new c6967cd37d1 [fix][broker] Fix namespace backlog quota check with
retention. (#17706)
c6967cd37d1 is described below
commit c6967cd37d11c01c9cc873233e157c7d1af70c6a
Author: JiangHaiting
AuthorDate: Mon Sep 19 10:29:51 2022 +0800
[fix][broker] Fix namespace backlog quota check with retention. (#17706)
---
.../apache/pulsar/broker/admin/AdminResource.java | 10 +--
.../apache/pulsar/broker/admin/AdminApiTest.java | 32 ++
2 files changed, 39 insertions(+), 3 deletions(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/AdminResource.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/AdminResource.java
index 1b05df826c9..47c69ecc698 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/AdminResource.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/AdminResource.java
@@ -370,17 +370,21 @@ public abstract class AdminResource extends
PulsarWebResource {
}
protected boolean checkBacklogQuota(BacklogQuota quota, RetentionPolicies
retention) {
-if (retention == null || retention.getRetentionSizeInMB() <= 0 ||
retention.getRetentionTimeInMinutes() <= 0) {
+if (retention == null
+|| (retention.getRetentionSizeInMB() <= 0 &&
retention.getRetentionTimeInMinutes() <= 0)) {
return true;
}
if (quota == null) {
quota =
pulsar().getBrokerService().getBacklogQuotaManager().getDefaultQuota();
}
-if (quota.getLimitSize() >= (retention.getRetentionSizeInMB() * 1024 *
1024)) {
+
+if (retention.getRetentionSizeInMB() > 0
+&& quota.getLimitSize() >= (retention.getRetentionSizeInMB() *
1024 * 1024)) {
return false;
}
// time based quota is in second
-if (quota.getLimitTime() >= (retention.getRetentionTimeInMinutes() *
60)) {
+if (retention.getRetentionTimeInMinutes() > 0
+&& quota.getLimitTime() >=
retention.getRetentionTimeInMinutes() * 60) {
return false;
}
return true;
diff --git
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/AdminApiTest.java
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/AdminApiTest.java
index 9e537d0d294..fe7c13057ba 100644
---
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/AdminApiTest.java
+++
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/AdminApiTest.java
@@ -3406,4 +3406,36 @@ public class AdminApiTest extends
MockedPulsarServiceBaseTest {
long value2 = partitionedStats.getEarliestMsgPublishTimeInBacklogs();
Assert.assertNotEquals(value2, 0);
}
+
+@Test
+public void testRetentionAndBacklogQuotaCheck() throws
PulsarAdminException {
+String namespace = "prop-xyz/ns1";
+//test size check.
+admin.namespaces().setRetention(namespace, new RetentionPolicies(-1,
10));
+admin.namespaces().setBacklogQuota(namespace,
BacklogQuota.builder().limitSize(9 * 1024 * 1024).build());
+
Assert.expectThrows(PulsarAdminException.PreconditionFailedException.class, ()
-> {
+admin.namespaces().setBacklogQuota(namespace,
BacklogQuota.builder().limitSize(100 * 1024 * 1024).build());
+});
+
+//test time check
+admin.namespaces().setRetention(namespace, new RetentionPolicies(10,
-1));
+admin.namespaces().setBacklogQuota(namespace,
BacklogQuota.builder().limitTime(9 * 60).build());
+
Assert.expectThrows(PulsarAdminException.PreconditionFailedException.class, ()
-> {
+admin.namespaces().setBacklogQuota(namespace,
BacklogQuota.builder().limitTime(11 * 60).build());
+});
+
+// test both size and time.
+admin.namespaces().setRetention(namespace, new RetentionPolicies(10,
10));
+admin.namespaces().setBacklogQuota(namespace,
BacklogQuota.builder().limitSize(9 * 1024 * 1024).build());
+admin.namespaces().setBacklogQuota(namespace,
BacklogQuota.builder().limitTime(9 * 60).build());
+admin.namespaces().setBacklogQuota(namespace,
BacklogQuota.builder().limitSize(9 * 1024 * 1024).
+limitTime(9 * 60).build());
+
Assert.expectThrows(PulsarAdminException.PreconditionFailedException.class, ()
-> {
+admin.namespaces().setBacklogQuota(namespace,
BacklogQuota.builder().limitSize(100 * 1024 * 1024).build());
+});
+
Assert.expectThrows(PulsarAdminException.PreconditionFailedException.class, ()
-> {
+admin.namespaces().setBacklogQuota(namespace,
BacklogQuota.builder().limitTime(100 * 60).build());
+});
+
+}
}
[GitHub] [pulsar] codelipenghui closed issue #17707: Why pulsar retention quota must exceed configured backlog quota
codelipenghui closed issue #17707: Why pulsar retention quota must exceed configured backlog quota URL: https://github.com/apache/pulsar/issues/17707 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui merged pull request #17706: [fix][broker] Fix namespace backlog quota check with retention.
codelipenghui merged PR #17706: URL: https://github.com/apache/pulsar/pull/17706 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch master updated: [flaky-test]Add information in ManagedLedgerBkTest to determine the problem (#17441)
This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 8d34b0f3836 [flaky-test]Add information in ManagedLedgerBkTest to
determine the problem (#17441)
8d34b0f3836 is described below
commit 8d34b0f38365b71c8c486a7464f2cbd90dcd4472
Author: fengyubiao
AuthorDate: Mon Sep 19 10:24:47 2022 +0800
[flaky-test]Add information in ManagedLedgerBkTest to determine the problem
(#17441)
---
.../mledger/impl/ManagedLedgerBkTest.java | 12 +---
.../mledger/util/ThrowableToStringUtil.java| 36 ++
2 files changed, 43 insertions(+), 5 deletions(-)
diff --git
a/managed-ledger/src/test/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerBkTest.java
b/managed-ledger/src/test/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerBkTest.java
index 9026c0f6ac4..3ad521eeb79 100644
---
a/managed-ledger/src/test/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerBkTest.java
+++
b/managed-ledger/src/test/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerBkTest.java
@@ -19,7 +19,6 @@
package org.apache.bookkeeper.mledger.impl;
import static org.testng.Assert.assertEquals;
-import static org.testng.Assert.assertFalse;
import static org.testng.Assert.assertNotNull;
import static org.testng.Assert.assertTrue;
import static org.testng.Assert.fail;
@@ -50,6 +49,7 @@ import org.apache.bookkeeper.mledger.ManagedLedgerFactory;
import org.apache.bookkeeper.mledger.ManagedLedgerFactoryConfig;
import org.apache.bookkeeper.mledger.Position;
import org.apache.bookkeeper.mledger.impl.cache.EntryCacheManager;
+import org.apache.bookkeeper.mledger.util.ThrowableToStringUtil;
import org.apache.bookkeeper.test.BookKeeperClusterTestCase;
import org.apache.pulsar.common.policies.data.PersistentOfflineTopicStats;
import org.testng.annotations.Test;
@@ -312,7 +312,7 @@ public class ManagedLedgerBkTest extends
BookKeeperClusterTestCase {
}
final CountDownLatch counter = new CountDownLatch(positions.size());
-final AtomicBoolean gotException = new AtomicBoolean(false);
+final AtomicReference gotException = new AtomicReference();
for (Position p : positions) {
cursor.asyncDelete(p, new DeleteCallback() {
@@ -324,8 +324,7 @@ public class ManagedLedgerBkTest extends
BookKeeperClusterTestCase {
@Override
public void deleteFailed(ManagedLedgerException exception,
Object ctx) {
-exception.printStackTrace();
-gotException.set(true);
+gotException.set(exception);
counter.countDown();
}
}, null);
@@ -336,7 +335,10 @@ public class ManagedLedgerBkTest extends
BookKeeperClusterTestCase {
cursor.close();
ledger.close();
-assertFalse(gotException.get());
+// Add information to determine the problem.
+if (gotException.get() != null){
+fail(ThrowableToStringUtil.toString(gotException.get()));
+}
}
/**
diff --git
a/managed-ledger/src/test/java/org/apache/bookkeeper/mledger/util/ThrowableToStringUtil.java
b/managed-ledger/src/test/java/org/apache/bookkeeper/mledger/util/ThrowableToStringUtil.java
new file mode 100644
index 000..d7540ad30ca
--- /dev/null
+++
b/managed-ledger/src/test/java/org/apache/bookkeeper/mledger/util/ThrowableToStringUtil.java
@@ -0,0 +1,36 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.bookkeeper.mledger.util;
+
+import org.apache.pulsar.common.util.FutureUtil;
+
+public class ThrowableToStringUtil {
+
+public static String toString(Throwable throwable){
+Throwable unwrapThrowable =
FutureUtil.unwrapCompletionException(throwable);
+StringBuilder stringBuilder = new StringBuilder();
+stringBuilder.append(unwrapThrowable.getClass().getName())
+.append(": ").append(unwrapThrowable.getMessage());
+for (StackTraceElement stackTraceElement :
[GitHub] [pulsar] codelipenghui merged pull request #17441: [flaky-test]Add information in ManagedLedgerBkTest to determine the problem
codelipenghui merged PR #17441: URL: https://github.com/apache/pulsar/pull/17441 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on pull request #17485: [feat][tiered-storage] Introduce offload throttling
codelipenghui commented on PR #17485: URL: https://github.com/apache/pulsar/pull/17485#issuecomment-1250477448 @tjiuming, Could you please create a proposal for this one? We will introduce a new mechanism, we need to consider how to operate and monitor the new feature. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #17173: [feature request] C++ and Python client support consumer memory limits.
github-actions[bot] commented on issue #17173: URL: https://github.com/apache/pulsar/issues/17173#issuecomment-1250475247 The issue had no activity for 30 days, mark with Stale label. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #17174: [feature request] C++ and Python client consumer add epoch.
github-actions[bot] commented on issue #17174: URL: https://github.com/apache/pulsar/issues/17174#issuecomment-1250475229 The issue had no activity for 30 days, mark with Stale label. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #17139: [feature request] C++ and Python client support Dead Letter Topic
github-actions[bot] commented on issue #17139: URL: https://github.com/apache/pulsar/issues/17139#issuecomment-1250475257 The issue had no activity for 30 days, mark with Stale label. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #17181: [feature request] C++ and Python support negative acknowledgment backoff
github-actions[bot] commented on issue #17181: URL: https://github.com/apache/pulsar/issues/17181#issuecomment-1250475212 The issue had no activity for 30 days, mark with Stale label. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #17186: [feature request] C++ and Python support inclusive seek
github-actions[bot] commented on issue #17186: URL: https://github.com/apache/pulsar/issues/17186#issuecomment-1250475201 The issue had no activity for 30 days, mark with Stale label. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch master updated: [improve][client-c++] support Exclusive Producer access mode for c++ (#17439)
This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 43de9f566d6 [improve][client-c++] support Exclusive Producer access
mode for c++ (#17439)
43de9f566d6 is described below
commit 43de9f566d60de9937bdb6a0b11a8137ee3d6df8
Author: Cong Zhao
AuthorDate: Mon Sep 19 10:16:00 2022 +0800
[improve][client-c++] support Exclusive Producer access mode for c++
(#17439)
---
.../include/pulsar/ProducerConfiguration.h | 26 ++
pulsar-client-cpp/lib/ClientConnection.cc | 5 +
pulsar-client-cpp/lib/ClientConnection.h | 1 +
pulsar-client-cpp/lib/Commands.cc | 7 +-
pulsar-client-cpp/lib/Commands.h | 3 ++-
pulsar-client-cpp/lib/PartitionedProducerImpl.cc | 8 ---
pulsar-client-cpp/lib/ProducerConfiguration.cc | 8 +++
pulsar-client-cpp/lib/ProducerConfigurationImpl.h | 1 +
pulsar-client-cpp/lib/ProducerImpl.cc | 16 +++--
pulsar-client-cpp/lib/ProducerImpl.h | 1 +
pulsar-client-cpp/tests/ProducerTest.cc| 24
11 files changed, 88 insertions(+), 12 deletions(-)
diff --git a/pulsar-client-cpp/include/pulsar/ProducerConfiguration.h
b/pulsar-client-cpp/include/pulsar/ProducerConfiguration.h
index 7c278dd6e91..fb331ea828c 100644
--- a/pulsar-client-cpp/include/pulsar/ProducerConfiguration.h
+++ b/pulsar-client-cpp/include/pulsar/ProducerConfiguration.h
@@ -79,6 +79,18 @@ class PULSAR_PUBLIC ProducerConfiguration {
*/
KeyBasedBatching
};
+enum ProducerAccessMode
+{
+/**
+ * By default multiple producers can publish on a topic.
+ */
+Shared = 0,
+
+/**
+ * Require exclusive access for producer. Fail immediately if there's
already a producer connected.
+ */
+Exclusive = 1
+};
ProducerConfiguration();
~ProducerConfiguration();
@@ -501,6 +513,20 @@ class PULSAR_PUBLIC ProducerConfiguration {
*/
bool isChunkingEnabled() const;
+/**
+ * Set the type of access mode that the producer requires on the topic.
+ *
+ * @see ProducerAccessMode
+ * @param accessMode
+ *The type of access to the topic that the producer requires
+ */
+ProducerConfiguration& setAccessMode(const ProducerAccessMode& accessMode);
+
+/**
+ * Get the type of access mode that the producer requires on the topic.
+ */
+ProducerAccessMode getAccessMode() const;
+
friend class PulsarWrapper;
private:
diff --git a/pulsar-client-cpp/lib/ClientConnection.cc
b/pulsar-client-cpp/lib/ClientConnection.cc
index 162c148a8dc..20e34586e00 100644
--- a/pulsar-client-cpp/lib/ClientConnection.cc
+++ b/pulsar-client-cpp/lib/ClientConnection.cc
@@ -1073,6 +1073,11 @@ void ClientConnection::handleIncomingCommand() {
if (producerSuccess.has_schema_version()) {
data.schemaVersion =
producerSuccess.schema_version();
}
+if (producerSuccess.has_topic_epoch()) {
+data.topicEpoch =
Optional::of(producerSuccess.topic_epoch());
+} else {
+data.topicEpoch = Optional::empty();
+}
requestData.promise.setValue(data);
requestData.timer->cancel();
}
diff --git a/pulsar-client-cpp/lib/ClientConnection.h
b/pulsar-client-cpp/lib/ClientConnection.h
index b615eaab2d0..418a5831397 100644
--- a/pulsar-client-cpp/lib/ClientConnection.h
+++ b/pulsar-client-cpp/lib/ClientConnection.h
@@ -77,6 +77,7 @@ struct ResponseData {
std::string producerName;
int64_t lastSequenceId;
std::string schemaVersion;
+Optional topicEpoch;
};
typedef std::shared_ptr> NamespaceTopicsPtr;
diff --git a/pulsar-client-cpp/lib/Commands.cc
b/pulsar-client-cpp/lib/Commands.cc
index 3a00d302e86..417e6e31a17 100644
--- a/pulsar-client-cpp/lib/Commands.cc
+++ b/pulsar-client-cpp/lib/Commands.cc
@@ -351,7 +351,8 @@ SharedBuffer Commands::newProducer(const std::string&
topic, uint64_t producerId
const std::string& producerName, uint64_t
requestId,
const std::map&
metadata,
const SchemaInfo& schemaInfo, uint64_t
epoch,
- bool userProvidedProducerName, bool
encrypted) {
+ bool userProvidedProducerName, bool
encrypted,
+ ProducerAccessMode accessMode,
Optional topicEpoch) {
BaseCommand cmd;
cmd.set_type(BaseCommand::PRODUCER);
[GitHub] [pulsar] codelipenghui merged pull request #17439: [improve][client-c++] support Exclusive Producer access mode for c++
codelipenghui merged PR #17439: URL: https://github.com/apache/pulsar/pull/17439 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] Jason918 commented on a diff in pull request #17706: [fix][broker] Fix namespace backlog quota check with retention.
Jason918 commented on code in PR #17706:
URL: https://github.com/apache/pulsar/pull/17706#discussion_r973821745
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/AdminResource.java:
##
@@ -370,17 +370,21 @@ protected CompletableFuture>
getTopicPoliciesAsyncWithRe
}
protected boolean checkBacklogQuota(BacklogQuota quota, RetentionPolicies
retention) {
-if (retention == null || retention.getRetentionSizeInMB() <= 0 ||
retention.getRetentionTimeInMinutes() <= 0) {
+if (retention == null
+|| (retention.getRetentionSizeInMB() <= 0 &&
retention.getRetentionTimeInMinutes() <= 0)) {
Review Comment:
And the case of `(retention.getRetentionSizeInMB() <= 0 &&
retention.getRetentionTimeInMinutes() <= 0)` is the default setting, it's
covered by previous unit tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] Jason918 commented on a diff in pull request #17706: [fix][broker] Fix namespace backlog quota check with retention.
Jason918 commented on code in PR #17706:
URL: https://github.com/apache/pulsar/pull/17706#discussion_r973820086
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/AdminResource.java:
##
@@ -370,17 +370,21 @@ protected CompletableFuture>
getTopicPoliciesAsyncWithRe
}
protected boolean checkBacklogQuota(BacklogQuota quota, RetentionPolicies
retention) {
-if (retention == null || retention.getRetentionSizeInMB() <= 0 ||
retention.getRetentionTimeInMinutes() <= 0) {
+if (retention == null
+|| (retention.getRetentionSizeInMB() <= 0 &&
retention.getRetentionTimeInMinutes() <= 0)) {
Review Comment:
@codelipenghui As mentioned in the issue. The test in L3417 won't fail
before this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] yaalsn commented on pull request #17382: [improve][CI] Generate unit test code coverage reports and upload to Codecov
yaalsn commented on PR #17382: URL: https://github.com/apache/pulsar/pull/17382#issuecomment-1250467533 /pulsarbot rerun-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar-site] branch update-docs-banner-texts updated (fae4c5cbac2 -> cc01984341d)
This is an automated email from the ASF dual-hosted git repository. urfree pushed a change to branch update-docs-banner-texts in repository https://gitbox.apache.org/repos/asf/pulsar-site.git from fae4c5cbac2 update add cc01984341d update No new revisions were added by this update. Summary of changes: site2/website-next/src/css/custom.css | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
[GitHub] [pulsar] codelipenghui commented on a diff in pull request #17706: [fix][broker] Fix namespace backlog quota check with retention.
codelipenghui commented on code in PR #17706:
URL: https://github.com/apache/pulsar/pull/17706#discussion_r973818490
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/AdminResource.java:
##
@@ -370,17 +370,21 @@ protected CompletableFuture>
getTopicPoliciesAsyncWithRe
}
protected boolean checkBacklogQuota(BacklogQuota quota, RetentionPolicies
retention) {
-if (retention == null || retention.getRetentionSizeInMB() <= 0 ||
retention.getRetentionTimeInMinutes() <= 0) {
+if (retention == null
+|| (retention.getRetentionSizeInMB() <= 0 &&
retention.getRetentionTimeInMinutes() <= 0)) {
Review Comment:
Looks like the added test can't cover this branch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar-site] urfreespace commented on pull request #207: fix docs banner text
urfreespace commented on PR #207: URL: https://github.com/apache/pulsar-site/pull/207#issuecomment-1250463108 As the new text is more explicit and precise so update the text from this Pr and keep the background color blue, are you all agree? @Anonymitaet @tisonkun @momo-jun -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] tisonkun opened a new pull request, #17709: [improve][ci] docbot comment only for open PRs
tisonkun opened a new pull request, #17709: URL: https://github.com/apache/pulsar/pull/17709 This closes https://github.com/apache/pulsar-test-infra/issues/73. cc @michaeljmarshall @nodece ### Documentation - [ ] `doc-required` (Your PR needs to update docs and you will update later) - [x] `doc-not-needed` (Please explain why) - [ ] `doc` (Your PR contains doc changes) - [ ] `doc-complete` (Docs have been already added) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on pull request #17151: [fix][broker] Added check for invisible characters for subscription name
codelipenghui commented on PR #17151: URL: https://github.com/apache/pulsar/pull/17151#issuecomment-1250461038 > I think the change makes sense. Is there a reason we don't do this validation for subscriptions created via auto subscription creation in the Pulsar Protocol? We should add the validation. It's better to keep consistent with the REST API and Pulsar Protocol. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch master updated: [improve][broker] Make cursor properties support modify single value concurrently. (#17164)
This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 2b85c436f08 [improve][broker] Make cursor properties support modify
single value concurrently. (#17164)
2b85c436f08 is described below
commit 2b85c436f08af969eb625406912621e98321f890
Author: Cong Zhao
AuthorDate: Mon Sep 19 09:42:00 2022 +0800
[improve][broker] Make cursor properties support modify single value
concurrently. (#17164)
---
.../apache/bookkeeper/mledger/ManagedCursor.java | 37 +++---
.../bookkeeper/mledger/impl/ManagedCursorImpl.java | 84 +++---
.../apache/bookkeeper/mledger/util/Futures.java| 25 +++
.../mledger/impl/ManagedCursorContainerTest.java | 17 -
.../mledger/impl/ManagedCursorPropertiesTest.java | 47 +++-
5 files changed, 172 insertions(+), 38 deletions(-)
diff --git
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/ManagedCursor.java
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/ManagedCursor.java
index 3f6852e4085..17dbac09a22 100644
---
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/ManagedCursor.java
+++
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/ManagedCursor.java
@@ -86,14 +86,35 @@ public interface ManagedCursor {
*/
Map getCursorProperties();
- /**
- * Updates the properties.
- * @param cursorProperties
- * @return a handle to the result of the operation
- */
- default CompletableFuture setCursorProperties(Map
cursorProperties) {
- return CompletableFuture.completedFuture(null);
- }
+/**
+ * Add a property associated with the cursor.
+ *
+ * Note: {@link ManagedLedgerException.BadVersionException} will be set in
this {@link CompletableFuture},
+ * if there are concurrent modification and store data has changed.
+ *
+ * @return a handle to the result of the operation
+ */
+CompletableFuture putCursorProperty(String key, String value);
+
+/**
+ * Set all properties associated with the cursor.
+ *
+ * Note: {@link ManagedLedgerException.BadVersionException} will be set in
this {@link CompletableFuture},
+ * if there are concurrent modification and store data has changed.
+ *
+ * @return a handle to the result of the operation
+ */
+CompletableFuture setCursorProperties(Map
cursorProperties);
+
+/**
+ * Remove a property associated with the cursor.
+ *
+ * Note: {@link ManagedLedgerException.BadVersionException} will be set in
this {@link CompletableFuture},
+ * if there are concurrent modification and store data has changed.
+ *
+ * @return a handle to the result of the operation
+ */
+CompletableFuture removeCursorProperty(String key);
/**
* Add a property associated with the last stored position.
diff --git
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java
index 2bfa4ebb7fe..6d595e76dc1 100644
---
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java
+++
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java
@@ -59,6 +59,7 @@ import java.util.concurrent.atomic.AtomicReference;
import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
import java.util.concurrent.locks.ReadWriteLock;
import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.function.Function;
import org.apache.bookkeeper.client.AsyncCallback.CloseCallback;
import org.apache.bookkeeper.client.AsyncCallback.OpenCallback;
import org.apache.bookkeeper.client.BKException;
@@ -119,6 +120,7 @@ public class ManagedCursorImpl implements ManagedCursor {
protected final ManagedLedgerConfig config;
protected final ManagedLedgerImpl ledger;
private final String name;
+
private volatile Map cursorProperties;
private final BookKeeper.DigestType digestType;
@@ -173,7 +175,10 @@ public class ManagedCursorImpl implements ManagedCursor {
private boolean isCursorLedgerReadOnly = true;
// Stat of the cursor z-node
+// NOTE: Don't update cursorLedgerStat alone,
+// please use updateCursorLedgerStat method to update cursorLedgerStat and
managedCursorInfo at the same time.
private volatile Stat cursorLedgerStat;
+private volatile ManagedCursorInfo managedCursorInfo;
private static final LongPairConsumer positionRangeConverter
= PositionImpl::new;
private static final LongPairConsumer
recyclePositionRangeConverter = (key, value) -> {
@@ -314,6 +319,11 @@ public class ManagedCursorImpl implements ManagedCursor {
this.mbean = new ManagedCursorMXBeanImpl(this);
}
+private void
[GitHub] [pulsar] codelipenghui merged pull request #17164: [improve][broker] Make cursor properties support modify single value concurrently.
codelipenghui merged PR #17164: URL: https://github.com/apache/pulsar/pull/17164 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on pull request #17700: [fix][metadata] Cleanup state when lock revalidation gets `LockBusyException`
mattisonchao commented on PR #17700: URL: https://github.com/apache/pulsar/pull/17700#issuecomment-1250441561 Add testing, convert to draft. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] zymap commented on a diff in pull request #17228: [fix][storage] refresh the ledgers map when the offload complete failed
zymap commented on code in PR #17228:
URL: https://github.com/apache/pulsar/pull/17228#discussion_r973809022
##
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java:
##
@@ -2353,6 +2353,42 @@ private void
maybeOffloadInBackground(CompletableFuture promise) {
}
}
+// Although we have caught the connection loss exception on the meta
store, to avoid other exceptions cause
+// the mismatch between meta store and in memory, we refresh the ledger
info list when the offload execute
+// failed by badversion
+private void asyncRefreshLedgersInfoOnBadVersion(ManagedLedgerException
exception) {
+if (!(exception instanceof BadVersionException)) {
+return;
+}
+if (!metadataMutex.tryLock()) {
+scheduledExecutor.schedule(
+() -> asyncRefreshLedgersInfoOnBadVersion(exception), 100,
TimeUnit.MILLISECONDS);
+return;
+}
+store.getManagedLedgerInfo(name, false, new MetaStoreCallback<>() {
+@Override
+public void operationComplete(ManagedLedgerInfo mlInfo, Stat stat)
{
+ledgersStat = stat;
+try {
+synchronized (ManagedLedgerImpl.this) {
+for (LedgerInfo li : mlInfo.getLedgerInfoList()) {
+long ledgerId = li.getLedgerId();
+ledgers.put(ledgerId, li);
Review Comment:
Because we trigger this operation after offloading failed. It shouldn't have
any remove operation on the metadata store. If it have remove, it should
succeed or fail before the offload, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[pulsar] branch branch-2.9 updated: Update proxy lookup throw exception type (#17600)
This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch branch-2.9
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-2.9 by this push:
new ab78a7e0c91 Update proxy lookup throw exception type (#17600)
ab78a7e0c91 is described below
commit ab78a7e0c912d014aa7f9ed3e396b63f9e6ba9b1
Author: Guangning E
AuthorDate: Wed Sep 14 09:47:28 2022 +0800
Update proxy lookup throw exception type (#17600)
(cherry picked from commit 52a380f7e930e633dcefcbdb29ee1460e867dcfe)
---
.../org/apache/pulsar/proxy/server/LookupProxyHandler.java | 14 +++---
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git
a/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/LookupProxyHandler.java
b/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/LookupProxyHandler.java
index d9d208085fd..cbebfa34fa1 100644
---
a/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/LookupProxyHandler.java
+++
b/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/LookupProxyHandler.java
@@ -151,7 +151,7 @@ public class LookupProxyHandler {
if (t != null) {
log.warn("[{}] Failed to lookup topic {}: {}",
clientAddress, topic, t.getMessage());
proxyConnection.ctx().writeAndFlush(
-
Commands.newLookupErrorResponse(ServerError.ServiceNotReady, t.getMessage(),
clientRequestId));
+Commands.newLookupErrorResponse(getServerError(t),
t.getMessage(), clientRequestId));
} else {
String brokerUrl = connectWithTLS ? r.brokerUrlTls :
r.brokerUrl;
if (r.redirect) {
@@ -179,7 +179,7 @@ public class LookupProxyHandler {
}).exceptionally(ex -> {
// Failed to connect to backend broker
proxyConnection.ctx().writeAndFlush(
-
Commands.newLookupErrorResponse(ServerError.ServiceNotReady, ex.getMessage(),
clientRequestId));
+Commands.newLookupErrorResponse(getServerError(ex),
ex.getMessage(), clientRequestId));
return null;
});
}
@@ -249,7 +249,7 @@ public class LookupProxyHandler {
});
}).exceptionally(ex -> {
// Failed to connect to backend broker
-
proxyConnection.ctx().writeAndFlush(Commands.newPartitionMetadataResponse(ServerError.ServiceNotReady,
+
proxyConnection.ctx().writeAndFlush(Commands.newPartitionMetadataResponse(getServerError(ex),
ex.getMessage(), clientRequestId));
return null;
});
@@ -322,7 +322,7 @@ public class LookupProxyHandler {
if (t != null) {
log.warn("[{}] Failed to get TopicsOfNamespace {}: {}",
clientAddress, namespaceName, t.getMessage());
proxyConnection.ctx().writeAndFlush(
-Commands.newError(clientRequestId,
ServerError.ServiceNotReady, t.getMessage()));
+Commands.newError(clientRequestId, getServerError(t),
t.getMessage()));
} else {
proxyConnection.ctx().writeAndFlush(
Commands.newGetTopicsOfNamespaceResponse(r,
clientRequestId));
@@ -333,7 +333,7 @@ public class LookupProxyHandler {
}).exceptionally(ex -> {
// Failed to connect to backend broker
proxyConnection.ctx().writeAndFlush(
-Commands.newError(clientRequestId,
ServerError.ServiceNotReady, ex.getMessage()));
+Commands.newError(clientRequestId, getServerError(ex),
ex.getMessage()));
return null;
});
}
@@ -376,7 +376,7 @@ public class LookupProxyHandler {
if (t != null) {
log.warn("[{}] Failed to get schema {}: {}",
clientAddress, topic, t);
proxyConnection.ctx().writeAndFlush(
-Commands.newError(clientRequestId,
ServerError.ServiceNotReady, t.getMessage()));
+Commands.newError(clientRequestId, getServerError(t),
t.getMessage()));
} else {
proxyConnection.ctx().writeAndFlush(
Commands.newGetSchemaResponse(clientRequestId, r));
@@ -387,7 +387,7 @@ public class LookupProxyHandler {
}).exceptionally(ex -> {
// Failed to connect to backend broker
proxyConnection.ctx().writeAndFlush(
-Commands.newError(clientRequestId,
ServerError.ServiceNotReady, ex.getMessage()));
+Commands.newError(clientRequestId, getServerError(ex),
ex.getMessage()));
return null;
});
[pulsar] branch branch-2.10 updated: Update proxy lookup throw exception type (#17600)
This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-2.10 by this push:
new 9fecb9e10ca Update proxy lookup throw exception type (#17600)
9fecb9e10ca is described below
commit 9fecb9e10ca8dd502fe07f46148ba7e8e4ff0156
Author: Guangning E
AuthorDate: Wed Sep 14 09:47:28 2022 +0800
Update proxy lookup throw exception type (#17600)
(cherry picked from commit 52a380f7e930e633dcefcbdb29ee1460e867dcfe)
---
.../org/apache/pulsar/proxy/server/LookupProxyHandler.java | 14 +++---
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git
a/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/LookupProxyHandler.java
b/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/LookupProxyHandler.java
index c242242d0c9..76cdddccad2 100644
---
a/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/LookupProxyHandler.java
+++
b/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/LookupProxyHandler.java
@@ -151,7 +151,7 @@ public class LookupProxyHandler {
if (t != null) {
log.warn("[{}] Failed to lookup topic {}: {}",
clientAddress, topic, t.getMessage());
proxyConnection.ctx().writeAndFlush(
-
Commands.newLookupErrorResponse(ServerError.ServiceNotReady, t.getMessage(),
clientRequestId));
+Commands.newLookupErrorResponse(getServerError(t),
t.getMessage(), clientRequestId));
} else {
String brokerUrl = connectWithTLS ? r.brokerUrlTls :
r.brokerUrl;
if (r.redirect) {
@@ -179,7 +179,7 @@ public class LookupProxyHandler {
}).exceptionally(ex -> {
// Failed to connect to backend broker
proxyConnection.ctx().writeAndFlush(
-
Commands.newLookupErrorResponse(ServerError.ServiceNotReady, ex.getMessage(),
clientRequestId));
+Commands.newLookupErrorResponse(getServerError(ex),
ex.getMessage(), clientRequestId));
return null;
});
}
@@ -249,7 +249,7 @@ public class LookupProxyHandler {
});
}).exceptionally(ex -> {
// Failed to connect to backend broker
-
proxyConnection.ctx().writeAndFlush(Commands.newPartitionMetadataResponse(ServerError.ServiceNotReady,
+
proxyConnection.ctx().writeAndFlush(Commands.newPartitionMetadataResponse(getServerError(ex),
ex.getMessage(), clientRequestId));
return null;
});
@@ -323,7 +323,7 @@ public class LookupProxyHandler {
log.warn("[{}] Failed to get TopicsOfNamespace {}: {}",
clientAddress, namespaceName, t.getMessage());
proxyConnection.ctx().writeAndFlush(
-Commands.newError(clientRequestId,
ServerError.ServiceNotReady, t.getMessage()));
+Commands.newError(clientRequestId, getServerError(t),
t.getMessage()));
} else {
proxyConnection.ctx().writeAndFlush(
Commands.newGetTopicsOfNamespaceResponse(r,
clientRequestId));
@@ -334,7 +334,7 @@ public class LookupProxyHandler {
}).exceptionally(ex -> {
// Failed to connect to backend broker
proxyConnection.ctx().writeAndFlush(
-Commands.newError(clientRequestId,
ServerError.ServiceNotReady, ex.getMessage()));
+Commands.newError(clientRequestId, getServerError(ex),
ex.getMessage()));
return null;
});
}
@@ -377,7 +377,7 @@ public class LookupProxyHandler {
if (t != null) {
log.warn("[{}] Failed to get schema {}: {}",
clientAddress, topic, t);
proxyConnection.ctx().writeAndFlush(
-Commands.newError(clientRequestId,
ServerError.ServiceNotReady, t.getMessage()));
+Commands.newError(clientRequestId, getServerError(t),
t.getMessage()));
} else {
proxyConnection.ctx().writeAndFlush(
Commands.newGetSchemaResponse(clientRequestId, r));
@@ -388,7 +388,7 @@ public class LookupProxyHandler {
}).exceptionally(ex -> {
// Failed to connect to backend broker
proxyConnection.ctx().writeAndFlush(
-Commands.newError(clientRequestId,
ServerError.ServiceNotReady, ex.getMessage()));
+Commands.newError(clientRequestId, getServerError(ex),
ex.getMessage()));
return null;
});
[pulsar] branch branch-2.11 updated: Update proxy lookup throw exception type (#17600)
This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch branch-2.11
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-2.11 by this push:
new 5bcf254b892 Update proxy lookup throw exception type (#17600)
5bcf254b892 is described below
commit 5bcf254b8922d255078a78e834675b7dc39fcc4f
Author: Guangning E
AuthorDate: Wed Sep 14 09:47:28 2022 +0800
Update proxy lookup throw exception type (#17600)
(cherry picked from commit 52a380f7e930e633dcefcbdb29ee1460e867dcfe)
---
.../org/apache/pulsar/proxy/server/LookupProxyHandler.java | 14 +++---
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git
a/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/LookupProxyHandler.java
b/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/LookupProxyHandler.java
index 1532b5c0f52..472de4327f9 100644
---
a/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/LookupProxyHandler.java
+++
b/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/LookupProxyHandler.java
@@ -151,7 +151,7 @@ public class LookupProxyHandler {
if (t != null) {
log.warn("[{}] Failed to lookup topic {}: {}",
clientAddress, topic, t.getMessage());
proxyConnection.ctx().writeAndFlush(
-
Commands.newLookupErrorResponse(ServerError.ServiceNotReady, t.getMessage(),
clientRequestId));
+Commands.newLookupErrorResponse(getServerError(t),
t.getMessage(), clientRequestId));
} else {
String brokerUrl = connectWithTLS ? r.brokerUrlTls :
r.brokerUrl;
if (r.redirect) {
@@ -179,7 +179,7 @@ public class LookupProxyHandler {
}).exceptionally(ex -> {
// Failed to connect to backend broker
proxyConnection.ctx().writeAndFlush(
-
Commands.newLookupErrorResponse(ServerError.ServiceNotReady, ex.getMessage(),
clientRequestId));
+Commands.newLookupErrorResponse(getServerError(ex),
ex.getMessage(), clientRequestId));
return null;
});
}
@@ -249,7 +249,7 @@ public class LookupProxyHandler {
});
}).exceptionally(ex -> {
// Failed to connect to backend broker
-
proxyConnection.ctx().writeAndFlush(Commands.newPartitionMetadataResponse(ServerError.ServiceNotReady,
+
proxyConnection.ctx().writeAndFlush(Commands.newPartitionMetadataResponse(getServerError(ex),
ex.getMessage(), clientRequestId));
return null;
});
@@ -330,7 +330,7 @@ public class LookupProxyHandler {
log.warn("[{}] Failed to get TopicsOfNamespace {}: {}",
clientAddress, namespaceName, t.getMessage());
proxyConnection.ctx().writeAndFlush(
-Commands.newError(clientRequestId,
ServerError.ServiceNotReady, t.getMessage()));
+Commands.newError(clientRequestId, getServerError(t),
t.getMessage()));
} else {
proxyConnection.ctx().writeAndFlush(
Commands.newGetTopicsOfNamespaceResponse(r.getTopics(), r.getTopicsHash(),
r.isFiltered(),
@@ -342,7 +342,7 @@ public class LookupProxyHandler {
}).exceptionally(ex -> {
// Failed to connect to backend broker
proxyConnection.ctx().writeAndFlush(
-Commands.newError(clientRequestId,
ServerError.ServiceNotReady, ex.getMessage()));
+Commands.newError(clientRequestId, getServerError(ex),
ex.getMessage()));
return null;
});
}
@@ -385,7 +385,7 @@ public class LookupProxyHandler {
if (t != null) {
log.warn("[{}] Failed to get schema {}: {}",
clientAddress, topic, t);
proxyConnection.ctx().writeAndFlush(
-Commands.newError(clientRequestId,
ServerError.ServiceNotReady, t.getMessage()));
+Commands.newError(clientRequestId, getServerError(t),
t.getMessage()));
} else {
proxyConnection.ctx().writeAndFlush(
Commands.newGetSchemaResponse(clientRequestId, r));
@@ -396,7 +396,7 @@ public class LookupProxyHandler {
}).exceptionally(ex -> {
// Failed to connect to backend broker
proxyConnection.ctx().writeAndFlush(
-Commands.newError(clientRequestId,
ServerError.ServiceNotReady, ex.getMessage()));
+Commands.newError(clientRequestId, getServerError(ex),
ex.getMessage()));
return null;
});
[GitHub] [pulsar] mattisonchao commented on a diff in pull request #17700: [fix][metadata] Cleanup state when lock revalidation gets `LockBusyException`
mattisonchao commented on code in PR #17700:
URL: https://github.com/apache/pulsar/pull/17700#discussion_r973804080
##
pulsar-metadata/src/main/java/org/apache/pulsar/metadata/coordination/impl/ResourceLockImpl.java:
##
@@ -184,38 +185,21 @@ synchronized void lockWasInvalidated() {
}
log.info("Lock on resource {} was invalidated", path);
-revalidate(value)
-.thenRun(() -> log.info("Successfully revalidated the lock on
{}", path))
-.exceptionally(ex -> {
-synchronized (ResourceLockImpl.this) {
-if (ex.getCause() instanceof BadVersionException) {
-log.warn("Failed to revalidate the lock at {}.
Marked as expired", path);
-state = State.Released;
-expiredFuture.complete(null);
-} else {
-// We failed to revalidate the lock due to
connectivity issue
-// Continue assuming we hold the lock, until we
can revalidate it, either
-// on Reconnected or SessionReestablished events.
-revalidateAfterReconnection = true;
-log.warn("Failed to revalidate the lock at {}.
Retrying later on reconnection {}", path,
-ex.getCause().getMessage());
-}
-}
-return null;
-});
+revalidate(value, true)
+.thenRun(() -> log.info("Successfully revalidated the lock on
{}", path));
}
synchronized CompletableFuture revalidateIfNeededAfterReconnection()
{
if (revalidateAfterReconnection) {
revalidateAfterReconnection = false;
log.warn("Revalidate lock at {} after reconnection", path);
-return revalidate(value);
+return revalidate(value, true);
} else {
return CompletableFuture.completedFuture(null);
}
}
-synchronized CompletableFuture revalidate(T newValue) {
+synchronized CompletableFuture revalidate(T newValue, boolean
retryWhenConnectionLost) {
Review Comment:
Add `retryWhenConnectionLost` to avoid useless resource lock acquire
distributed lock.
e.g. when we invoke the `acquire` method, if we got a connection problem, we
will return the failed future immediately and discard the resource lock. if we
continue re-try when reconnecting, we probably get the distributed lock for
useless resource lock.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] AnonHxy commented on pull request #15033: [improvement][client-java] Avoid too large memory preallocation for batch message.
AnonHxy commented on PR #15033: URL: https://github.com/apache/pulsar/pull/15033#issuecomment-1250422496 > > This patch broken `org.apache.pulsar.client.impl.BatchMessageContainerImplTest.recoveryAfterOom` test. Please help fix it , thanks @tjiuming > > @AnonHxy Yes, I've fixed, PTAL Also have some checkstyle failure. Please make sure that CI could be pass in your own fork repo first~ @tjiuming -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on a diff in pull request #17700: [fix][metadata] Cleanup state when lock revalidation gets `LockBusyException`
mattisonchao commented on code in PR #17700:
URL: https://github.com/apache/pulsar/pull/17700#discussion_r973535614
##
pulsar-metadata/src/main/java/org/apache/pulsar/metadata/coordination/impl/ResourceLockImpl.java:
##
@@ -232,6 +217,23 @@ synchronized CompletableFuture revalidate(T
newValue) {
});
revalidateFuture = newFuture;
}
+revalidateFuture.exceptionally(ex -> {
+Throwable realCause = FutureUtil.unwrapCompletionException(ex);
+synchronized (ResourceLockImpl.this) {
+if (realCause instanceof BadVersionException || realCause
instanceof LockBusyException) {
+log.warn("Failed to revalidate the lock at {}. Marked as
expired", path);
+state = State.Released;
+expiredFuture.complete(null);
+} else {
+// We failed to revalidate the lock due to connectivity
issue
+// Continue assuming we hold the lock, until we can
revalidate it, either
+// on Reconnected or SessionReestablished events.
+log.warn("Failed to revalidate the lock at {}. Retrying
later on reconnection {}", path,
Review Comment:
there may lost update `revalidateAfterReconnection`, the fix PR #17664
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on pull request #17700: [fix][metadata] Cleanup state when lock revalidation gets `LockBusyException`
mattisonchao commented on PR #17700: URL: https://github.com/apache/pulsar/pull/17700#issuecomment-1250419110 > If I understand this right, the root cause this case is that we missed handling revalidate failure in `revalidateIfNeededAfterReconnection`. Can we just add it there, like `lockWasInvalidated` did? I don't want to invoke exception handling everywhere. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] AnonHxy closed pull request #15033: [improvement][client-java] Avoid too large memory preallocation for batch message.
AnonHxy closed pull request #15033: [improvement][client-java] Avoid too large memory preallocation for batch message. URL: https://github.com/apache/pulsar/pull/15033 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] tjiuming opened a new pull request, #15033: [improvement][client-java] Avoid too large memory preallocation for batch message.
tjiuming opened a new pull request, #15033: URL: https://github.com/apache/pulsar/pull/15033 Fixes (https://github.com/apache/pulsar/issues/14943) ### Motivation Allocate small memory for batch message first, and let it grow(with no memory resizes and memory copies). ### Does this pull request potentially affect one of the following parts: *If `yes` was chosen, please highlight the changes* - Dependencies (does it add or upgrade a dependency): (no) - The public API: (no) - The schema: (no) - The default values of configurations: (no) - The wire protocol: (no) - The rest endpoints: (no) - The admin cli options: (no) - Anything that affects deployment: (no) ### Documentation Check the box below or label this PR directly (if you have committer privilege). Need to update docs? - [ ] `doc-required` (If you need help on updating docs, create a doc issue) - [x] `no-need-doc` (Please explain why) - [ ] `doc` (If this PR contains doc changes) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar-site] branch main updated: Docs sync done from apache/pulsar(#c40c7ee)
This is an automated email from the ASF dual-hosted git repository. urfree pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/pulsar-site.git The following commit(s) were added to refs/heads/main by this push: new 18a431e8d6a Docs sync done from apache/pulsar(#c40c7ee) 18a431e8d6a is described below commit 18a431e8d6a4035774bf57c9b26764ee48443356 Author: Pulsar Site Updater AuthorDate: Mon Sep 19 00:02:23 2022 + Docs sync done from apache/pulsar(#c40c7ee)
[pulsar-site] branch main updated: Docs sync done from apache/pulsar(#c40c7ee)
This is an automated email from the ASF dual-hosted git repository. urfree pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/pulsar-site.git The following commit(s) were added to refs/heads/main by this push: new a80fe6344f5 Docs sync done from apache/pulsar(#c40c7ee) a80fe6344f5 is described below commit a80fe6344f55dd4304c78b89d753aa1fd8f208c1 Author: Pulsar Site Updater AuthorDate: Sun Sep 18 18:02:10 2022 + Docs sync done from apache/pulsar(#c40c7ee)
[GitHub] [pulsar] tjiuming commented on a diff in pull request #17041: [monitoring][broker][metadata] add metadata store metrics
tjiuming commented on code in PR #17041:
URL: https://github.com/apache/pulsar/pull/17041#discussion_r973726806
##
pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/PrometheusMetricsTest.java:
##
@@ -1522,6 +1524,81 @@ public void testSplitTopicAndPartitionLabel() throws
Exception {
consumer2.close();
}
+
+@Test
+public void testMetadataStoreStats() throws Exception {
Review Comment:
sorry I missed ur comment
##
pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/PrometheusMetricsTest.java:
##
@@ -1522,6 +1524,81 @@ public void testSplitTopicAndPartitionLabel() throws
Exception {
consumer2.close();
}
+
+@Test
+public void testMetadataStoreStats() throws Exception {
Review Comment:
@asafm sorry I missed ur comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] tjiuming commented on pull request #15033: [improvement][client-java] Avoid too large memory preallocation for batch message.
tjiuming commented on PR #15033: URL: https://github.com/apache/pulsar/pull/15033#issuecomment-1250311640 > This patch broken `org.apache.pulsar.client.impl.BatchMessageContainerImplTest.recoveryAfterOom` test. Please help fix it , thanks @tjiuming @AnonHxy Yes, I've fixed, PTAL -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch master updated: [fix][metadata] Set revalidateAfterReconnection true for certain failures (#17664)
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new c40c7ee9fb3 [fix][metadata] Set revalidateAfterReconnection true for
certain failures (#17664)
c40c7ee9fb3 is described below
commit c40c7ee9fb35eafea9c9923dfcf62706ea5d36bf
Author: Michael Marshall
AuthorDate: Sun Sep 18 06:27:59 2022 -0700
[fix][metadata] Set revalidateAfterReconnection true for certain failures
(#17664)
---
.../org/apache/pulsar/metadata/coordination/impl/ResourceLockImpl.java | 1 +
1 file changed, 1 insertion(+)
diff --git
a/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/coordination/impl/ResourceLockImpl.java
b/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/coordination/impl/ResourceLockImpl.java
index bc3f47d41dc..22eaccc278b 100644
---
a/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/coordination/impl/ResourceLockImpl.java
+++
b/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/coordination/impl/ResourceLockImpl.java
@@ -196,6 +196,7 @@ public class ResourceLockImpl implements ResourceLock
{
// We failed to revalidate the lock due to
connectivity issue
// Continue assuming we hold the lock, until we
can revalidate it, either
// on Reconnected or SessionReestablished events.
+revalidateAfterReconnection = true;
log.warn("Failed to revalidate the lock at {}.
Retrying later on reconnection {}", path,
ex.getCause().getMessage());
}
[GitHub] [pulsar] eolivelli merged pull request #17664: [fix][metadata] Set revalidateAfterReconnection true for certain failures
eolivelli merged PR #17664: URL: https://github.com/apache/pulsar/pull/17664 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] yaalsn commented on pull request #17382: [improve][CI] Generate unit test code coverage reports and upload to Codecov
yaalsn commented on PR #17382: URL: https://github.com/apache/pulsar/pull/17382#issuecomment-1250262380 /pulsarbot rerun-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar-site] branch main updated: Docs sync done from apache/pulsar(#4ec1009)
This is an automated email from the ASF dual-hosted git repository. urfree pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/pulsar-site.git The following commit(s) were added to refs/heads/main by this push: new 9fe120e2b0e Docs sync done from apache/pulsar(#4ec1009) 9fe120e2b0e is described below commit 9fe120e2b0e946518248c97cd8dfd8492f236161 Author: Pulsar Site Updater AuthorDate: Sun Sep 18 12:02:04 2022 + Docs sync done from apache/pulsar(#4ec1009)
[GitHub] [pulsar] tisonkun commented on pull request #17693: [feat][CI] Add approval solution to reduce GitHub Actions resource consumption
tisonkun commented on PR #17693: URL: https://github.com/apache/pulsar/pull/17693#issuecomment-1250241072 @lhotari Perhaps we can also always should these instructions on pull requests created. All new pull requests should encounter this case. The effect looks like https://github.com/pingcap/tidb/pull/37911#issuecomment-1250238271 and we can implement it with GitHub Actions (It's about up to 10~30 new lightweight workflow runs ([metric](https://play.clickhouse.com/play?user=play#U0VMRUNUIHRvU3RhcnRPZkRheShjcmVhdGVkX2F0KSBkLCBDT1VOVCgqKSB0CkZST00gZ2l0aHViX2V2ZW50cwpXSEVSRSByZXBvX25hbWUgPSAnYXBhY2hlL3B1bHNhcicKICBBTkQgZXZlbnRfdHlwZSA9ICdQdWxsUmVxdWVzdEV2ZW50JwogIEFORCBhY3Rpb24gPSAnb3BlbmVkJwpHUk9VUCBCWSBkCk9SREVSIEJZIGQgREVTQwpMSU1JVCAxMA==))). The reason is that the summary in check result doesn't show in the main thread of the pull request. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] pjfanning commented on pull request #17457: [fix][sec] bump snakeyaml to 1.31 fix CVE-2022-25857
pjfanning commented on PR #17457: URL: https://github.com/apache/pulsar/pull/17457#issuecomment-1250236732 snakeyaml v1.32 fixes a 2nd similar issue - https://github.com/advisories/GHSA-9w3m-gqgf-c4p9 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] tisonkun commented on a diff in pull request #17669: [fix][doc] Fix M1 JVM Installation Instructions
tisonkun commented on code in PR #17669: URL: https://github.com/apache/pulsar/pull/17669#discussion_r973690909 ## README.md: ## @@ -138,6 +138,47 @@ Requirements: > Note: this project includes a [Maven > Wrapper](https://maven.apache.org/wrapper/) that can be used instead of a > system installed Maven. > Use it by replacing `mvn` by `./mvnw` on Linux and `mvnw.cmd` on Windows in > the commands below. +### Install JDK on M1 +In the current version, Pulsar uses a BookKeeper version which in turn uses RocksDB. RocksDB is compiled to work on x86 architecture and not ARM. Therefore, Pulsar can only work with x86 JDK. This is planned to be fixed in future versions of Pulsar. Review Comment: And perhaps after we release 2.11.0 and users start to use the new version, it will be less and less useful. > M1 is notorious to have many quirks. Anything we can do to help will be greatly appreciated. I agree. So +0 on this patch now. FYI - I encounter several other issues, and I prefer to resolve troubleshootings in discussions instead of a manual try to cover most of them: * https://github.com/apache/pulsar/issues/16639 * https://github.com/apache/pulsar/issues/16652 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] tisonkun commented on a diff in pull request #17669: [fix][doc] Fix M1 JVM Installation Instructions
tisonkun commented on code in PR #17669: URL: https://github.com/apache/pulsar/pull/17669#discussion_r973687982 ## README.md: ## @@ -138,6 +138,47 @@ Requirements: > Note: this project includes a [Maven > Wrapper](https://maven.apache.org/wrapper/) that can be used instead of a > system installed Maven. > Use it by replacing `mvn` by `./mvnw` on Linux and `mvnw.cmd` on Windows in > the commands below. +### Install JDK on M1 +In the current version, Pulsar uses a BookKeeper version which in turn uses RocksDB. RocksDB is compiled to work on x86 architecture and not ARM. Therefore, Pulsar can only work with x86 JDK. This is planned to be fixed in future versions of Pulsar. Review Comment: This doesn't stand any more on the nightly version. See also https://github.com/apache/pulsar/issues/12166#issuecomment-1237601981. I agree that there can be a few troubleshooting to work on a different environment with older version, and I'd prefer a discussion thread (https://github.com/apache/pulsar/issues/12166 seems a good place to comment these instructions and closed) or a mac specific section like [Hadoop does](https://github.com/apache/hadoop/blob/a73c4804d882d9bc1a424080f20c4902cb9db832/BUILDING.txt#L389-L418) (but still, on <= 2.10 branches?). ## README.md: ## @@ -138,6 +138,47 @@ Requirements: > Note: this project includes a [Maven > Wrapper](https://maven.apache.org/wrapper/) that can be used instead of a > system installed Maven. > Use it by replacing `mvn` by `./mvnw` on Linux and `mvnw.cmd` on Windows in > the commands below. +### Install JDK on M1 +In the current version, Pulsar uses a BookKeeper version which in turn uses RocksDB. RocksDB is compiled to work on x86 architecture and not ARM. Therefore, Pulsar can only work with x86 JDK. This is planned to be fixed in future versions of Pulsar. Review Comment: This doesn't stand any more on the nightly version. See also https://github.com/apache/pulsar/issues/12166#issuecomment-1237601981. I agree that there can be a few troubleshooting to work on a different environment with older version, and I'd prefer a discussion thread (https://github.com/apache/pulsar/issues/12166 seems a good place to comment these instructions and closed). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] tisonkun commented on a diff in pull request #17669: [fix][doc] Fix M1 JVM Installation Instructions
tisonkun commented on code in PR #17669: URL: https://github.com/apache/pulsar/pull/17669#discussion_r973688501 ## README.md: ## @@ -138,6 +138,47 @@ Requirements: > Note: this project includes a [Maven > Wrapper](https://maven.apache.org/wrapper/) that can be used instead of a > system installed Maven. > Use it by replacing `mvn` by `./mvnw` on Linux and `mvnw.cmd` on Windows in > the commands below. +### Install JDK on M1 +In the current version, Pulsar uses a BookKeeper version which in turn uses RocksDB. RocksDB is compiled to work on x86 architecture and not ARM. Therefore, Pulsar can only work with x86 JDK. This is planned to be fixed in future versions of Pulsar. Review Comment: And I just notice that it misleads me because it's not about "Install JDK on M1" - it's about "Install X86 JDK on M1" (if you get a RocksDB JNI load exception). https://www.google.com/search?q=Install+x86+JDK+on+M1 gives you: * https://itnext.io/how-to-install-x86-and-arm-jdks-on-the-mac-m1-apple-silicon-using-sdkman-872a5adc050d * https://github.com/sdkman/sdkman-cli/issues/1068 That can help. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] tisonkun commented on a diff in pull request #17669: [fix][doc] Fix M1 JVM Installation Instructions
tisonkun commented on code in PR #17669: URL: https://github.com/apache/pulsar/pull/17669#discussion_r973689315 ## README.md: ## @@ -138,6 +138,47 @@ Requirements: > Note: this project includes a [Maven > Wrapper](https://maven.apache.org/wrapper/) that can be used instead of a > system installed Maven. > Use it by replacing `mvn` by `./mvnw` on Linux and `mvnw.cmd` on Windows in > the commands below. +### Install JDK on M1 +In the current version, Pulsar uses a BookKeeper version which in turn uses RocksDB. RocksDB is compiled to work on x86 architecture and not ARM. Therefore, Pulsar can only work with x86 JDK. This is planned to be fixed in future versions of Pulsar. Review Comment: So the meaningful information here is, that if you fail to run Pulsar on M1 due to RocksDB JNI error (with a typical stack trace), the reason is that you need an X86 JDK for older versions. Then mention some methods to achieve it as you like. I get confused the first time: I'm an M1 user, and this section seems written for me. But it's not. It's for those who running Pulsar <= 2.10 on M1 and get the RocksDB JNI load exception. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] tisonkun commented on a diff in pull request #17669: [fix][doc] Fix M1 JVM Installation Instructions
tisonkun commented on code in PR #17669: URL: https://github.com/apache/pulsar/pull/17669#discussion_r973689315 ## README.md: ## @@ -138,6 +138,47 @@ Requirements: > Note: this project includes a [Maven > Wrapper](https://maven.apache.org/wrapper/) that can be used instead of a > system installed Maven. > Use it by replacing `mvn` by `./mvnw` on Linux and `mvnw.cmd` on Windows in > the commands below. +### Install JDK on M1 +In the current version, Pulsar uses a BookKeeper version which in turn uses RocksDB. RocksDB is compiled to work on x86 architecture and not ARM. Therefore, Pulsar can only work with x86 JDK. This is planned to be fixed in future versions of Pulsar. Review Comment: So the meaningful information here is, that if you fail to run Pulsar on M1 due to RocksDB JNI error (with a typical stack trace), the reason is that you need an X86 JDK for older versions. Then mention some methods to achieve it as you like. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] tisonkun commented on a diff in pull request #17669: [fix][doc] Fix M1 JVM Installation Instructions
tisonkun commented on code in PR #17669: URL: https://github.com/apache/pulsar/pull/17669#discussion_r973688501 ## README.md: ## @@ -138,6 +138,47 @@ Requirements: > Note: this project includes a [Maven > Wrapper](https://maven.apache.org/wrapper/) that can be used instead of a > system installed Maven. > Use it by replacing `mvn` by `./mvnw` on Linux and `mvnw.cmd` on Windows in > the commands below. +### Install JDK on M1 +In the current version, Pulsar uses a BookKeeper version which in turn uses RocksDB. RocksDB is compiled to work on x86 architecture and not ARM. Therefore, Pulsar can only work with x86 JDK. This is planned to be fixed in future versions of Pulsar. Review Comment: And I just notice that it misleads me because it's not about "Install JDK on M1" - it's about "Install X86 JDK on M1". https://www.google.com/search?q=Install+x86+JDK+on+M1 gives you: * https://itnext.io/how-to-install-x86-and-arm-jdks-on-the-mac-m1-apple-silicon-using-sdkman-872a5adc050d * https://github.com/sdkman/sdkman-cli/issues/1068 That can help. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] tisonkun commented on a diff in pull request #17669: [fix][doc] Fix M1 JVM Installation Instructions
tisonkun commented on code in PR #17669: URL: https://github.com/apache/pulsar/pull/17669#discussion_r973687982 ## README.md: ## @@ -138,6 +138,47 @@ Requirements: > Note: this project includes a [Maven > Wrapper](https://maven.apache.org/wrapper/) that can be used instead of a > system installed Maven. > Use it by replacing `mvn` by `./mvnw` on Linux and `mvnw.cmd` on Windows in > the commands below. +### Install JDK on M1 +In the current version, Pulsar uses a BookKeeper version which in turn uses RocksDB. RocksDB is compiled to work on x86 architecture and not ARM. Therefore, Pulsar can only work with x86 JDK. This is planned to be fixed in future versions of Pulsar. Review Comment: This doesn't stand any more on the nightly version. See also https://github.com/apache/pulsar/issues/12166#issuecomment-1237601981. I agree that there can be a few troubleshooting to work on a different environment with older version, and I'd prefer a discussion thread or a mac specific section like [Hadoop does](https://github.com/apache/hadoop/blob/a73c4804d882d9bc1a424080f20c4902cb9db832/BUILDING.txt#L389-L418) (but still, on < 2.10 branches?). ## README.md: ## @@ -138,6 +138,47 @@ Requirements: > Note: this project includes a [Maven > Wrapper](https://maven.apache.org/wrapper/) that can be used instead of a > system installed Maven. > Use it by replacing `mvn` by `./mvnw` on Linux and `mvnw.cmd` on Windows in > the commands below. +### Install JDK on M1 +In the current version, Pulsar uses a BookKeeper version which in turn uses RocksDB. RocksDB is compiled to work on x86 architecture and not ARM. Therefore, Pulsar can only work with x86 JDK. This is planned to be fixed in future versions of Pulsar. Review Comment: This doesn't stand any more on the nightly version. See also https://github.com/apache/pulsar/issues/12166#issuecomment-1237601981. I agree that there can be a few troubleshooting to work on a different environment with older version, and I'd prefer a discussion thread or a mac specific section like [Hadoop does](https://github.com/apache/hadoop/blob/a73c4804d882d9bc1a424080f20c4902cb9db832/BUILDING.txt#L389-L418) (but still, on <= 2.10 branches?). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] tisonkun commented on a diff in pull request #17669: [fix][doc] Fix M1 JVM Installation Instructions
tisonkun commented on code in PR #17669: URL: https://github.com/apache/pulsar/pull/17669#discussion_r973687982 ## README.md: ## @@ -138,6 +138,47 @@ Requirements: > Note: this project includes a [Maven > Wrapper](https://maven.apache.org/wrapper/) that can be used instead of a > system installed Maven. > Use it by replacing `mvn` by `./mvnw` on Linux and `mvnw.cmd` on Windows in > the commands below. +### Install JDK on M1 +In the current version, Pulsar uses a BookKeeper version which in turn uses RocksDB. RocksDB is compiled to work on x86 architecture and not ARM. Therefore, Pulsar can only work with x86 JDK. This is planned to be fixed in future versions of Pulsar. Review Comment: This doesn't stand any more on the nightly version. See also https://github.com/apache/pulsar/issues/12166#issuecomment-1237601981. I agree that there can be a few troubleshooting to work on a different environment with older version, and I'd prefer a discussion thread or a mac specific section like [Hadoop does](https://github.com/apache/hadoop/blob/a73c4804d882d9bc1a424080f20c4902cb9db832/BUILDING.txt#L389-L418). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] asafm commented on a diff in pull request #17041: [monitoring][broker][metadata] add metadata store metrics
asafm commented on code in PR #17041:
URL: https://github.com/apache/pulsar/pull/17041#discussion_r973681673
##
pulsar-metadata/src/main/java/org/apache/pulsar/metadata/api/MetadataStoreConfig.java:
##
@@ -29,6 +29,9 @@
@Getter
@ToString
public class MetadataStoreConfig {
+public static final String LOCAL_METADATA_STORE = "local-metadata-store";
+public static final String STATE_METADATA_STORE = "state-metadata-store";
Review Comment:
>And I would suggest changing "local-metadata-store" -> "metadata-store",
just to keep consistent with the configuration name that we have in the
`broker.conf`.
I agree with that. Today the user knows this from configuration:
`metadataStoreUrl` and `configurationMetadataStoreUrl` so when they see a label
name, they need to be able to match it with what they configured, no?
##
pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/AbstractMetadataStore.java:
##
@@ -80,9 +85,9 @@ public abstract class AbstractMetadataStore implements
MetadataStoreExtended, Co
protected abstract CompletableFuture existsFromStore(String path);
-protected AbstractMetadataStore() {
-this.executor = Executors
-.newSingleThreadScheduledExecutor(new
DefaultThreadFactory("metadata-store"));
+protected AbstractMetadataStore(String metadataStoreName) {
+final String poolName = "metadata-store";
+this.executor = new ScheduledThreadPoolExecutor(1, new
DefaultThreadFactory(poolName));
Review Comment:
Shouldn't the poolName be as metadataStore name?
##
pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/PrometheusMetricsTest.java:
##
@@ -1522,6 +1524,81 @@ public void testSplitTopicAndPartitionLabel() throws
Exception {
consumer2.close();
}
+
+@Test
+public void testMetadataStoreStats() throws Exception {
Review Comment:
You resolved without answering
##
pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/stats/MetadataStoreStats.java:
##
@@ -0,0 +1,110 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.metadata.impl.stats;
+
+import io.prometheus.client.Counter;
+import io.prometheus.client.Histogram;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+public final class MetadataStoreStats implements AutoCloseable {
+private static final double[] BUCKETS = new double[]{1, 3, 5, 10, 20, 50,
100, 200, 500, 1000, 2000, 5000};
+private static final String OPS_TYPE_LABEL_NAME = "type";
+private static final String METADATA_STORE_LABEL_NAME = "name";
+private static final String STATUS = "status";
+
+private static final String OPS_TYPE_GET = "get";
+private static final String OPS_TYPE_DEL = "del";
+private static final String OPS_TYPE_PUT = "put";
+private static final String STATUS_SUCCESS = "success";
+private static final String STATUS_FAIL = "fail";
+
+protected static final String PREFIX = "pulsar_metadata_store_";
+
+private static final Histogram OPS_LATENCY = Histogram
+.build(PREFIX + "ops_latency", "-")
+.unit("ms")
+.buckets(BUCKETS)
+.labelNames(METADATA_STORE_LABEL_NAME, OPS_TYPE_LABEL_NAME, STATUS)
+.register();
+private static final Counter PUT_BYTES = Counter
+.build(PREFIX + "put", "-")
+.unit("bytes")
+.labelNames(METADATA_STORE_LABEL_NAME)
+.register();
+
+private final Histogram.Child getOpsSucceedChild;
+private final Histogram.Child delOpsSucceedChild;
+private final Histogram.Child putOpsSucceedChild;
+private final Histogram.Child getOpsFailedChild;
+private final Histogram.Child delOpsFailedChild;
+private final Histogram.Child putOpsFailedChild;
+private final Counter.Child putBytesChild;
+private final String metadataStoreName;
+private final AtomicBoolean closed = new AtomicBoolean(false);
+
+public MetadataStoreStats(String metadataStoreName) {
+this.metadataStoreName = metadataStoreName;
+
+this.getOpsSucceedChild = OPS_LATENCY.labels(metadataStoreName,
[GitHub] [pulsar] asafm commented on a diff in pull request #17669: [fix][doc] Fix M1 JVM Installation Instructions
asafm commented on code in PR #17669: URL: https://github.com/apache/pulsar/pull/17669#discussion_r973678845 ## README.md: ## @@ -138,6 +138,47 @@ Requirements: > Note: this project includes a [Maven > Wrapper](https://maven.apache.org/wrapper/) that can be used instead of a > system installed Maven. > Use it by replacing `mvn` by `./mvnw` on Linux and `mvnw.cmd` on Windows in > the commands below. +### Install JDK on M1 Review Comment: This issue happened to me and 5 other engineers I know of just as an example. Our approach, especially on getting started should be giving you everything you need to get started - in the README case, it's building the product. M1 is notorious to have many quirks. Anything we can do to help will be greatly appreciated. Just mentioning a few people recently running into those issues @geomagilles @teabot @nahguam just to name a few Bear in mind, that they can ignore this recommendation. Most newcomers will appreciate it as much as I did. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] shink commented on pull request #16003: [feature][pulsar-io-mongo] Add support for full message synchronization
shink commented on PR #16003: URL: https://github.com/apache/pulsar/pull/16003#issuecomment-1250204320 /pulsarbot rerun-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar-site] branch main updated: Docs sync done from apache/pulsar(#4ec1009)
This is an automated email from the ASF dual-hosted git repository. urfree pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/pulsar-site.git The following commit(s) were added to refs/heads/main by this push: new 451d35dbd77 Docs sync done from apache/pulsar(#4ec1009) 451d35dbd77 is described below commit 451d35dbd77faa98bb50bb96a1b2e18b344d2372 Author: Pulsar Site Updater AuthorDate: Sun Sep 18 06:03:56 2022 + Docs sync done from apache/pulsar(#4ec1009)
