[GitHub] [pulsar] zbentley commented on a diff in pull request #16535: [fix] [python client] Better Python garbage collection management for C++-owned objects
zbentley commented on code in PR #16535: URL: https://github.com/apache/pulsar/pull/16535#discussion_r947479658 ## pulsar-client-cpp/lib/checksum/crc32c_arm.h: ## @@ -37,11 +37,11 @@ #define crc32c_u16(crc, v) __crc32ch(crc, v) #define crc32c_u32(crc, v) __crc32cw(crc, v) #define crc32c_u64(crc, v) __crc32cd(crc, v) -#define PREF4X64L1(buffer, PREF_OFFSET, ITR) \ Review Comment: @BewareMyPower sure, but as I recall, if I do then the clang-format build fails. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] aloyszhang commented on pull request #17122: [proxy] remove useless code
aloyszhang commented on PR #17122: URL: https://github.com/apache/pulsar/pull/17122#issuecomment-1217493569 /pulsarbot run-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] AnonHxy commented on a diff in pull request #17084: [improve][broker]add ServerCnx state check before server handle request
AnonHxy commented on code in PR #17084:
URL: https://github.com/apache/pulsar/pull/17084#discussion_r947461907
##
pulsar-broker/src/test/java/org/apache/pulsar/broker/service/ServerCnxTest.java:
##
@@ -2153,4 +2167,157 @@ public void
testHandleAuthResponseWithoutClientVersion() {
verify(authResponse, times(1)).hasClientVersion();
verify(authResponse, times(0)).getClientVersion();
}
+
+@Test(expectedExceptions = IllegalArgumentException.class)
+void shouldFailHandleLookup() throws Exception {
+resetChannel();
+Field stateUpdater = ServerCnx.class.getDeclaredField("state");
+stateUpdater.setAccessible(true);
+stateUpdater.set(serverCnx, ServerCnx.State.Failed);
Review Comment:
This modification will break `serverCnx` state, which is a common component
in the test class. I think it is better to mock a new `ServerCnx` without
using the true `serverCnx ` instance
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] AnonHxy commented on a diff in pull request #17084: [improve][broker]add ServerCnx state check before server handle request
AnonHxy commented on code in PR #17084:
URL: https://github.com/apache/pulsar/pull/17084#discussion_r947463138
##
pulsar-broker/src/test/java/org/apache/pulsar/broker/service/ServerCnxTest.java:
##
@@ -2153,4 +2167,157 @@ public void
testHandleAuthResponseWithoutClientVersion() {
verify(authResponse, times(1)).hasClientVersion();
verify(authResponse, times(0)).getClientVersion();
}
+
+@Test(expectedExceptions = IllegalArgumentException.class)
+void shouldFailHandleLookup() throws Exception {
+resetChannel();
+Field stateUpdater = ServerCnx.class.getDeclaredField("state");
+stateUpdater.setAccessible(true);
+stateUpdater.set(serverCnx, ServerCnx.State.Failed);
+
+CommandLookupTopic commandLookupTopic = spy(CommandLookupTopic.class);
+serverCnx.handleLookup(commandLookupTopic);
Review Comment:
Maybe `serverCnx.handleLookup(any(CommandLookupTopic.class))` is better :)
##
pulsar-broker/src/test/java/org/apache/pulsar/broker/service/ServerCnxTest.java:
##
@@ -2153,4 +2167,157 @@ public void
testHandleAuthResponseWithoutClientVersion() {
verify(authResponse, times(1)).hasClientVersion();
verify(authResponse, times(0)).getClientVersion();
}
+
+@Test(expectedExceptions = IllegalArgumentException.class)
+void shouldFailHandleLookup() throws Exception {
+resetChannel();
+Field stateUpdater = ServerCnx.class.getDeclaredField("state");
+stateUpdater.setAccessible(true);
+stateUpdater.set(serverCnx, ServerCnx.State.Failed);
Review Comment:
This modification will change `serverCnx` state, which is a common
component in the test class. I think it is better to use mock a `ServerCnx` here
##
pulsar-broker/src/test/java/org/apache/pulsar/broker/service/ServerCnxTest.java:
##
@@ -2153,4 +2167,157 @@ public void
testHandleAuthResponseWithoutClientVersion() {
verify(authResponse, times(1)).hasClientVersion();
verify(authResponse, times(0)).getClientVersion();
}
+
+@Test(expectedExceptions = IllegalArgumentException.class)
+void shouldFailHandleLookup() throws Exception {
+resetChannel();
Review Comment:
I think the test method should be public
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on a diff in pull request #17125: [improve][client c++] Support KeyValue Schema.
BewareMyPower commented on code in PR #17125:
URL: https://github.com/apache/pulsar/pull/17125#discussion_r947437276
##
pulsar-client-cpp/tests/MessageTest.cc:
##
@@ -99,3 +99,24 @@ TEST(MessageTest, testMessageBuilder) {
ASSERT_EQ(msg.getData(), originalAddress);
}
}
+
+TEST(MessageTest, testMessageBuilderSetKeyValueContent) {
+std::string keyContent = "keyContent";
+std::string valueContent = "valueContent";
+
+// test inline encoding type.
+{
+KeyValue keyValue(keyContent, valueContent, INLINE);
+const Message& message = MessageBuilder().setContent(keyValue).build();
+ASSERT_EQ(message.getDataAsString(), keyValue.getContent());
+ASSERT_EQ(message.getPartitionKey(), "");
+}
+
+// test inline encoding type.
+{
+KeyValue keyValue(keyContent, valueContent, SEPARATED);
+const Message& message = MessageBuilder().setContent(keyValue).build();
+ASSERT_EQ(message.getDataAsString(), valueContent);
+ASSERT_EQ(message.getPartitionKey(), keyContent);
+}
+}
Review Comment:
It's better to add a newline at the end of file. If you're using a IDE of
JetBrains, you might go to "Editor - General" and select "Ensure every saved
file ends with a line break".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] codelipenghui opened a new issue, #17132: [test] Integration test coverage
codelipenghui opened a new issue, #17132: URL: https://github.com/apache/pulsar/issues/17132 ### Search before asking - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar. ### Motivation Improve integration test coverage ### Solution _No response_ ### Alternatives _No response_ ### Anything else? _No response_ ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #16578: Flaky-test: PulsarSinkE2ETest.testPulsarSinkDLQ
codelipenghui commented on issue #16578: URL: https://github.com/apache/pulsar/issues/16578#issuecomment-1217432017 Another one https://github.com/apache/pulsar/runs/7870706984?check_suite_focus=true -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui opened a new issue, #17131: Flaky-test: TopicTerminationTest.testTerminateWhilePublishing
codelipenghui opened a new issue, #17131: URL: https://github.com/apache/pulsar/issues/17131 ### Search before asking - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar. ### Example failure https://github.com/apache/pulsar/runs/7870706938?check_suite_focus=true ### Exception stacktrace Error: Tests run: 15, Failures: 1, Errors: 0, Skipped: 3, Time elapsed: 107.905 s <<< FAILURE! - in org.apache.pulsar.broker.service.TopicTerminationTest Error: testTerminateWhilePublishing(org.apache.pulsar.broker.service.TopicTerminationTest) Time elapsed: 20.068 s <<< FAILURE! org.testng.internal.thread.ThreadTimeoutException: Method org.apache.pulsar.broker.service.TopicTerminationTest.testTerminateWhilePublishing() didn't finish within the time-out 2 at org.testng.internal.MethodInvocationHelper.invokeWithTimeoutWithNewExecutor(MethodInvocationHelper.java:371) at org.testng.internal.MethodInvocationHelper.invokeWithTimeout(MethodInvocationHelper.java:282) at org.testng.internal.TestInvoker.invokeMethod(TestInvoker.java:605) at org.testng.internal.TestInvoker.retryFailed(TestInvoker.java:214) at org.testng.internal.MethodRunner.runInSequence(MethodRunner.java:58) at org.testng.internal.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:822) at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:147) at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146) at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128) at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) at org.testng.TestRunner.privateRun(TestRunner.java:764) at org.testng.TestRunner.run(TestRunner.java:585) at org.testng.SuiteRunner.runTest(SuiteRunner.java:384) at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:378) at org.testng.SuiteRunner.privateRun(SuiteRunner.java:337) at org.testng.SuiteRunner.run(SuiteRunner.java:286) at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53) at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96) at org.testng.TestNG.runSuitesSequentially(TestNG.java:1218) at org.testng.TestNG.runSuitesLocally(TestNG.java:1140) at org.testng.TestNG.runSuites(TestNG.java:1069) at org.testng.TestNG.run(TestNG.java:1037) at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:135) at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeSingleClass(TestNGDirectoryTestSuite.java:112) at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeLazy(TestNGDirectoryTestSuite.java:123) at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:90) at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:146) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418) ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] momo-jun opened a new pull request, #17130: [fix][doc] Sync recent changes on versioned docs
momo-jun opened a new pull request, #17130: URL: https://github.com/apache/pulsar/pull/17130 ### Modifications * Sync recent doc changes to 2.8.x/2.9.x/2.10.x docs during the switchover time window of https://github.com/apache/pulsar/pull/17074. * #17030 * #17039 * #16315 * #17057 ### Documentation - [ ] `doc` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] nodece opened a new pull request, #17129: [improve][docker] Switch to Temurin JDK
nodece opened a new pull request, #17129: URL: https://github.com/apache/pulsar/pull/17129 Signed-off-by: Zixuan Liu ### Motivation I noticed we are using OpenJDK in our Docker image, I suggest that we switch to the Temurin JDK, because our CI runs on the Temurin JDK, we need to keep the same JDK everywhere to avoid unexpected problems. The Temurin JDK is OpenJDK distribution from Adoptium, the old JDK from Ubuntu, they should all be built on the OpenJDK open source project, and should be fully compatible. The Temurin JDK is safe and reliable. > The Eclipse Temurin™ project provides code and processes that support the building of runtime binaries and associated technologies that are high performance, enterprise-caliber, cross-platform, open-source licensed, and Java SE TCK-tested for general use across the Java ecosystem. https://projects.eclipse.org/projects/adoptium.temurin If we need to discuss this more, see https://lists.apache.org/thread/t8l4r36vdyo7stkwcml7p1s22f466x8o. ### Modifications - Update test image - Update build image - Update pulsar image ### Documentation Check the box below or label this PR directly. Need to update docs? - [ ] `doc-required` (Your PR needs to update docs and you will update later) - [x] `doc-not-needed` (Please explain why) - [ ] `doc` (Your PR contains doc changes) - [ ] `doc-complete` (Docs have been already added) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] HQebupt commented on pull request #17084: [improve][broker]add ServerCnx state check before server handle request
HQebupt commented on PR #17084: URL: https://github.com/apache/pulsar/pull/17084#issuecomment-1217410565 /pulsarbot run-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] RobertIndie opened a new pull request, #17128: [feature][python] PIP 37: Add chunking support for python client
RobertIndie opened a new pull request, #17128: URL: https://github.com/apache/pulsar/pull/17128 ### Motivation This is the python client feature catch up for [PIP 37](https://github.com/apache/pulsar/wiki/PIP-37%3A-Large-message-size-handling-in-Pulsar) ### Modifications * Add `chunking_enabled ` to the producer configuration * Add `max_pending_chunked_message` and `auto_ack_oldest_chunked_message_on_queue_full ` to the consumer configuration ### Documentation Check the box below or label this PR directly. Need to update docs? - [x] `doc-required` Need to add `Enable Chunking` in the [Python client section](https://pulsar.apache.org/docs/next/client-libraries-python). - [ ] `doc-not-needed` (Please explain why) - [ ] `doc` (Your PR contains doc changes) - [ ] `doc-complete` (Docs have been already added) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] HQebupt commented on pull request #17084: [improve][broker]add ServerCnx state check before server handle request
HQebupt commented on PR #17084: URL: https://github.com/apache/pulsar/pull/17084#issuecomment-1217397460 /pulsarbot run-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] SignorMercurio commented on pull request #15775: [fix][Admin CLI docs] Sync the description and default value of `--processing-guarantees`
SignorMercurio commented on PR #15775: URL: https://github.com/apache/pulsar/pull/15775#issuecomment-1217397167 @momo-jun Of course yes. Note that there's a leading whitespace in `" #Java, Python, Go"`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] HQebupt commented on pull request #16756: [fix][broker]optimize the shutdown sequence of broker service when it close
HQebupt commented on PR #16756: URL: https://github.com/apache/pulsar/pull/16756#issuecomment-1217395702 /pulsarbot run-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] momo-jun commented on pull request #15775: [fix][Admin CLI docs] Sync the description and default value of `--processing-guarantees`
momo-jun commented on PR #15775: URL: https://github.com/apache/pulsar/pull/15775#issuecomment-1217395786 @SignorMercurio - just a double-check to resolve the conflict. Can I keep my changes and append the line298 `+ " #Java, Python, Go"` you added in #16853? https://user-images.githubusercontent.com/60642177/185023581-f8f4a8f3-3f68-4d72-9f33-a17e6d98dcc2.png;> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] tjiuming commented on a diff in pull request #17041: [monitoring][broker][metadata] add metadata store metrics
tjiuming commented on code in PR #17041:
URL: https://github.com/apache/pulsar/pull/17041#discussion_r947395766
##
pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/stats/MetadataStoreStats.java:
##
@@ -24,41 +24,42 @@
public final class MetadataStoreStats implements AutoCloseable {
private static final double[] BUCKETS = new double[]{1, 3, 5, 10, 20, 50,
100, 200, 500, 1000, 2000, 5000};
-private static final String LABEL_NAME = "metadata_store_name";
+private static final String LABEL_NAME = "name";
+protected static final String PREFIX = "pulsar_metadata_store_";
private static final Histogram GET_OPS_SUCCEED = Histogram
-.build("pulsar_metadata_store_get_ops_latency", "-")
+.build(PREFIX + "get_latency", "-")
.unit("ms")
.buckets(BUCKETS)
.labelNames(LABEL_NAME)
.register();
private static final Histogram DEL_OPS_SUCCEED = Histogram
-.build("pulsar_metadata_store_del_ops_latency", "-")
+.build(PREFIX + "del_latency", "-")
Review Comment:
I don't think so, it just changes
```
pulsar_metadata_store_get_latency_ms{name = 'xx'}
pulsar_metadata_store_del_latency_ms{name = 'xx'}
pulsar_metadata_store_put_latency_ms{name = 'xx'}
```
to
```
pulsar_metadata_store_latency_ms{name = 'xx', type = 'get'}
pulsar_metadata_store_latency_ms{name = 'xx', type = 'del'}
pulsar_metadata_store_latency_ms{name = 'xx', type = 'put'}
```
It comes no benefits and makes metrics data longer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] tisonkun commented on issue #16613: [improve][client] Optimize Third-Party Client Matrix Page
tisonkun commented on issue #16613: URL: https://github.com/apache/pulsar/issues/16613#issuecomment-1217387683 This issue can be closed by https://github.com/apache/pulsar/pull/17010. For further improvement, I notice that @momo-jun create a draft proposal #17098 which can supersede this one. @dave2wave @Anonymitaet please help close this issue :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #16416: The pulsar-admin doesn't set SNI properly when `tlsEnableHostnameVerification` is `false`
github-actions[bot] commented on issue #16416: URL: https://github.com/apache/pulsar/issues/16416#issuecomment-1217383026 The issue had no activity for 30 days, mark with Stale label. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #16613: [improve][client] Optimize Third-Party Client Matrix Page
github-actions[bot] commented on issue #16613: URL: https://github.com/apache/pulsar/issues/16613#issuecomment-1217382798 The issue had no activity for 30 days, mark with Stale label. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on pull request #16623: rename pulsar env variables for pulsar tools
github-actions[bot] commented on PR #16623: URL: https://github.com/apache/pulsar/pull/16623#issuecomment-1217382756 The pr had no activity for 30 days, mark with Stale label. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #16631: create function with restful API，which has the error "415 Unsupported Media Type"
github-actions[bot] commented on issue #16631: URL: https://github.com/apache/pulsar/issues/16631#issuecomment-1217382716 The issue had no activity for 30 days, mark with Stale label. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on pull request #17116: [fix][flaky-test]ConsumedLedgersTrimTest
poorbarcode commented on PR #17116: URL: https://github.com/apache/pulsar/pull/17116#issuecomment-1217382100 /pulsarbot rerun-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] 02/03: [fix][broker] Fix calculate avg message per entry (#17046)
This is an automated email from the ASF dual-hosted git repository.
technoboy pushed a commit to branch branch-2.11
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 72810a08cce89c07b61d134a287d2a32b31ff7ad
Author: Cong Zhao
AuthorDate: Wed Aug 17 09:02:35 2022 +0800
[fix][broker] Fix calculate avg message per entry (#17046)
---
.../org/apache/pulsar/broker/service/Consumer.java | 21 ++--
.../service/plugin/EntryFilterProducerTest.java| 62 +++
.../pulsar/broker/stats/ConsumerStatsTest.java | 124 +++--
3 files changed, 186 insertions(+), 21 deletions(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Consumer.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Consumer.java
index 5c7646921fb..85df9ff107c 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Consumer.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Consumer.java
@@ -290,13 +290,16 @@ public class Consumer {
return writePromise;
}
int unackedMessages = totalMessages;
-// Note
-// Must ensure that the message is written to the pendingAcks before
sent is first, because this consumer
-// is possible to disconnect at this time.
-if (pendingAcks != null) {
-for (int i = 0; i < entries.size(); i++) {
-Entry entry = entries.get(i);
-if (entry != null) {
+int totalEntries = 0;
+
+for (int i = 0; i < entries.size(); i++) {
+Entry entry = entries.get(i);
+if (entry != null) {
+totalEntries++;
+// Note
+// Must ensure that the message is written to the pendingAcks
before sent is first,
+// because this consumer is possible to disconnect at this
time.
+if (pendingAcks != null) {
int batchSize = batchSizes.getBatchSize(i);
int stickyKeyHash = getStickyKeyHash(entry);
long[] ackSet =
getCursorAckSet(PositionImpl.get(entry.getLedgerId(), entry.getEntryId()));
@@ -317,10 +320,10 @@ public class Consumer {
// calculate avg message per entry
if (avgMessagesPerEntry.get() < 1) { //valid avgMessagesPerEntry
should always >= 1
// set init value.
-avgMessagesPerEntry.set(1.0 * totalMessages / entries.size());
+avgMessagesPerEntry.set(1.0 * totalMessages / totalEntries);
} else {
avgMessagesPerEntry.set(avgMessagesPerEntry.get() * avgPercent
-+ (1 - avgPercent) * totalMessages / entries.size());
++ (1 - avgPercent) * totalMessages / totalEntries);
}
// reduce permit and increment unackedMsg count with total number of
messages in batch-msgs
diff --git
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/plugin/EntryFilterProducerTest.java
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/plugin/EntryFilterProducerTest.java
new file mode 100644
index 000..5973b7fe54b
--- /dev/null
+++
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/plugin/EntryFilterProducerTest.java
@@ -0,0 +1,62 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.service.plugin;
+
+
+import java.util.Collections;
+import java.util.Map;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.pulsar.broker.service.Consumer;
+
+@Slf4j
+public class EntryFilterProducerTest implements EntryFilter {
+@Override
+public FilterResult filterEntry(Entry entry, FilterContext context) {
+if (context.getMsgMetadata() == null) {
+return FilterResult.ACCEPT;
+}
+Consumer consumer = context.getConsumer();
+Map metadata = consumer != null ?
consumer.getMetadata() : Collections.emptyMap();
+log.info("filterEntry for {}", metadata);
+String matchValueAccept = metadata.getOrDefault("matchValueAccept",
"ACCEPT");
+String matchValueReject = metadata.getOrDefault("matchValueReject",
[pulsar] 03/03: [imp][managed-ledger] Optimize filterReadEntries usage of Entry interface ledger (#17108)
This is an automated email from the ASF dual-hosted git repository.
technoboy pushed a commit to branch branch-2.11
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 5fbc911ae380c6126e4d0150357d58315c09c20e
Author: Michael Marshall
AuthorDate: Tue Aug 16 13:47:00 2022 -0500
[imp][managed-ledger] Optimize filterReadEntries usage of Entry interface
ledger (#17108)
---
.../java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java| 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
diff --git
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java
index 4c688b2d9be..78ec42b7183 100644
---
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java
+++
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java
@@ -2288,9 +2288,7 @@ public class ManagedCursorImpl implements ManagedCursor {
} else {
// Remove from the entry list all the entries that were
already marked for deletion
return Lists.newArrayList(Collections2.filter(entries, entry
-> {
-boolean includeEntry = !individualDeletedMessages.contains(
-((PositionImpl) entry.getPosition()).getLedgerId(),
-((PositionImpl) entry.getPosition()).getEntryId());
+boolean includeEntry =
!individualDeletedMessages.contains(entry.getLedgerId(), entry.getEntryId());
if (!includeEntry) {
if (log.isDebugEnabled()) {
log.debug("[{}] [{}] Filtering entry at {} -
already deleted", ledger.getName(), name,
[pulsar] branch branch-2.11 updated (28f53b0214c -> 5fbc911ae38)
This is an automated email from the ASF dual-hosted git repository.
technoboy pushed a change to branch branch-2.11
in repository https://gitbox.apache.org/repos/asf/pulsar.git
from 28f53b0214c fix: bundle-data metadata leak because of bundlestats was
not clean (#17095)
new 7b871fd6a3c [fix][flaky-test]LedgerOffloaderMetricsTest (#17106)
new 72810a08cce [fix][broker] Fix calculate avg message per entry (#17046)
new 5fbc911ae38 [imp][managed-ledger] Optimize filterReadEntries usage of
Entry interface ledger (#17108)
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../bookkeeper/mledger/impl/ManagedCursorImpl.java | 4 +-
.../org/apache/pulsar/broker/service/Consumer.java | 21 ++--
...ilterTest.java => EntryFilterProducerTest.java} | 33 +++---
.../pulsar/broker/stats/ConsumerStatsTest.java | 124 +++--
.../broker/stats/LedgerOffloaderMetricsTest.java | 59 --
5 files changed, 139 insertions(+), 102 deletions(-)
copy
pulsar-broker/src/test/java/org/apache/pulsar/broker/service/plugin/{EntryFilterTest.java
=> EntryFilterProducerTest.java} (61%)
[pulsar] 01/03: [fix][flaky-test]LedgerOffloaderMetricsTest (#17106)
This is an automated email from the ASF dual-hosted git repository.
technoboy pushed a commit to branch branch-2.11
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 7b871fd6a3c3c32ccdbefbb01b160eb645ce0919
Author: fengyubiao
AuthorDate: Wed Aug 17 09:47:34 2022 +0800
[fix][flaky-test]LedgerOffloaderMetricsTest (#17106)
---
.../broker/stats/LedgerOffloaderMetricsTest.java | 59 --
1 file changed, 59 deletions(-)
diff --git
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/LedgerOffloaderMetricsTest.java
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/LedgerOffloaderMetricsTest.java
index 1d49d8d6e4e..f9ff2d2081d 100644
---
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/LedgerOffloaderMetricsTest.java
+++
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/LedgerOffloaderMetricsTest.java
@@ -18,30 +18,16 @@
*/
package org.apache.pulsar.broker.stats;
-import static org.mockito.Mockito.doReturn;
-import static org.mockito.Mockito.spy;
import static org.testng.Assert.assertEquals;
import static org.testng.Assert.assertTrue;
import java.util.ArrayList;
import java.util.HashMap;
-import java.util.LinkedList;
import java.util.Map;
import java.util.List;
-import java.util.Optional;
-import java.util.Queue;
import java.util.UUID;
-import java.util.concurrent.CompletableFuture;
import java.util.concurrent.TimeUnit;
-import org.apache.bookkeeper.mledger.LedgerOffloader;
-import org.apache.bookkeeper.mledger.ManagedLedger;
-import org.apache.bookkeeper.mledger.ManagedLedgerConfig;
import org.apache.bookkeeper.mledger.impl.LedgerOffloaderStatsImpl;
-import org.apache.pulsar.broker.service.BrokerService;
import org.apache.pulsar.broker.service.BrokerTestBase;
-import org.apache.pulsar.broker.service.Topic;
-import org.apache.pulsar.broker.service.persistent.PersistentTopic;
-import org.mockito.Mockito;
-import org.mockito.stubbing.Answer;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.Test;
@@ -67,30 +53,9 @@ public class LedgerOffloaderMetricsTest extends
BrokerTestBase {
String []topics = new String[3];
LedgerOffloaderStatsImpl offloaderStats = (LedgerOffloaderStatsImpl)
pulsar.getOffloaderStats();
-
-LedgerOffloader offloader = Mockito.mock(LedgerOffloader.class);
-Topic topic = Mockito.mock(PersistentTopic.class);
-CompletableFuture> topicFuture = new
CompletableFuture<>();
-Optional topicOptional = Optional.of(topic);
-topicFuture.complete(topicOptional);
-BrokerService brokerService = spy(pulsar.getBrokerService());
-doReturn(brokerService).when(pulsar).getBrokerService();
-
-
for (int i = 0; i < 3; i++) {
String topicName = "persistent://prop/ns-abc1/testMetrics" +
UUID.randomUUID();
topics[i] = topicName;
-admin.topics().createNonPartitionedTopic(topicName);
-
-
doReturn(topicFuture).when(brokerService).getTopicIfExists(topicName);
-assertTrue(topic instanceof PersistentTopic);
-
-ManagedLedger ledgerM = Mockito.mock(ManagedLedger.class);
-doReturn(ledgerM).when(((PersistentTopic)
topic)).getManagedLedger();
-ManagedLedgerConfig config =
Mockito.mock(ManagedLedgerConfig.class);
-doReturn(config).when(ledgerM).getConfig();
-doReturn(offloader).when(config).getLedgerOffloader();
-
offloaderStats.recordOffloadError(topicName);
offloaderStats.recordOffloadError(topicName);
offloaderStats.recordOffloadBytes(topicName, 100);
@@ -123,15 +88,6 @@ public class LedgerOffloaderMetricsTest extends
BrokerTestBase {
String ns2 = "prop/ns-abc2";
LedgerOffloaderStatsImpl offloaderStats = (LedgerOffloaderStatsImpl)
pulsar.getOffloaderStats();
-
-LedgerOffloader offloader = Mockito.mock(LedgerOffloader.class);
-Topic topic = Mockito.mock(PersistentTopic.class);
-CompletableFuture> topicFuture = new
CompletableFuture<>();
-Optional topicOptional = Optional.of(topic);
-topicFuture.complete(topicOptional);
-BrokerService brokerService = spy(pulsar.getBrokerService());
-doReturn(brokerService).when(pulsar).getBrokerService();
-Queue queue = new LinkedList<>();
Map> namespace2Topics = new HashMap<>();
for (int s = 0; s < 2; s++) {
String nameSpace = ns1;
@@ -146,20 +102,6 @@ public class LedgerOffloaderMetricsTest extends
BrokerTestBase {
String topicName = baseTopic1 + UUID.randomUUID();
List topicList = namespace2Topics.get(nameSpace);
topicList.add(topicName);
-
-queue.add(topicName);
-admin.topics().createNonPartitionedTopic(topicName);
-
doReturn(topicFuture).when(brokerService).getTopicIfExists(topicName);
-
[GitHub] [pulsar] HQebupt commented on pull request #16756: [fix][broker]optimize the shutdown sequence of broker service when it close
HQebupt commented on PR #16756: URL: https://github.com/apache/pulsar/pull/16756#issuecomment-1217374906 /pulsarbot run-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] HQebupt commented on pull request #17084: [improve][broker]add ServerCnx state check before server handle request
HQebupt commented on PR #17084: URL: https://github.com/apache/pulsar/pull/17084#issuecomment-1217374291 /pulsarbot run-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] Technoboy- opened a new issue, #17127: Flaky-test: ClientCnxRequestTimeoutQueueTest. setupClientCnx
Technoboy- opened a new issue, #17127: URL: https://github.com/apache/pulsar/issues/17127 ### Search before asking - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar. ### Example failure https://github.com/apache/pulsar/runs/7870157500?check_suite_focus=true ### Exception stacktrace ``` Error: Tests run: 8, Failures: 1, Errors: 0, Skipped: 7, Time elapsed: 0.061 s <<< FAILURE! - in org.apache.pulsar.client.impl.ClientCnxRequestTimeoutQueueTest Error: setupClientCnx(org.apache.pulsar.client.impl.ClientCnxRequestTimeoutQueueTest) Time elapsed: 0.018 s <<< FAILURE! java.lang.NoClassDefFoundError: Could not initialize class org.apache.pulsar.common.protocol.Commands at org.apache.pulsar.client.impl.ClientCnx.(ClientCnx.java:220) at org.apache.pulsar.client.impl.ClientCnxRequestTimeoutQueueTest.setupClientCnx(ClientCnxRequestTimeoutQueueTest.java:63) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132) at org.testng.internal.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:61) at org.testng.internal.ConfigInvoker.invokeConfigurationMethod(ConfigInvoker.java:366) at org.testng.internal.ConfigInvoker.invokeConfigurations(ConfigInvoker.java:320) at org.testng.TestRunner.invokeTestConfigurations(TestRunner.java:617) ``` ``` Error: Tests run: 6, Failures: 3, Errors: 0, Skipped: 3, Time elapsed: 0.341 s <<< FAILURE! - in org.apache.pulsar.client.impl.BinaryProtoLookupServiceTest Error: maxLookupRedirectsTest1(org.apache.pulsar.client.impl.BinaryProtoLookupServiceTest) Time elapsed: 0.003 s <<< FAILURE! java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: Could not initialize class org.apache.pulsar.common.protocol.Commands at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) at org.apache.pulsar.client.impl.BinaryProtoLookupServiceTest.maxLookupRedirectsTest1(BinaryProtoLookupServiceTest.java:83) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132) at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45) at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:70) at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.apache.pulsar.common.protocol.Commands at org.apache.pulsar.client.impl.BinaryProtoLookupService.lambda$findBroker$2(BinaryProtoLookupService.java:117) ``` ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] Technoboy- opened a new issue, #17126: Flaky-test: RGUsageMTAggrWaitForAllMsgsTest.testMTProduceConsumeRGUsageNonPersistentTopicNamesSameTenant
Technoboy- opened a new issue, #17126: URL: https://github.com/apache/pulsar/issues/17126 ### Search before asking - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar. ### Example failure https://github.com/apache/pulsar/runs/7870050421?check_suite_focus=true ### Exception stacktrace ``` Error: Tests run: 11, Failures: 1, Errors: 0, Skipped: 9, Time elapsed: 26.062 s <<< FAILURE! - in org.apache.pulsar.broker.resourcegroup.RGUsageMTAggrWaitForAllMsgsTest Error: testMTProduceConsumeRGUsageNonPersistentTopicNamesSameTenant(org.apache.pulsar.broker.resourcegroup.RGUsageMTAggrWaitForAllMsgsTest) Time elapsed: 0.581 s <<< FAILURE! java.lang.AssertionError: expected [400] but found [0] at org.testng.Assert.fail(Assert.java:99) at org.testng.Assert.failNotEquals(Assert.java:1037) at org.testng.Assert.assertEqualsImpl(Assert.java:140) at org.testng.Assert.assertEquals(Assert.java:122) at org.testng.Assert.assertEquals(Assert.java:907) at org.testng.Assert.assertEquals(Assert.java:917) at org.apache.pulsar.broker.resourcegroup.RGUsageMTAggrWaitForAllMsgsTest.testProduceConsumeUsageOnRG(RGUsageMTAggrWaitForAllMsgsTest.java:463) at org.apache.pulsar.broker.resourcegroup.RGUsageMTAggrWaitForAllMsgsTest.testMTProduceConsumeRGUsageNonPersistentTopicNamesSameTenant(RGUsageMTAggrWaitForAllMsgsTest.java:108) ``` ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on pull request #17116: [fix][flaky-test]ConsumedLedgersTrimTest
poorbarcode commented on PR #17116: URL: https://github.com/apache/pulsar/pull/17116#issuecomment-1217367836 /pulsarbot rerun-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch master updated: [fix][flaky-test]LedgerOffloaderMetricsTest (#17106)
This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 535415302ef [fix][flaky-test]LedgerOffloaderMetricsTest (#17106)
535415302ef is described below
commit 535415302ef6d1a9017f6ec25b87b24afd081155
Author: fengyubiao
AuthorDate: Wed Aug 17 09:47:34 2022 +0800
[fix][flaky-test]LedgerOffloaderMetricsTest (#17106)
---
.../broker/stats/LedgerOffloaderMetricsTest.java | 59 --
1 file changed, 59 deletions(-)
diff --git
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/LedgerOffloaderMetricsTest.java
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/LedgerOffloaderMetricsTest.java
index 1d49d8d6e4e..f9ff2d2081d 100644
---
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/LedgerOffloaderMetricsTest.java
+++
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/LedgerOffloaderMetricsTest.java
@@ -18,30 +18,16 @@
*/
package org.apache.pulsar.broker.stats;
-import static org.mockito.Mockito.doReturn;
-import static org.mockito.Mockito.spy;
import static org.testng.Assert.assertEquals;
import static org.testng.Assert.assertTrue;
import java.util.ArrayList;
import java.util.HashMap;
-import java.util.LinkedList;
import java.util.Map;
import java.util.List;
-import java.util.Optional;
-import java.util.Queue;
import java.util.UUID;
-import java.util.concurrent.CompletableFuture;
import java.util.concurrent.TimeUnit;
-import org.apache.bookkeeper.mledger.LedgerOffloader;
-import org.apache.bookkeeper.mledger.ManagedLedger;
-import org.apache.bookkeeper.mledger.ManagedLedgerConfig;
import org.apache.bookkeeper.mledger.impl.LedgerOffloaderStatsImpl;
-import org.apache.pulsar.broker.service.BrokerService;
import org.apache.pulsar.broker.service.BrokerTestBase;
-import org.apache.pulsar.broker.service.Topic;
-import org.apache.pulsar.broker.service.persistent.PersistentTopic;
-import org.mockito.Mockito;
-import org.mockito.stubbing.Answer;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.Test;
@@ -67,30 +53,9 @@ public class LedgerOffloaderMetricsTest extends
BrokerTestBase {
String []topics = new String[3];
LedgerOffloaderStatsImpl offloaderStats = (LedgerOffloaderStatsImpl)
pulsar.getOffloaderStats();
-
-LedgerOffloader offloader = Mockito.mock(LedgerOffloader.class);
-Topic topic = Mockito.mock(PersistentTopic.class);
-CompletableFuture> topicFuture = new
CompletableFuture<>();
-Optional topicOptional = Optional.of(topic);
-topicFuture.complete(topicOptional);
-BrokerService brokerService = spy(pulsar.getBrokerService());
-doReturn(brokerService).when(pulsar).getBrokerService();
-
-
for (int i = 0; i < 3; i++) {
String topicName = "persistent://prop/ns-abc1/testMetrics" +
UUID.randomUUID();
topics[i] = topicName;
-admin.topics().createNonPartitionedTopic(topicName);
-
-
doReturn(topicFuture).when(brokerService).getTopicIfExists(topicName);
-assertTrue(topic instanceof PersistentTopic);
-
-ManagedLedger ledgerM = Mockito.mock(ManagedLedger.class);
-doReturn(ledgerM).when(((PersistentTopic)
topic)).getManagedLedger();
-ManagedLedgerConfig config =
Mockito.mock(ManagedLedgerConfig.class);
-doReturn(config).when(ledgerM).getConfig();
-doReturn(offloader).when(config).getLedgerOffloader();
-
offloaderStats.recordOffloadError(topicName);
offloaderStats.recordOffloadError(topicName);
offloaderStats.recordOffloadBytes(topicName, 100);
@@ -123,15 +88,6 @@ public class LedgerOffloaderMetricsTest extends
BrokerTestBase {
String ns2 = "prop/ns-abc2";
LedgerOffloaderStatsImpl offloaderStats = (LedgerOffloaderStatsImpl)
pulsar.getOffloaderStats();
-
-LedgerOffloader offloader = Mockito.mock(LedgerOffloader.class);
-Topic topic = Mockito.mock(PersistentTopic.class);
-CompletableFuture> topicFuture = new
CompletableFuture<>();
-Optional topicOptional = Optional.of(topic);
-topicFuture.complete(topicOptional);
-BrokerService brokerService = spy(pulsar.getBrokerService());
-doReturn(brokerService).when(pulsar).getBrokerService();
-Queue queue = new LinkedList<>();
Map> namespace2Topics = new HashMap<>();
for (int s = 0; s < 2; s++) {
String nameSpace = ns1;
@@ -146,20 +102,6 @@ public class LedgerOffloaderMetricsTest extends
BrokerTestBase {
String topicName = baseTopic1 + UUID.randomUUID();
List topicList = namespace2Topics.get(nameSpace);
topicList.add(topicName);
-
-queue.add(topicName);
-
[GitHub] [pulsar] codelipenghui closed issue #16858: Flaky-test: LedgerOffloaderMetricsTest.cleanup
codelipenghui closed issue #16858: Flaky-test: LedgerOffloaderMetricsTest.cleanup URL: https://github.com/apache/pulsar/issues/16858 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui merged pull request #17106: [fix][flaky-test]LedgerOffloaderMetricsTest
codelipenghui merged PR #17106: URL: https://github.com/apache/pulsar/pull/17106 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] tisonkun commented on a diff in pull request #16993: [cleanup][broker] Add Maven Modernizer plugin in pulsar-broker and fix violation in `test: broker/admin`
tisonkun commented on code in PR #16993: URL: https://github.com/apache/pulsar/pull/16993#discussion_r947376228 ## pulsar-broker/pom.xml: ## @@ -425,6 +425,25 @@ + +org.gaul +modernizer-maven-plugin + + false Review Comment: @MarvinCai please check this comment https://github.com/apache/pulsar/pull/16993#discussion_r945935655 I'm OK since this patch does improve the situation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] momo-jun commented on a diff in pull request #16843: [improve][doc] Add more concepts/tasks for bookie isolation
momo-jun commented on code in PR #16843: URL: https://github.com/apache/pulsar/pull/16843#discussion_r947375066 ## site2/docs/administration-isolation-bookie.md: ## @@ -10,24 +10,230 @@ import TabItem from '@theme/TabItem'; -A namespace can be isolated into user-defined groups of bookies, which guarantees all the data that belongs to the namespace is stored in desired bookies. The bookie affinity group uses the BookKeeper [rack-aware placement policy](https://bookkeeper.apache.org/docs/latest/api/javadoc/org/apache/bookkeeper/client/EnsemblePlacementPolicy.html) and it is a way to feed rack information which is stored as JSON format in znode. +Isolating bookies equals isolating message storage, which is a data storage mechanism that provides isolation and safety for specific topics. -You can set a bookie affinity group using one of the following methods. +Bookie isolation is controlled by BookKeeper clients. For Pulsar, there are two kinds of BookKeeper clients to read and write data. +* BookKeeper clients on the broker side + Pulsar brokers use these BookKeeper clients to read and write topic messages. Review Comment: Yes. Nice catch! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] momo-jun commented on a diff in pull request #16843: [improve][doc] Add more concepts/tasks for bookie isolation
momo-jun commented on code in PR #16843: URL: https://github.com/apache/pulsar/pull/16843#discussion_r947374994 ## site2/docs/administration-isolation-bookie.md: ## @@ -10,24 +10,230 @@ import TabItem from '@theme/TabItem'; -A namespace can be isolated into user-defined groups of bookies, which guarantees all the data that belongs to the namespace is stored in desired bookies. The bookie affinity group uses the BookKeeper [rack-aware placement policy](https://bookkeeper.apache.org/docs/latest/api/javadoc/org/apache/bookkeeper/client/EnsemblePlacementPolicy.html) and it is a way to feed rack information which is stored as JSON format in znode. +Isolating bookies equals isolating message storage, which is a data storage mechanism that provides isolation and safety for specific topics. -You can set a bookie affinity group using one of the following methods. +Bookie isolation is controlled by BookKeeper clients. For Pulsar, there are two kinds of BookKeeper clients to read and write data. +* BookKeeper clients on the broker side + Pulsar brokers use these BookKeeper clients to read and write topic messages. +* BookKeeper clients on the bookie auto-recovery side + * The bookie auditor checks whether ledger replicas fulfill the configured isolation policy; + * The bookie replication worker writes ledger replicas to target bookies according to the configured isolation policy. + +To isolate bookies, you need to complete the following tasks. +1. Select a [data isolation policy](#understand-bookie-data-isolation-policies) based on your requirements. +2. [Enable the policy on BookKeeper clients](#enable-bookie-data-placement-policy). +3. [Configure the policy on bookie instances](#configure-data-placement-policy-on-bookie-instances). + + +## Understand bookie data isolation policy + +Bookie data isolation policy is built on top of the existing BookKeeper rack-aware placement policy. The “rack” concept can be anything, for example, racks, regions, availability zones. It writes the configured isolation policy into the metadata store. Both BookKeeper clients on the broker and bookie auto-recovery side read the configured isolation policy from the metadata store and apply it when choosing bookies to store messages. + +BookKeeper provides three kinds of data isolation policy for disaster tolerance. +* Rack-aware placement policy (default) +* Region-aware placement policy +* Zone-aware placement policy + +:::tip + +* Both [rack-aware placement policy](#rack-aware-placement-policy) and [region-aware placement policy](#region-aware-placement-policy) can be used in all kinds of deployments where racks are a subset of a region. The major difference between the two policies is: + * With `RackawareEnsemblePlacementPolicy` configured, the BookKeeper client chooses bookies from different **racks** to reduce the single-point-of-failure. If there is only one rack available, the policy falls back on choosing a random bookie across available ones. + * With `RegionAwareEnsemblePlacementPolicy` configured, the BookKeeper client chooses bookies from different **regions**; for the selected region, it chooses bookies from different racks if more than one ensemble falls into the same region. + +* Zone-aware placement policy (`ZoneAwareEnsemblePlacementPolicy`) can be used in a public cloud infrastructure where Availability Zones (AZs) are isolated locations within the data center regions that public cloud services originate from and operate in. + +::: + +### Rack-aware placement policy + +Rack-aware placement policy enforces different data replicas to be placed in different racks to guarantee the rack-level disaster tolerance for your production environment. A data center usually has a lot of racks, and each rack has many storage nodes. You can use `RackAwareEnsemblePlacementPolicy` to configure the rack information for each bookie. + + Qualified rack size of bookies + +When the available rack size of bookies can meet the requirements configured on a topic, the rack-aware placement policy can work well and you don’t need any extra configurations. + +For example, the BookKeeper cluster has 4 racks and 13 bookie instances as shown the following diagram. When a topic is configured with `EnsembleSize=3, WriteQuorum=3, AckQuorum=2`, the BookKeeper client chooses one bookie instance from three different racks to write data to, such as Bookie2, Bookie8, and Bookie12. + + + + + Enforced minimum rack size of bookies + +When the available rack size of bookies cannot meet the requirements configured on a topic, the strategy that the BookKeeper client chooses bookies to recover old ledgers and create new ledgers depends on whether the enforced minimum rack size of bookies is configured. + +In this case, if you want to make the rack-aware placement policy work as usual, you need to
[GitHub] [pulsar] codelipenghui commented on pull request #17049: [fix][broker] Fix schema does not replicate successfully
codelipenghui commented on PR #17049: URL: https://github.com/apache/pulsar/pull/17049#issuecomment-1217358836 /pulsarbot run-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] shibd opened a new pull request, #17125: [improve][client c++] Support KeyValue Schema.
shibd opened a new pull request, #17125: URL: https://github.com/apache/pulsar/pull/17125 ### Motivation C++ client Support KeyValue Schema. ### Modifications - A new constructor is added in `SchemaInfo` to combine key and value schemas. - Add a new `KeyValue` class, to help users merge and parse key and value data. ### Documentation The current Schema documentation is relatively simple, I will open a new PR to add more content and sample code - [x] `doc-required` (Your PR needs to update docs and you will update later) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] MarvinCai commented on a diff in pull request #16993: [cleanup][broker] Add Maven Modernizer plugin in pulsar-broker and fix violation in `test: broker/admin`
MarvinCai commented on code in PR #16993: URL: https://github.com/apache/pulsar/pull/16993#discussion_r947368626 ## pulsar-broker/pom.xml: ## @@ -425,6 +425,25 @@ + +org.gaul +modernizer-maven-plugin + + false Review Comment: should be true after as we've fixed all the violation? ## pulsar-broker/pom.xml: ## @@ -425,6 +425,25 @@ + +org.gaul +modernizer-maven-plugin + + false Review Comment: should be true as we've fixed all the violation? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on pull request #17116: [fix][flaky-test]ConsumedLedgersTrimTest
poorbarcode commented on PR #17116: URL: https://github.com/apache/pulsar/pull/17116#issuecomment-1217347494 /pulsarbot rerun-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on pull request #17059: [test][broker]auto relealse connection when use proxy
poorbarcode commented on PR #17059: URL: https://github.com/apache/pulsar/pull/17059#issuecomment-1217347387 /pulsarbot rerun-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on pull request #17106: [fix][flaky-test]LedgerOffloaderMetricsTest
poorbarcode commented on PR #17106: URL: https://github.com/apache/pulsar/pull/17106#issuecomment-1217347118 /pulsarbot rerun-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch master updated: [fix][broker] Fix calculate avg message per entry (#17046)
This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 2c2b75edf3c [fix][broker] Fix calculate avg message per entry (#17046)
2c2b75edf3c is described below
commit 2c2b75edf3c33a1258c070e12e7d175ae83a1c66
Author: Cong Zhao
AuthorDate: Wed Aug 17 09:02:35 2022 +0800
[fix][broker] Fix calculate avg message per entry (#17046)
---
.../org/apache/pulsar/broker/service/Consumer.java | 21 ++--
.../service/plugin/EntryFilterProducerTest.java| 62 +++
.../pulsar/broker/stats/ConsumerStatsTest.java | 124 +++--
3 files changed, 186 insertions(+), 21 deletions(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Consumer.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Consumer.java
index 20f3d3f74d8..5add5829174 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Consumer.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Consumer.java
@@ -290,13 +290,16 @@ public class Consumer {
return writePromise;
}
int unackedMessages = totalMessages;
-// Note
-// Must ensure that the message is written to the pendingAcks before
sent is first, because this consumer
-// is possible to disconnect at this time.
-if (pendingAcks != null) {
-for (int i = 0; i < entries.size(); i++) {
-Entry entry = entries.get(i);
-if (entry != null) {
+int totalEntries = 0;
+
+for (int i = 0; i < entries.size(); i++) {
+Entry entry = entries.get(i);
+if (entry != null) {
+totalEntries++;
+// Note
+// Must ensure that the message is written to the pendingAcks
before sent is first,
+// because this consumer is possible to disconnect at this
time.
+if (pendingAcks != null) {
int batchSize = batchSizes.getBatchSize(i);
int stickyKeyHash = getStickyKeyHash(entry);
long[] ackSet =
getCursorAckSet(PositionImpl.get(entry.getLedgerId(), entry.getEntryId()));
@@ -317,10 +320,10 @@ public class Consumer {
// calculate avg message per entry
if (avgMessagesPerEntry.get() < 1) { //valid avgMessagesPerEntry
should always >= 1
// set init value.
-avgMessagesPerEntry.set(1.0 * totalMessages / entries.size());
+avgMessagesPerEntry.set(1.0 * totalMessages / totalEntries);
} else {
avgMessagesPerEntry.set(avgMessagesPerEntry.get() * avgPercent
-+ (1 - avgPercent) * totalMessages / entries.size());
++ (1 - avgPercent) * totalMessages / totalEntries);
}
// reduce permit and increment unackedMsg count with total number of
messages in batch-msgs
diff --git
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/plugin/EntryFilterProducerTest.java
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/plugin/EntryFilterProducerTest.java
new file mode 100644
index 000..5973b7fe54b
--- /dev/null
+++
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/plugin/EntryFilterProducerTest.java
@@ -0,0 +1,62 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.service.plugin;
+
+
+import java.util.Collections;
+import java.util.Map;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.pulsar.broker.service.Consumer;
+
+@Slf4j
+public class EntryFilterProducerTest implements EntryFilter {
+@Override
+public FilterResult filterEntry(Entry entry, FilterContext context) {
+if (context.getMsgMetadata() == null) {
+return FilterResult.ACCEPT;
+}
+Consumer consumer = context.getConsumer();
+Map metadata = consumer != null ?
consumer.getMetadata() : Collections.emptyMap();
+log.info("filterEntry for {}",
[GitHub] [pulsar] codelipenghui merged pull request #17046: [fix][broker] Fix calculate avg message per entry
codelipenghui merged PR #17046: URL: https://github.com/apache/pulsar/pull/17046 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on a diff in pull request #16843: [improve][doc] Add more concepts/tasks for bookie isolation
Anonymitaet commented on code in PR #16843: URL: https://github.com/apache/pulsar/pull/16843#discussion_r947344330 ## site2/docs/administration-isolation-bookie.md: ## @@ -10,24 +10,230 @@ import TabItem from '@theme/TabItem'; -A namespace can be isolated into user-defined groups of bookies, which guarantees all the data that belongs to the namespace is stored in desired bookies. The bookie affinity group uses the BookKeeper [rack-aware placement policy](https://bookkeeper.apache.org/docs/latest/api/javadoc/org/apache/bookkeeper/client/EnsemblePlacementPolicy.html) and it is a way to feed rack information which is stored as JSON format in znode. +Isolating bookies equals isolating message storage, which is a data storage mechanism that provides isolation and safety for specific topics. -You can set a bookie affinity group using one of the following methods. +Bookie isolation is controlled by BookKeeper clients. For Pulsar, there are two kinds of BookKeeper clients to read and write data. +* BookKeeper clients on the broker side + Pulsar brokers use these BookKeeper clients to read and write topic messages. +* BookKeeper clients on the bookie auto-recovery side + * The bookie auditor checks whether ledger replicas fulfill the configured isolation policy; + * The bookie replication worker writes ledger replicas to target bookies according to the configured isolation policy. + +To isolate bookies, you need to complete the following tasks. +1. Select a [data isolation policy](#understand-bookie-data-isolation-policies) based on your requirements. +2. [Enable the policy on BookKeeper clients](#enable-bookie-data-placement-policy). +3. [Configure the policy on bookie instances](#configure-data-placement-policy-on-bookie-instances). + + +## Understand bookie data isolation policy + +Bookie data isolation policy is built on top of the existing BookKeeper rack-aware placement policy. The “rack” concept can be anything, for example, racks, regions, availability zones. It writes the configured isolation policy into the metadata store. Both BookKeeper clients on the broker and bookie auto-recovery side read the configured isolation policy from the metadata store and apply it when choosing bookies to store messages. + +BookKeeper provides three kinds of data isolation policy for disaster tolerance. Review Comment: ```suggestion BookKeeper provides three kinds of data isolation policies for disaster tolerance. ``` ## site2/docs/administration-isolation-bookie.md: ## @@ -10,24 +10,230 @@ import TabItem from '@theme/TabItem'; -A namespace can be isolated into user-defined groups of bookies, which guarantees all the data that belongs to the namespace is stored in desired bookies. The bookie affinity group uses the BookKeeper [rack-aware placement policy](https://bookkeeper.apache.org/docs/latest/api/javadoc/org/apache/bookkeeper/client/EnsemblePlacementPolicy.html) and it is a way to feed rack information which is stored as JSON format in znode. +Isolating bookies equals isolating message storage, which is a data storage mechanism that provides isolation and safety for specific topics. -You can set a bookie affinity group using one of the following methods. +Bookie isolation is controlled by BookKeeper clients. For Pulsar, there are two kinds of BookKeeper clients to read and write data. +* BookKeeper clients on the broker side + Pulsar brokers use these BookKeeper clients to read and write topic messages. +* BookKeeper clients on the bookie auto-recovery side + * The bookie auditor checks whether ledger replicas fulfill the configured isolation policy; + * The bookie replication worker writes ledger replicas to target bookies according to the configured isolation policy. + +To isolate bookies, you need to complete the following tasks. +1. Select a [data isolation policy](#understand-bookie-data-isolation-policies) based on your requirements. +2. [Enable the policy on BookKeeper clients](#enable-bookie-data-placement-policy). +3. [Configure the policy on bookie instances](#configure-data-placement-policy-on-bookie-instances). + + +## Understand bookie data isolation policy + +Bookie data isolation policy is built on top of the existing BookKeeper rack-aware placement policy. The “rack” concept can be anything, for example, racks, regions, availability zones. It writes the configured isolation policy into the metadata store. Both BookKeeper clients on the broker and bookie auto-recovery side read the configured isolation policy from the metadata store and apply it when choosing bookies to store messages. + +BookKeeper provides three kinds of data isolation policy for disaster tolerance. +* Rack-aware placement policy (default) +* Region-aware placement policy +* Zone-aware placement policy + +:::tip + +* Both [rack-aware placement policy](#rack-aware-placement-policy) and [region-aware
[pulsar-site] branch main updated (aa9e358ce1a -> 1207da6f281)
This is an automated email from the ASF dual-hosted git repository. urfree pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/pulsar-site.git from aa9e358ce1a Docs sync done from apache/pulsar(#7ec1d54) new 6d12b818e12 fix: remove exec command in pulsar-config-doc-gen.sh new 1207da6f281 Merge pull request #164 from SignorMercurio/fix/config-docs-migrate The 1009 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: site2/tools/pulsar-config-doc-gen.sh | 10 -- 1 file changed, 4 insertions(+), 6 deletions(-)
[GitHub] [pulsar] leizhiyuan commented on issue #17115: [Bug] the retention policies of health topic
leizhiyuan commented on issue #17115: URL: https://github.com/apache/pulsar/issues/17115#issuecomment-1217304112 see https://lists.apache.org/thread/c4b262w1o5g0l34dxtb3pqvt17w8tg8q -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] leizhiyuan closed issue #17087: [Bug] when delete namespace bundle-data in metastore was not deleted
leizhiyuan closed issue #17087: [Bug] when delete namespace bundle-data in metastore was not deleted URL: https://github.com/apache/pulsar/issues/17087 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar-site] branch main updated: Docs sync done from apache/pulsar(#7ec1d54)
This is an automated email from the ASF dual-hosted git repository. urfree pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/pulsar-site.git The following commit(s) were added to refs/heads/main by this push: new aa9e358ce1a Docs sync done from apache/pulsar(#7ec1d54) aa9e358ce1a is described below commit aa9e358ce1a91f9afc897cb831b5487aa70783e7 Author: Pulsar Site Updater AuthorDate: Wed Aug 17 00:02:07 2022 + Docs sync done from apache/pulsar(#7ec1d54)
[GitHub] [pulsar-client-go] tisonkun commented on pull request #802: [oauth2] Remove oauth2 go.mod and go.sum
tisonkun commented on PR #802: URL: https://github.com/apache/pulsar-client-go/pull/802#issuecomment-1217262908 @pgier I read your comment now. And it seems my statement above holds: > As long as the whole codebase is small and it isn't polluted in the linking stage, it's fine. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] eolivelli commented on pull request #17118: [fix][functions] Make mandatory to provide a schema in Context::newOutputRecordBuilder
eolivelli commented on PR #17118: URL: https://github.com/apache/pulsar/pull/17118#issuecomment-1217162313 /pulsar-bot rerun-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch master updated: [imp][managed-ledger] Optimize filterReadEntries usage of Entry interface ledger (#17108)
This is an automated email from the ASF dual-hosted git repository.
mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 7ec1d543280 [imp][managed-ledger] Optimize filterReadEntries usage of
Entry interface ledger (#17108)
7ec1d543280 is described below
commit 7ec1d5432804a75f76d66a9d13186f299c317168
Author: Michael Marshall
AuthorDate: Tue Aug 16 13:47:00 2022 -0500
[imp][managed-ledger] Optimize filterReadEntries usage of Entry interface
ledger (#17108)
---
.../java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java| 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
diff --git
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java
index 8e74f32037f..ef1c25f458e 100644
---
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java
+++
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java
@@ -2288,9 +2288,7 @@ public class ManagedCursorImpl implements ManagedCursor {
} else {
// Remove from the entry list all the entries that were
already marked for deletion
return Lists.newArrayList(Collections2.filter(entries, entry
-> {
-boolean includeEntry = !individualDeletedMessages.contains(
-((PositionImpl) entry.getPosition()).getLedgerId(),
-((PositionImpl) entry.getPosition()).getEntryId());
+boolean includeEntry =
!individualDeletedMessages.contains(entry.getLedgerId(), entry.getEntryId());
if (!includeEntry) {
if (log.isDebugEnabled()) {
log.debug("[{}] [{}] Filtering entry at {} -
already deleted", ledger.getName(), name,
[GitHub] [pulsar] merlimat merged pull request #17108: [imp][managed-ledger] Optimize getting ledger and entry id from entry
merlimat merged PR #17108: URL: https://github.com/apache/pulsar/pull/17108 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar-site] branch main updated: Docs sync done from apache/pulsar(#4d7b1ac)
This is an automated email from the ASF dual-hosted git repository. urfree pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/pulsar-site.git The following commit(s) were added to refs/heads/main by this push: new 2cc1a0238b3 Docs sync done from apache/pulsar(#4d7b1ac) 2cc1a0238b3 is described below commit 2cc1a0238b38c96ce5bbd32b9e6dab12adc6ce9e Author: Pulsar Site Updater AuthorDate: Tue Aug 16 18:01:48 2022 + Docs sync done from apache/pulsar(#4d7b1ac)
[GitHub] [pulsar] michaeljmarshall commented on issue #15985: Add getLastMessageId feature to Python Client
michaeljmarshall commented on issue #15985: URL: https://github.com/apache/pulsar/issues/15985#issuecomment-1216946011 Fixed in https://github.com/apache/pulsar/pull/16182 and https://github.com/apache/pulsar/pull/16255 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] michaeljmarshall closed issue #15985: Add getLastMessageId feature to Python Client
michaeljmarshall closed issue #15985: Add getLastMessageId feature to Python Client URL: https://github.com/apache/pulsar/issues/15985 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] lhotari commented on issue #11351: Fix precise publish rate limiting
lhotari commented on issue #11351: URL: https://github.com/apache/pulsar/issues/11351#issuecomment-1216914500 This issue was fixed by #11446 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] lhotari closed issue #11351: Fix precise publish rate limiting
lhotari closed issue #11351: Fix precise publish rate limiting URL: https://github.com/apache/pulsar/issues/11351 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] marksilcox commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
marksilcox commented on PR #15558: URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1216874652 @asafm Not sure I will have the bandwidth to look at `FunctionsStatsGenerator` anytime soon -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] horizonzy commented on a diff in pull request #16590: [PIP-186] Introduce two phase deletion protocol based on system topic
horizonzy commented on code in PR #16590:
URL: https://github.com/apache/pulsar/pull/16590#discussion_r947003851
##
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/deletion/LedgerDeletionService.java:
##
@@ -0,0 +1,117 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.bookkeeper.mledger.deletion;
+
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import org.apache.bookkeeper.mledger.proto.MLDataFormats;
+import
org.apache.bookkeeper.mledger.proto.MLDataFormats.ManagedLedgerInfo.LedgerInfo;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.PulsarClientException;
+
+public interface LedgerDeletionService {
+
+/**
+ * Start.
+ */
+void start() throws PulsarClientException, PulsarAdminException;
+
+/**
+ * @param topicName topicName
+ * @param ledgerId ledgerId
+ * @param context ledgerInfo
+ * @param component managed_ledger, managed_cursor, schema_storage
+ * @param type ledger, offload_ledger
+ * @param properties properties
+ * @return
+ */
+CompletableFuture appendPendingDeleteLedger(String topicName, long
ledgerId, LedgerInfo context,
+ LedgerComponent component,
LedgerType type, Map properties);
+
+/**
+ *
+ * @param topicName topicName
+ * @param ledgerId ledgerId
+ * @param component managed_ledger, managed_cursor, schema_storage
+ * @param isBelievedDelete isBelievedDelete, if false, we should check the
param is match the ledger metadata.
+ * @return
+ */
+CompletableFuture asyncDeleteLedger(String topicName, long ledgerId,
LedgerComponent component,
+ boolean isBelievedDelete);
+
+/**
+ *
+ * @param topicName topicName
+ * @param ledgerId ledgerId
+ * @param offloadContext offloadContext
+ * @return
+ */
+CompletableFuture asyncDeleteOffloadedLedger(String topicName, long
ledgerId,
+
MLDataFormats.OffloadContext offloadContext);
+
+/**
+ * Close.
+ */
+void close() throws Exception;
+
+/**
+ * Async close.
+ */
+CompletableFuture asyncClose();
+
+class LedgerDeletionServiceDisable implements LedgerDeletionService {
+
+@Override
+public void start() {
+//No op
+}
+
+private static final CompletableFuture COMPLETABLE_FUTURE =
CompletableFuture.completedFuture(null);
Review Comment:
Fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] horizonzy commented on a diff in pull request #16590: [PIP-186] Introduce two phase deletion protocol based on system topic
horizonzy commented on code in PR #16590:
URL: https://github.com/apache/pulsar/pull/16590#discussion_r947003565
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v2/PersistentTopics.java:
##
@@ -1079,6 +1080,34 @@ public void deleteTopic(
});
}
+@POST
+@Path("/{tenant}/{namespace}/{topic}/ledger/delete")
+@ApiOperation(value = "Delete a topic.",
+notes = "The topic cannot be deleted if delete is not forcefully
and there's any active "
++ "subscription or producer connected to the it. "
++ "Force delete ignores connected clients and deletes
topic by explicitly closing them.")
+@ApiResponses(value = {
+@ApiResponse(code = 307, message = "Current broker doesn't serve
the namespace of this topic"),
+@ApiResponse(code = 401, message = "Don't have permission to
administrate resources on this tenant"),
+@ApiResponse(code = 403, message = "Don't have admin permission"),
+@ApiResponse(code = 404, message = "Topic does not exist"),
+@ApiResponse(code = 412, message = "Topic has active
producers/subscriptions"),
+@ApiResponse(code = 500, message = "Internal server error")})
+public void deleteLedger(
Review Comment:
If there are 2 brokers, it will exists two system topic consumers(A and B).
If the topic onwership is A, when B received pending delete ledger msg, it will
send delete ledger command to A. So introduce rest api in server side and
PusarAdmin.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on pull request #17049: [fix][broker] Fix schema does not replicate successfully
codelipenghui commented on PR #17049: URL: https://github.com/apache/pulsar/pull/17049#issuecomment-1216870713 There should be another issue related to this part https://github.com/apache/pulsar/blob/9f40cc1d1104900c450a599676ca446b1f096a00/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentReplicator.java#L367 Here might break the message replication order. I'm working on a fix for now. We should cherry-pick this one along with the ordering issue fix PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] horizonzy commented on a diff in pull request #16590: [PIP-186] Introduce two phase deletion protocol based on system topic
horizonzy commented on code in PR #16590:
URL: https://github.com/apache/pulsar/pull/16590#discussion_r94760
##
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java:
##
@@ -2570,70 +2651,107 @@ void internalTrimLedgers(boolean isTruncate,
CompletableFuture promise) {
advanceCursorsIfNecessary(ledgersToDelete);
PositionImpl currentLastConfirmedEntry = lastConfirmedEntry;
-// Update metadata
-for (LedgerInfo ls : ledgersToDelete) {
-if (currentLastConfirmedEntry != null && ls.getLedgerId() ==
currentLastConfirmedEntry.getLedgerId()) {
-// this info is relevant because the lastMessageId won't
be available anymore
-log.info("[{}] Ledger {} contains the current last
confirmed entry {}, and it is going to be "
- + "deleted", name, ls.getLedgerId(),
currentLastConfirmedEntry);
-}
-
-invalidateReadHandle(ls.getLedgerId());
-
-ledgers.remove(ls.getLedgerId());
-NUMBER_OF_ENTRIES_UPDATER.addAndGet(this, -ls.getEntries());
-TOTAL_SIZE_UPDATER.addAndGet(this, -ls.getSize());
-entryCache.invalidateAllEntries(ls.getLedgerId());
-}
-for (LedgerInfo ls : offloadedLedgersToDelete) {
-LedgerInfo.Builder newInfoBuilder = ls.toBuilder();
-
newInfoBuilder.getOffloadContextBuilder().setBookkeeperDeleted(true);
-String driverName = OffloadUtils.getOffloadDriverName(ls,
-config.getLedgerOffloader().getOffloadDriverName());
-Map driverMetadata =
OffloadUtils.getOffloadDriverMetadata(ls,
-
config.getLedgerOffloader().getOffloadDriverMetadata());
-OffloadUtils.setOffloadDriverMetadata(newInfoBuilder,
driverName, driverMetadata);
-ledgers.put(ls.getLedgerId(), newInfoBuilder.build());
-}
+// Update metadata
+// Mark deletable ledgers
+Set deletableLedgers =
+Stream.concat(ledgersToDelete.stream().filter(ls ->
!ls.getOffloadContext().getBookkeeperDeleted()),
+
offloadedLedgersToDelete.stream()).map(LedgerInfo::getLedgerId).collect(Collectors.toSet());
+
+// Mark deletable offloaded ledgers
+Set deletableOffloadedLedgers = ledgersToDelete.stream()
+.filter(ls -> ls.getOffloadContext().hasUidMsb())
+.map(LedgerInfo::getLedgerId).collect(Collectors.toSet());
+
+CompletableFuture appendDeleteLedgerFuture =
+appendPendingDeleteLedger(deletableLedgers,
deletableOffloadedLedgers);
+appendDeleteLedgerFuture.thenAccept(ignore -> {
+believedDeleteIds.addAll(deletableLedgers);
+for (LedgerInfo ls : ledgersToDelete) {
+if (currentLastConfirmedEntry != null
+&& ls.getLedgerId() ==
currentLastConfirmedEntry.getLedgerId()) {
+// this info is relevant because the lastMessageId
won't be available anymore
+log.info("[{}] Ledger {} contains the current last
confirmed entry {}, and it is going to be "
++ "deleted", name, ls.getLedgerId(),
currentLastConfirmedEntry);
+}
+invalidateReadHandle(ls.getLedgerId());
+ledgers.remove(ls.getLedgerId());
+entryCache.invalidateAllEntries(ls.getLedgerId());
-if (log.isDebugEnabled()) {
-log.debug("[{}] Updating of ledgers list after trimming",
name);
-}
+NUMBER_OF_ENTRIES_UPDATER.addAndGet(this,
-ls.getEntries());
+TOTAL_SIZE_UPDATER.addAndGet(this, -ls.getSize());
+}
+for (LedgerInfo ls : offloadedLedgersToDelete) {
+LedgerInfo.Builder newInfoBuilder = ls.toBuilder();
+
newInfoBuilder.getOffloadContextBuilder().setBookkeeperDeleted(true);
+String driverName = OffloadUtils.getOffloadDriverName(ls,
+
config.getLedgerOffloader().getOffloadDriverName());
+Map driverMetadata =
OffloadUtils.getOffloadDriverMetadata(ls,
+
config.getLedgerOffloader().getOffloadDriverMetadata());
+OffloadUtils.setOffloadDriverMetadata(newInfoBuilder,
driverName, driverMetadata);
+ledgers.put(ls.getLedgerId(), newInfoBuilder.build());
+}
-store.asyncUpdateLedgerIds(name, getManagedLedgerInfo(),
ledgersStat, new MetaStoreCallback() {
-@Override
-public void operationComplete(Void result, Stat
[GitHub] [pulsar] codelipenghui commented on a diff in pull request #17049: [fix][broker] Fix schema does not replicate successfully
codelipenghui commented on code in PR #17049:
URL: https://github.com/apache/pulsar/pull/17049#discussion_r946998918
##
pulsar-broker/src/test/java/org/apache/pulsar/broker/service/ReplicatorTest.java:
##
@@ -1412,22 +1424,16 @@ public void testReplicatorWithFailedAck() throws
Exception {
producer1.produce(10);
-Position deletedPos = cursor.getMarkDeletedPosition();
-Position readPos = cursor.getReadPosition();
-
-Awaitility.await().timeout(30, TimeUnit.SECONDS).until(
-() -> cursor.getMarkDeletedPosition().getEntryId() !=
(cursor.getReadPosition().getEntryId() - 1));
-
-assertNotEquals((readPos.getEntryId() - 1), deletedPos.getEntryId());
Review Comment:
Removed here because the producer might be closed with pending messages.
After the cursor change to the close state, the producer will also be closed
after a read entries failure. So that all the pending messages will fail with
the `Producer Already Closed` exception, then the cursor will rewind
https://github.com/apache/pulsar/blob/9f40cc1d1104900c450a599676ca446b1f096a00/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentReplicator.java#L425
This will make the check failed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on pull request #16832: [feature][broker] Support cgroup v2 by using `jdk.internal.platform.Metrics` in Pulsar Loadbalancer
heesung-sn commented on PR #16832: URL: https://github.com/apache/pulsar/pull/16832#issuecomment-1216854483 > @heesung-sn What do you think? According to this article, https://developers.redhat.com/articles/2022/04/19/java-17-whats-new-openjdks-container-awareness#recent_changes_in_openjdk_s_container_awareness_code It seems like `OperatingSystemMXBean` already provides `cpu usage in percentage relative to cgroup (for both v1 and v2)` (I will call it as `cpu_usage_in_percent_cgroup` for our discussion) I agree with you. From my understanding, load balancer cares about `cpu_usage_in_percent_cgroup` in the end for the cpu usage computation. However, LB also requires other signals such as `memory_usage_in_percent_cgroup`, `network_usage_in_percent_cgroup`, which require separate limits for their percentage computation. As you pointed out, the problem is that LB uses the same generic code (requiring limit) to compute the resource percentage. Maybe we can tweak the code to ignore the limit and use the signal as-is if they are already in the `*_usage_in_percent_cgroup` form. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] nodece opened a new pull request, #17124: [improve][test] Verify original auth data on proxy
nodece opened a new pull request, #17124: URL: https://github.com/apache/pulsar/pull/17124 Signed-off-by: Zixuan Liu ### Motivation Add a test to verify the original auth data on the proxy. ### Documentation Check the box below or label this PR directly. Need to update docs? - [ ] `doc-required` (Your PR needs to update docs and you will update later) - [x] `doc-not-needed` (Please explain why) - [ ] `doc` (Your PR contains doc changes) - [ ] `doc-complete` (Docs have been already added) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] michaeljmarshall opened a new pull request, #17123: [fix][broker] Pass subscriptionName to auth service
michaeljmarshall opened a new pull request, #17123: URL: https://github.com/apache/pulsar/pull/17123 ### Motivation Pass subscription name to the auth service. ### Modifications * Update several methods that need to pass subscription name. ### Verifying this change This change is a trivial rework / code cleanup without any test coverage. ### Documentation - [x] `doc-not-needed` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] eolivelli commented on a diff in pull request #16590: [PIP-186] Introduce two phase deletion protocol based on system topic
eolivelli commented on code in PR #16590:
URL: https://github.com/apache/pulsar/pull/16590#discussion_r946919802
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v2/PersistentTopics.java:
##
@@ -1079,6 +1080,34 @@ public void deleteTopic(
});
}
+@POST
+@Path("/{tenant}/{namespace}/{topic}/ledger/delete")
+@ApiOperation(value = "Delete a topic.",
+notes = "The topic cannot be deleted if delete is not forcefully
and there's any active "
++ "subscription or producer connected to the it. "
++ "Force delete ignores connected clients and deletes
topic by explicitly closing them.")
+@ApiResponses(value = {
+@ApiResponse(code = 307, message = "Current broker doesn't serve
the namespace of this topic"),
+@ApiResponse(code = 401, message = "Don't have permission to
administrate resources on this tenant"),
+@ApiResponse(code = 403, message = "Don't have admin permission"),
+@ApiResponse(code = 404, message = "Topic does not exist"),
+@ApiResponse(code = 412, message = "Topic has active
producers/subscriptions"),
+@ApiResponse(code = 500, message = "Internal server error")})
+public void deleteLedger(
Review Comment:
why do we need a new API?
##
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java:
##
@@ -2570,70 +2651,107 @@ void internalTrimLedgers(boolean isTruncate,
CompletableFuture promise) {
advanceCursorsIfNecessary(ledgersToDelete);
PositionImpl currentLastConfirmedEntry = lastConfirmedEntry;
-// Update metadata
-for (LedgerInfo ls : ledgersToDelete) {
-if (currentLastConfirmedEntry != null && ls.getLedgerId() ==
currentLastConfirmedEntry.getLedgerId()) {
-// this info is relevant because the lastMessageId won't
be available anymore
-log.info("[{}] Ledger {} contains the current last
confirmed entry {}, and it is going to be "
- + "deleted", name, ls.getLedgerId(),
currentLastConfirmedEntry);
-}
-
-invalidateReadHandle(ls.getLedgerId());
-
-ledgers.remove(ls.getLedgerId());
-NUMBER_OF_ENTRIES_UPDATER.addAndGet(this, -ls.getEntries());
-TOTAL_SIZE_UPDATER.addAndGet(this, -ls.getSize());
-entryCache.invalidateAllEntries(ls.getLedgerId());
-}
-for (LedgerInfo ls : offloadedLedgersToDelete) {
-LedgerInfo.Builder newInfoBuilder = ls.toBuilder();
-
newInfoBuilder.getOffloadContextBuilder().setBookkeeperDeleted(true);
-String driverName = OffloadUtils.getOffloadDriverName(ls,
-config.getLedgerOffloader().getOffloadDriverName());
-Map driverMetadata =
OffloadUtils.getOffloadDriverMetadata(ls,
-
config.getLedgerOffloader().getOffloadDriverMetadata());
-OffloadUtils.setOffloadDriverMetadata(newInfoBuilder,
driverName, driverMetadata);
-ledgers.put(ls.getLedgerId(), newInfoBuilder.build());
-}
+// Update metadata
+// Mark deletable ledgers
+Set deletableLedgers =
+Stream.concat(ledgersToDelete.stream().filter(ls ->
!ls.getOffloadContext().getBookkeeperDeleted()),
+
offloadedLedgersToDelete.stream()).map(LedgerInfo::getLedgerId).collect(Collectors.toSet());
+
+// Mark deletable offloaded ledgers
+Set deletableOffloadedLedgers = ledgersToDelete.stream()
+.filter(ls -> ls.getOffloadContext().hasUidMsb())
+.map(LedgerInfo::getLedgerId).collect(Collectors.toSet());
+
+CompletableFuture appendDeleteLedgerFuture =
+appendPendingDeleteLedger(deletableLedgers,
deletableOffloadedLedgers);
+appendDeleteLedgerFuture.thenAccept(ignore -> {
+believedDeleteIds.addAll(deletableLedgers);
+for (LedgerInfo ls : ledgersToDelete) {
+if (currentLastConfirmedEntry != null
+&& ls.getLedgerId() ==
currentLastConfirmedEntry.getLedgerId()) {
+// this info is relevant because the lastMessageId
won't be available anymore
+log.info("[{}] Ledger {} contains the current last
confirmed entry {}, and it is going to be "
++ "deleted", name, ls.getLedgerId(),
currentLastConfirmedEntry);
+}
+invalidateReadHandle(ls.getLedgerId());
+ledgers.remove(ls.getLedgerId());
+entryCache.invalidateAllEntries(ls.getLedgerId());
-if
[GitHub] [pulsar] codelipenghui commented on issue #16770: Flaky-test: PersistentSubscriptionMessageDispatchStreamingDispatcherThrottlingTest.testRelativeMessageRateLimitingThrottling
codelipenghui commented on issue #16770: URL: https://github.com/apache/pulsar/issues/16770#issuecomment-1216788418 Another one: https://github.com/apache/pulsar/runs/7859284770?check_suite_focus=true -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-go] severinson opened a new pull request, #829: Enable seeking individual topic partitions
severinson opened a new pull request, #829: URL: https://github.com/apache/pulsar-client-go/pull/829 Currently, seeking on a consumer with a KeyShared subscription fails. This PR removes an unnecessary check to seek the underlying partitionConsumer responsible for the topic. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] eolivelli commented on a diff in pull request #16590: [PIP-186] Introduce two phase deletion protocol based on system topic
eolivelli commented on code in PR #16590:
URL: https://github.com/apache/pulsar/pull/16590#discussion_r946912131
##
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/deletion/LedgerDeletionService.java:
##
@@ -0,0 +1,117 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.bookkeeper.mledger.deletion;
+
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import org.apache.bookkeeper.mledger.proto.MLDataFormats;
+import
org.apache.bookkeeper.mledger.proto.MLDataFormats.ManagedLedgerInfo.LedgerInfo;
+import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.PulsarClientException;
+
+public interface LedgerDeletionService {
+
+/**
+ * Start.
+ */
+void start() throws PulsarClientException, PulsarAdminException;
+
+/**
+ * @param topicName topicName
+ * @param ledgerId ledgerId
+ * @param context ledgerInfo
+ * @param component managed_ledger, managed_cursor, schema_storage
+ * @param type ledger, offload_ledger
+ * @param properties properties
+ * @return
+ */
+CompletableFuture appendPendingDeleteLedger(String topicName, long
ledgerId, LedgerInfo context,
+ LedgerComponent component,
LedgerType type, Map properties);
+
+/**
+ *
+ * @param topicName topicName
+ * @param ledgerId ledgerId
+ * @param component managed_ledger, managed_cursor, schema_storage
+ * @param isBelievedDelete isBelievedDelete, if false, we should check the
param is match the ledger metadata.
+ * @return
+ */
+CompletableFuture asyncDeleteLedger(String topicName, long ledgerId,
LedgerComponent component,
+ boolean isBelievedDelete);
+
+/**
+ *
+ * @param topicName topicName
+ * @param ledgerId ledgerId
+ * @param offloadContext offloadContext
+ * @return
+ */
+CompletableFuture asyncDeleteOffloadedLedger(String topicName, long
ledgerId,
+
MLDataFormats.OffloadContext offloadContext);
+
+/**
+ * Close.
+ */
+void close() throws Exception;
+
+/**
+ * Async close.
+ */
+CompletableFuture asyncClose();
+
+class LedgerDeletionServiceDisable implements LedgerDeletionService {
+
+@Override
+public void start() {
+//No op
+}
+
+private static final CompletableFuture COMPLETABLE_FUTURE =
CompletableFuture.completedFuture(null);
Review Comment:
you cannot cache CompletableFutures
the behaviour will be unpredictable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] aloyszhang opened a new pull request, #17122: [proxy] remove useless code
aloyszhang opened a new pull request, #17122: URL: https://github.com/apache/pulsar/pull/17122 ### Motivation Minor update for proxy `metadataStoreUrl` configuration ### Modifications remove useless code ### Documentation - [ ] `doc-not-needed` (Please explain why) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] alpreu opened a new issue, #17121: [Bug][pulsar-client-admin-api] Topics.getInternalInfo() throws 500 instead of 404 if topic does not exist
alpreu opened a new issue, #17121:
URL: https://github.com/apache/pulsar/issues/17121
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues)
and found nothing similar.
### Version
macOS, master branch, commit bc68f02a
### Minimal reproduce step
1) Create a PulsarAdmin object
2) Use `admin.topics().getInternalInfo("this-topic-does-not-exist")
### What did you expect to see?
Response is HTTP 404 Not Found (as described in the docs)
### What did you see instead?
Response is HTTP 500 Internal Server Error
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] HQebupt commented on pull request #16756: [fix][broker]optimize the shutdown sequence of broker service when it close
HQebupt commented on PR #16756: URL: https://github.com/apache/pulsar/pull/16756#issuecomment-1216761089 > > > If the EventLoop is closed first, the client reconnection will fail. > > > > > > The reconnection won't retry to other brokers? @Technoboy- > > Bundle is not unloaded at the moment. True. Closing protocol handler first is reasonable because the outside service maybe depends on broker service. How about it ? @Technoboy- @poorbarcode -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] HQebupt commented on a diff in pull request #16756: [fix][broker]optimize the shutdown sequence of broker service when it close
HQebupt commented on code in PR #16756:
URL: https://github.com/apache/pulsar/pull/16756#discussion_r946890686
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java:
##
@@ -731,6 +731,13 @@ public CompletableFuture
closeAndRemoveReplicationClient(String clusterNam
public CompletableFuture closeAsync() {
try {
log.info("Shutting down Pulsar Broker service");
+log.info("Event loops shutting down gracefully...");
+List> shutdownEventLoops = new ArrayList<>();
+shutdownEventLoops.add(shutdownEventLoopGracefully(acceptorGroup));
+shutdownEventLoops.add(shutdownEventLoopGracefully(workerGroup));
Review Comment:
True. Closing protocol handler first is reasonable because the protocol
handler service maybe depends on broker service. How about it ? @Technoboy-
https://user-images.githubusercontent.com/4970972/184911740-760e9eb3-563c-4732-9dcd-11eaccc4b996.png;>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[pulsar] branch branch-2.10 updated: [improve][proxy] Consolidate Netty channel flushes to mitigate syscall overhead (#16372)
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-2.10 by this push:
new ba15416c126 [improve][proxy] Consolidate Netty channel flushes to
mitigate syscall overhead (#16372)
ba15416c126 is described below
commit ba15416c126a178e2569dbead7bcaf41c777817b
Author: lipenghui
AuthorDate: Tue Jul 5 14:29:04 2022 +0800
[improve][proxy] Consolidate Netty channel flushes to mitigate syscall
overhead (#16372)
### Motivation
Follow change for https://github.com/apache/pulsar/pull/16361 which
commented at https://github.com/apache/pulsar/pull/16361#issuecomment-1173241406
(cherry picked from commit 10db821e8e369efd6cba05eabda0e41ef346cab5)
---
.../main/java/org/apache/pulsar/proxy/server/DirectProxyHandler.java | 3 +++
.../java/org/apache/pulsar/proxy/server/ServiceChannelInitializer.java | 3 +++
2 files changed, 6 insertions(+)
diff --git
a/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/DirectProxyHandler.java
b/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/DirectProxyHandler.java
index 9edb16a5e65..ca9e6cdb2ae 100644
---
a/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/DirectProxyHandler.java
+++
b/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/DirectProxyHandler.java
@@ -37,6 +37,7 @@ import io.netty.handler.codec.haproxy.HAProxyCommand;
import io.netty.handler.codec.haproxy.HAProxyMessage;
import io.netty.handler.codec.haproxy.HAProxyProtocolVersion;
import io.netty.handler.codec.haproxy.HAProxyProxiedProtocol;
+import io.netty.handler.flush.FlushConsolidationHandler;
import io.netty.handler.ssl.SslContext;
import io.netty.handler.ssl.SslHandler;
import io.netty.handler.ssl.SslProvider;
@@ -178,6 +179,8 @@ public class DirectProxyHandler {
b.handler(new ChannelInitializer() {
@Override
protected void initChannel(SocketChannel ch) {
+ch.pipeline().addLast("consolidation", new
FlushConsolidationHandler(1024,
+true));
if (tlsEnabledWithBroker) {
String host = targetBrokerAddress.getHostString();
int port = targetBrokerAddress.getPort();
diff --git
a/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/ServiceChannelInitializer.java
b/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/ServiceChannelInitializer.java
index 2fd0156155d..fc7c78d6a24 100644
---
a/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/ServiceChannelInitializer.java
+++
b/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/ServiceChannelInitializer.java
@@ -21,6 +21,7 @@ package org.apache.pulsar.proxy.server;
import io.netty.channel.ChannelInitializer;
import io.netty.channel.socket.SocketChannel;
import io.netty.handler.codec.LengthFieldBasedFrameDecoder;
+import io.netty.handler.flush.FlushConsolidationHandler;
import io.netty.handler.ssl.SslContext;
import io.netty.handler.ssl.SslHandler;
import io.netty.handler.ssl.SslProvider;
@@ -91,6 +92,8 @@ public class ServiceChannelInitializer extends
ChannelInitializer
@Override
protected void initChannel(SocketChannel ch) throws Exception {
+ch.pipeline().addLast("consolidation", new
FlushConsolidationHandler(1024,
+true));
if (serverSslCtxRefresher != null && this.enableTls) {
SslContext sslContext = serverSslCtxRefresher.get();
if (sslContext != null) {
[GitHub] [pulsar] github-actions[bot] commented on pull request #17118: [fix][functions] Make mandatory to provide a schema in Context::newOutputRecordBuilder
github-actions[bot] commented on PR #17118: URL: https://github.com/apache/pulsar/pull/17118#issuecomment-1216687198 @cbornet Please provide a correct documentation label for your PR. Instructions see [Pulsar Documentation Label Guide](https://docs.google.com/document/d/1Qw7LHQdXWBW9t2-r-A7QdFDBwmZh6ytB4guwMoXHqc0). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on issue #17111: Flaky-test: AdminApi2Test.testDeleteNamespace
poorbarcode commented on issue #17111: URL: https://github.com/apache/pulsar/issues/17111#issuecomment-1216678124 assign to me -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] coderzc commented on a diff in pull request #17092: [improve][broker]: DelayedDeliveryTracker init and addMessage
coderzc commented on code in PR #17092:
URL: https://github.com/apache/pulsar/pull/17092#discussion_r946806761
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/delayed/InMemoryDelayedDeliveryTracker.java:
##
@@ -99,7 +100,7 @@ private long getCutoffTime() {
}
@Override
-public boolean addMessage(long ledgerId, long entryId, long deliverAt) {
+public synchronized boolean addMessage(long ledgerId, long entryId, long
deliverAt) {
Review Comment:
I think it is best to use the same lock as `getScheduledMessages`. Keep use
`synchronized(dispatcher)`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] codelipenghui opened a new issue, #17120: Flaky-test: ResourceGroupUsageAggregationTest.testProduceConsumeUsageOnRG
codelipenghui opened a new issue, #17120: URL: https://github.com/apache/pulsar/issues/17120 ### Search before asking - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar. ### Example failure https://github.com/apache/pulsar/runs/7856747228?check_suite_focus=true ### Exception stacktrace ``` Error: Tests run: 4, Failures: 1, Errors: 0, Skipped: 3, Time elapsed: 7.957 s <<< FAILURE! - in org.apache.pulsar.broker.resourcegroup.ResourceGroupUsageAggregationTest Error: testProduceConsumeUsageOnRG(org.apache.pulsar.broker.resourcegroup.ResourceGroupUsageAggregationTest) Time elapsed: 0.194 s <<< FAILURE! java.lang.AssertionError: expected [7] but found [10] at org.testng.Assert.fail(Assert.java:99) at org.testng.Assert.failNotEquals(Assert.java:1037) at org.testng.Assert.assertEqualsImpl(Assert.java:140) at org.testng.Assert.assertEquals(Assert.java:122) at org.testng.Assert.assertEquals(Assert.java:797) at org.testng.Assert.assertEquals(Assert.java:807) at org.apache.pulsar.broker.resourcegroup.ResourceGroupUsageAggregationTest.verifyStats(ResourceGroupUsageAggregationTest.java:205) at org.apache.pulsar.broker.resourcegroup.ResourceGroupUsageAggregationTest.testProduceConsumeUsageOnRG(ResourceGroupUsageAggregationTest.java:174) at org.apache.pulsar.broker.resourcegroup.ResourceGroupUsageAggregationTest.testProduceConsumeUsageOnRG(ResourceGroupUsageAggregationTest.java:82) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132) at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45) at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73) at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) ``` ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] Technoboy- commented on a diff in pull request #15955: PIP-174: New managed ledger entry cache implementation
Technoboy- commented on code in PR #15955:
URL: https://github.com/apache/pulsar/pull/15955#discussion_r946799085
##
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/cache/SharedCacheSegmentBufferCopy.java:
##
@@ -0,0 +1,99 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.bookkeeper.mledger.impl.cache;
+
+import io.netty.buffer.ByteBuf;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.pulsar.common.allocator.PulsarByteBufAllocator;
+import org.apache.pulsar.common.util.collections.ConcurrentLongLongPairHashMap;
+
+class SharedCacheSegmentBufferCopy implements AutoCloseable,
SharedCacheSegment {
+
+private final ByteBuf cacheBuffer;
+private final AtomicInteger currentOffset = new AtomicInteger();
+private final ConcurrentLongLongPairHashMap index;
+private final int segmentSize;
+
+private static final int ALIGN_64_MASK = ~(64 - 1);
+
+SharedCacheSegmentBufferCopy(int segmentSize) {
+this.segmentSize = segmentSize;
+this.cacheBuffer = PulsarByteBufAllocator.DEFAULT.buffer(segmentSize,
segmentSize);
+this.cacheBuffer.writerIndex(segmentSize - 1);
+this.index = ConcurrentLongLongPairHashMap.newBuilder()
+// We are going to often clear() the map, with the expectation
that it's going to get filled again
+// immediately after. In these conditions it does not make
sense to shrink it each time.
+.autoShrink(false)
+.concurrencyLevel(Runtime.getRuntime().availableProcessors() *
8)
+.build();
+}
+
+@Override
+public boolean insert(long ledgerId, long entryId, ByteBuf entry) {
+int entrySize = entry.readableBytes();
+int alignedSize = align64(entrySize);
+int offset = currentOffset.getAndAdd(alignedSize);
+
+if (offset + entrySize > segmentSize) {
+// The segment is full
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] Technoboy- commented on a diff in pull request #15955: PIP-174: New managed ledger entry cache implementation
Technoboy- commented on code in PR #15955:
URL: https://github.com/apache/pulsar/pull/15955#discussion_r946798736
##
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/cache/SharedCacheSegmentBufferRefCount.java:
##
@@ -0,0 +1,91 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.bookkeeper.mledger.impl.cache;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.util.IllegalReferenceCountException;
+import java.util.concurrent.atomic.AtomicInteger;
+import
org.apache.pulsar.common.util.collections.ConcurrentLongPairObjectHashMap;
+
+class SharedCacheSegmentBufferRefCount implements SharedCacheSegment {
+
+private final AtomicInteger currentSize = new AtomicInteger();
+private final ConcurrentLongPairObjectHashMap index;
+private final int segmentSize;
+
+SharedCacheSegmentBufferRefCount(int segmentSize) {
+this.segmentSize = segmentSize;
+this.index = ConcurrentLongPairObjectHashMap.newBuilder()
+// We are going to often clear() the map, with the expectation
that it's going to get filled again
+// immediately after. In these conditions it does not make
sense to shrink it each time.
+.autoShrink(false)
+.concurrencyLevel(Runtime.getRuntime().availableProcessors() *
2)
+.build();
+}
+
+@Override
+public boolean insert(long ledgerId, long entryId, ByteBuf entry) {
+int newSize = currentSize.addAndGet(entry.readableBytes());
+
+if (newSize > segmentSize) {
+// The segment is full
+return false;
+} else {
+// Insert entry into read cache segment
+ByteBuf oldValue = index.putIfAbsent(ledgerId, entryId,
entry.retain());
+if (oldValue != null) {
+entry.release();
+return false;
+} else {
+return true;
+}
+}
+}
+
+@Override
+public ByteBuf get(long ledgerId, long entryId) {
+ByteBuf entry = index.get(ledgerId, entryId);
+if (entry != null) {
+try {
+return entry.retain();
+} catch (IllegalReferenceCountException e) {
+// Entry was removed between the get() and the retain() calls
+return null;
+}
+} else {
+return null;
+}
+}
+
+@Override
+public int getSize() {
+return currentSize.get();
+}
+
+@Override
+public void close() {
+clear();
+}
+
+@Override
+public void clear() {
+index.forEach((ledgerId, entryId, e) -> e.release());
+index.clear();
Review Comment:
fixed
##
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/cache/SharedCacheSegmentBufferRefCount.java:
##
@@ -0,0 +1,91 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.bookkeeper.mledger.impl.cache;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.util.IllegalReferenceCountException;
+import java.util.concurrent.atomic.AtomicInteger;
+import
org.apache.pulsar.common.util.collections.ConcurrentLongPairObjectHashMap;
+
+class SharedCacheSegmentBufferRefCount implements SharedCacheSegment {
+
+private final AtomicInteger currentSize = new AtomicInteger();
+private final ConcurrentLongPairObjectHashMap index;
+private
[GitHub] [pulsar] codelipenghui commented on issue #16516: Flaky-test: BrokerInterceptorTest.testProducerCreation
codelipenghui commented on issue #16516: URL: https://github.com/apache/pulsar/issues/16516#issuecomment-1216656084 Another one: https://github.com/apache/pulsar/runs/7850960846?check_suite_focus=true -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] Anonymitaet opened a new issue, #17119: [Doc] add explanations of "topics can be compacted on topic level automatically"
Anonymitaet opened a new issue, #17119: URL: https://github.com/apache/pulsar/issues/17119 ### Search before asking - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar. ### What issue do you find in Pulsar docs? Alexander Preuss: The docs say that we can only automatically compact topics on a namespace level [(source)](https://pulsar.apache.org/docs/next/cookbooks-compaction/#configuring-compaction-to-run-automatically). Is this correct and is there a reason we cannot auto-compact on an individual topic level? @codelipenghui: Also supported in topic level, but need to enable topic policies. After 2.11.0, topic policies will be enabled by default. We should update the document. ### What is your suggestion? Add explanations as suggested by @codelipenghui ### Any reference? _No response_ ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #15764: Flaky-test: ConnectionTimeoutTest.testLowTimeout
codelipenghui commented on issue #15764: URL: https://github.com/apache/pulsar/issues/15764#issuecomment-1216651802 The issue doesn't fix yet, https://github.com/apache/pulsar/runs/7856446588?check_suite_focus=true ``` Error: Tests run: 2, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 7.226 s <<< FAILURE! - in org.apache.pulsar.client.impl.ConnectionTimeoutTest Error: testLowTimeout(org.apache.pulsar.client.impl.ConnectionTimeoutTest) Time elapsed: 3.055 s <<< FAILURE! java.lang.AssertionError: expected [class io.netty.channel.ConnectTimeoutException] but found [class io.netty.channel.StacklessClosedChannelException] at org.testng.Assert.fail(Assert.java:99) at org.testng.Assert.failNotEquals(Assert.java:1037) at org.testng.Assert.assertEqualsImpl(Assert.java:140) at org.testng.Assert.assertEquals(Assert.java:122) at org.testng.Assert.assertEquals(Assert.java:617) at org.apache.pulsar.client.impl.ConnectionTimeoutTest.testLowTimeout(ConnectionTimeoutTest.java:76) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132) at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45) at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73) at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] michaeljmarshall opened a new issue, #15764: Flaky-test: ConnectionTimeoutTest.testLowTimeout
michaeljmarshall opened a new issue, #15764: URL: https://github.com/apache/pulsar/issues/15764 ConnectionTimeoutTest.testLowTimeout is flaky. It fails sporadically. https://github.com/apache/pulsar/runs/6582172714 ``` Error: testLowTimeout(org.apache.pulsar.client.impl.ConnectionTimeoutTest) Time elapsed: 3.042 s <<< FAILURE! java.lang.AssertionError: expected [class io.netty.channel.ConnectTimeoutException] but found [class io.netty.channel.StacklessClosedChannelException] at org.testng.Assert.fail(Assert.java:99) at org.testng.Assert.failNotEquals(Assert.java:1037) at org.testng.Assert.assertEqualsImpl(Assert.java:140) at org.testng.Assert.assertEquals(Assert.java:122) at org.testng.Assert.assertEquals(Assert.java:617) at org.apache.pulsar.client.impl.ConnectionTimeoutTest.testLowTimeout(ConnectionTimeoutTest.java:44) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132) at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45) at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73) at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #16516: Flaky-test: BrokerInterceptorTest.testProducerCreation
codelipenghui commented on issue #16516: URL: https://github.com/apache/pulsar/issues/16516#issuecomment-1216649215 #16742 doesn't fix the issue yet, https://github.com/apache/pulsar/runs/7856446693?check_suite_focus=true ``` Error: Tests run: 18, Failures: 3, Errors: 0, Skipped: 9, Time elapsed: 120.353 s <<< FAILURE! - in org.apache.pulsar.broker.intercept.BrokerInterceptorTest Error: testProducerCreation(org.apache.pulsar.broker.intercept.BrokerInterceptorTest) Time elapsed: 0.151 s <<< FAILURE! java.lang.AssertionError: expected [1] but found [0] at org.testng.Assert.fail(Assert.java:99) at org.testng.Assert.failNotEquals(Assert.java:1037) at org.testng.Assert.assertEqualsImpl(Assert.java:140) at org.testng.Assert.assertEquals(Assert.java:122) at org.testng.Assert.assertEquals(Assert.java:907) at org.testng.Assert.assertEquals(Assert.java:917) at org.apache.pulsar.broker.intercept.BrokerInterceptorTest.testProducerCreation(BrokerInterceptorTest.java:145) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132) at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45) at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73) at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui opened a new issue, #16516: Flaky-test: BrokerInterceptorTest.testProducerCreation
codelipenghui opened a new issue, #16516: URL: https://github.com/apache/pulsar/issues/16516 [example failure](https://github.com/apache/pulsar/runs/7274700601?check_suite_focus=true) ``` Error: Tests run: 18, Failures: 3, Errors: 0, Skipped: 9, Time elapsed: 128.822 s <<< FAILURE! - in org.apache.pulsar.broker.intercept.BrokerInterceptorTest Error: testProducerCreation(org.apache.pulsar.broker.intercept.BrokerInterceptorTest) Time elapsed: 0.247 s <<< FAILURE! java.lang.AssertionError: expected [true] but found [false] at org.testng.Assert.fail(Assert.java:99) at org.testng.Assert.failNotEquals(Assert.java:1037) at org.testng.Assert.assertTrue(Assert.java:45) at org.testng.Assert.assertTrue(Assert.java:55) at org.apache.pulsar.broker.intercept.BrokerInterceptorTest.testProducerCreation(BrokerInterceptorTest.java:145) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132) at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45) at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73) at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) ```
[GitHub] [pulsar] codelipenghui commented on issue #16795: Flaky-test: SimpleProducerConsumerTest.testSharedSamePriorityConsumer
codelipenghui commented on issue #16795: URL: https://github.com/apache/pulsar/issues/16795#issuecomment-1216645459 Another one: https://github.com/apache/pulsar/runs/7856447331?check_suite_focus=true -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] cbornet opened a new pull request, #17118: [fix][functions] Make mandatory to provide a schema in Context::newOutputRecordBuilder
cbornet opened a new pull request, #17118: URL: https://github.com/apache/pulsar/pull/17118 Fixes #17114 ### Modifications Change `newOutputRecordBuilder` to `newOutputRecordBuilder(Schema schema)` ### Verifying this change - [ ] Make sure that the change passes the CI checks. This change is already covered by existing tests, such as ContextImplTest and PulsarFunctionsJavaTest::testRecordFunctionTest ### Does this pull request potentially affect one of the following parts: *If `yes` was chosen, please highlight the changes* - The public API: yes ### Documentation Check the box below or label this PR directly. Need to update docs? - [ ] `doc-required` (Your PR needs to update docs and you will update later) - [x] `doc-not-needed` fix - [ ] `doc` (Your PR contains doc changes) - [ ] `doc-complete` (Docs have been already added) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] nicoloboschi opened a new pull request, #17117: [feat][connector] ElasticSearch Sink: add a option to copy key fields into the value
nicoloboschi opened a new pull request, #17117: URL: https://github.com/apache/pulsar/pull/17117 ### Motivation Add the ability to add the key fields to the document. With this option the resulting document will contain both key and value fields if schema is enabled and the message is JSON/Avro format. ### Modifications * New option `copyKeyFields` default to `false`. If true, the key fields will be copied to the document. In order to be activated it's required to: - set `copyKeyFields` to true - set `schemaEnable` to true - the message format be in JSON or AVRO - message value is not null - [x] `doc` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on a diff in pull request #16756: [fix][broker]optimize the shutdown sequence of broker service when it close
poorbarcode commented on code in PR #16756:
URL: https://github.com/apache/pulsar/pull/16756#discussion_r946722398
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java:
##
@@ -731,6 +731,13 @@ public CompletableFuture
closeAndRemoveReplicationClient(String clusterNam
public CompletableFuture closeAsync() {
try {
log.info("Shutting down Pulsar Broker service");
+log.info("Event loops shutting down gracefully...");
+List> shutdownEventLoops = new ArrayList<>();
+shutdownEventLoops.add(shutdownEventLoopGracefully(acceptorGroup));
+shutdownEventLoops.add(shutdownEventLoopGracefully(workerGroup));
Review Comment:
Other close tasks rely on EventLoop. E.g.
`unloadNamespaceBundlesGracefully`, I don't recommend this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] mattisonchao closed pull request #15811: [fix][broker] Avoid using `thenAccept` invoke async method.
mattisonchao closed pull request #15811: [fix][broker] Avoid using `thenAccept` invoke async method. URL: https://github.com/apache/pulsar/pull/15811 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] mattisonchao opened a new pull request, #15811: [fix][broker] Avoid using `thenAccept` invoke async method.
mattisonchao opened a new pull request, #15811: URL: https://github.com/apache/pulsar/pull/15811 ### Motivation Avoid invoking async methods with `thenAccept`. It will ignore the exception and do not waiting for the inner `CompletableFuture`. ### Modifications - Use `thenCompose` to instead of `thenAccept` ### Verifying this change - [x] Make sure that the change passes the CI checks. ### Documentation - [x] `doc-not-needed` (Please explain why) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] tisonkun commented on pull request #15767: [pulsar-broker] Change dispatcher blocked/unblocked log level to debug
tisonkun commented on PR #15767: URL: https://github.com/apache/pulsar/pull/15767#issuecomment-1216567234 @Shawyeok Thanks for your contribution! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] poorbarcode opened a new pull request, #17116: [fix][flaky-test]ConsumedLedgersTrimTest
poorbarcode opened a new pull request, #17116: URL: https://github.com/apache/pulsar/pull/17116 Fixes - #11145 - #10380 - #17044 Master Issue: #11145 #10380 #17044 ### Motivation https://github.com/apache/pulsar/blob/4d7b1acff35a1a131ffeff34e8cec6007f6a2ec9/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/ConsumedLedgersTrimTest.java#L145-L150 In addition to calling `internalTrimLedgers` directly in this unit test, the Broker also has scheduled tasks `consumedLedgersMonitor` calling `internalTrimLedgers`. (High light)See the code above, If `consumedLedgersMonitor` call `internalTrimLedgers` before `line:148`: the command `topics.getLastMessageId` will return `-1:-1:-1`, then the #11145 #10380 occurs. https://github.com/apache/pulsar/blob/4d7b1acff35a1a131ffeff34e8cec6007f6a2ec9/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/ConsumedLedgersTrimTest.java#L152-L161 (High light)See the code above, If `consumedLedgersMonitor` call `internalTrimLedgers` before `line:161`: the command `managedLedger.getLedgersInfoAsList().size()` will return `1`, then the #17044 occurs. ### Modifications - Fix flaky test: Make `internalTrimLedgers` never trigger by `consumedLedgersMonitor` before `consumedLedgersMonitor` is triggered manually: disabled policy retention. https://github.com/apache/pulsar/blob/4d7b1acff35a1a131ffeff34e8cec6007f6a2ec9/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/ConsumedLedgersTrimTest.java#L94-L99 - Try to solve another hidden Falky test: See the code above, I think `line:99` may be flaky, because we can't guarantee that `consumedLedgersMonitor --> internalTrimLedgers` will be executed later than `line:99`, so delete `line:99` ### Documentation - [ ] `doc-required` (Your PR needs to update docs and you will update later) - [x] `doc-not-needed` (Please explain why) - [ ] `doc` (Your PR contains doc changes) - [ ] `doc-complete` (Docs have been already added) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] GuoHaoZai commented on pull request #17092: [improve][broker]: DelayedDeliveryTracker init and addMessage
GuoHaoZai commented on PR #17092: URL: https://github.com/apache/pulsar/pull/17092#issuecomment-1216559200 > modify Motivation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] leizhiyuan commented on issue #17115: [Bug] the retention policies of health topic
leizhiyuan commented on issue #17115:
URL: https://github.com/apache/pulsar/issues/17115#issuecomment-1216556275
addition
stats
```
{
"entriesAddedCounter" : 75477,
"numberOfEntries" : 77,
"totalSize" : 8316,
"currentLedgerEntries" : 77,
"currentLedgerSize" : 8316,
"lastLedgerCreatedTimestamp" : "2022-08-16T18:18:07.747+08:00",
"lastLedgerCreationFailureTimestamp" : "2022-06-09T21:34:32.892+08:00",
"waitingCursorsCount" : 1,
"pendingAddEntriesCount" : 0,
"lastConfirmedEntry" : "12461620:76",
"state" : "LedgerOpened",
"ledgers" : [ {
"ledgerId" : 12461620,
"entries" : 0,
"size" : 0,
"offloaded" : false,
"underReplicated" : false
} ],
"cursors" : { },
"schemaLedgers" : [ {
"ledgerId" : 151979,
"entries" : 1,
"size" : 117,
"offloaded" : false,
"underReplicated" : false
} ],
"compactedLedger" : {
"ledgerId" : -1,
"entries" : -1,
"size" : -1,
"offloaded" : false,
"underReplicated" : false
}
}
```
internal-stats
```
{
"msgRateIn" : 0.0,
"msgThroughputIn" : 0.0,
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"bytesInCounter" : 8148350,
"msgInCounter" : 75585,
"bytesOutCounter" : 0,
"msgOutCounter" : 0,
"averageMsgSize" : 0.0,
"msgChunkPublished" : false,
"storageSize" : 8316,
"backlogSize" : 0,
"offloadedStorageSize" : 0,
"publishers" : [ ],
"subscriptions" : { },
"replication" : { },
"deduplicationStatus" : "Disabled",
"nonContiguousDeletedMessagesRanges" : 0,
"nonContiguousDeletedMessagesRangesSerializedSize" : 0,
"publishRateLimitedTimes" : 0
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
