[pulsar] branch master updated (23e90d65244 -> 889966c6a20)
This is an automated email from the ASF dual-hosted git repository. lhotari pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git from 23e90d65244 [improve][fn] Support SubscriptionPosition in python instance (#18431) add 889966c6a20 [fix][broker] Fix possible race condition in completing topic list watcher (#18624) No new revisions were added by this update. Summary of changes: .../java/org/apache/pulsar/broker/service/TopicListService.java | 6 +- 1 file changed, 5 insertions(+), 1 deletion(-)
[GitHub] [pulsar] lhotari merged pull request #18624: [fix][broker] Fix possible race condition in completing topic list watcher
lhotari merged PR #18624: URL: https://github.com/apache/pulsar/pull/18624 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] lhotari opened a new pull request, #18628: [fix][test] Fix flaky NamespacesTest.testSetOffloadThreshold
lhotari opened a new pull request, #18628: URL: https://github.com/apache/pulsar/pull/18628 Fixes #18588 ### Motivation NamespacesTest.testSetOffloadThreshold is flaky. ### Modifications - Use awaitility to poll for changes (which are asynchronously applied in Pulsar, there is no strong consistency) - Drop the validation for default values since that was ignored any ways. ### Documentation - [ ] `doc` - [ ] `doc-required` - [x] `doc-not-needed` - [ ] `doc-complete` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codecov-commenter commented on pull request #18621: [fix][monitor] Fix reporting pulsar_subscription_blocked_on_unacked_messages metric
codecov-commenter commented on PR #18621: URL: https://github.com/apache/pulsar/pull/18621#issuecomment-1327124291 # [Codecov](https://codecov.io/gh/apache/pulsar/pull/18621?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#18621](https://codecov.io/gh/apache/pulsar/pull/18621?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (3a8802d) into [master](https://codecov.io/gh/apache/pulsar/commit/cd85a67bd3a880732678b0dfd5f7769ad90a90ef?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (cd85a67) will **decrease** coverage by `10.41%`. > The diff coverage is `14.28%`. [](https://codecov.io/gh/apache/pulsar/pull/18621?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #18621 +/- ## = - Coverage 47.39% 36.98% -10.42% + Complexity10479 1962 -8517 = Files 698 209 -489 Lines 6807014420-53650 Branches 7279 1573 -5706 = - Hits 32264 5333-26931 + Misses32228 8506-23722 + Partials 3578 581 -2997 ``` | Flag | Coverage Δ | | |---|---|---| | unittests | `36.98% <14.28%> (-10.42%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/pulsar/pull/18621?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...pache/pulsar/client/impl/TableViewBuilderImpl.java](https://codecov.io/gh/apache/pulsar/pull/18621/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL1RhYmxlVmlld0J1aWxkZXJJbXBsLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...a/org/apache/pulsar/client/impl/TableViewImpl.java](https://codecov.io/gh/apache/pulsar/pull/18621/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL1RhYmxlVmlld0ltcGwuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...pulsar/client/impl/TableViewConfigurationData.java](https://codecov.io/gh/apache/pulsar/pull/18621/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL1RhYmxlVmlld0NvbmZpZ3VyYXRpb25EYXRhLmphdmE=) | `33.33% <50.00%> (+33.33%)` | :arrow_up: | | [...apache/pulsar/client/impl/AutoClusterFailover.java](https://codecov.io/gh/apache/pulsar/pull/18621/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL0F1dG9DbHVzdGVyRmFpbG92ZXIuamF2YQ==) | `70.00% <0.00%> (-6.12%)` | :arrow_down: | | [...he/pulsar/client/impl/PartitionedProducerImpl.java](https://codecov.io/gh/apache/pulsar/pull/18621/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL1BhcnRpdGlvbmVkUHJvZHVjZXJJbXBsLmphdmE=) | `30.34% <0.00%> (-5.13%)` | :arrow_down: | | [.../pulsar/client/impl/ControlledClusterFailover.java](https://codecov.io/gh/apache/pulsar/pull/18621/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL0NvbnRyb2xsZWRDbHVzdGVyRmFpbG92ZXIuamF2YQ==) | `58.87% <0.00%> (-3.74%)` | :arrow_down: | |
[GitHub] [pulsar] BewareMyPower commented on a diff in pull request #18625: [improve][broker] Reduce object creation in TopicResources.handleNotification
BewareMyPower commented on code in PR #18625:
URL: https://github.com/apache/pulsar/pull/18625#discussion_r1032077333
##
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/resources/TopicResources.java:
##
@@ -127,10 +125,14 @@ public CompletableFuture
clearTenantPersistence(String tenant) {
}
void handleNotification(Notification notification) {
+if (topicListeners.isEmpty()) {
+return;
+}
if (notification.getPath().startsWith(MANAGED_LEDGER_PATH)
Review Comment:
I think we can merge the early return guard into the `if` block like:
```java
if (!topicListeners.isEmpty()
&& notification.getPath().startsWith(MANAGED_LEDGER_PATH)
&& (notification.getType() == NotificationType.Created
|| notification.getType() == NotificationType.Deleted)) {
```
It will be more consistent. Or you can use a merged guard like:
```java
if (topicListeners.isEmpty()
|| !notification.getPath().startsWith(MANAGED_LEDGER_PATH)
|| (notification.getType() != NotificationType.Created
&& notification.getType() != NotificationType.Deleted)) {
return;
}
```
But it will bring more code changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] jiangpengcheng opened a new issue, #18627: expose pulsar-client in python instance
jiangpengcheng opened a new issue, #18627: URL: https://github.com/apache/pulsar/issues/18627 ### Search before asking - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar. ### Motivation since java instance has exposed it, it's better to expose it for python too ### Solution _No response_ ### Alternatives _No response_ ### Anything else? _No response_ ### Are you willing to submit a PR? - [X] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] jiangpengcheng commented on pull request #18432: [improve][fn] Support schema in python instance
jiangpengcheng commented on PR #18432: URL: https://github.com/apache/pulsar/pull/18432#issuecomment-1327098797 /pulsarbot rerun-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] jiangpengcheng opened a new issue, #18626: Support e2e in python function
jiangpengcheng opened a new issue, #18626: URL: https://github.com/apache/pulsar/issues/18626 ### Search before asking - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar. ### Motivation python instance doesn't support e2e cryption ### Solution _No response_ ### Alternatives _No response_ ### Anything else? _No response_ ### Are you willing to submit a PR? - [X] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] nodece commented on issue #17933: Flaky-test: V1_ProxyAuthenticationTest.anonymousSocketTest
nodece commented on issue #17933: URL: https://github.com/apache/pulsar/issues/17933#issuecomment-1327082491 I tried to fix this test but still is a flaky test. Do you have any ideas? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codecov-commenter commented on pull request #18625: [improve][broker] Reduce object creation in TopicResources.handleNotification
codecov-commenter commented on PR #18625: URL: https://github.com/apache/pulsar/pull/18625#issuecomment-1327081362 # [Codecov](https://codecov.io/gh/apache/pulsar/pull/18625?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#18625](https://codecov.io/gh/apache/pulsar/pull/18625?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (ee8849c) into [master](https://codecov.io/gh/apache/pulsar/commit/cd85a67bd3a880732678b0dfd5f7769ad90a90ef?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (cd85a67) will **decrease** coverage by `10.26%`. > The diff coverage is `14.28%`. [](https://codecov.io/gh/apache/pulsar/pull/18625?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #18625 +/- ## = - Coverage 47.39% 37.13% -10.27% + Complexity10479 1971 -8508 = Files 698 209 -489 Lines 6807014420-53650 Branches 7279 1573 -5706 = - Hits 32264 5355-26909 + Misses32228 8479-23749 + Partials 3578 586 -2992 ``` | Flag | Coverage Δ | | |---|---|---| | unittests | `37.13% <14.28%> (-10.27%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/pulsar/pull/18625?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...pache/pulsar/client/impl/TableViewBuilderImpl.java](https://codecov.io/gh/apache/pulsar/pull/18625/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL1RhYmxlVmlld0J1aWxkZXJJbXBsLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...a/org/apache/pulsar/client/impl/TableViewImpl.java](https://codecov.io/gh/apache/pulsar/pull/18625/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL1RhYmxlVmlld0ltcGwuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...pulsar/client/impl/TableViewConfigurationData.java](https://codecov.io/gh/apache/pulsar/pull/18625/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL1RhYmxlVmlld0NvbmZpZ3VyYXRpb25EYXRhLmphdmE=) | `33.33% <50.00%> (+33.33%)` | :arrow_up: | | [...apache/pulsar/client/impl/AutoClusterFailover.java](https://codecov.io/gh/apache/pulsar/pull/18625/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL0F1dG9DbHVzdGVyRmFpbG92ZXIuamF2YQ==) | `70.00% <0.00%> (-6.12%)` | :arrow_down: | | [.../pulsar/client/impl/ControlledClusterFailover.java](https://codecov.io/gh/apache/pulsar/pull/18625/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL0NvbnRyb2xsZWRDbHVzdGVyRmFpbG92ZXIuamF2YQ==) | `58.87% <0.00%> (-3.74%)` | :arrow_down: | | [...pache/pulsar/broker/admin/v2/PersistentTopics.java](https://codecov.io/gh/apache/pulsar/pull/18625/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9hZG1pbi92Mi9QZXJzaXN0ZW50VG9waWNzLmphdmE=) | | | |
[GitHub] [pulsar] lhotari commented on issue #17933: Flaky-test: V1_ProxyAuthenticationTest.anonymousSocketTest
lhotari commented on issue #17933: URL: https://github.com/apache/pulsar/issues/17933#issuecomment-1327077407 @nodece would you like to try to address this flaky test? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] lhotari commented on issue #17933: Flaky-test: V1_ProxyAuthenticationTest.anonymousSocketTest
lhotari commented on issue #17933: URL: https://github.com/apache/pulsar/issues/17933#issuecomment-1327077255 Reopening. Problem persists: https://github.com/apache/pulsar/actions/runs/3545605005/jobs/5953957450#step:10:719 ``` Error: Tests run: 36, Failures: 1, Errors: 0, Skipped: 35, Time elapsed: 35.034 s <<< FAILURE! - in org.apache.pulsar.websocket.proxy.v1.V1_ProxyAuthenticationTest Error: anonymousSocketTest(org.apache.pulsar.websocket.proxy.v1.V1_ProxyAuthenticationTest) Time elapsed: 10.066 s <<< FAILURE! org.testng.internal.thread.ThreadTimeoutException: Method org.apache.pulsar.websocket.proxy.v1.V1_ProxyAuthenticationTest.anonymousSocketTest() didn't finish within the time-out 1 at java.base@17.0.5/java.lang.Thread.sleep(Native Method) at java.base@17.0.5/java.lang.Thread.sleep(Thread.java:337) at java.base@17.0.5/java.util.concurrent.TimeUnit.sleep(TimeUnit.java:446) at app//org.awaitility.core.Uninterruptibles.sleepUninterruptibly(Uninterruptibles.java:35) at app//org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:117) at app//org.awaitility.core.AssertionCondition.await(AssertionCondition.java:119) at app//org.awaitility.core.AssertionCondition.await(AssertionCondition.java:31) at app//org.awaitility.core.ConditionFactory.until(ConditionFactory.java:985) at app//org.awaitility.core.ConditionFactory.untilAsserted(ConditionFactory.java:769) at app//org.apache.pulsar.websocket.proxy.v1.V1_ProxyAuthenticationTest.socketTest(V1_ProxyAuthenticationTest.java:134) at app//org.apache.pulsar.websocket.proxy.v1.V1_ProxyAuthenticationTest.anonymousSocketTest(V1_ProxyAuthenticationTest.java:147) at java.base@17.0.5/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base@17.0.5/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base@17.0.5/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base@17.0.5/java.lang.reflect.Method.invoke(Method.java:568) at app//org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139) at app//org.testng.internal.invokers.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:47) at app//org.testng.internal.invokers.InvokeMethodRunnable.call(InvokeMethodRunnable.java:76) at app//org.testng.internal.invokers.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11) at java.base@17.0.5/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base@17.0.5/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base@17.0.5/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base@17.0.5/java.lang.Thread.run(Thread.java:833) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] lhotari opened a new issue, #17933: Flaky-test: V1_ProxyAuthenticationTest.anonymousSocketTest
lhotari opened a new issue, #17933: URL: https://github.com/apache/pulsar/issues/17933 ### Search before asking - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar. ### Example failure https://github.com/apache/pulsar/actions/runs/3184651648/jobs/5193461550#step:10:1372 ### Exception stacktrace ``` Error: Tests run: 16, Failures: 1, Errors: 0, Skipped: 15, Time elapsed: 64.371 s <<< FAILURE! - in org.apache.pulsar.websocket.proxy.v1.V1_ProxyAuthenticationTest Error: anonymousSocketTest(org.apache.pulsar.websocket.proxy.v1.V1_ProxyAuthenticationTest) Time elapsed: 2.403 s <<< FAILURE! java.lang.AssertionError: lists don't have the same size expected [5] but found [10] at org.testng.Assert.fail(Assert.java:99) at org.testng.Assert.failNotEquals(Assert.java:1037) at org.testng.Assert.assertEqualsImpl(Assert.java:140) at org.testng.Assert.assertEquals(Assert.java:122) at org.testng.Assert.assertEquals(Assert.java:907) at org.testng.Assert.assertEquals(Assert.java:1089) at org.testng.Assert.assertEquals(Assert.java:1065) at org.apache.pulsar.websocket.proxy.v1.V1_ProxyAuthenticationTest.socketTest(V1_ProxyAuthenticationTest.java:149) at org.apache.pulsar.websocket.proxy.v1.V1_ProxyAuthenticationTest.anonymousSocketTest(V1_ProxyAuthenticationTest.java:159) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132) at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45) at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73) at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) ``` ### Are you willing to submit a PR? - [X] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] lhotari commented on issue #18365: Flaky-test: V1_ProxyAuthenticationTest.authenticatedSocketTest
lhotari commented on issue #18365: URL: https://github.com/apache/pulsar/issues/18365#issuecomment-1327076930 This issue duplicates #17933 . I'll close this one and re-open #17933 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] lhotari closed issue #18365: Flaky-test: V1_ProxyAuthenticationTest.authenticatedSocketTest
lhotari closed issue #18365: Flaky-test: V1_ProxyAuthenticationTest.authenticatedSocketTest URL: https://github.com/apache/pulsar/issues/18365 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] lhotari commented on issue #18365: Flaky-test: V1_ProxyAuthenticationTest.authenticatedSocketTest
lhotari commented on issue #18365: URL: https://github.com/apache/pulsar/issues/18365#issuecomment-1327074053 @nodece Would you mind fixing this issue? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] lhotari commented on issue #18365: Flaky-test: V1_ProxyAuthenticationTest.authenticatedSocketTest
lhotari commented on issue #18365: URL: https://github.com/apache/pulsar/issues/18365#issuecomment-1327071679 Another one: https://github.com/apache/pulsar/actions/runs/3545605005/jobs/5953957450#step:10:719 ``` Error: Tests run: 36, Failures: 1, Errors: 0, Skipped: 35, Time elapsed: 35.034 s <<< FAILURE! - in org.apache.pulsar.websocket.proxy.v1.V1_ProxyAuthenticationTest Error: anonymousSocketTest(org.apache.pulsar.websocket.proxy.v1.V1_ProxyAuthenticationTest) Time elapsed: 10.066 s <<< FAILURE! org.testng.internal.thread.ThreadTimeoutException: Method org.apache.pulsar.websocket.proxy.v1.V1_ProxyAuthenticationTest.anonymousSocketTest() didn't finish within the time-out 1 at java.base@17.0.5/java.lang.Thread.sleep(Native Method) at java.base@17.0.5/java.lang.Thread.sleep(Thread.java:337) at java.base@17.0.5/java.util.concurrent.TimeUnit.sleep(TimeUnit.java:446) at app//org.awaitility.core.Uninterruptibles.sleepUninterruptibly(Uninterruptibles.java:35) at app//org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:117) at app//org.awaitility.core.AssertionCondition.await(AssertionCondition.java:119) at app//org.awaitility.core.AssertionCondition.await(AssertionCondition.java:31) at app//org.awaitility.core.ConditionFactory.until(ConditionFactory.java:985) at app//org.awaitility.core.ConditionFactory.untilAsserted(ConditionFactory.java:769) at app//org.apache.pulsar.websocket.proxy.v1.V1_ProxyAuthenticationTest.socketTest(V1_ProxyAuthenticationTest.java:134) at app//org.apache.pulsar.websocket.proxy.v1.V1_ProxyAuthenticationTest.anonymousSocketTest(V1_ProxyAuthenticationTest.java:147) at java.base@17.0.5/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base@17.0.5/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base@17.0.5/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base@17.0.5/java.lang.reflect.Method.invoke(Method.java:568) at app//org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139) at app//org.testng.internal.invokers.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:47) at app//org.testng.internal.invokers.InvokeMethodRunnable.call(InvokeMethodRunnable.java:76) at app//org.testng.internal.invokers.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11) at java.base@17.0.5/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base@17.0.5/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base@17.0.5/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base@17.0.5/java.lang.Thread.run(Thread.java:833) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-cpp] erobot opened a new pull request, #125: [fix] Fix PartitionedProducerImpl::closeAsync to close sub-producers properly
erobot opened a new pull request, #125:
URL: https://github.com/apache/pulsar-client-cpp/pull/125
### Motivation
PartitionedProducerImpl do not close sub-producers properly when any
sub-producer creation fails. Continuing to retry creating producer will
eventually reach the maximum producer limit. It seems a regression caused by
#54.
When sub-producer creation fails, state_ is set to Failed.
PartitionedProducerImpl::closeAsync only do cleanup when state_==Ready and
sub-producers do not close when state_==Failed.
https://github.com/apache/pulsar-client-cpp/blob/f0268ecd29a6d0030b7d07379ec609884b4c14ff/lib/PartitionedProducerImpl.cc#L273-L276
### Modifications
Close sub-producers when state != Closed.
### Verifying this change
test code:
Client client("pulsar://localhost:6650");
Producer producer;
while (true) {
Result result =
client.createProducer("persistent://public/default/test", producer);
if (result != ResultOk) {
cerr << "Error creating producer: " << result << endl;
std::this_thread::sleep_for(std::chrono::seconds(1));
continue;
}
break;
}
test cmd:
pulsar-admin namespaces set-max-producers -p 10 public/default
pulsar-admin topics create-partition-topic -n 2 public/default/test
pulsar-perf produce -r 1 -n 10 public/default/test-partition-0
test procedure:
1. set namespace max producer limit to 10
2. create partitoned topic public/default/test with 2 partitions
3. use pulsar-perf to create 10 prodcuers on test-partition-0
4. run test code to create producer for public/default/test, will fail
because test-partition-0 already have 10 producer
5.
* without fix, test-partition-1 will have 10 producers because of
sub-producer not close
* with fix, test-partition-1 will have not more than 1 producer
### Documentation
- [ ] `doc-required`
(Your PR needs to update docs and you will update later)
- [x] `doc-not-needed`
bug-fix only
- [ ] `doc`
(Your PR contains doc changes)
- [ ] `doc-complete`
(Docs have been already added)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on a diff in pull request #18084: [improve][broker] PIP-192: Define new load manager base interfaces
BewareMyPower commented on code in PR #18084:
URL: https://github.com/apache/pulsar/pull/18084#discussion_r1032042101
##
pulsar-common/src/main/java/org/apache/pulsar/policies/data/loadbalancer/LocalBrokerData.java:
##
@@ -268,7 +268,7 @@ public double
getMaxResourceUsageWithWeightWithinLimit(final double cpuWeight, f
bandwidthOut.percentUsage() * bandwidthOutWeight) / 100;
}
-private static double max(double... args) {
+ public static double max(double... args) {
Review Comment:
```suggestion
public static double max(double... args) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #18625: [improve][broker] Reduce object creation in TopicResources.handleNotification
lhotari commented on PR #18625: URL: https://github.com/apache/pulsar/pull/18625#issuecomment-1327068447 @andrasbeni please review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] lhotari opened a new pull request, #18625: [improve][broker] Reduce object creation in TopicResources.handleNotification
lhotari opened a new pull request, #18625: URL: https://github.com/apache/pulsar/pull/18625 ### Motivation Follow "zero garbage" style on hot code paths. * In this case, the usage of EnumSet isn't justified since there's only 2 enums to check * topicListeners is a ConcurrentHashMap, no need for a copy * skip checks when topicListeners is empty ### Modifications * optimize TopicResources.handleNotification method. ### Documentation - [ ] `doc` - [ ] `doc-required` - [x] `doc-not-needed` - [ ] `doc-complete` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] nodece commented on pull request #18512: [fix][broker] namespace not found will cause request timeout
nodece commented on PR #18512: URL: https://github.com/apache/pulsar/pull/18512#issuecomment-1327067534 @eolivelli Could you review this PR again? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on issue #18598: Discuss about reverting #16937 "skip mis-configured resource usage(>100%) in load balancer"
heesung-sn commented on issue #18598: URL: https://github.com/apache/pulsar/issues/18598#issuecomment-1327066548 Note: If resourceUsage > 100% becomes the winner, the moving avg function will decay it more slowly than others( unfair signal treatment). But I assume this is the intention of the weighted configs too. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar-site] branch main updated: Docs sync done from apache/pulsar(#23e90d6)
This is an automated email from the ASF dual-hosted git repository. urfree pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/pulsar-site.git The following commit(s) were added to refs/heads/main by this push: new 6d95ad9a4c4 Docs sync done from apache/pulsar(#23e90d6) 6d95ad9a4c4 is described below commit 6d95ad9a4c45ccffe08aa722765de0395163fb4e Author: Pulsar Site Updater AuthorDate: Fri Nov 25 06:01:40 2022 + Docs sync done from apache/pulsar(#23e90d6)
[GitHub] [pulsar] labuladong commented on pull request #18512: [fix][broker] namespace not found will cause request timeout
labuladong commented on PR #18512: URL: https://github.com/apache/pulsar/pull/18512#issuecomment-1327051014 /pulsarbot rerun-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codecov-commenter commented on pull request #18624: [fix][broker] Fix possible race condition in completing topic list watcher
codecov-commenter commented on PR #18624: URL: https://github.com/apache/pulsar/pull/18624#issuecomment-1327049095 # [Codecov](https://codecov.io/gh/apache/pulsar/pull/18624?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#18624](https://codecov.io/gh/apache/pulsar/pull/18624?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (9f25960) into [master](https://codecov.io/gh/apache/pulsar/commit/cd85a67bd3a880732678b0dfd5f7769ad90a90ef?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (cd85a67) will **decrease** coverage by `10.31%`. > The diff coverage is `14.28%`. [](https://codecov.io/gh/apache/pulsar/pull/18624?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #18624 +/- ## = - Coverage 47.39% 37.08% -10.32% + Complexity10479 1968 -8511 = Files 698 209 -489 Lines 6807014420-53650 Branches 7279 1573 -5706 = - Hits 32264 5348-26916 + Misses32228 8486-23742 + Partials 3578 586 -2992 ``` | Flag | Coverage Δ | | |---|---|---| | unittests | `37.08% <14.28%> (-10.32%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/pulsar/pull/18624?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...pache/pulsar/client/impl/TableViewBuilderImpl.java](https://codecov.io/gh/apache/pulsar/pull/18624/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL1RhYmxlVmlld0J1aWxkZXJJbXBsLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...a/org/apache/pulsar/client/impl/TableViewImpl.java](https://codecov.io/gh/apache/pulsar/pull/18624/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL1RhYmxlVmlld0ltcGwuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...pulsar/client/impl/TableViewConfigurationData.java](https://codecov.io/gh/apache/pulsar/pull/18624/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL1RhYmxlVmlld0NvbmZpZ3VyYXRpb25EYXRhLmphdmE=) | `33.33% <50.00%> (+33.33%)` | :arrow_up: | | [...apache/pulsar/client/impl/AutoClusterFailover.java](https://codecov.io/gh/apache/pulsar/pull/18624/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL0F1dG9DbHVzdGVyRmFpbG92ZXIuamF2YQ==) | `70.00% <0.00%> (-6.12%)` | :arrow_down: | | [.../pulsar/client/impl/ControlledClusterFailover.java](https://codecov.io/gh/apache/pulsar/pull/18624/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL0NvbnRyb2xsZWRDbHVzdGVyRmFpbG92ZXIuamF2YQ==) | `58.87% <0.00%> (-3.74%)` | :arrow_down: | | [.../org/apache/pulsar/client/impl/ConnectionPool.java](https://codecov.io/gh/apache/pulsar/pull/18624/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL0Nvbm5lY3Rpb25Qb29sLmphdmE=) | `37.43% <0.00%> (-1.03%)` | :arrow_down: | |
[GitHub] [pulsar] codecov-commenter commented on pull request #18622: [fix][proxy] Fix logging in shutdown when proxy shutdown
codecov-commenter commented on PR #18622: URL: https://github.com/apache/pulsar/pull/18622#issuecomment-1327047690 # [Codecov](https://codecov.io/gh/apache/pulsar/pull/18622?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#18622](https://codecov.io/gh/apache/pulsar/pull/18622?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (9b4c1db) into [master](https://codecov.io/gh/apache/pulsar/commit/be1d07e16119fafbe489315ab68c4751bf03c31d?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (be1d07e) will **increase** coverage by `0.31%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/pulsar/pull/18622?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #18622 +/- ## + Coverage 36.82% 37.13% +0.31% + Complexity 7842 1971-5871 Files 698 209 -489 Lines 6806014420 -53640 Branches 7277 1573-5704 - Hits 25062 5355 -19707 + Misses39675 8479 -31196 + Partials 3323 586-2737 ``` | Flag | Coverage Δ | | |---|---|---| | unittests | `37.13% <ø> (+0.31%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/pulsar/pull/18622?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...apache/pulsar/client/impl/AutoClusterFailover.java](https://codecov.io/gh/apache/pulsar/pull/18622/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL0F1dG9DbHVzdGVyRmFpbG92ZXIuamF2YQ==) | `70.00% <0.00%> (-6.12%)` | :arrow_down: | | [.../pulsar/client/impl/ControlledClusterFailover.java](https://codecov.io/gh/apache/pulsar/pull/18622/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL0NvbnRyb2xsZWRDbHVzdGVyRmFpbG92ZXIuamF2YQ==) | `58.87% <0.00%> (-3.74%)` | :arrow_down: | | [...va/org/apache/pulsar/client/impl/HandlerState.java](https://codecov.io/gh/apache/pulsar/pull/18622/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL0hhbmRsZXJTdGF0ZS5qYXZh) | `67.56% <0.00%> (-2.71%)` | :arrow_down: | | [...a/org/apache/pulsar/client/impl/TableViewImpl.java](https://codecov.io/gh/apache/pulsar/pull/18622/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL1RhYmxlVmlld0ltcGwuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...pache/pulsar/client/impl/TableViewBuilderImpl.java](https://codecov.io/gh/apache/pulsar/pull/18622/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL1RhYmxlVmlld0J1aWxkZXJJbXBsLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...pache/pulsar/proxy/server/ProxyServiceStarter.java](https://codecov.io/gh/apache/pulsar/pull/18622/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLXByb3h5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9wdWxzYXIvcHJveHkvc2VydmVyL1Byb3h5U2VydmljZVN0YXJ0ZXIuamF2YQ==) | | | |
[GitHub] [pulsar] lhotari commented on pull request #18624: [fix][broker] Fix possible race condition in completing topic list watcher
lhotari commented on PR #18624: URL: https://github.com/apache/pulsar/pull/18624#issuecomment-1327039127 @andrasbeni Please review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] lhotari opened a new pull request, #18624: [fix][broker] Fix possible race condition in completing topic list watcher
lhotari opened a new pull request, #18624: URL: https://github.com/apache/pulsar/pull/18624 ### Motivation While reading code in TopicListService, I noticed a race condition that could happen in the case where the connection is closing and there's a TopicListWatcher that is in being initialized. ### Modifications Add check to see if the watcherFuture can be completed. If not, deregister the watcher. ### Documentation - [ ] `doc` - [ ] `doc-required` - [x] `doc-not-needed` - [ ] `doc-complete` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] coderzc closed pull request #18622: [fix][proxy] Fix logging in shutdown when proxy shutdown
coderzc closed pull request #18622: [fix][proxy] Fix logging in shutdown when proxy shutdown URL: https://github.com/apache/pulsar/pull/18622 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] labuladong commented on pull request #18512: [fix][broker] namespace not found will cause request timeout
labuladong commented on PR #18512: URL: https://github.com/apache/pulsar/pull/18512#issuecomment-1327027840 /pulsarbot rerun-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #18599: [improve][misc] Upgrade Netty to 4.1.85.Final and Netty Tcnative to 2.0.54.Final
lhotari commented on PR #18599: URL: https://github.com/apache/pulsar/pull/18599#issuecomment-1327021041 There are odd test failures with newer Netty version. I was investigating the issue with [`org.apache.pulsar.client.api.PatternMultiTopicsConsumerTest` ](https://github.com/apache/pulsar/actions/runs/3543677928/jobs/5950405326#step:10:790). In some cases, messages are not delivered to the consumer. This happens in the testNotifications test case. Since the topic is subscribed instantly after it's created, this could be some sort of race condition. Perhaps the latency is reduced with the newer Netty version slightly, just enough to trigger the race condition? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on pull request #18623: [fix][branch-2.9] Enable branch-2.9 force push
github-actions[bot] commented on PR #18623: URL: https://github.com/apache/pulsar/pull/18623#issuecomment-1327018106 @congbobo184 Please add the following content to your PR description and select a checkbox: ``` - [ ] `doc` - [ ] `doc-required` - [ ] `doc-not-needed` - [ ] `doc-complete` ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] congbobo184 opened a new pull request, #18623: [fix][branch-2.9] Enable branch-2.9 force push
congbobo184 opened a new pull request, #18623: URL: https://github.com/apache/pulsar/pull/18623 reason : https://lists.apache.org/thread/jcrq9q0k6kh9rvb71dwb9s3mvo5c5dk5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch branch-2.9-backup created (now e3611cb8517)
This is an automated email from the ASF dual-hosted git repository. penghui pushed a change to branch branch-2.9-backup in repository https://gitbox.apache.org/repos/asf/pulsar.git at e3611cb8517 fix `DnsResolverTest` license header No new revisions were added by this update.
[GitHub] [pulsar] heesung-sn commented on issue #18598: Discuss about reverting #16937 "skip mis-configured resource usage(>100%) in load balancer"
heesung-sn commented on issue #18598: URL: https://github.com/apache/pulsar/issues/18598#issuecomment-1326997463 Sure. If resourceUsage * weight is expected to be more than 100%, we should revert this change. The motivation does not hold. Thanks for taking care of this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] Technoboy- commented on a diff in pull request #18603: [improve][broker] System topic writer/reader connection not counted
Technoboy- commented on code in PR #18603:
URL: https://github.com/apache/pulsar/pull/18603#discussion_r1031972743
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java:
##
@@ -470,6 +470,9 @@ protected void unregisterTopicPolicyListener() {
}
protected boolean isSameAddressProducersExceeded(Producer producer) {
+if (isSystemTopic()) {
Review Comment:
|| producer.isRemote()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[pulsar] branch master updated (4545b082a32 -> 23e90d65244)
This is an automated email from the ASF dual-hosted git repository. xyz pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git from 4545b082a32 [fix][doc] fix rest api param description of setting backlog quota on topic (#18592) add 23e90d65244 [improve][fn] Support SubscriptionPosition in python instance (#18431) No new revisions were added by this update. Summary of changes: pulsar-functions/instance/src/main/python/python_instance.py | 6 ++ 1 file changed, 6 insertions(+)
[GitHub] [pulsar] BewareMyPower closed issue #18424: Support SubscriptionInitialPosition for python function
BewareMyPower closed issue #18424: Support SubscriptionInitialPosition for python function URL: https://github.com/apache/pulsar/issues/18424 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] BewareMyPower merged pull request #18431: [improve][fn] Support SubscriptionPosition in python instance
BewareMyPower merged PR #18431: URL: https://github.com/apache/pulsar/pull/18431 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] momo-jun commented on pull request #18582: [improve][doc] Add doc for the HTTP Sink
momo-jun commented on PR #18582: URL: https://github.com/apache/pulsar/pull/18582#issuecomment-1326962825 @cbornet thanks for adding the docs. It deserves a `feature` in the PR title. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] momo-jun commented on a diff in pull request #18582: [improve][doc] Add doc for the HTTP Sink
momo-jun commented on code in PR #18582: URL: https://github.com/apache/pulsar/pull/18582#discussion_r1031959039 ## site2/docs/io-http-sink.md: ## @@ -0,0 +1,56 @@ +--- +id: io-http-sink +title: HTTTP sink connector +sidebar_label: "HTTP sink connector" +--- + +The HTTP sink connector pulls the records from Pulsar topics and makes a POST request to a configurable HTTP url (webhook). + +The body of the HTTP request is the JSON representation of the record value. The header `Content-Type: application/json` is added to the HTTP request. + +Some other HTTP headers are added to the HTTP request: + +* `PulsarTopic`: the topic of the record +* `PulsarKey`: the key of the record +* `PulsarEventTime`: the event time of the record +* `PulsarPublishTime`: the publish time of the record +* `PulsarMessageId`: the ID of the message contained in the record +* `PulsarProperties`-*: each record property is passed with the property name prefixed by `PulsarProperties-` + +## Configuration + +The configuration of the HTTP sink connector has the following properties. + +### Property + +| Name | Type | Required | Default | Description | +|---||--|--|-| +| `url` | String | false| http://localhost | The URL of the HTTP server | +| `headers` | Map| false| empty map|The HDFS directory where files read from or written to. | Review Comment: ```suggestion | `headers` | Map| false| empty map|The HDFS directory where files are read from or written to. | ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] momo-jun commented on a diff in pull request #18582: [improve][doc] Add doc for the HTTP Sink
momo-jun commented on code in PR #18582: URL: https://github.com/apache/pulsar/pull/18582#discussion_r1031958418 ## site2/docs/io-http-sink.md: ## @@ -0,0 +1,56 @@ +--- +id: io-http-sink +title: HTTTP sink connector +sidebar_label: "HTTP sink connector" +--- + +The HTTP sink connector pulls the records from Pulsar topics and makes a POST request to a configurable HTTP url (webhook). Review Comment: ```suggestion The HTTP sink connector pulls the records from Pulsar topics and makes a POST request to a configurable HTTP URL (webhook). ``` ## site2/docs/io-http-sink.md: ## @@ -0,0 +1,56 @@ +--- +id: io-http-sink +title: HTTTP sink connector +sidebar_label: "HTTP sink connector" +--- + +The HTTP sink connector pulls the records from Pulsar topics and makes a POST request to a configurable HTTP url (webhook). + +The body of the HTTP request is the JSON representation of the record value. The header `Content-Type: application/json` is added to the HTTP request. + +Some other HTTP headers are added to the HTTP request: + +* `PulsarTopic`: the topic of the record +* `PulsarKey`: the key of the record +* `PulsarEventTime`: the event time of the record +* `PulsarPublishTime`: the publish time of the record +* `PulsarMessageId`: the ID of the message contained in the record +* `PulsarProperties`-*: each record property is passed with the property name prefixed by `PulsarProperties-` + +## Configuration + +The configuration of the HTTP sink connector has the following properties. + +### Property + +| Name | Type | Required | Default | Description | +|---||--|--|-| +| `url` | String | false| http://localhost | The URL of the HTTP server | +| `headers` | Map| false| empty map|The HDFS directory where files read from or written to. | Review Comment: ```suggestion | `headers` | Map | false| empty map|The HDFS directory where files are read from or written to. | ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] Demogorgon314 commented on a diff in pull request #18084: [improve][broker] PIP-192: Define new load manager base interfaces
Demogorgon314 commented on code in PR #18084:
URL: https://github.com/apache/pulsar/pull/18084#discussion_r1031953243
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/data/BrokerLoadData.java:
##
@@ -0,0 +1,188 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.data;
+
+import lombok.Data;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.policies.data.loadbalancer.ResourceUsage;
+import org.apache.pulsar.policies.data.loadbalancer.SystemResourceUsage;
+
+/**
+ * Contains all the data that is maintained locally on each broker.
+ *
+ * Migrate from {@link
org.apache.pulsar.policies.data.loadbalancer.LocalBrokerData}.
+ * And removed the lookup data, see {@link BrokerLookupData}
+ */
+@Data
+public class BrokerLoadData {
+private static final double gigaBitToByte = 128 * 1024 * 1024.0;
Review Comment:
I have removed some temporary unused methods from `BrokerLoadData`. I think
we can add it when we use it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on a diff in pull request #18620: [fix][broker] If ledger lost, cursor mark delete position can not forward
poorbarcode commented on code in PR #18620:
URL: https://github.com/apache/pulsar/pull/18620#discussion_r1031951904
##
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java:
##
@@ -2661,6 +2708,34 @@ void setReadPosition(Position newReadPositionInt) {
}
}
+void clearIncompleteAckedRecordsByLedgerId(final long ledgerId){
+lock.writeLock().lock();
+try {
+List rangeListToDelete =
individualDeletedMessages.asRanges().stream()
+.filter(range -> range.lowerEndpoint().getLedgerId() ==
ledgerId)
+.map(range -> Range.openClosed(
+new LongPairRangeSet.LongPair(ledgerId,
range.lowerEndpoint().getEntryId()),
+new LongPairRangeSet.LongPair(ledgerId,
range.upperEndpoint().getEntryId())
+)).collect(Collectors.toList());
+if (!rangeListToDelete.isEmpty()) {
+rangeListToDelete.forEach(individualDeletedMessages::remove);
+}
+
+if (batchDeletedIndexes != null) {
+Set batchedIndexesToDelete =
batchDeletedIndexes.keySet().stream()
+.filter(position -> position.getLedgerId() == ledgerId)
+.collect(Collectors.toSet());
+if (!batchedIndexesToDelete.isEmpty()) {
+
batchedIndexesToDelete.forEach(batchDeletedIndexes::remove);
+}
+}
+} catch (Exception e){
+e.printStackTrace();
Review Comment:
This was added during debugging, and I forgot to delete it， already fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on a diff in pull request #18620: [fix][broker] If ledger lost, cursor mark delete position can not forward
poorbarcode commented on code in PR #18620:
URL: https://github.com/apache/pulsar/pull/18620#discussion_r1031951717
##
pulsar-broker/src/test/java/org/apache/pulsar/broker/LedgerLostTest.java:
##
@@ -0,0 +1,429 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker;
+
+import java.net.Inet4Address;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import lombok.AllArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.client.BookKeeper;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedCursorImpl;
+import org.apache.pulsar.broker.service.Topic;
+import org.apache.pulsar.broker.service.persistent.PersistentSubscription;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.client.admin.PulsarAdmin;
+import org.apache.pulsar.client.api.Consumer;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.PulsarClient;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.SubscriptionType;
+import org.apache.pulsar.client.impl.BatchMessageIdImpl;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.common.policies.data.ClusterData;
+import org.apache.pulsar.common.policies.data.TenantInfo;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble;
+import org.testcontainers.shaded.org.awaitility.Awaitility;
+import org.testcontainers.shaded.org.awaitility.reflect.WhiteboxImpl;
+import org.testng.Assert;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.DataProvider;
+import org.testng.annotations.Test;
+
+@Slf4j
+@Test(groups = "broker")
+public class LedgerLostTest {
+
+// prefer inet4.
+private static final String LOCALHOST =
Inet4Address.getLoopbackAddress().getHostAddress();
+private static final String CLUSTER = "broken_ledger_test";
+private static final String DEFAULT_TENANT = "public";
+private static final String DEFAULT_NAMESPACE = DEFAULT_TENANT +
"/default";
+
+protected LocalBookkeeperEnsemble bkEnsemble;
+protected ServiceConfiguration pulsarConfig;
+protected PulsarService pulsarService;
+protected int brokerWebServicePort;
+protected int brokerServicePort;
+protected String metadataServiceUri;
+protected BookKeeper bookKeeperClient;
+protected String brokerUrl;
+protected String brokerServiceUrl;
+protected PulsarAdmin pulsarAdmin;
+protected PulsarClient pulsarClient;
+
+@BeforeClass
+protected void setup() throws Exception {
+log.info("--- Start cluster ---");
+startLocalBookie();
+initPulsarConfig();
+startPulsar();
+}
+
+@AfterClass
+protected void cleanup() throws Exception {
+log.info("--- Shutting down ---");
+silentStopPulsar();
+stopLocalBookie();
+}
+
+protected void startLocalBookie() throws Exception{
+log.info("===> Start bookie ");
+bkEnsemble = new LocalBookkeeperEnsemble(3, 0, () -> 0);
+bkEnsemble.start();
+metadataServiceUri = String.format("zk:%s:%s", LOCALHOST,
bkEnsemble.getZookeeperPort());
+initBookieClient();
+}
+
+protected void initBookieClient() throws Exception {
+bookKeeperClient = new BookKeeper(String.format("%s:%s", LOCALHOST,
bkEnsemble.getZookeeperPort()));
+}
+
+protected void stopLocalBookie() {
+log.info("===> Close bookie client");
+try {
+bookKeeperClient.close();
+} catch (Exception e){
+log.error("Close bookie client fail", e);
+}
+
[GitHub] [pulsar] github-actions[bot] commented on pull request #18073: [fix][txn] Port for master "Use PulsarClient HashWheelTimer to schedule producer batch trigger task"
github-actions[bot] commented on PR #18073: URL: https://github.com/apache/pulsar/pull/18073#issuecomment-1326948797 The pr had no activity for 30 days, mark with Stale label. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #18173: PIP-217: LoadShedding Strategy Improment
github-actions[bot] commented on issue #18173: URL: https://github.com/apache/pulsar/issues/18173#issuecomment-1326948687 The issue had no activity for 30 days, mark with Stale label. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on pull request #18186: [improve] [broker] AvgShedder
github-actions[bot] commented on PR #18186: URL: https://github.com/apache/pulsar/pull/18186#issuecomment-1326948655 The pr had no activity for 30 days, mark with Stale label. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] hangc0276 commented on issue #18598: Discuss about reverting #16937 "skip mis-configured resource usage(>100%) in load balancer"
hangc0276 commented on issue #18598: URL: https://github.com/apache/pulsar/issues/18598#issuecomment-1326947253 In the Pulsar load balance strategy, including OverloadShedder and ThresholdShedder, the weight of each resource is not ensured in [0, 1]. The total `resourceUsage` of each broker won't ensure to be less than 100%. > Incorrectly scaled resource load usage(cpu, memory, network usages bigger than 100%) can harm the load computation in the load balancer logics, as the load balancer computation expects all resource usages are normalized to the 100% scale. For the motivation of https://github.com/apache/pulsar/pull/16937, the weight of the resource is not misconfigured, and it will break the old behavior, and lead to load balance not working after applying this PR in their cluster. I support reverting this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] coderzc opened a new pull request, #18622: [fix][proxy] Fix logging in shutdown when proxy shutdown
coderzc opened a new pull request, #18622: URL: https://github.com/apache/pulsar/pull/18622 ### Motivation & Modifications We have set log4j.shutdownHookEnabled=false, so we need add finally LogManager.shutdown() invoke in PulsarProxy Runtime ShutdownHook. ### Documentation - [ ] `doc` - [ ] `doc-required` - [x] `doc-not-needed` - [ ] `doc-complete` ### Matching PR in forked repository PR in forked repository: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on a diff in pull request #18084: [improve][broker] PIP-192: Define new load manager base interfaces
codelipenghui commented on code in PR #18084:
URL: https://github.com/apache/pulsar/pull/18084#discussion_r1031939571
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/data/BrokerLoadData.java:
##
@@ -0,0 +1,188 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.loadbalance.extensions.data;
+
+import lombok.Data;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.policies.data.loadbalancer.ResourceUsage;
+import org.apache.pulsar.policies.data.loadbalancer.SystemResourceUsage;
+
+/**
+ * Contains all the data that is maintained locally on each broker.
+ *
+ * Migrate from {@link
org.apache.pulsar.policies.data.loadbalancer.LocalBrokerData}.
+ * And removed the lookup data, see {@link BrokerLookupData}
+ */
+@Data
+public class BrokerLoadData {
+private static final double gigaBitToByte = 128 * 1024 * 1024.0;
+
+// Most recently available system resource usage.
+private ResourceUsage cpu;
+private ResourceUsage memory;
+private ResourceUsage directMemory;
+
+private ResourceUsage bandwidthIn;
+private ResourceUsage bandwidthOut;
+
+// Message data from the most recent namespace bundle stats.
+private double msgThroughputIn;
+private ResourceUsage msgThroughputInUsage;
+private double msgThroughputOut;
+private ResourceUsage msgThroughputOutUsage;
+private double msgRateIn;
+private double msgRateOut;
+
+public BrokerLoadData() {
+cpu = new ResourceUsage();
+memory = new ResourceUsage();
+directMemory = new ResourceUsage();
+bandwidthIn = new ResourceUsage();
+bandwidthOut = new ResourceUsage();
+msgThroughputInUsage = new ResourceUsage();
+msgThroughputOutUsage = new ResourceUsage();
+}
+
+/**
+ * Using the system resource usage and bundle stats acquired from the
Pulsar client, update this LocalBrokerData.
+ *
+ * @param systemResourceUsage
+ *System resource usage (cpu, memory, and direct memory).
+ */
+public void update(final SystemResourceUsage systemResourceUsage) {
+updateSystemResourceUsage(systemResourceUsage);
+}
+
+/**
+ * Using another LocalBrokerData, update this.
+ *
+ * @param other
+ *LocalBrokerData to update from.
+ */
+public void update(final BrokerLoadData other) {
+updateSystemResourceUsage(other.cpu, other.memory, other.directMemory,
other.bandwidthIn, other.bandwidthOut);
+}
+
+// Set the cpu, memory, and direct memory to that of the new system
resource usage data.
+private void updateSystemResourceUsage(final SystemResourceUsage
systemResourceUsage) {
+updateSystemResourceUsage(systemResourceUsage.cpu,
systemResourceUsage.memory, systemResourceUsage.directMemory,
+systemResourceUsage.bandwidthIn,
systemResourceUsage.bandwidthOut);
+}
+
+// Update resource usage given each individual usage.
+private void updateSystemResourceUsage(final ResourceUsage cpu, final
ResourceUsage memory,
+ final ResourceUsage directMemory,
final ResourceUsage bandwidthIn,
+ final ResourceUsage bandwidthOut) {
+this.cpu = cpu;
+this.memory = memory;
+this.directMemory = directMemory;
+this.bandwidthIn = bandwidthIn;
+this.bandwidthOut = bandwidthOut;
+}
+
+public double getMaxResourceUsage() {
+return max(cpu.percentUsage(), directMemory.percentUsage(),
bandwidthIn.percentUsage(),
+bandwidthOut.percentUsage()) / 100;
+}
+
+public double getMaxResourceUsageWithWeight(final double cpuWeight, final
double memoryWeight,
+final double
directMemoryWeight, final double bandwidthInWeight,
+final double
bandwidthOutWeight) {
+return max(cpu.percentUsage() * cpuWeight, memory.percentUsage() *
memoryWeight,
+directMemory.percentUsage() * directMemoryWeight,
bandwidthIn.percentUsage() *
[GitHub] [pulsar] Technoboy- commented on issue #18598: Discuss about reverting #16937 "skip mis-configured resource usage(>100%) in load balancer"
Technoboy- commented on issue #18598: URL: https://github.com/apache/pulsar/issues/18598#issuecomment-1326941643 > I am not sure that this is a real problem or not, it looks like that setting loadBalancerCPUResourceWeight= 100 and see CPU usage as 2000% is like a hack. I think the demo test is not clear, I have updated the test. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #18598: Discuss about reverting #16937 "skip mis-configured resource usage(>100%) in load balancer"
codelipenghui commented on issue #18598: URL: https://github.com/apache/pulsar/issues/18598#issuecomment-1326940134 I discussed this issue with Hang a few days ago. Hang is the initial designer of the threshold shedder. The weight is not required to <= 1. I think we mistakenly merged https://github.com/apache/pulsar/pull/16937. Users can use any non-negative number for the weight. I support reverting the PR first and revisiting the issue that @heesung-sn wants to fix. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] Technoboy- closed pull request #18620: [fix][broker] If ledger lost, cursor mark delete position can not forward
Technoboy- closed pull request #18620: [fix][broker] If ledger lost, cursor mark delete position can not forward URL: https://github.com/apache/pulsar/pull/18620 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch master updated: [fix][doc] fix rest api param description of setting backlog quota on topic (#18592)
This is an automated email from the ASF dual-hosted git repository.
liuyu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 4545b082a32 [fix][doc] fix rest api param description of setting
backlog quota on topic (#18592)
4545b082a32 is described below
commit 4545b082a32c2a869c07a936adf35342ce15e9b1
Author: Li Li
AuthorDate: Fri Nov 25 09:15:35 2022 +0800
[fix][doc] fix rest api param description of setting backlog quota on topic
(#18592)
---
.../java/org/apache/pulsar/broker/admin/v2/PersistentTopics.java | 7 ---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v2/PersistentTopics.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v2/PersistentTopics.java
index a7a42533e4c..9ba408d1b52 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v2/PersistentTopics.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v2/PersistentTopics.java
@@ -2109,7 +2109,8 @@ public class PersistentTopics extends
PersistentTopicsBase {
@ApiParam(value = "Whether leader broker redirected this call to
this broker. For internal use.")
@QueryParam("authoritative") @DefaultValue("false") boolean
authoritative,
@QueryParam("isGlobal") @DefaultValue("false") boolean isGlobal,
-@QueryParam("backlogQuotaType") BacklogQuotaType backlogQuotaType,
BacklogQuotaImpl backlogQuota) {
+@QueryParam("backlogQuotaType") BacklogQuotaType backlogQuotaType,
+@ApiParam(value = "backlog quota policies for the specified
topic") BacklogQuotaImpl backlogQuota) {
validateTopicName(tenant, namespace, encodedTopic);
preValidation(authoritative)
.thenCompose(__ -> internalSetBacklogQuota(backlogQuotaType,
backlogQuota, isGlobal))
@@ -2274,7 +2275,7 @@ public class PersistentTopics extends
PersistentTopicsBase {
@PathParam("tenant") String tenant,
@PathParam("namespace") String namespace,
@PathParam("topic") @Encoded String encodedTopic,
-@ApiParam(value = "TTL in seconds for the specified namespace",
required = true)
+@ApiParam(value = "TTL in seconds for the specified topic",
required = true)
@QueryParam("messageTTL") Integer messageTTL,
@QueryParam("isGlobal") @DefaultValue("false") boolean isGlobal,
@ApiParam(value = "Whether leader broker redirected this call to
this broker. For internal use.")
@@ -2436,7 +2437,7 @@ public class PersistentTopics extends
PersistentTopicsBase {
@ApiParam(value = "Whether leader broker redirected this call to
this broker. For internal use.")
@QueryParam("authoritative") @DefaultValue("false") boolean
authoritative,
@QueryParam("isGlobal") @DefaultValue("false") boolean isGlobal,
-@ApiParam(value = "Retention policies for the specified
namespace") RetentionPolicies retention) {
+@ApiParam(value = "Retention policies for the specified topic")
RetentionPolicies retention) {
validateTopicName(tenant, namespace, encodedTopic);
preValidation(authoritative)
.thenCompose(__ -> internalSetRetention(retention, isGlobal))
[GitHub] [pulsar] Anonymitaet closed issue #18591: [Doc] the body param declaration for the Restful API about setting a backlog quota for a topic is missing
Anonymitaet closed issue #18591: [Doc] the body param declaration for the Restful API about setting a backlog quota for a topic is missing URL: https://github.com/apache/pulsar/issues/18591 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] Anonymitaet merged pull request #18592: [fix][doc] fix rest api param description of setting backlog quota on topic
Anonymitaet merged PR #18592: URL: https://github.com/apache/pulsar/pull/18592 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on pull request #18582: [improve][doc] Add doc for the HTTP Sink
Anonymitaet commented on PR #18582: URL: https://github.com/apache/pulsar/pull/18582#issuecomment-1326929855 Thanks for your contribution! Writers will review this PR from technical writing pespectives after it gets technical reviews from engineers. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-helm-chart] johndlang opened a new issue, #337: nil pointer evaluating interface {}.bookkeeper
johndlang opened a new issue, #337:
URL: https://github.com/apache/pulsar-helm-chart/issues/337
**Describe the bug**
When tying to install v3.0.0 I get a error when defining values for
bookkeeper
**To Reproduce**
Steps to reproduce the behavior:
helm install pulsar-dev -f values/pulsar_values_dev.yaml apache/pulsar
--version v3.0.0
with values.yaml: >
bookkeeper:
configData:
PULSAR_MEM: |
-Xms512m -Xmx1g -XX:MaxDirectMemorySize=2g
replicaCount: 3
resources:
requests:
cpu: 1
memory: 5Gi
volumes:
journal:
name: journal
size: 20Gi
local_storage: false
useMultiVolumes: true
multiVolumes:
- name: journal0
size: 10Gi
mountPath: /pulsar/data/bookkeeper/journal0
- name: journal1
size: 10Gi
mountPath: /pulsar/data/bookkeeper/journal1
ledgers:
name: ledgers
size: 100Gi
local_storage: false
useMultiVolumes: true
multiVolumes:
- name: ledgers0
size: 50Gi
mountPath: /pulsar/data/bookkeeper/ledgers0
- name: ledgers1
size: 50Gi
mountPath: /pulsar/data/bookkeeper/ledgers1
Error: INSTALLATION FAILED: template:
pulsar/templates/bookkeeper-statefulset.yaml:270:20: executing
"pulsar/templates/bookkeeper-statefulset.yaml" at
<.Values.bookkeeper.volumes.journal.selector>: nil pointer evaluating interface
{}.bookkeeper
**Expected behavior**
Chart to install sucessfully
**Screenshots**
If applicable, add screenshots to help explain your problem.
**Desktop (please complete the following information):**
Helm v3.10.0 EKS 1.24
**Additional context**
Add any other context about the problem here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on issue #18614: Remove `Timeline` since there's no clear plan
Anonymitaet commented on issue #18614: URL: https://github.com/apache/pulsar/issues/18614#issuecomment-1326922856 > "Coming soon" is easy to be a liar. I don't find any clear plan or contributors active working on this page: https://pulsar.apache.org/release-notes/timeline > > If there's any, I'm happy to close this issue and follow that initiative. OK -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on pull request #18621: [fix][monitor] Fix reporting pulsar_subscription_blocked_on_unacked_messages metric
github-actions[bot] commented on PR #18621: URL: https://github.com/apache/pulsar/pull/18621#issuecomment-1326920345 @aymkhalil Please add the following content to your PR description and select a checkbox: ``` - [ ] `doc` - [ ] `doc-required` - [ ] `doc-not-needed` - [ ] `doc-complete` ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] aymkhalil opened a new pull request, #18621: [fix][monitor] Fix reporting pulsar_subscription_blocked_on_unacked_messages metric
aymkhalil opened a new pull request, #18621: URL: https://github.com/apache/pulsar/pull/18621 ### Motivation The pulsar_subscription_blocked_on_unacked_messages metric was not scrapped by Prometheus although the dispatcher can be blocked on number of unacked messages: ``` Dispatcher read is blocked due to unackMessages 200010 reached to max 20 ``` ### Modifications When Prometheus scrapes stats topics, the previous logic was relying on the consumer flag to update the subscription although the subscription configs and logic is independent of the attached consumer. The related config for subscription is `maxUnackedMessagesPerSubscription` and for consumers is `maxUnackedMessagesPerConsumer` This change just copied over the value from the `SubscriptionStats` to the `AggregatedSubscriptionStats` ### Verifying this change - [X] Make sure that the change passes the CI checks. *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. I couldn't add unit tests for `NamespaceStatsAggregator` but tested the changes locally with Pulsar standalone + Prometheous:  ### Does this pull request potentially affect one of the following parts: *If the box was checked, please highlight the changes* - [ ] Dependencies (add or upgrade a dependency) - [ ] The public API - [ ] The schema - [ ] The default values of configurations - [ ] The threading model - [ ] The binary protocol - [ ] The REST endpoints - [ ] The admin CLI options - [ ] Anything that affects deployment ### Documentation - [ ] `doc` - [ ] `doc-required` - [ ] `doc-not-needed` - [ ] `doc-complete` ### Matching PR in forked repository PR in forked repository: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on a diff in pull request #18620: [fix] [broker] If ledger lost, cursor makr delete position can not forward
mattisonchao commented on code in PR #18620:
URL: https://github.com/apache/pulsar/pull/18620#discussion_r1031913744
##
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpReadEntry.java:
##
@@ -102,9 +102,11 @@ public void readEntriesFailed(ManagedLedgerException
exception, Object ctx) {
readPosition, exception.getMessage());
final ManagedLedgerImpl ledger = (ManagedLedgerImpl)
cursor.getManagedLedger();
Position nexReadPosition;
+Long lostLedger = null;
if (exception instanceof
ManagedLedgerException.LedgerNotExistException) {
// try to find and move to next valid ledger
nexReadPosition =
cursor.getNextLedgerPosition(readPosition.getLedgerId());
+lostLedger = readPosition.ledgerId;
} else {
// Skip this read operation
nexReadPosition =
ledger.getValidPositionAfterSkippedEntries(readPosition, count);
Review Comment:
I think we also need to check here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on a diff in pull request #18620: [fix] [broker] If ledger lost, cursor makr delete position can not forward
mattisonchao commented on code in PR #18620:
URL: https://github.com/apache/pulsar/pull/18620#discussion_r1031919726
##
pulsar-broker/src/test/java/org/apache/pulsar/broker/LedgerLostTest.java:
##
@@ -0,0 +1,429 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker;
+
+import java.net.Inet4Address;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import lombok.AllArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.client.BookKeeper;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedCursorImpl;
+import org.apache.pulsar.broker.service.Topic;
+import org.apache.pulsar.broker.service.persistent.PersistentSubscription;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.client.admin.PulsarAdmin;
+import org.apache.pulsar.client.api.Consumer;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.PulsarClient;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.SubscriptionType;
+import org.apache.pulsar.client.impl.BatchMessageIdImpl;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.common.policies.data.ClusterData;
+import org.apache.pulsar.common.policies.data.TenantInfo;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble;
+import org.testcontainers.shaded.org.awaitility.Awaitility;
+import org.testcontainers.shaded.org.awaitility.reflect.WhiteboxImpl;
+import org.testng.Assert;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.DataProvider;
+import org.testng.annotations.Test;
+
+@Slf4j
+@Test(groups = "broker")
+public class LedgerLostTest {
+
+// prefer inet4.
+private static final String LOCALHOST =
Inet4Address.getLoopbackAddress().getHostAddress();
+private static final String CLUSTER = "broken_ledger_test";
+private static final String DEFAULT_TENANT = "public";
+private static final String DEFAULT_NAMESPACE = DEFAULT_TENANT +
"/default";
+
+protected LocalBookkeeperEnsemble bkEnsemble;
+protected ServiceConfiguration pulsarConfig;
+protected PulsarService pulsarService;
+protected int brokerWebServicePort;
+protected int brokerServicePort;
+protected String metadataServiceUri;
+protected BookKeeper bookKeeperClient;
+protected String brokerUrl;
+protected String brokerServiceUrl;
+protected PulsarAdmin pulsarAdmin;
+protected PulsarClient pulsarClient;
+
+@BeforeClass
+protected void setup() throws Exception {
+log.info("--- Start cluster ---");
+startLocalBookie();
+initPulsarConfig();
+startPulsar();
+}
+
+@AfterClass
+protected void cleanup() throws Exception {
+log.info("--- Shutting down ---");
+silentStopPulsar();
+stopLocalBookie();
+}
+
+protected void startLocalBookie() throws Exception{
+log.info("===> Start bookie ");
+bkEnsemble = new LocalBookkeeperEnsemble(3, 0, () -> 0);
+bkEnsemble.start();
+metadataServiceUri = String.format("zk:%s:%s", LOCALHOST,
bkEnsemble.getZookeeperPort());
+initBookieClient();
+}
+
+protected void initBookieClient() throws Exception {
+bookKeeperClient = new BookKeeper(String.format("%s:%s", LOCALHOST,
bkEnsemble.getZookeeperPort()));
+}
+
+protected void stopLocalBookie() {
+log.info("===> Close bookie client");
+try {
+bookKeeperClient.close();
+} catch (Exception e){
+log.error("Close bookie client fail", e);
+}
+
[GitHub] [pulsar] mattisonchao commented on a diff in pull request #18620: [fix] [broker] If ledger lost, cursor makr delete position can not forward
mattisonchao commented on code in PR #18620:
URL: https://github.com/apache/pulsar/pull/18620#discussion_r1031919293
##
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpReadEntry.java:
##
@@ -117,6 +119,9 @@ public void readEntriesFailed(ManagedLedgerException
exception, Object ctx) {
return;
}
updateReadPosition(nexReadPosition);
+if (lostLedger != null) {
+clearIncompleteAckedRecordsFromLedger(lostLedger);
Review Comment:
Call `cursor.getManagedLedger().removeNonRecoverableLedger(ledgerId);`
directly to avoid adding a useless stack frame.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on a diff in pull request #18620: [fix] [broker] If ledger lost, cursor makr delete position can not forward
mattisonchao commented on code in PR #18620:
URL: https://github.com/apache/pulsar/pull/18620#discussion_r1031918767
##
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java:
##
@@ -1729,6 +1729,18 @@ synchronized void ledgerClosed(final LedgerHandle lh) {
}
}
+@Override
+public void removeNonRecoverableLedger(long ledgerId){
+ledgers.remove(ledgerId);
+Iterator managedCursorIterator = cursors.iterator();
+while (managedCursorIterator.hasNext()){
+ManagedCursor managedCursor = managedCursorIterator.next();
+if (managedCursor instanceof ManagedCursorImpl managedCursorImpl){
+
managedCursorImpl.clearIncompleteAckedRecordsByLedgerId(ledgerId);
Review Comment:
Using interface implementations may break encapsulation, but I'm not sure
it's justified here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on a diff in pull request #18620: [fix] [broker] If ledger lost, cursor makr delete position can not forward
mattisonchao commented on code in PR #18620:
URL: https://github.com/apache/pulsar/pull/18620#discussion_r1031917592
##
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java:
##
@@ -2661,6 +2708,34 @@ void setReadPosition(Position newReadPositionInt) {
}
}
+void clearIncompleteAckedRecordsByLedgerId(final long ledgerId){
+lock.writeLock().lock();
+try {
+List rangeListToDelete =
individualDeletedMessages.asRanges().stream()
+.filter(range -> range.lowerEndpoint().getLedgerId() ==
ledgerId)
+.map(range -> Range.openClosed(
Review Comment:
Why can't we return the range direct?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[pulsar-site] branch main updated: Docs sync done from apache/pulsar(#a0a1c35)
This is an automated email from the ASF dual-hosted git repository. urfree pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/pulsar-site.git The following commit(s) were added to refs/heads/main by this push: new 03ed9ebe90a Docs sync done from apache/pulsar(#a0a1c35) 03ed9ebe90a is described below commit 03ed9ebe90ae2032fa42725e25440b83ad43845f Author: Pulsar Site Updater AuthorDate: Fri Nov 25 00:01:39 2022 + Docs sync done from apache/pulsar(#a0a1c35)
[GitHub] [pulsar] mattisonchao commented on a diff in pull request #18620: [fix] [broker] If ledger lost, cursor makr delete position can not forward
mattisonchao commented on code in PR #18620:
URL: https://github.com/apache/pulsar/pull/18620#discussion_r1031916835
##
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java:
##
@@ -2661,6 +2708,34 @@ void setReadPosition(Position newReadPositionInt) {
}
}
+void clearIncompleteAckedRecordsByLedgerId(final long ledgerId){
+lock.writeLock().lock();
+try {
+List rangeListToDelete =
individualDeletedMessages.asRanges().stream()
+.filter(range -> range.lowerEndpoint().getLedgerId() ==
ledgerId)
+.map(range -> Range.openClosed(
+new LongPairRangeSet.LongPair(ledgerId,
range.lowerEndpoint().getEntryId()),
+new LongPairRangeSet.LongPair(ledgerId,
range.upperEndpoint().getEntryId())
Review Comment:
`range.upperEndpoint().getEntryId() -1` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on a diff in pull request #18620: [fix] [broker] If ledger lost, cursor makr delete position can not forward
mattisonchao commented on code in PR #18620:
URL: https://github.com/apache/pulsar/pull/18620#discussion_r1031915619
##
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java:
##
@@ -2661,6 +2708,34 @@ void setReadPosition(Position newReadPositionInt) {
}
}
+void clearIncompleteAckedRecordsByLedgerId(final long ledgerId){
Review Comment:
Access Modifiers?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on a diff in pull request #18620: [fix] [broker] If ledger lost, cursor makr delete position can not forward
mattisonchao commented on code in PR #18620:
URL: https://github.com/apache/pulsar/pull/18620#discussion_r1031915208
##
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java:
##
@@ -480,7 +482,24 @@ public void operationComplete(ManagedCursorInfo info, Stat
stat) {
// Need to proceed and read the last entry in the
specified ledger to find out the last position
log.info("[{}] Consumer {} meta-data recover from ledger
{}", ledger.getName(), name,
info.getCursorsLedgerId());
-recoverFromLedger(info, callback);
+recoverFromLedger(info, new VoidCallback() {
+@Override
+public void operationComplete() {
+LinkedHashSet ledgersDeletedButNotAcked =
checkLedgersDeletedButNotAcked();
Review Comment:
Why do we need this checking logic? It looks like we already handled at
cursor got `NonRecoverableLedgerException`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on a diff in pull request #18620: [fix] [broker] If ledger lost, cursor makr delete position can not forward
mattisonchao commented on code in PR #18620:
URL: https://github.com/apache/pulsar/pull/18620#discussion_r1031915208
##
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java:
##
@@ -480,7 +482,24 @@ public void operationComplete(ManagedCursorInfo info, Stat
stat) {
// Need to proceed and read the last entry in the
specified ledger to find out the last position
log.info("[{}] Consumer {} meta-data recover from ledger
{}", ledger.getName(), name,
info.getCursorsLedgerId());
-recoverFromLedger(info, callback);
+recoverFromLedger(info, new VoidCallback() {
+@Override
+public void operationComplete() {
+LinkedHashSet ledgersDeletedButNotAcked =
checkLedgersDeletedButNotAcked();
Review Comment:
Why do we need this checking logic? It looks like we already processed at
cursor got `NonRecoverableLedgerException`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on a diff in pull request #18620: [fix] [broker] If ledger lost, cursor makr delete position can not forward
mattisonchao commented on code in PR #18620:
URL: https://github.com/apache/pulsar/pull/18620#discussion_r1031914094
##
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java:
##
@@ -2661,6 +2708,34 @@ void setReadPosition(Position newReadPositionInt) {
}
}
+void clearIncompleteAckedRecordsByLedgerId(final long ledgerId){
+lock.writeLock().lock();
+try {
+List rangeListToDelete =
individualDeletedMessages.asRanges().stream()
+.filter(range -> range.lowerEndpoint().getLedgerId() ==
ledgerId)
+.map(range -> Range.openClosed(
+new LongPairRangeSet.LongPair(ledgerId,
range.lowerEndpoint().getEntryId()),
+new LongPairRangeSet.LongPair(ledgerId,
range.upperEndpoint().getEntryId())
+)).collect(Collectors.toList());
+if (!rangeListToDelete.isEmpty()) {
+rangeListToDelete.forEach(individualDeletedMessages::remove);
+}
+
+if (batchDeletedIndexes != null) {
+Set batchedIndexesToDelete =
batchDeletedIndexes.keySet().stream()
+.filter(position -> position.getLedgerId() == ledgerId)
+.collect(Collectors.toSet());
+if (!batchedIndexesToDelete.isEmpty()) {
+
batchedIndexesToDelete.forEach(batchDeletedIndexes::remove);
+}
+}
+} catch (Exception e){
+e.printStackTrace();
Review Comment:
why `e.printStackTrace();`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on a diff in pull request #18620: [fix] [broker] If ledger lost, cursor makr delete position can not forward
mattisonchao commented on code in PR #18620:
URL: https://github.com/apache/pulsar/pull/18620#discussion_r1031913744
##
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpReadEntry.java:
##
@@ -102,9 +102,11 @@ public void readEntriesFailed(ManagedLedgerException
exception, Object ctx) {
readPosition, exception.getMessage());
final ManagedLedgerImpl ledger = (ManagedLedgerImpl)
cursor.getManagedLedger();
Position nexReadPosition;
+Long lostLedger = null;
if (exception instanceof
ManagedLedgerException.LedgerNotExistException) {
// try to find and move to next valid ledger
nexReadPosition =
cursor.getNextLedgerPosition(readPosition.getLedgerId());
+lostLedger = readPosition.ledgerId;
} else {
// Skip this read operation
nexReadPosition =
ledger.getValidPositionAfterSkippedEntries(readPosition, count);
Review Comment:
I think we also need to check here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] eolivelli commented on pull request #18594: [improve][broker] Add an option to return 0 when querying partitions of a nonexistent topic
eolivelli commented on PR #18594: URL: https://github.com/apache/pulsar/pull/18594#issuecomment-1326869063 @rdhabalia PTAL -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch master updated (f89a2fac959 -> a0a1c357d16)
This is an automated email from the ASF dual-hosted git repository. eolivelli pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git from f89a2fac959 [fix][proxy] Remove duplicate test dependency org.awaitility:awaitility in pulsar-proxy (#18606) add a0a1c357d16 [fix][misc] Fix NoClassDefFoundError: io/netty/incubator/channel/uring/IOUringEventLoopGroup (#18609) No new revisions were added by this update. Summary of changes: pom.xml | 5 + pulsar-common/pom.xml | 5 + 2 files changed, 10 insertions(+)
[GitHub] [pulsar] eolivelli closed issue #18530: [Bug] bin/pulsar-client NoClassDefFoundError: io/netty/incubator/channel/uring/IOUringEventLoopGroup
eolivelli closed issue #18530: [Bug] bin/pulsar-client NoClassDefFoundError: io/netty/incubator/channel/uring/IOUringEventLoopGroup URL: https://github.com/apache/pulsar/issues/18530 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] eolivelli merged pull request #18609: [fix][misc] Fix NoClassDefFoundError: io/netty/incubator/channel/uring/IOUringEventLoopGroup
eolivelli merged PR #18609: URL: https://github.com/apache/pulsar/pull/18609 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] eolivelli commented on a diff in pull request #18609: [fix][misc] Fix NoClassDefFoundError: io/netty/incubator/channel/uring/IOUringEventLoopGroup
eolivelli commented on code in PR #18609: URL: https://github.com/apache/pulsar/pull/18609#discussion_r1031888905 ## pulsar-common/pom.xml: ## @@ -131,6 +131,11 @@ netty-tcnative-boringssl-static + + io.netty.incubator + netty-incubator-transport-classes-io_uring Review Comment: Maybe it depends on your maven version -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] poorbarcode opened a new pull request, #18620: [fix] [broker] If ledger lost, cursor makr delete position can not forward
poorbarcode opened a new pull request, #18620:
URL: https://github.com/apache/pulsar/pull/18620
### Motivation
Configuration `autoSkipNonRecoverableData` is designed to turn this feature
on if we can accept partial data loss. When a ledger is lost, the broker will
still work. But now we have this problem: If a ledger is lost, consumer and
producer can work, but the cursor mark delete position can not forward.
For example: `topic-1` has 3 ledgers:
```
{
ledegr-1 : [1:1],[1:2],[1:3],[1:4],[1:5]
ledegr-2: [2:1],[2:2],[2:3],[2:4],[2:5]
ledegr-3: [3:1],[3:2],[3:3],[3:4],[3:5]
}
```
### Modifications
### Documentation
- [ ] `doc`
- [ ] `doc-required`
- [x] `doc-not-needed`
- [ ] `doc-complete`
### Matching PR in forked repository
PR in forked repository:
- https://github.com/poorbarcode/pulsar/pull/44
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] moweonlee commented on issue #13167: Not enough non-faulty bookies available
moweonlee commented on issue #13167: URL: https://github.com/apache/pulsar/issues/13167#issuecomment-1326852831 @arpillai89 Here is the configuration that I used for my pulsar cluster. I didn't think about how much storage volume is necessary to store all data incoming. Now it is clear that my case was pretty simple. So it will be good to see Grafana's storage volume chart first as long as Pulsar is deployed in K8s ``` ./bin/pulsar-admin namespaces set-retention public/default --size 8G --time -1 ./bin/pulsar-admin namespaces get-retention public/default ./bin/pulsar-admin namespaces set-message-ttl public/default --messageTTL 60 ./bin/pulsar-admin namespaces get-message-ttl public/defaul ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] nicoloboschi closed pull request #18609: [fix][misc] Fix NoClassDefFoundError: io/netty/incubator/channel/uring/IOUringEventLoopGroup
nicoloboschi closed pull request #18609: [fix][misc] Fix NoClassDefFoundError: io/netty/incubator/channel/uring/IOUringEventLoopGroup URL: https://github.com/apache/pulsar/pull/18609 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] nicoloboschi closed pull request #18608: [improve][cli] Pulsar shell: allow cloning an existing config
nicoloboschi closed pull request #18608: [improve][cli] Pulsar shell: allow cloning an existing config URL: https://github.com/apache/pulsar/pull/18608 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch master updated: [fix][proxy] Remove duplicate test dependency org.awaitility:awaitility in pulsar-proxy (#18606)
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new f89a2fac959 [fix][proxy] Remove duplicate test dependency
org.awaitility:awaitility in pulsar-proxy (#18606)
f89a2fac959 is described below
commit f89a2fac95947a9c9087d3aafdc0f058f86899d6
Author: Lari Hotari
AuthorDate: Thu Nov 24 21:56:51 2022 +0200
[fix][proxy] Remove duplicate test dependency org.awaitility:awaitility in
pulsar-proxy (#18606)
---
pulsar-proxy/pom.xml | 5 -
1 file changed, 5 deletions(-)
diff --git a/pulsar-proxy/pom.xml b/pulsar-proxy/pom.xml
index a870dbecded..0fcc2c5c7aa 100644
--- a/pulsar-proxy/pom.xml
+++ b/pulsar-proxy/pom.xml
@@ -180,11 +180,6 @@
ipaddress
${seancfoley.ipaddress.version}
-
- org.awaitility
- awaitility
- test
-
[GitHub] [pulsar] lhotari merged pull request #18606: [fix][proxy] Remove duplicate test dependency org.awaitility:awaitility in pulsar-proxy
lhotari merged PR #18606: URL: https://github.com/apache/pulsar/pull/18606 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] nicoloboschi opened a new pull request, #18619: [fix][docs] Fix elasticsearch sink layout
nicoloboschi opened a new pull request, #18619: URL: https://github.com/apache/pulsar/pull/18619 ### Motivation  ### Modifications * Removed whitespaces - [x] `doc` - [ ] `doc-required` - [ ] `doc-not-needed` - [ ] `doc-complete` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] 01/01: [fix][docs] Fix elasticsearch sink layout
This is an automated email from the ASF dual-hosted git repository. nicoloboschi pushed a commit to branch fix-es-doc in repository https://gitbox.apache.org/repos/asf/pulsar.git commit 65c9174d6dd4f13805449084910d723b67c71327 Author: Nicolò Boschi AuthorDate: Thu Nov 24 20:48:54 2022 +0100 [fix][docs] Fix elasticsearch sink layout --- site2/docs/io-elasticsearch-sink.md | 12 ++-- 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/site2/docs/io-elasticsearch-sink.md b/site2/docs/io-elasticsearch-sink.md index 26e04569ef7..ba593e62ad1 100644 --- a/site2/docs/io-elasticsearch-sink.md +++ b/site2/docs/io-elasticsearch-sink.md @@ -33,17 +33,17 @@ entry.The `DELETE` operation can be performed when the primary key i not empty and the remaining value is empty. Use the `nullValueAction` to configure this behavior. The default configuration simply ignores such empty values. - + ### Map multiple indexes - + Since Pulsar 2.9.0, the `indexName` property is no more required. If you omit it, the sink writes to an index name after the Pulsar topic name. - + ### Enable bulk writes - + Since Pulsar 2.9.0, you can use bulk writes by setting the `bulkEnabled` property to `true`. - + ### Enable secure connections via TLS - + Since Pulsar 2.9.0, you can enable secure connections with TLS. ## Configuration
[pulsar] branch fix-es-doc created (now 65c9174d6dd)
This is an automated email from the ASF dual-hosted git repository. nicoloboschi pushed a change to branch fix-es-doc in repository https://gitbox.apache.org/repos/asf/pulsar.git at 65c9174d6dd [fix][docs] Fix elasticsearch sink layout This branch includes the following new commits: new 65c9174d6dd [fix][docs] Fix elasticsearch sink layout The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[pulsar] branch master updated: [fix][test] Close KafkaProducer in integration tests to prevent thread leak (#18618)
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 86f4e0c6825 [fix][test] Close KafkaProducer in integration tests to
prevent thread leak (#18618)
86f4e0c6825 is described below
commit 86f4e0c6825a0d54362bb81380b4646eb6ae13af
Author: Lari Hotari
AuthorDate: Thu Nov 24 21:10:46 2022 +0200
[fix][test] Close KafkaProducer in integration tests to prevent thread leak
(#18618)
---
.../integration/io/sources/KafkaSourceTester.java | 33 +++---
1 file changed, 17 insertions(+), 16 deletions(-)
diff --git
a/tests/integration/src/test/java/org/apache/pulsar/tests/integration/io/sources/KafkaSourceTester.java
b/tests/integration/src/test/java/org/apache/pulsar/tests/integration/io/sources/KafkaSourceTester.java
index 772e2662df6..9eab0084091 100644
---
a/tests/integration/src/test/java/org/apache/pulsar/tests/integration/io/sources/KafkaSourceTester.java
+++
b/tests/integration/src/test/java/org/apache/pulsar/tests/integration/io/sources/KafkaSourceTester.java
@@ -118,29 +118,30 @@ public class KafkaSourceTester extends
SourceTester {
@Override
public Map produceSourceMessages(int numMessages) throws
Exception{
-KafkaProducer producer = new KafkaProducer<>(
+try (KafkaProducer producer = new KafkaProducer<>(
ImmutableMap.of(
ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,
kafkaContainer.getBootstrapServers(),
ProducerConfig.CLIENT_ID_CONFIG,
UUID.randomUUID().toString()
),
new StringSerializer(),
new StringSerializer()
-);
-LinkedHashMap kvs = new LinkedHashMap<>();
-for (int i = 0; i < numMessages; i++) {
-String key = "key-" + i;
-String value = "value-" + i;
-ProducerRecord record = new ProducerRecord<>(
-kafkaTopicName,
-key,
-value
-);
-kvs.put(key, value);
-producer.send(record).get();
+)) {
+LinkedHashMap kvs = new LinkedHashMap<>();
+for (int i = 0; i < numMessages; i++) {
+String key = "key-" + i;
+String value = "value-" + i;
+ProducerRecord record = new ProducerRecord<>(
+kafkaTopicName,
+key,
+value
+);
+kvs.put(key, value);
+producer.send(record).get();
+}
+
+log.info("Successfully produced {} messages to kafka topic {}",
numMessages, kafkaTopicName);
+return kvs;
}
-
-log.info("Successfully produced {} messages to kafka topic {}",
numMessages, kafkaTopicName);
-return kvs;
}
@Override
[GitHub] [pulsar] lhotari closed issue #15853: Pulsar IO tests leak resources in Kafka related tests
lhotari closed issue #15853: Pulsar IO tests leak resources in Kafka related tests URL: https://github.com/apache/pulsar/issues/15853 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] lhotari merged pull request #18618: [fix][test] Close KafkaProducer in integration tests to prevent thread leak
lhotari merged PR #18618: URL: https://github.com/apache/pulsar/pull/18618 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] lhotari closed pull request #18599: [improve][misc] Upgrade Netty to 4.1.85.Final and Netty Tcnative to 2.0.54.Final
lhotari closed pull request #18599: [improve][misc] Upgrade Netty to 4.1.85.Final and Netty Tcnative to 2.0.54.Final URL: https://github.com/apache/pulsar/pull/18599 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch master updated: [fix][ci] Increase thread stack size to 1500k for Maven JVM to prevent StackOverflowError in compilation (#18602)
This is an automated email from the ASF dual-hosted git repository. lhotari pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/master by this push: new b3d09850f01 [fix][ci] Increase thread stack size to 1500k for Maven JVM to prevent StackOverflowError in compilation (#18602) b3d09850f01 is described below commit b3d09850f014b7e2d5467711ecb06023636f8a17 Author: Lari Hotari AuthorDate: Thu Nov 24 20:37:33 2022 +0200 [fix][ci] Increase thread stack size to 1500k for Maven JVM to prevent StackOverflowError in compilation (#18602) --- .github/workflows/ci-go-functions.yaml | 2 +- .github/workflows/ci-maven-cache-update.yaml | 2 +- .github/workflows/ci-owasp-dependency-check.yaml | 2 +- .github/workflows/pulsar-ci-flaky.yaml | 2 +- .github/workflows/pulsar-ci.yaml | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-) diff --git a/.github/workflows/ci-go-functions.yaml b/.github/workflows/ci-go-functions.yaml index 57c806b4d88..8ff8023fe46 100644 --- a/.github/workflows/ci-go-functions.yaml +++ b/.github/workflows/ci-go-functions.yaml @@ -32,7 +32,7 @@ concurrency: cancel-in-progress: true env: - MAVEN_OPTS: -Dhttp.keepAlive=false -Dmaven.wagon.http.pool=false -Dmaven.wagon.http.retryHandler.class=standard -Dmaven.wagon.http.retryHandler.count=3 + MAVEN_OPTS: -Xss1500k -Dhttp.keepAlive=false -Dmaven.wagon.http.pool=false -Dmaven.wagon.http.retryHandler.class=standard -Dmaven.wagon.http.retryHandler.count=3 jobs: preconditions: diff --git a/.github/workflows/ci-maven-cache-update.yaml b/.github/workflows/ci-maven-cache-update.yaml index bf7f70bb032..daba3dcc032 100644 --- a/.github/workflows/ci-maven-cache-update.yaml +++ b/.github/workflows/ci-maven-cache-update.yaml @@ -42,7 +42,7 @@ on: - cron: '30 */12 * * *' env: - MAVEN_OPTS: -Dhttp.keepAlive=false -Dmaven.wagon.http.pool=false -Dmaven.wagon.http.retryHandler.class=standard -Dmaven.wagon.http.retryHandler.count=3 + MAVEN_OPTS: -Xss1500k -Dhttp.keepAlive=false -Dmaven.wagon.http.pool=false -Dmaven.wagon.http.retryHandler.class=standard -Dmaven.wagon.http.retryHandler.count=3 jobs: update-maven-dependencies-cache: diff --git a/.github/workflows/ci-owasp-dependency-check.yaml b/.github/workflows/ci-owasp-dependency-check.yaml index 595d82f4ce0..38bd7f9eec4 100644 --- a/.github/workflows/ci-owasp-dependency-check.yaml +++ b/.github/workflows/ci-owasp-dependency-check.yaml @@ -23,7 +23,7 @@ on: - cron: '15 0 * * *' env: - MAVEN_OPTS: -Dhttp.keepAlive=false -Dmaven.wagon.http.pool=false -Dmaven.wagon.http.retryHandler.class=standard -Dmaven.wagon.http.retryHandler.count=3 + MAVEN_OPTS: -Xss1500k -Dhttp.keepAlive=false -Dmaven.wagon.http.pool=false -Dmaven.wagon.http.retryHandler.class=standard -Dmaven.wagon.http.retryHandler.count=3 jobs: run-owasp-dependency-check: diff --git a/.github/workflows/pulsar-ci-flaky.yaml b/.github/workflows/pulsar-ci-flaky.yaml index b7f142f8891..9c52cfe77d6 100644 --- a/.github/workflows/pulsar-ci-flaky.yaml +++ b/.github/workflows/pulsar-ci-flaky.yaml @@ -29,7 +29,7 @@ concurrency: cancel-in-progress: true env: - MAVEN_OPTS: -Xmx1024m -Dhttp.keepAlive=false -Dmaven.wagon.http.pool=false -Dmaven.wagon.http.retryHandler.class=standard -Dmaven.wagon.http.retryHandler.count=3 + MAVEN_OPTS: -Xss1500k -Xmx1024m -Dhttp.keepAlive=false -Dmaven.wagon.http.pool=false -Dmaven.wagon.http.retryHandler.class=standard -Dmaven.wagon.http.retryHandler.count=3 # defines the retention period for the intermediate build artifacts needed for rerunning a failed build job # it's possible to rerun individual failed jobs when the build artifacts are available # if the artifacts have already been expired, the complete workflow can be rerun by closing and reopening the PR or by rebasing the PR diff --git a/.github/workflows/pulsar-ci.yaml b/.github/workflows/pulsar-ci.yaml index dad5d55c967..ea518b58043 100644 --- a/.github/workflows/pulsar-ci.yaml +++ b/.github/workflows/pulsar-ci.yaml @@ -31,7 +31,7 @@ concurrency: cancel-in-progress: true env: - MAVEN_OPTS: -Xmx1024m -Dhttp.keepAlive=false -Dmaven.wagon.http.pool=false -Dmaven.wagon.http.retryHandler.class=standard -Dmaven.wagon.http.retryHandler.count=3 + MAVEN_OPTS: -Xss1500k -Xmx1024m -Dhttp.keepAlive=false -Dmaven.wagon.http.pool=false -Dmaven.wagon.http.retryHandler.class=standard -Dmaven.wagon.http.retryHandler.count=3 # defines the retention period for the intermediate build artifacts needed for rerunning a failed build job # it's possible to rerun individual failed jobs when the build artifacts are available # if the artifacts have already been expired, the complete workflow can be rerun by closing and reopening the PR or by rebasing the PR
[GitHub] [pulsar] lhotari merged pull request #18602: [fix][ci] Increase thread stack size to 1500k for Maven JVM to prevent StackOverflowError in compilation
lhotari merged PR #18602: URL: https://github.com/apache/pulsar/pull/18602 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch master updated: [improve][misc] Upgrade Lombok to 1.18.24 (#18604)
This is an automated email from the ASF dual-hosted git repository. lhotari pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/master by this push: new 00650b01c30 [improve][misc] Upgrade Lombok to 1.18.24 (#18604) 00650b01c30 is described below commit 00650b01c30399053776748731a93faa7c9b5df5 Author: Lari Hotari AuthorDate: Thu Nov 24 20:36:41 2022 +0200 [improve][misc] Upgrade Lombok to 1.18.24 (#18604) --- pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pom.xml b/pom.xml index d02329a841d..1c38442041d 100644 --- a/pom.xml +++ b/pom.xml @@ -208,7 +208,7 @@ flexible messaging model and an intuitive client API. 0.9.1 2.1.0 3.18.1 -1.18.22 +1.18.24 1.3.2 2.3.1 1.2.0
[GitHub] [pulsar] lhotari merged pull request #18604: [improve][misc] Upgrade Lombok to 1.18.24
lhotari merged PR #18604: URL: https://github.com/apache/pulsar/pull/18604 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] lhotari commented on issue #18588: Flaky-test: NamespacesTest.testSetOffloadThreshold
lhotari commented on issue #18588: URL: https://github.com/apache/pulsar/issues/18588#issuecomment-1326760564 This seems to be caused by #17875 changes. I'll take a look. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] lhotari closed pull request #18606: [fix][proxy] Remove duplicate test dependency org.awaitility:awaitility in pulsar-proxy
lhotari closed pull request #18606: [fix][proxy] Remove duplicate test dependency org.awaitility:awaitility in pulsar-proxy URL: https://github.com/apache/pulsar/pull/18606 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar-site] branch main updated: Docs sync done from apache/pulsar(#bf3ee85)
This is an automated email from the ASF dual-hosted git repository. urfree pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/pulsar-site.git The following commit(s) were added to refs/heads/main by this push: new 105974789c8 Docs sync done from apache/pulsar(#bf3ee85) 105974789c8 is described below commit 105974789c89fa0ac54e9cfcc4b28fded4acc3cf Author: Pulsar Site Updater AuthorDate: Thu Nov 24 18:01:40 2022 + Docs sync done from apache/pulsar(#bf3ee85) --- site2/website-next/docs/client-libraries-java.md | 1 + .../docs/functions-deploy-cluster-builtin.md | 16 .../website-next/docs/functions-deploy-cluster.md | 1 + site2/website-next/docs/functions-package-java.md | 101 - site2/website-next/sidebars.json | 3 +- 5 files changed, 120 insertions(+), 2 deletions(-) diff --git a/site2/website-next/docs/client-libraries-java.md b/site2/website-next/docs/client-libraries-java.md index 7127368cf24..b35d7c13d91 100644 --- a/site2/website-next/docs/client-libraries-java.md +++ b/site2/website-next/docs/client-libraries-java.md @@ -1129,6 +1129,7 @@ You can use the available parameters in the `loadConf` configuration or related |---|---|---|---|--- | `topic` | string | yes | The topic name of the TableView. | N/A | `autoUpdatePartitionInterval` | int | no | The interval to check for newly added partitions. | 60 (seconds) +| `subscriptionName` | string | no | The subscription name of the TableView. | null ### Register listeners diff --git a/site2/website-next/docs/functions-deploy-cluster-builtin.md b/site2/website-next/docs/functions-deploy-cluster-builtin.md new file mode 100644 index 000..2e0e3b3cd38 --- /dev/null +++ b/site2/website-next/docs/functions-deploy-cluster-builtin.md @@ -0,0 +1,16 @@ +--- +id: functions-deploy-cluster-builtin +title: Use built-in functions +sidebar_label: "Use built-in functions" +--- + +Similar to built-in connectors, the code of Java functions [packaged as NAR](functions-package-java.md) that are placed in the `functions` directory of the function worker are loaded at startup and can be referenced when creating a function. + +For instance if you have a built-in function with name `exclamation` in its `pulsar-io.yaml`, you can create a function instance with: + +```bash +bin/pulsar-admin functions create \ + --function-type exclamation \ + --inputs persistent://public/default/input-1 \ + --output persistent://public/default/output-1 +``` diff --git a/site2/website-next/docs/functions-deploy-cluster.md b/site2/website-next/docs/functions-deploy-cluster.md index 042d5a3d2ed..ccfd36e63de 100644 --- a/site2/website-next/docs/functions-deploy-cluster.md +++ b/site2/website-next/docs/functions-deploy-cluster.md @@ -31,4 +31,5 @@ bin/pulsar-admin functions update \ * [Enable parallel processing](functions-deploy-cluster-parallelism.md) * [Enable end-to-end encryption](functions-deploy-cluster-encryption.md) * [Enable package management service](functions-deploy-cluster-package.md) +* [Use built-in functions](functions-deploy-cluster-builtin.md) diff --git a/site2/website-next/docs/functions-package-java.md b/site2/website-next/docs/functions-package-java.md index f2187a14357..daa3bf8aa4c 100644 --- a/site2/website-next/docs/functions-package-java.md +++ b/site2/website-next/docs/functions-package-java.md @@ -4,13 +4,25 @@ title: Package Java Functions sidebar_label: "Package Java Functions" --- +There are two methods to package Java Functions, that is [uber JAR](#package-as-jar) and [NAR](#package-as-nar). + :::note +If you plan to package and distribute your function for others to use, you are obligated to +license and copyright your own code properly. Remember to add the license and copyright to +all libraries your code uses and to your distribution. + +If you use the [NAR](#package-as-nar) method, the NAR plugin +automatically creates a `DEPENDENCIES` file in the generated NAR package, including the proper +licensing and copyrights of all libraries of your function. + For the runtime Java version, refer to [Pulsar Runtime Java Version Recommendation](https://github.com/apache/pulsar/blob/master/README.md#pulsar-runtime-java-version-recommendation) according to your target Pulsar version. ::: -To package a Java function, complete the following steps. +## Package as JAR + +To package a Java function as JAR, complete the following steps. 1. Create a new maven project with a pom file. In the following code sample, the value of `mainClass` is your package name. @@ -106,3 +118,90 @@ To package a Java function, complete the following steps. 07:55:03.724 [main] INFO org.apache.pulsar.functions.runtime.ProcessRuntime - Started process successfully ... ``` + +## Package as NAR + +To package a Java function as NAR, complete the following steps. + +1. Create a new maven project with a pom file. + + ```xml
