[GitHub] [pulsar-client-go] fracasula commented on issue #995: When a new release?
fracasula commented on issue #995: URL: https://github.com/apache/pulsar-client-go/issues/995#issuecomment-1501048063 Thanks @RobertIndie :+1: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-go] liangyepianzhou closed pull request #1002: [feat][txn]Implement transactional consumer/producer API
liangyepianzhou closed pull request #1002: [feat][txn]Implement transactional consumer/producer API URL: https://github.com/apache/pulsar-client-go/pull/1002 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar-client-go] branch txn_API updated (b68967b -> fafc99e)
This is an automated email from the ASF dual-hosted git repository. xiangying pushed a change to branch txn_API in repository https://gitbox.apache.org/repos/asf/pulsar-client-go.git discard b68967b debug discard 91acf23 debug discard 849ae36 fix race condition discard 61d2f1d fix race condition add fafc99e fix test This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (b68967b) \ N -- N -- N refs/heads/txn_API (fafc99e) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: pulsar/transaction_test.go | 13 ++--- 1 file changed, 2 insertions(+), 11 deletions(-)
[pulsar-client-go] branch txn_API updated (91acf23 -> b68967b)
This is an automated email from the ASF dual-hosted git repository. xiangying pushed a change to branch txn_API in repository https://gitbox.apache.org/repos/asf/pulsar-client-go.git from 91acf23 debug add b68967b debug No new revisions were added by this update. Summary of changes: pulsar/transaction_test.go | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
[pulsar-client-go] branch txn_API updated (849ae36 -> 91acf23)
This is an automated email from the ASF dual-hosted git repository. xiangying pushed a change to branch txn_API in repository https://gitbox.apache.org/repos/asf/pulsar-client-go.git from 849ae36 fix race condition add 91acf23 debug No new revisions were added by this update. Summary of changes: pulsar/transaction_test.go | 11 ++- 1 file changed, 10 insertions(+), 1 deletion(-)
[pulsar-client-go] branch txn_API updated (61d2f1d -> 849ae36)
This is an automated email from the ASF dual-hosted git repository. xiangying pushed a change to branch txn_API in repository https://gitbox.apache.org/repos/asf/pulsar-client-go.git from 61d2f1d fix race condition add 849ae36 fix race condition No new revisions were added by this update. Summary of changes: pulsar/transaction_test.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
[GitHub] [pulsar] Apurva007 commented on pull request #20035: [Fix][Doc] Update docs for Partition increase command
Apurva007 commented on PR #20035: URL: https://github.com/apache/pulsar/pull/20035#issuecomment-1501033501 Tested workflow in 2 Pulsar clusters connected via geo-replication:   -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] Apurva007 commented on issue #17878: [Doc] Fix Admin Interface docs for Managing partitioned topics
Apurva007 commented on issue #17878: URL: https://github.com/apache/pulsar/issues/17878#issuecomment-1501032116 Fixed the CI checks using the codecov suggestion by adding a token. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar-client-go] branch txn_API updated (f5e1893 -> 61d2f1d)
This is an automated email from the ASF dual-hosted git repository. xiangying pushed a change to branch txn_API in repository https://gitbox.apache.org/repos/asf/pulsar-client-go.git from f5e1893 rollback add 61d2f1d fix race condition No new revisions were added by this update. Summary of changes: pulsar/transaction_test.go | 17 + 1 file changed, 9 insertions(+), 8 deletions(-)
[GitHub] [pulsar] yyj8 commented on pull request #20048: [improve][broker]PIP-255 Add topic metrics for the number of production data requests to add a topic and the average number of messages per req
yyj8 commented on PR #20048: URL: https://github.com/apache/pulsar/pull/20048#issuecomment-1501018944 /pulsarbot run-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #18963: Add metrics for failed topic load operation
github-actions[bot] commented on issue #18963: URL: https://github.com/apache/pulsar/issues/18963#issuecomment-1501018542 The issue had no activity for 30 days, mark with Stale label. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on pull request #19501: [fix][broker] unify topic-level policies enable judgment conditions
github-actions[bot] commented on PR #19501: URL: https://github.com/apache/pulsar/pull/19501#issuecomment-1501018210 The pr had no activity for 30 days, mark with Stale label. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #19601: PIP-248: Add metrics for backlog quota event
github-actions[bot] commented on issue #19601: URL: https://github.com/apache/pulsar/issues/19601#issuecomment-1501018142 The issue had no activity for 30 days, mark with Stale label. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar-site] branch main updated: Docs sync done from apache/pulsar (#90b9dd4)
This is an automated email from the ASF dual-hosted git repository. urfree pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/pulsar-site.git The following commit(s) were added to refs/heads/main by this push: new fd7e6fc83e4 Docs sync done from apache/pulsar (#90b9dd4) fd7e6fc83e4 is described below commit fd7e6fc83e4f683b11ed1d5968223fd84e2fc955 Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> AuthorDate: Sun Apr 9 01:37:36 2023 + Docs sync done from apache/pulsar (#90b9dd4) --- static/reference/next/config/reference-configuration-broker.md | 4 ++-- static/reference/next/config/reference-configuration-standalone.md | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/static/reference/next/config/reference-configuration-broker.md b/static/reference/next/config/reference-configuration-broker.md index a20fe7d2209..df4bd03c796 100644 --- a/static/reference/next/config/reference-configuration-broker.md +++ b/static/reference/next/config/reference-configuration-broker.md @@ -957,12 +957,12 @@ Interval to flush dynamic resource quota to ZooKeeper **Category**: Load Balancer -### loadBalancerServiceUnitStateCleanUpDelayTimeInSeconds +### loadBalancerServiceUnitStateTombstoneDelayTimeInSeconds After this delay, the service-unit state channel tombstones any service units (e.g., bundles) in semi-terminal states. For example, after splits, parent bundles will be `deleted`, and then after this delay, the parent bundles' state will be `tombstoned` in the service-unit state channel. Pulsar does not immediately remove such semi-terminal states to avoid unnecessary system confusion, as the bundles in the `tombstoned` state might temporarily look available to reassign. Rarely, one coul [...] **Type**: `long` -**Default**: `604800` +**Default**: `3600` **Dynamic**: `false` diff --git a/static/reference/next/config/reference-configuration-standalone.md b/static/reference/next/config/reference-configuration-standalone.md index a20fe7d2209..df4bd03c796 100644 --- a/static/reference/next/config/reference-configuration-standalone.md +++ b/static/reference/next/config/reference-configuration-standalone.md @@ -957,12 +957,12 @@ Interval to flush dynamic resource quota to ZooKeeper **Category**: Load Balancer -### loadBalancerServiceUnitStateCleanUpDelayTimeInSeconds +### loadBalancerServiceUnitStateTombstoneDelayTimeInSeconds After this delay, the service-unit state channel tombstones any service units (e.g., bundles) in semi-terminal states. For example, after splits, parent bundles will be `deleted`, and then after this delay, the parent bundles' state will be `tombstoned` in the service-unit state channel. Pulsar does not immediately remove such semi-terminal states to avoid unnecessary system confusion, as the bundles in the `tombstoned` state might temporarily look available to reassign. Rarely, one coul [...] **Type**: `long` -**Default**: `604800` +**Default**: `3600` **Dynamic**: `false`
[GitHub] [pulsar] Apurva007 commented on issue #17878: [Doc] Fix Admin Interface docs for Managing partitioned topics
Apurva007 commented on issue #17878: URL: https://github.com/apache/pulsar/issues/17878#issuecomment-1501001777 Created the PRs. Waiting for the CIs to pass. Currently the CI checks are failing due to #19952 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch master updated (421d707419e -> 90b9dd4c432)
This is an automated email from the ASF dual-hosted git repository. mmarshall pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git from 421d707419e [improve][broker] PIP-192 made split handler idempotent (#19988) add 90b9dd4c432 [fix][fn] Make /version return correct version (#20047) No new revisions were added by this update. Summary of changes: .../functions/worker/rest/ConfigurationResource.java | 15 +-- 1 file changed, 5 insertions(+), 10 deletions(-)
[GitHub] [pulsar] michaeljmarshall merged pull request #20047: [fix][fn] Make /version return correct version
michaeljmarshall merged PR #20047: URL: https://github.com/apache/pulsar/pull/20047 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on pull request #20044: [improve][client] Add retry when readTailMessages fails
heesung-sn commented on PR #20044: URL: https://github.com/apache/pulsar/pull/20044#issuecomment-1500970075 @lifepuzzlefun I updated the code to "retry-forever" upon `readTailMessages` failure. I assume the current "fail-stop" is not expected. Another thing to note is that since the reader acks messages immediately, this "retry" will not retry the handling logic with the same message. (It will move on and retry the next message) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] heesung-sn closed pull request #20045: [improve][broker] PIP-192 Add TableView Monitor
heesung-sn closed pull request #20045: [improve][broker] PIP-192 Add TableView Monitor URL: https://github.com/apache/pulsar/pull/20045 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on pull request #20045: [improve][broker] PIP-192 Add TableView Monitor
heesung-sn commented on PR #20045: URL: https://github.com/apache/pulsar/pull/20045#issuecomment-1500965842 Closing this PR, as the table-view is supposed to retry forever. https://github.com/apache/pulsar/pull/20044 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on pull request #20044: [improve][client] Add retry when readTailMessages fails
heesung-sn commented on PR #20044: URL: https://github.com/apache/pulsar/pull/20044#issuecomment-1500964944 @lifepuzzlefun I updated the code to "retry-forever" upon `readTailMessages` failure. I assume the current "fail-stop" is a bug. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on pull request #20044: [improve][client] Add retry when readTailMessages fails
heesung-sn commented on PR #20044: URL: https://github.com/apache/pulsar/pull/20044#issuecomment-1500963337 I updated the code to retry upon failure. @lifepuzzlefun -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codecov-commenter commented on pull request #20047: [fix][fn] Make /version return correct version
codecov-commenter commented on PR #20047: URL: https://github.com/apache/pulsar/pull/20047#issuecomment-1500953817 ## [Codecov](https://codecov.io/gh/apache/pulsar/pull/20047?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#20047](https://codecov.io/gh/apache/pulsar/pull/20047?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (b28f95c) into [master](https://codecov.io/gh/apache/pulsar/commit/3b118b6bd54cd8e592b8c2cd329a3fa3dfd18a11?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (3b118b6) will **increase** coverage by `0.05%`. > The diff coverage is `44.39%`. [](https://codecov.io/gh/apache/pulsar/pull/20047?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #20047 +/- ## + Coverage 72.86% 72.91% +0.05% - Complexity3162831649 +21 Files 1861 1858 -3 Lines137500 137420 -80 Branches 1514115107 -34 + Hits 100187 100203 +16 + Misses2935129236 -115 - Partials 7962 7981 +19 ``` | Flag | Coverage Δ | | |---|---|---| | inttests | `24.42% <0.95%> (+0.02%)` | :arrow_up: | | systests | `25.01% <20.04%> (-0.01%)` | :arrow_down: | | unittests | `72.18% <35.32%> (+0.05%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/pulsar/pull/20047?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...a/org/apache/pulsar/broker/admin/v2/Functions.java](https://codecov.io/gh/apache/pulsar/pull/20047?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9hZG1pbi92Mi9GdW5jdGlvbnMuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...java/org/apache/pulsar/broker/admin/v2/Worker.java](https://codecov.io/gh/apache/pulsar/pull/20047?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9hZG1pbi92Mi9Xb3JrZXIuamF2YQ==) | `11.76% <0.00%> (ø)` | | | [...org/apache/pulsar/broker/admin/v2/WorkerStats.java](https://codecov.io/gh/apache/pulsar/pull/20047?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9hZG1pbi92Mi9Xb3JrZXJTdGF0cy5qYXZh) | `40.00% <0.00%> (ø)` | | | [...r/functions/worker/rest/ConfigurationResource.java](https://codecov.io/gh/apache/pulsar/pull/20047?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWZ1bmN0aW9ucy93b3JrZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3B1bHNhci9mdW5jdGlvbnMvd29ya2VyL3Jlc3QvQ29uZmlndXJhdGlvblJlc291cmNlLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...ons/worker/rest/api/v2/FunctionsApiV2Resource.java](https://codecov.io/gh/apache/pulsar/pull/20047?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWZ1bmN0aW9ucy93b3JrZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3B1bHNhci9mdW5jdGlvbnMvd29ya2VyL3Jlc3QvYXBpL3YyL0Z1bmN0aW9uc0FwaVYyUmVzb3VyY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...s/worker/rest/api/v2/WorkerStatsApiV2Resource.java](https://codecov.io/gh/apache/pulsar/pull/20047?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWZ1bmN0aW9ucy93b3JrZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3B1bHNhci9mdW5jdGlvbnMvd29ya2VyL3Jlc3QvYXBpL3YyL1dvcmtlclN0YXRzQXBpVjJSZXNvdXJjZS5qYXZh) | `0.00% <0.00%> (ø)` | | |
[GitHub] [pulsar] heesung-sn commented on pull request #20044: [improve][client] Add isInterrupted func on TableView
heesung-sn commented on PR #20044: URL: https://github.com/apache/pulsar/pull/20044#issuecomment-1500947438 To expand this discussion a little further, I think the current tableview works under best-effort delivery and needs some improvement in the following areas for mission-critical data replication in production. - guarantee liveness(better error handling. e.g infinite retry) - handle slowness (how to handle slow consumer, e.g self-kill if too slow. If the consumer barely receives messages due to bugs, network, and etc.) - expose operational metrics(table view state(retrying, active), consume latency, highest messageId(highest logical sequence number)) Currently, it is hard or impossible for callers to check them outside of tableview. And this pr is trying to expose the worst-case tableview state, where tableview is interrupted and stopped to consume messages. We could change the func name to isAlive, isConnected, isClosed, isStopped, or other. I found isInterrupted is a good choice here, as it tells if tableview is "interrupted" by some unknown reason. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] yyj8 opened a new pull request, #20048: [improve][broker]PIP-255 Add topic metrics for the number of production data requests to add a topic and the average number of messages per re
yyj8 opened a new pull request, #20048: URL: https://github.com/apache/pulsar/pull/20048 ### Motivation In the current topic level monitoring, there is no metrics to monitor the number of send data requests and the number of messages per send request (i.e., each entry) for a topic. This is inconvenient for us to troubleshoot the following issues: 1. In cases where the client does not perform batch data writing or the batch written messages are not very reasonable, we cannot intuitively reflect through monitoring and cannot detect unreasonable business usage in a timely manner; It has a significant impact on production and consumption performance, increasing cluster load, reducing cluster performance, and increasing overall costs; 2. When we enable the production flow limiting strategy, if the flow limiting condition is triggered by the number of requests on a single connection, it is difficult to accurately locate which topic's high-frequency send data is causing it; ### Modifications Add four metrics: `pulsar_request_rate_in`: Topic send request rate. `pulsar_average_msg_per_request`: Average number of messages per send request for topic. `pulsar_producer_request_in`: producer send request rate. `pulsar_producer_avg_msg_per_request`: Average number of messages per send request for producer. ### Does this pull request potentially affect one of the following parts: *If the box was checked, please highlight the changes* - [ ] Dependencies (add or upgrade a dependency) - [ ] The public API - [ ] The schema - [ ] The default values of configurations - [ ] The threading model - [ ] The binary protocol - [ ] The REST endpoints - [ ] The admin CLI options - [x] The metrics - [ ] Anything that affects deployment ### Documentation - [ ] `doc` - [x] `doc-required` - [ ] `doc-not-needed` - [ ] `doc-complete` ### Matching PR in forked repository PR in forked repository: https://github.com/yyj8/pulsar/pull/6 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch master updated: [improve][broker] PIP-192 made split handler idempotent (#19988)
This is an automated email from the ASF dual-hosted git repository.
kwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 421d707419e [improve][broker] PIP-192 made split handler idempotent
(#19988)
421d707419e is described below
commit 421d707419ee42378b9c5b37812bd5c617ef2c9a
Author: Heesung Sohn <103456639+heesung...@users.noreply.github.com>
AuthorDate: Sat Apr 8 03:49:16 2023 -0700
[improve][broker] PIP-192 made split handler idempotent (#19988)
Master Issue: https://github.com/apache/pulsar/issues/16691
### Motivation
Raising a PR to implement: https://github.com/apache/pulsar/issues/16691
### Modifications
This PR
- Makes split handler idempotent .
- Makes Leader's orphan monitor keep trying to send split msg until
finished.
- Select bundle boundaries at the SplitScheduler to have the same split
boundaries for each Split handler retry.
- Adds a split condition to check if the parent's Splitting state has moved.
- Made Admin Unload command forceful to unload any bundles in invalid
states.
---
conf/broker.conf | 2 +-
.../apache/pulsar/broker/ServiceConfiguration.java | 2 +-
.../extensions/ExtensibleLoadManagerImpl.java | 2 +-
.../channel/ServiceUnitStateChannelImpl.java | 263 +-
.../extensions/channel/ServiceUnitStateData.java | 15 +
.../loadbalance/extensions/models/Split.java | 6 +-
.../loadbalance/extensions/models/Unload.java | 10 +-
.../DefaultNamespaceBundleSplitStrategyImpl.java | 76 +++-
.../pulsar/broker/namespace/NamespaceService.java | 10 +-
.../common/naming/NamespaceBundleFactory.java | 8 +-
.../naming/NamespaceBundleSplitAlgorithm.java | 3 +
.../SpecifiedPositionsBundleSplitAlgorithm.java| 47 ++-
.../extensions/ExtensibleLoadManagerImplTest.java | 1 +
.../channel/ServiceUnitStateChannelTest.java | 399 ++---
.../extensions/channel/models/SplitTest.java | 8 +-
.../extensions/scheduler/SplitSchedulerTest.java | 19 +-
.../DefaultNamespaceBundleSplitStrategyTest.java | 81 -
...SpecifiedPositionsBundleSplitAlgorithmTest.java | 15 +
18 files changed, 767 insertions(+), 200 deletions(-)
diff --git a/conf/broker.conf b/conf/broker.conf
index 1bbb12313f9..414756abcb7 100644
--- a/conf/broker.conf
+++ b/conf/broker.conf
@@ -1455,7 +1455,7 @@
loadBalancerNamespaceBundleSplitConditionHitCountThreshold=3
# the service-unit state channel when there are a large number of bundles.
# minimum value = 30 secs
# (only used in load balancer extension logics)
-loadBalancerServiceUnitStateCleanUpDelayTimeInSeconds=604800
+loadBalancerServiceUnitStateTombstoneDelayTimeInSeconds=3600
### --- Replication --- ###
diff --git
a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
index 52cebe15f6a..c86358b9b5b 100644
---
a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
+++
b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
@@ -2609,7 +2609,7 @@ public class ServiceConfiguration implements
PulsarConfiguration {
+ "minimum value = 30 secs"
+ "(only used in load balancer extension logics)"
)
-private long loadBalancerServiceUnitStateCleanUpDelayTimeInSeconds =
604800;
+private long loadBalancerServiceUnitStateTombstoneDelayTimeInSeconds =
3600;
@FieldContext(
category = CATEGORY_LOAD_BALANCER,
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/ExtensibleLoadManagerImpl.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/ExtensibleLoadManagerImpl.java
index c1234b6dab2..e054a1bc662 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/ExtensibleLoadManagerImpl.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/ExtensibleLoadManagerImpl.java
@@ -449,7 +449,7 @@ public class ExtensibleLoadManagerImpl implements
ExtensibleLoadManager {
log.warn(msg);
throw new IllegalArgumentException(msg);
}
-Unload unload = new Unload(sourceBroker,
bundle.toString(), destinationBroker);
+Unload unload = new Unload(sourceBroker,
bundle.toString(), destinationBroker, true);
UnloadDecision unloadDecision =
new UnloadDecision(unload,
UnloadDecision.Label.Success, UnloadDecision.Reason.Admin);
return unloadAsync(unloadDecision,
diff --git
[GitHub] [pulsar] Demogorgon314 merged pull request #19988: [improve][broker] PIP-192 made split handler idempotent
Demogorgon314 merged PR #19988: URL: https://github.com/apache/pulsar/pull/19988 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on pull request #19988: [improve][broker] PIP-192 made split handler idempotent
heesung-sn commented on PR #19988: URL: https://github.com/apache/pulsar/pull/19988#issuecomment-1500824518 @Demogorgon314 @Technoboy- plz merge if looking good -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-go] Gleiphir2769 opened a new pull request, #1003: [fix]: Fix flaky test TestMaxPendingChunkMessages()
Gleiphir2769 opened a new pull request, #1003: URL: https://github.com/apache/pulsar-client-go/pull/1003 Fixes #983 ### Motivation Old TestMaxPendingChunkMessages() uses the concurrent message publish to make the consumer discard unavailable chunk. And it's flaky. So the `sendSingeChunk()` is introduced to manual create scenarios where old chunks should be discarded. ### Modifications - Fix `TestMaxPendingChunkMessages()` ### Verifying this change - [x] Make sure that the change passes the CI checks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] lhotari commented on issue #16601: Add support for cgroup v2 in Pulsar Loadbalancer
lhotari commented on issue #16601: URL: https://github.com/apache/pulsar/issues/16601#issuecomment-1500815350 " AKS Kubernetes 1.25+ uses cgroups v2." https://github.com/Azure/AKS/releases/tag/2023-03-05 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-go] RobertIndie commented on issue #995: When a new release?
RobertIndie commented on issue #995: URL: https://github.com/apache/pulsar-client-go/issues/995#issuecomment-1500814711 0.10.0 has been released: https://github.com/apache/pulsar-client-go/releases/tag/v0.10.0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-go] RobertIndie closed issue #995: When a new release?
RobertIndie closed issue #995: When a new release? URL: https://github.com/apache/pulsar-client-go/issues/995 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-go] Gleiphir2769 commented on issue #983: Flaky Test: TestMaxPendingChunkMessages
Gleiphir2769 commented on issue #983: URL: https://github.com/apache/pulsar-client-go/issues/983#issuecomment-1500814304 I'm this flaky test author. I will fix it. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on pull request #20044: [improve][client] Add isInterrupted func on TableView
heesung-sn commented on PR #20044: URL: https://github.com/apache/pulsar/pull/20044#issuecomment-1500813602 For the option 3, by this pr, callers could create new tableviews, by constantly monitoring if the table view is interrupted or not. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on pull request #20044: [improve][client] Add isInterrupted func on TableView
heesung-sn commented on PR #20044: URL: https://github.com/apache/pulsar/pull/20044#issuecomment-1500813083 1. retry forever upon failures. 2. throw RuntimeException from the data accessor apis, if already interrupted.(get, entrySet, keySet) 3. expose table view state(e.g isInterrupted) It appears that we have the above options. If tableview is supposed to guarantee the liveness, I think we can go to the option 1 instead. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-node] RobertIndie commented on a diff in pull request #309: [Fix] Fix message listener doesn't respect receiver queue size
RobertIndie commented on code in PR #309:
URL:
https://github.com/apache/pulsar-client-node/pull/309#discussion_r1161075834
##
src/Consumer.cc:
##
@@ -55,21 +56,34 @@ void Consumer::Init(Napi::Env env, Napi::Object exports) {
struct MessageListenerProxyData {
std::shared_ptr cMessage;
Consumer *consumer;
+ std::function callback;
- MessageListenerProxyData(std::shared_ptr cMessage,
Consumer *consumer)
- : cMessage(cMessage), consumer(consumer) {}
+ MessageListenerProxyData(std::shared_ptr cMessage,
Consumer *consumer,
+ std::function callback)
+ : cMessage(cMessage), consumer(consumer), callback(callback) {}
};
void MessageListenerProxy(Napi::Env env, Napi::Function jsCallback,
MessageListenerProxyData *data) {
Napi::Object msg = Message::NewInstance({}, data->cMessage);
Consumer *consumer = data->consumer;
- delete data;
// `consumer` might be null in certain cases, segmentation fault might
happend without this null check. We
// need to handle this rare case in future.
if (consumer) {
-jsCallback.Call({msg, consumer->Value()});
+Napi::Value ret = jsCallback.Call({msg, consumer->Value()});
Review Comment:
Yes. That's right.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] lifepuzzlefun commented on pull request #20044: [improve][client] Add isInterrupted func on TableView
lifepuzzlefun commented on PR #20044: URL: https://github.com/apache/pulsar/pull/20044#issuecomment-1500809025 I understand the idea of this patch. but from a api user's view. it is confusing that what is an `interrupted`, and how should the api user react to the interrupted. I think should choose how to expose the internal exception on TableView to the api user. or we handle the exception inside the client with some retry method. : - ) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-site] markacola opened a new pull request, #508: Update typo omissions concepts-messaging.md
markacola opened a new pull request, #508: URL: https://github.com/apache/pulsar-site/pull/508 ### Documentation - [x ] `doc` - [ ] `doc-required` - [ ] `doc-not-needed` - [ ] `doc-complete` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] svabra commented on issue #20041: [Bug] Failed to pull-up docker-compose for broker, bookie, zk and pulsar-sql
svabra commented on issue #20041: URL: https://github.com/apache/pulsar/issues/20041#issuecomment-1500806701 I was wondering. If the CLI is triggering the Gateway, which includes the REST service according to this [presentation](https://youtu.be/DnXu0Q0mvTM?t=962), then only the exposition of the port 8081 should be done. Should we not address this in the deployment documentation? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] lifepuzzlefun commented on a diff in pull request #19944: [improve][client] Support multi-topic messageId deserialization to ack messages
lifepuzzlefun commented on code in PR #19944:
URL: https://github.com/apache/pulsar/pull/19944#discussion_r1161072141
##
pulsar-common/src/main/proto/PulsarApi.proto:
##
@@ -62,9 +62,9 @@ message MessageIdData {
optional int32 batch_index = 4 [default = -1];
repeated int64 ack_set = 5;
optional int32 batch_size = 6;
-
// For the chunk message id, we need to specify the first chunk message id.
optional MessageIdData first_chunk_message_id = 7;
+optional string topicName = 8;
Review Comment:
hi, i come from Kafka. and i think when client enable group commit offset.
the field can be group by topicName and we can have much smaller RPC message to
send on the wire. see kafka OffsetCommit RPC
https://kafka.apache.org/protocol.html#The_Messages_OffsetCommit : - )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] lhotari commented on issue #16601: Add support for cgroup v2 in Pulsar Loadbalancer
lhotari commented on issue #16601: URL: https://github.com/apache/pulsar/issues/16601#issuecomment-1500805110 It looks like Kubernetes 1.25.2+ defaults to cgroups v2 based on https://twitter.com/brunoborges/status/1644413308571721750?t=YLdUH-DamtbgA43dLCBbdw=19 and https://github.com/Azure/AKS/issues/3443 . See https://kubernetes.io/blog/2022/08/31/cgroupv2-ga-1-25/ . Pulsar load balancer doesn't balance memory consumption in that case since it doesn't support cgroups V2. /Cc @nicoloboschi @michaeljmarshall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
