[GitHub] [pulsar] mattisonchao merged pull request #19240: [fix][broker] Support deleting partitioned topics with the keyword `-partition-` (#19230)
mattisonchao merged PR #19240: URL: https://github.com/apache/pulsar/pull/19240 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] nodece commented on pull request #19203: [improve][websocket] Add ping support
nodece commented on PR #19203: URL: https://github.com/apache/pulsar/pull/19203#issuecomment-1384035503 /pulsarbot rerun-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on pull request #19158: [improve][client] PIP-224 Part 1: Add TopicMessageId for seek and acknowledge
BewareMyPower commented on PR #19158: URL: https://github.com/apache/pulsar/pull/19158#issuecomment-1384038345 @codelipenghui @congbobo184 @eolivelli @hangc0276 PTAL -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-node] BewareMyPower closed issue #284: Document minimum required libstdc++ version
BewareMyPower closed issue #284: Document minimum required libstdc++ version URL: https://github.com/apache/pulsar-client-node/issues/284 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-node] BewareMyPower commented on issue #284: Document minimum required libstdc++ version
BewareMyPower commented on issue #284: URL: https://github.com/apache/pulsar-client-node/issues/284#issuecomment-1384024450 I have tested Rocky Linux 8, CentOS 8 with latest master and they both work well. But CentOS 7 is not supported because its highest `GLIBCXX` version is 3.4.19. I've opened an issue: https://github.com/apache/pulsar-client-node/issues/286. So I will close this issue for now. You can reopen it if you have other questions. BTW, the `libstdc++` version might not be user-friendly because users have to check the libstdc++ versions for the systems they use. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-node] BewareMyPower opened a new issue, #286: Support CentOS 7
BewareMyPower opened a new issue, #286:
URL: https://github.com/apache/pulsar-client-node/issues/286
### Motivation
The EOL of CentOS 7 is 2024-06-30, see
https://wiki.centos.org/About/Product. However, the Node.js client does not
support CentOS 7.
### Reproduce
First, build the `napi-xxx.tar.gz` for x86_64 glibc linux and run the
`centos:7` container.
```bash
docker build --build-arg PLATFORM=x86_64 --build-arg ARCH=x86_64 --build-arg
NODE_VERSION=18 \
-t node-build-x86_64 -f pkg/linux/Dockerfile_linux_glibc .
docker run -i -v $PWD:/pulsar-client-node node-build-x86_64 \
/pulsar-client-node/pkg/linux/build-napi-inside-docker.sh
docker run -v $PWD:/pulsar-client-node -it centos:7 /bin/bash
```
Then, run the following commands inside the container:
```bash
yum update -y
curl -O -L
https://nodejs.org/download/release/v16.19.0/node-v16.19.0-linux-x64.tar.gz
tar zxf node-v16.19.0-linux-x64.tar.gz
export PATH=$PWD/node-v16.19.0-linux-x64/bin:$PATH
tar zxf
/pulsar-client-node/build/stage/pulsar-client-node-1.9.0-rc.0/napi-linux-glibc-x64.tar.gz
```
Now, we can see `Pulsar.node` can be loaded because of the high `GLIBCXX`
version for CentOS 7.
```bash
# node
Welcome to Node.js v16.19.0.
Type ".help" for more information.
> const Pulsar = require('./binding/Pulsar.node')
Uncaught:
Error: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required
by /root/binding/Pulsar.node)
at Object.Module._extensions..node
(node:internal/modules/cjs/loader:1249:18)
at Module.load (node:internal/modules/cjs/loader:1043:32)
at Function.Module._load (node:internal/modules/cjs/loader:878:12)
at Module.require (node:internal/modules/cjs/loader:1067:19)
at require (node:internal/modules/cjs/helpers:103:18) {
code: 'ERR_DLOPEN_FAILED'
}
```
We can see the supported GLIBCXX versions of `libstdc++.so` on CentOS 7 are:
```bash
# strings /usr/lib64/libstdc++.so.6 | grep ^GLIBCXX_[0-9]
GLIBCXX_3.4
GLIBCXX_3.4.1
GLIBCXX_3.4.2
GLIBCXX_3.4.3
GLIBCXX_3.4.4
GLIBCXX_3.4.5
GLIBCXX_3.4.6
GLIBCXX_3.4.7
GLIBCXX_3.4.8
GLIBCXX_3.4.9
GLIBCXX_3.4.10
GLIBCXX_3.4.11
GLIBCXX_3.4.12
GLIBCXX_3.4.13
GLIBCXX_3.4.14
GLIBCXX_3.4.15
GLIBCXX_3.4.16
GLIBCXX_3.4.17
GLIBCXX_3.4.18
GLIBCXX_3.4.19
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] Nicklee007 opened a new pull request, #19244: [fix][broker] Avoid bundle unload destination broker be set as an inactive broker.
Nicklee007 opened a new pull request, #19244: URL: https://github.com/apache/pulsar/pull/19244 ### Motivation we should check the `destinationBroker` whether is an inactive broker and reject the unload request when the `destinationBroker` is set as an inactive destination. Otherwise the unload result not satisfied our expectation which will cause the bundle need unload more times. ### Modifications 1. check `destinationBroker` before `setNamespaceBundleAffinity` in `unloadNamespaceBundle`; 2. add some unit test. ### Documentation - [X] `doc-not-needed` ### Matching PR in forked repository PR in forked repository: https://github.com/Nicklee007/pulsar/pull/7 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-dotpulsar] entvex opened a new pull request, #133: Cleaned up the entire project using profile "DotPulsar: Full Cleanup"
entvex opened a new pull request, #133: URL: https://github.com/apache/pulsar-dotpulsar/pull/133 # Description Cleaned up the entire project using profile "DotPulsar: Full Cleanup". # Testing Checked that it builds -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-node] BewareMyPower commented on issue #284: Document minimum required libstdc++ version
BewareMyPower commented on issue #284: URL: https://github.com/apache/pulsar-client-node/issues/284#issuecomment-1383945985 #285 fixes the issue for Debian buster, but the libstdc++ version is still too high for RHEL-based distributions. Here are the `GLIBCXX_` symbols from the `Pulsar.node` built in latest master. ``` GLIBCXX_3.4 GLIBCXX_3.4.11 GLIBCXX_3.4.14 GLIBCXX_3.4.15 GLIBCXX_3.4.18 GLIBCXX_3.4.19 GLIBCXX_3.4.20 GLIBCXX_3.4.21 GLIBCXX_3.4.22 GLIBCXX_3.4.9 ``` Compared with 1.8.0, the master only reduces the `GLIBCXX_3.4.26` string. The root cause is that the `libpulsarwithdeps.a` is installed from a deb package on Debian-based distributions: https://github.com/apache/pulsar-client-node/blob/dc2c3c5feb1d51e317215ae183ccf0a1b7306353/pkg/linux/download-cpp-client.sh#L43-L46 Not matter it's Debian 10 or Debian 11, the deb package was built on Debian 9: https://github.com/apache/pulsar-client-cpp/blob/cedc0c11b75b137537ada3ff782f0e91482e58b6/pkg/deb/Dockerfile#L22 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codecov-commenter commented on pull request #19237: [fix][ci] Fix codecov reporting by configuring to wait for all builds sending coverage
codecov-commenter commented on PR #19237: URL: https://github.com/apache/pulsar/pull/19237#issuecomment-1383941922 # [Codecov](https://codecov.io/gh/apache/pulsar/pull/19237?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#19237](https://codecov.io/gh/apache/pulsar/pull/19237?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (71230c7) into [master](https://codecov.io/gh/apache/pulsar/commit/246c2701e5c43e02e9783c82d4d107d06b019951?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (246c270) will **decrease** coverage by `11.14%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/pulsar/pull/19237?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #19237 +/- ## = - Coverage 43.01% 31.86% -11.15% + Complexity10167 6153 -4014 = Files 747 633 -114 Lines 7225559929-12326 Branches 7786 6251 -1535 = - Hits 3108019099-11981 - Misses3757738205 +628 + Partials 3598 2625 -973 ``` | Flag | Coverage Δ | | |---|---|---| | unittests | `31.86% <ø> (-11.15%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/pulsar/pull/19237?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...ava/org/apache/pulsar/broker/admin/v1/Brokers.java](https://codecov.io/gh/apache/pulsar/pull/19237?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9hZG1pbi92MS9Ccm9rZXJzLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...va/org/apache/pulsar/broker/admin/v1/Clusters.java](https://codecov.io/gh/apache/pulsar/pull/19237?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9hZG1pbi92MS9DbHVzdGVycy5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/pulsar/broker/admin/v1/Properties.java](https://codecov.io/gh/apache/pulsar/pull/19237?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9hZG1pbi92MS9Qcm9wZXJ0aWVzLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../apache/pulsar/broker/admin/v2/ResourceGroups.java](https://codecov.io/gh/apache/pulsar/pull/19237?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9hZG1pbi92Mi9SZXNvdXJjZUdyb3Vwcy5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...sar/broker/stats/metrics/ManagedLedgerMetrics.java](https://codecov.io/gh/apache/pulsar/pull/19237?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9zdGF0cy9tZXRyaWNzL01hbmFnZWRMZWRnZXJNZXRyaWNzLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...ar/common/naming/PartitionedManagedLedgerInfo.java](https://codecov.io/gh/apache/pulsar/pull/19237?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NvbW1vbi9uYW1pbmcvUGFydGl0aW9uZWRNYW5hZ2VkTGVkZ2VySW5mby5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [pulsar] lhotari commented on pull request #19237: [fix][ci] Fix codecov reporting by configuring to wait for all builds sending coverage
lhotari commented on PR #19237:
URL: https://github.com/apache/pulsar/pull/19237#issuecomment-1383830109
codecov has a known issue
https://github.com/codecov/codecov-action/issues/598 where uploading to codecov
might fail with this error
```
[2023-01-16T10:14:31.630Z] ['info'] Pinging Codecov:
https://codecov.io/upload/v4?package=github-action-3.1.1-uploader-0.3.2=***=lh-fix-codecov-config-to-wait-for-all-unit-tests=3928945879_url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Factions%2Fruns%2F3928945879=b8189e0dfe212686a6aba0319866ee8f996dc796=Pulsar+CI=19237=github-actions=apache%2Fpulsar===unittests=
[2023-01-16T10:14:32.646Z] ['error'] There was an error running the
uploader: Error uploading to [https://codecov.io:](https://codecov.io/) Error:
There was an error fetching the storage URL during POST: 404 - {'detail':
ErrorDetail(string='Unable to locate build via Github Actions API. Please
upload with the Codecov repository upload token to resolve issue.',
code='not_found')}
```
It's too bad that there isn't a way to retry uploading in this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar-client-node] shibd commented on issue #281: Failed to create producer: ConnectError when upgrading to 1.8.0
shibd commented on issue #281:
URL:
https://github.com/apache/pulsar-client-node/issues/281#issuecomment-1383830282
Hi, Let's continue the discussion here.
https://github.com/apache/pulsar-client-node/pull/282#issuecomment-1378497083
I build from source code following Master's `README` and it worked.
I suspect maybe you're using a different server environment?
Can you test by following the steps below:
1. Refer to [How to
build](https://github.com/apache/pulsar-client-node/tree/v1.8.0#how-to-build)
of master README to build from the source
2. Copy this
[demo](https://github.com/shibd/pulsar-node-oauth2-ssl-test/blob/main/producer-oauth2.js)
to
[example](https://github.com/apache/pulsar-client-node/tree/master/examples) on
your host.
3. Change require
```git
const Pulsar = require('shibaodi-pulsar-client');
// change to
const Pulsar = require('../');
```
4. Run this demo.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on a diff in pull request #19242: [fix] [broker] Counter of pending send messages in Replicator incorrect if schema future not complete
poorbarcode commented on code in PR #19242:
URL: https://github.com/apache/pulsar/pull/19242#discussion_r1071067949
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/GeoPersistentReplicator.java:
##
@@ -152,6 +154,7 @@ protected boolean replicateEntries(List entries) {
// Increment pending messages for messages produced locally
PENDING_MESSAGES_UPDATER.incrementAndGet(this);
Review Comment:
Already fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar-client-node] shibd commented on pull request #282: [fix] Use OAuth2 on macOX arm64 failed.
shibd commented on PR #282: URL: https://github.com/apache/pulsar-client-node/pull/282#issuecomment-1383818178 It's not a good idea, I'll close it first. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-node] shibd closed pull request #282: [fix] Use OAuth2 on macOX arm64 failed.
shibd closed pull request #282: [fix] Use OAuth2 on macOX arm64 failed. URL: https://github.com/apache/pulsar-client-node/pull/282 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on pull request #19242: [fix] [broker] Counter of pending send messages in Replicator incorrect if schema future not complete
poorbarcode commented on PR #19242: URL: https://github.com/apache/pulsar/pull/19242#issuecomment-1383818058 @codelipenghui @mattisonchao > Why not only add the pending messages after producer.sendAsync is called? > Why can't we move this line to else branch? Great suggestion! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-node] BewareMyPower commented on issue #284: Document minimum required libstdc++ version
BewareMyPower commented on issue #284: URL: https://github.com/apache/pulsar-client-node/issues/284#issuecomment-1383815349 https://github.com/apache/pulsar-client-node/pull/285 might fix this issue, I will perform more verifications in my local env. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #19161: [improve][misc] Register Jackson Java 8 support modules by default
lhotari commented on PR #19161: URL: https://github.com/apache/pulsar/pull/19161#issuecomment-1383814879 Since a PIP was requested, I have created #19243 which is already in voting since there was already a discussion. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] lhotari opened a new issue, #19243: PIP-243: Register Jackson Java 8 support modules by default
lhotari opened a new issue, #19243: URL: https://github.com/apache/pulsar/issues/19243 ### Motivation Jackson has a separate Java 8 support modules for adding support for proper serialization and deserialization of new classes that were added in Java 8 (Java 8 was released in 2014). These Jackson Java 8 support modules haven't been used in the Pulsar code base. This is a pity. This causes a lot of pain when using Java Time classes in Pulsar applications or Pulsar Functions. There are ways to get the classes working for applications, but the documentation is missing. It would make things easier if the Java 8 support modules for Jackson would be included and registered by default. ### Goal Register Jackson Java 8 support modules by default ### API Changes _No response_ ### Implementation Register Jackson Java 8 support modules by default ### Alternatives _No response_ ### Anything else? _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on a diff in pull request #19242: [fix] [broker] Counter of pending send messages in Replicator incorrect if schema future not complete
mattisonchao commented on code in PR #19242:
URL: https://github.com/apache/pulsar/pull/19242#discussion_r1071052160
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/GeoPersistentReplicator.java:
##
@@ -152,6 +154,7 @@ protected boolean replicateEntries(List entries) {
// Increment pending messages for messages produced locally
PENDING_MESSAGES_UPDATER.incrementAndGet(this);
Review Comment:
Why can't we move this line to else branch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] yuruguo commented on pull request #19048: [improve][client] Add unified newTableView method in PulsarClient
yuruguo commented on PR #19048: URL: https://github.com/apache/pulsar/pull/19048#issuecomment-1383793429 > We should also change the test to point to the new method `newTableView(Schema schema)` > > And we should also add the test for the new added method `newTableView()` Test has been modified and added, PTAL :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] lhotari merged pull request #19228: [improve][misc] Follow up on ObjectMapper sharing changes
lhotari merged PR #19228: URL: https://github.com/apache/pulsar/pull/19228 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] poorbarcode opened a new pull request, #19242: [fix] [broker] Counter of pending send messages in Replicator incorre…
poorbarcode opened a new pull request, #19242: URL: https://github.com/apache/pulsar/pull/19242 ### Motivation The Replication task execution process works like this: - read messages from the origin cluster - load schema - send to the target cluster Because the network across the cluster is unstable, the sending task may be slower, so if the number of messages being sent is too large, the task will be suspended. There is a counter(`pendingMessages `) in `Replicator` indicating the count of sending messages. (Highlight)There is a problem that causes the counter to be inaccurate when the scheme future is incomplete. ### Modifications - Make that the counter is correct when exceptions occur or the `schemaFuture` is incomplete - Same change for `ShadowReplicator` ### Documentation - [ ] `doc` - [ ] `doc-required` - [x] `doc-not-needed` - [ ] `doc-complete` ### Matching PR in forked repository PR in forked repository: - https://github.com/poorbarcode/pulsar/pull/52 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on a diff in pull request #19094: [feat][admin] PIP-219 Part-1 Add admin API for trimming topic
codelipenghui commented on code in PR #19094: URL: https://github.com/apache/pulsar/pull/19094#discussion_r1071017707 ## pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v2/PersistentTopics.java: ## @@ -3267,6 +3267,44 @@ public void getLastMessageId( } } +@PUT Review Comment: And is it better to change to @POST? As I understand, the @PUT method usually used to create some resources. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on a diff in pull request #19094: [feat][admin] PIP-219 Part-1 Add admin API for trimming topic
codelipenghui commented on code in PR #19094:
URL: https://github.com/apache/pulsar/pull/19094#discussion_r1071001411
##
pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdTopics.java:
##
@@ -3107,4 +3109,15 @@ void run() throws PulsarAdminException {
getAdmin().topics().setSchemaValidationEnforced(topic, enable);
}
}
+@Parameters(commandDescription = "Trim a topic")
+private class TrimTopic extends CliCommand {
+@Parameter(description = "tenant/namespace", required = true)
Review Comment:
It should be `topic name`?
##
pulsar-common/src/main/java/org/apache/pulsar/common/policies/data/TopicOperation.java:
##
@@ -55,4 +55,5 @@ public enum TopicOperation {
SET_REPLICATED_SUBSCRIPTION_STATUS,
GET_REPLICATED_SUBSCRIPTION_STATUS,
+TRIM_TOPIC,
Review Comment:
I think we also need to add the new operation to
https://github.com/apache/pulsar/blob/efb251cec6ff7dd7a92bc8ff99e8b2fe04d1eb00/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/authorization/PulsarAuthorizationProvider.java#L548-L556
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/PersistentTopicsBase.java:
##
@@ -4655,6 +4655,68 @@ protected void internalGetLastMessageId(AsyncResponse
asyncResponse, boolean aut
return null;
});
}
+protected CompletableFuture internalTrimTopic(AsyncResponse
asyncResponse, boolean authoritative) {
+return validateTopicOwnershipAsync(topicName, authoritative)
Review Comment:
We should get the partitions first? If it's a partitioned topic, it doesn't
make sense to check the ownership of the partitioned topic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar-client-python] BewareMyPower closed issue #64: [Doc] Update the release process about how to release Python client
BewareMyPower closed issue #64: [Doc] Update the release process about how to release Python client URL: https://github.com/apache/pulsar-client-python/issues/64 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-python] BewareMyPower merged pull request #68: Add the release process for Python client
BewareMyPower merged PR #68: URL: https://github.com/apache/pulsar-client-python/pull/68 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] coderzc commented on a diff in pull request #19138: [feat][broker][PIP-195] Implement delayed message index bucket snapshot(merge/delete) - part8
coderzc commented on code in PR #19138:
URL: https://github.com/apache/pulsar/pull/19138#discussion_r1070985106
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/delayed/bucket/BucketDelayedDeliveryTracker.java:
##
@@ -243,6 +263,53 @@ public synchronized boolean addMessage(long ledgerId, long
entryId, long deliver
return true;
}
+private synchronized CompletableFuture asyncMergeBucketSnapshot() {
+List values =
immutableBuckets.asMapOfRanges().values().stream().toList();
+long minNumberMessages = Long.MAX_VALUE;
+int minIndex = -1;
+for (int i = 0; i + 1 < values.size(); i++) {
+ImmutableBucket bucketL = values.get(i);
+ImmutableBucket bucketR = values.get(i + 1);
+long numberMessages = bucketL.numberBucketDelayedMessages +
bucketR.numberBucketDelayedMessages;
+if (numberMessages < minNumberMessages) {
+minNumberMessages = (int) numberMessages;
+minIndex = i;
+}
+}
+return asyncMergeBucketSnapshot(values.get(minIndex),
values.get(minIndex + 1));
+}
+
+private synchronized CompletableFuture
asyncMergeBucketSnapshot(ImmutableBucket bucketA,
+
ImmutableBucket bucketB) {
+immutableBuckets.remove(Range.closed(bucketA.startLedgerId,
bucketA.endLedgerId));
+immutableBuckets.remove(Range.closed(bucketB.startLedgerId,
bucketB.endLedgerId));
+
+CompletableFuture snapshotCreateFutureA =
+
bucketA.getSnapshotCreateFuture().orElse(CompletableFuture.completedFuture(null));
+CompletableFuture snapshotCreateFutureB =
+
bucketB.getSnapshotCreateFuture().orElse(CompletableFuture.completedFuture(null));
+
+return CompletableFuture.allOf(snapshotCreateFutureA,
snapshotCreateFutureB).thenCompose(__ -> {
+
CompletableFuture>
futureA =
+bucketA.getRemainSnapshotSegment();
+
CompletableFuture>
futureB =
+bucketB.getRemainSnapshotSegment();
+return futureA.thenCombine(futureB,
CombinedSegmentDelayedIndexQueue::wrap)
+.thenCompose(combinedDelayedIndexQueue -> {
+CompletableFuture removeAFuture =
bucketA.asyncDeleteBucketSnapshot();
+CompletableFuture removeBFuture =
bucketB.asyncDeleteBucketSnapshot();
+
+return CompletableFuture.allOf(removeAFuture,
removeBFuture).thenRun(() -> {
Review Comment:
> Can we mark the old segment as deleting or merging? So that we can
continue the merge operation after the broker crashes.
This will introduce more state in metadata, we can improve it when we really
need it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] thetumbled commented on pull request #19201: [fix] [broker] fix timeout transaction.
thetumbled commented on PR #19201: URL: https://github.com/apache/pulsar/pull/19201#issuecomment-1383694030 > @thetumbled It looks like the change is not about the issue that you described in the PR details. When reading the motivation of the PR, I thought it was related to the transaction buffer snapshot, but after checking the changes, it looks like the fix is for the transaction coordinator. When I was troubleshooting why transaction recovery took a long time, I found that the root cause was transaction timeout. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on pull request #19201: [fix] [broker] fix timeout transaction.
codelipenghui commented on PR #19201: URL: https://github.com/apache/pulsar/pull/19201#issuecomment-1383687447 @thetumbled It looks like the change is not about the issue that you described in the PR details. When reading the motivation of the PR, I thought it was related to the transaction buffer snapshot, but after checking the changes, it looks like the fix is for the transaction coordinator. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] mattisonchao closed pull request #19240: [fix][broker] Support deleting partitioned topics with the keyword `-partition-` (#19230)
mattisonchao closed pull request #19240: [fix][broker] Support deleting partitioned topics with the keyword `-partition-` (#19230) URL: https://github.com/apache/pulsar/pull/19240 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] mattisonchao merged pull request #19241: [branch-2.9] Fix compile error caused by cherry-pick #19167.
mattisonchao merged PR #19241: URL: https://github.com/apache/pulsar/pull/19241 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on pull request #19240: [fix][broker] Support deleting partitioned topics with the keyword `-partition-` (#19230)
mattisonchao commented on PR #19240: URL: https://github.com/apache/pulsar/pull/19240#issuecomment-1383663480 We need to wait for PR #19241 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] mattisonchao opened a new pull request, #19241: [branch-2.9] Fix compile error caused by cherry-pick #19167.
mattisonchao opened a new pull request, #19241: URL: https://github.com/apache/pulsar/pull/19241 ## Motivation Fix compile error caused by cherry-pick #19167 The commit is https://github.com/apache/pulsar/commit/e310713be433e6e36a4191deb74ae8e5d0a03820 https://user-images.githubusercontent.com/74767115/212631756-2f2be172-2ecd-41a9-8e1e-b9c7b75a73ec.png;> ## Modification - Use the constructor instead of the builder pattern. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] mattisonchao opened a new pull request, #19240: [fix][broker] Support deleting partitioned topics with the keyword `-partition-` (#19230)
mattisonchao opened a new pull request, #19240: URL: https://github.com/apache/pulsar/pull/19240 ### Motivation (cherry picked from commit fc4bca6823282c4f5375e5d48552c1f0e114b5ee) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] nodece commented on pull request #19203: [improve][websocket] Add ping support
nodece commented on PR #19203: URL: https://github.com/apache/pulsar/pull/19203#issuecomment-1383636936 /pulsarbot rerun-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] Demogorgon314 commented on a diff in pull request #19153: [improve] PIP-241: add TopicEventListener / topic events for the BrokerService
Demogorgon314 commented on code in PR #19153:
URL: https://github.com/apache/pulsar/pull/19153#discussion_r1070934942
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java:
##
@@ -283,6 +286,8 @@ public class BrokerService implements Closeable {
private Set
brokerEntryMetadataInterceptors;
private Set brokerEntryPayloadProcessors;
+private final List topicEventListeners = new
CopyOnWriteArrayList<>();
Review Comment:
Can we add a separate class like `TopicEventListeners` or
`TopicEventService` to package all operations to let the `BrokerService`
duty more clear?
What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #19237: [fix][ci] Fix codecov reporting by configuring to wait for all builds sending coverage
lhotari commented on PR #19237: URL: https://github.com/apache/pulsar/pull/19237#issuecomment-1383630982 > @yaalsn Do you remember the reason for doing this? I found this description in #18081 "Except flaky test group to run code coverage, otherwise we cannot get an accurate base coverage." Why would that make it more accurate? We have a lot of tests in the flaky test group. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #19237: [fix][ci] Fix codecov reporting by configuring to wait for all builds sending coverage
lhotari commented on PR #19237: URL: https://github.com/apache/pulsar/pull/19237#issuecomment-1383626356 One more attempt. The coverage profile wasn't activated for the BROKER_FLAKY unit test group: https://github.com/apache/pulsar/blob/f23363c6f50b89b7dcb658c06ffaaf496bb65a86/build/run_unit_group.sh#L36-L40 @yaalsn Do you remember the reason for doing this? In addition, the coverage profile wasn't activated when the install target (maven goal) was used instead of verify. I fixed the issues in the run_unit_group.sh script and also upgraded jacoco-maven-plugin to 0.8.8 . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] coderzc commented on a diff in pull request #19138: [feat][broker][PIP-195] Implement delayed message index bucket snapshot(merge/delete) - part8
coderzc commented on code in PR #19138:
URL: https://github.com/apache/pulsar/pull/19138#discussion_r1070929495
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/delayed/bucket/CombinedSegmentDelayedIndexQueue.java:
##
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.delayed.bucket;
+
+import java.util.List;
+import javax.annotation.concurrent.NotThreadSafe;
+import
org.apache.pulsar.broker.delayed.proto.DelayedMessageIndexBucketSnapshotFormat.DelayedIndex;
+import
org.apache.pulsar.broker.delayed.proto.DelayedMessageIndexBucketSnapshotFormat.SnapshotSegment;
+
+@NotThreadSafe
+public class CombinedSegmentDelayedIndexQueue implements DelayedIndexQueue {
Review Comment:
Sorry, I missed this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] lhotari merged pull request #19238: Fix flaky test CompactionReaderImplTest(wait until the consumerFuture completes)
lhotari merged PR #19238: URL: https://github.com/apache/pulsar/pull/19238 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] lhotari closed issue #19214: Flaky-test: CompactionReaderImplTest.test
lhotari closed issue #19214: Flaky-test: CompactionReaderImplTest.test URL: https://github.com/apache/pulsar/issues/19214 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] lhotari commented on issue #13923: [Proxy] Race condition in Pulsar Proxy that causes UnsupportedOperationExceptions in Proxy logs
lhotari commented on issue #13923: URL: https://github.com/apache/pulsar/issues/13923#issuecomment-1383604095 One interesting detail is that there's `hasProxyToBrokerUrl: false` when the problem occurs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-node] BewareMyPower commented on pull request #282: [fix] Use OAuth2 on macOX arm64 failed.
BewareMyPower commented on PR #282:
URL:
https://github.com/apache/pulsar-client-node/pull/282#issuecomment-1383595543
From
https://github.com/apache/pulsar-client-node/pull/282#discussion_r1066658153 I
see
> I tested and the parameter doesn't work if compiled on x86_64.
Then I tried the following patch, which adds the same compile flags as this
PR does for arm64 macOS.
```diff
diff --git a/pkg/mac/build-cpp-deps-lib.sh b/pkg/mac/build-cpp-deps-lib.sh
index e2078d1..aa4e04d 100755
--- a/pkg/mac/build-cpp-deps-lib.sh
+++ b/pkg/mac/build-cpp-deps-lib.sh
@@ -167,12 +167,12 @@ if [ ! -f curl-${CURL_VERSION}.done ]; then
tar xfz curl-${CURL_VERSION}.tar.gz
pushd curl-${CURL_VERSION}
CFLAGS="-fPIC -arch ${ARCH}
-mmacosx-version-min=${MACOSX_DEPLOYMENT_TARGET}" \
-./configure --with-ssl=$PREFIX \
+./configure \
--without-nghttp2 \
--without-libidn2 \
--disable-ldap \
--without-brotli \
- --without-secure-transport \
+ --with-secure-transport \
--disable-ipv6 \
--prefix=$PREFIX \
--host=$ARCH-apple-darwin
```
Then, build from source.
```bash
pkg/mac/build-cpp-deps-lib.sh
pkg/mac/build-cpp-lib.sh
npm install
```
After that, I reproduced the same issue with
https://github.com/apache/pulsar-client-node/pull/282#issuecomment-1381722708
successfully
```
% node examples/producer
node:internal/modules/cjs/loader:1210
return process.dlopen(module, path.toNamespacedPath(filename));
^
Error:
dlopen(/Users/xuyunze/node-demo/pulsar-client-node/lib/binding/Pulsar.node,
0x0001): symbol not found in flat namespace (_kSecAttrLabel)
at Object.Module._extensions..node
(node:internal/modules/cjs/loader:1210:18)
at Module.load (node:internal/modules/cjs/loader:1004:32)
at Function.Module._load (node:internal/modules/cjs/loader:839:12)
at Module.require (node:internal/modules/cjs/loader:1028:19)
at require (node:internal/modules/cjs/helpers:102:18)
at Object.
(/Users/xuyunze/node-demo/pulsar-client-node/src/pulsar-binding.js:24:17)
at Module._compile (node:internal/modules/cjs/loader:1126:14)
at Object.Module._extensions..js
(node:internal/modules/cjs/loader:1180:10)
at Module.load (node:internal/modules/cjs/loader:1004:32)
at Function.Module._load (node:internal/modules/cjs/loader:839:12) {
code: 'ERR_DLOPEN_FAILED'
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] lhotari commented on issue #13923: [Proxy] Race condition in Pulsar Proxy that causes UnsupportedOperationExceptions in Proxy logs
lhotari commented on issue #13923: URL: https://github.com/apache/pulsar/issues/13923#issuecomment-1383592139 This problem remains. This log message appears in production environments and also in integration tests. recent example in https://github.com/apache/pulsar/actions/runs/3927573395/jobs/6714826271#step:12:7081 ``` 2023-01-16T06:43:38,959 - INFO - [docker-java-stream--1658439674:DockerUtils$4@383] - DOCKER.exec(pulsar-proxy-pulsar-proxy:tail -f /var/log/pulsar/proxy.log): STDOUT: 023-01-16T06:43:38,718+ [pulsar-proxy-io-2-1] WARN org.apache.pulsar.proxy.server.ProxyConnection - [/172.18.0.1:36040] Got exception UnsupportedOperationException : Message: null State: ProxyLookupRequests java.lang.UnsupportedOperationException: null at org.apache.pulsar.common.protocol.PulsarDecoder.handleSubscribe(PulsarDecoder.java:532) ~[org.apache.pulsar-pulsar-common-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT] at org.apache.pulsar.common.protocol.PulsarDecoder.channelRead(PulsarDecoder.java:243) ~[org.apache.pulsar-pulsar-common-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT] at org.apache.pulsar.proxy.server.ProxyConnection.channelRead(ProxyConnection.java:254) ~[org.apache.pulsar-pulsar-proxy-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[io.netty-netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.nett 2023-01-16T06:43:38,959 - INFO - [docker-java-stream--1658439674:DockerUtils$4@383] - DOCKER.exec(pulsar-proxy-pulsar-proxy:tail -f /var/log/pulsar/proxy.log): STDOUT: y-netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[io.netty-netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[io.netty-netty-codec-4.1.86.Final.jar:4.1.86.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[io.netty-netty-codec-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[io.netty-netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty-netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[io.netty-netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.ne 2023-01-16T06:43:38,959 - INFO - [docker-java-stream--1658439674:DockerUtils$4@383] - DOCKER.exec(pulsar-proxy-pulsar-proxy:tail -f /var/log/pulsar/proxy.log): STDOUT: tty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) ~[io.netty-netty-handler-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[io.netty-netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty-netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[io.netty-netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.handler.flush.FlushConsolidationHandler.channelRead(FlushConsolidationHandler.java:152) ~[io.netty-netty-handler-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[io.netty-netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead( 2023-01-16T06:43:38,959 - INFO - [docker-java-stream--1658439674:DockerUtils$4@383] - DOCKER.exec(pulsar-proxy-pulsar-proxy:tail -f /var/log/pulsar/proxy.log): STDOUT: AbstractChannelHandlerContext.java:420) ~[io.netty-netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[io.netty-netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[io.netty-netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[io.netty-netty-transport-4.1.86.Final.jar:4.1.86.Final] at
[GitHub] [pulsar] lhotari opened a new issue, #13923: [Proxy] Race condition in Pulsar Proxy that causes UnsupportedOperationExceptions in Proxy logs
lhotari opened a new issue, #13923: URL: https://github.com/apache/pulsar/issues/13923 **Describe the bug** It is common that UnsupportedOperationExceptions appear on the Proxy logs. This particular issue was reproduced very often when Geo-replication was configured between 2 clusters. ``` 16:57:50.305 [pulsar-proxy-io-2-3] INFO org.apache.pulsar.proxy.server.ProxyConnection - [/10.34.1.169:47600] New connection opened 16:57:50.329 [pulsar-proxy-io-2-3] INFO org.apache.pulsar.proxy.server.ProxyConnection - [/10.34.1.169:47600] complete connection, init proxy handler. authenticated with token role superuser, hasProxyToBrokerUrl: false 16:57:50.331 [pulsar-proxy-io-2-3] WARN io.netty.channel.DefaultChannelPipeline - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception. java.lang.UnsupportedOperationException: null at org.apache.pulsar.common.protocol.PulsarDecoder.handleProducer(PulsarDecoder.java:479) at org.apache.pulsar.common.protocol.PulsarDecoder.channelRead(PulsarDecoder.java:193) at org.apache.pulsar.proxy.server.ProxyConnection.channelRead(ProxyConnection.java:193) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [io.netty-netty-transport-4.1.72.Final.jar:4.1.72.Fi nal] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [io.netty-netty-transport-4.1.72.Final.jar:4.1.72.Fi nal] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [io.netty-netty-transport-4.1.72.Final.jar:4.1.72.Fina l] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) [io.netty-netty-codec-4.1.72.Final.jar:4.1.72.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) [io.netty-netty-codec-4.1.72.Final.jar:4.1.72.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [io.netty-netty-transport-4.1.72.Final.jar:4.1.72.Fi nal] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [io.netty-netty-transport-4.1.72.Final.jar:4.1.72.Fi nal] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [io.netty-netty-transport-4.1.72.Final.jar:4.1.72.Fina l] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1371) [io.netty-netty-handler-4.1.72.Final.jar:4.1.72.Final] at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1245) [io.netty-netty-handler-4.1.72.Final.jar:4.1.72.Final] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285) [io.netty-netty-handler-4.1.72.Final.jar:4.1.72.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) [io.netty-netty-codec-4.1.72.Final.jar:4.1.72.Final ] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) [io.netty-netty-codec-4.1.72.Final.jar:4.1.72.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) [io.netty-netty-codec-4.1.72.Final.jar:4.1.72.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [io.netty-netty-transport-4.1.72.Final.jar:4.1.72.Fi nal] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [io.netty-netty-transport-4.1.72.Final.jar:4.1.72.Fi nal] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [io.netty-netty-transport-4.1.72.Final.jar:4.1.72.Fina l] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [io.netty-netty-transport-4.1.72.Final.jar:4.1.72.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [io.netty-netty-transport-4.1.72.Final.jar:4.1.72.Fi nal] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [io.netty-netty-transport-4.1.72.Final.jar:4.1.72.Fi nal] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [io.netty-netty-transport-4.1.72.Final.jar:4.1.72.Final] at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795) [io.netty-netty-transport-classes-epoll
[GitHub] [pulsar] yuruguo commented on pull request #19048: [improve][client] Add unified newTableView method in PulsarClient
yuruguo commented on PR #19048: URL: https://github.com/apache/pulsar/pull/19048#issuecomment-1383591669 /pulsarbot rerun-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-node] BewareMyPower commented on pull request #282: [fix] Use OAuth2 on macOX arm64 failed.
BewareMyPower commented on PR #282:
URL:
https://github.com/apache/pulsar-client-node/pull/282#issuecomment-1383587647
@ericallam Could this error be reproduced by running the following code?
```javascript
const Pulsar =
require('./node_modules/shibaodi-pulsar-client/lib/binding/Pulsar.node')
```
You can change the path in `require` to the actual path of the
`Pulsar.node`, which could be downloaded from
https://github.com/shibd/pulsar-client-node/actions/runs/3904096695 (uncompress
`macos-18-arm64`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] nodece commented on pull request #19203: [improve][websocket] Add ping support
nodece commented on PR #19203: URL: https://github.com/apache/pulsar/pull/19203#issuecomment-1383573526 /pulsarbot rerun-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] lhotari commented on issue #16978: Flaky-test: FilterEntryTest.testEntryFilterRescheduleMessageDependingOnConsumerSharedSubscription
lhotari commented on issue #16978: URL: https://github.com/apache/pulsar/issues/16978#issuecomment-1383556933 Another one https://github.com/apache/pulsar/actions/runs/3927573395/jobs/6714418502#step:11:1062 ``` Error: Tests run: 17, Failures: 1, Errors: 0, Skipped: 16, Time elapsed: 129.697 s <<< FAILURE! - in org.apache.pulsar.broker.service.plugin.FilterEntryTest Error: testEntryFilterRescheduleMessageDependingOnConsumerSharedSubscription(org.apache.pulsar.broker.service.plugin.FilterEntryTest) Time elapsed: 60.835 s <<< FAILURE! java.util.concurrent.TimeoutException at java.base/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1960) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2095) at org.apache.pulsar.broker.service.plugin.FilterEntryTest.testEntryFilterRescheduleMessageDependingOnConsumerSharedSubscription(FilterEntryTest.java:441) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139) at org.testng.internal.invokers.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:47) at org.testng.internal.invokers.InvokeMethodRunnable.call(InvokeMethodRunnable.java:76) at org.testng.internal.invokers.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #19237: [fix][ci] Fix codecov reporting by configuring to wait for all builds sending coverage
lhotari commented on PR #19237: URL: https://github.com/apache/pulsar/pull/19237#issuecomment-1383527048 > The final codecov result of comparison in PR comment is not accurated because the `master` branch CI should run after every PR merged, but now it doesn't because pulsar's CI runner resource is not sufficient and it needs long time to finish. Maybe the first thing is that we need to shorten the unit test running time. My assumption is that the main reason for the false comparison results have been the missing `after_n_builds` setting. It's true that it won't be exactly accurate if master branch results are outdated. Let's see if this assumption holds. The codecov upload was still missing from the 10th build job that resides in ci-pulsar-flaky.yaml and that's why the results didn't show up. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] yuruguo commented on pull request #19048: [improve][client] Add unified newTableView method in PulsarClient
yuruguo commented on PR #19048: URL: https://github.com/apache/pulsar/pull/19048#issuecomment-1383518020 /pulsarbot rerun-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codecov-commenter commented on pull request #19203: [improve][websocket] Add ping support
codecov-commenter commented on PR #19203: URL: https://github.com/apache/pulsar/pull/19203#issuecomment-1383513716 # [Codecov](https://codecov.io/gh/apache/pulsar/pull/19203?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#19203](https://codecov.io/gh/apache/pulsar/pull/19203?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (27b92cd) into [master](https://codecov.io/gh/apache/pulsar/commit/246c2701e5c43e02e9783c82d4d107d06b019951?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (246c270) will **decrease** coverage by `5.74%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/pulsar/pull/19203?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #19203 +/- ## - Coverage 43.01% 37.27% -5.75% + Complexity10167 1990-8177 Files 747 209 -538 Lines 7225514442 -57813 Branches 7786 1578-6208 - Hits 31080 5383 -25697 + Misses37577 8477 -29100 + Partials 3598 582-3016 ``` | Flag | Coverage Δ | | |---|---|---| | unittests | `37.27% <ø> (-5.75%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/pulsar/pull/19203?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...g/apache/pulsar/client/impl/ConnectionHandler.java](https://codecov.io/gh/apache/pulsar/pull/19203?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL0Nvbm5lY3Rpb25IYW5kbGVyLmphdmE=) | `50.00% <0.00%> (-5.32%)` | :arrow_down: | | [.../org/apache/pulsar/client/impl/ConnectionPool.java](https://codecov.io/gh/apache/pulsar/pull/19203?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL0Nvbm5lY3Rpb25Qb29sLmphdmE=) | `37.43% <0.00%> (-1.03%)` | :arrow_down: | | [.../timeout/TransactionTimeoutTrackerFactoryImpl.java](https://codecov.io/gh/apache/pulsar/pull/19203?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci90cmFuc2FjdGlvbi90aW1lb3V0L1RyYW5zYWN0aW9uVGltZW91dFRyYWNrZXJGYWN0b3J5SW1wbC5qYXZh) | | | | [...lsar/broker/loadbalance/BrokerFilterException.java](https://codecov.io/gh/apache/pulsar/pull/19203?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9sb2FkYmFsYW5jZS9Ccm9rZXJGaWx0ZXJFeGNlcHRpb24uamF2YQ==) | | | | [...tion/buffer/impl/InMemTransactionBufferReader.java](https://codecov.io/gh/apache/pulsar/pull/19203?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci90cmFuc2FjdGlvbi9idWZmZXIvaW1wbC9Jbk1lbVRyYW5zYWN0aW9uQnVmZmVyUmVhZGVyLmphdmE=) | | | | [...r/broker/service/schema/SchemaRegistryService.java](https://codecov.io/gh/apache/pulsar/pull/19203?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9zZXJ2aWNlL3NjaGVtYS9TY2hlbWFSZWdpc3RyeVNlcnZpY2UuamF2YQ==) | | | |
[GitHub] [pulsar] gaozhangmin commented on a diff in pull request #19204: [improve][broker] Add metric to indicates the max size of managed ledger cache size
gaozhangmin commented on code in PR #19204:
URL: https://github.com/apache/pulsar/pull/19204#discussion_r1070836262
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/metrics/ManagedLedgerCacheMetrics.java:
##
@@ -48,6 +48,7 @@ public synchronized List generate() {
m.put("brk_ml_count", mlCacheStats.getNumberOfManagedLedgers());
m.put("brk_ml_cache_used_size", mlCacheStats.getCacheUsedSize());
+m.put("brk_ml_cache_total_size", mlCacheStats.getCacheMaxSize());
Review Comment:
This configuration is dynamic, we can override it through api.
And there so many examples like `jvm_memory_bytes_max` is also a fixed
value, but exposed as a metrics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] thetumbled commented on pull request #19201: [fix] [broker] fix timeout transaction.
thetumbled commented on PR #19201:
URL: https://github.com/apache/pulsar/pull/19201#issuecomment-1383474511
> @thetumbled
>
>
https://github.com/apache/pulsar/blob/246c2701e5c43e02e9783c82d4d107d06b019951/pulsar-broker/src/main/java/org/apache/pulsar/broker/TransactionMetadataStoreService.java#L136-L137
>
>
> the future `thenAccept` change the executing thread, so the problem has
never been truth fixed.
> so its better to change code like this can slove this problem
>
> ```
>
openTransactionMetadataStore(tcId).thenAccept((store) -> {
> stores.put(tcId, store);
> internalPinnedExecutor.execute(() -> {
> ```
good idea. i have change implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on a diff in pull request #19138: [feat][broker][PIP-195] Implement delayed message index bucket snapshot(merge/delete) - part8
codelipenghui commented on code in PR #19138:
URL: https://github.com/apache/pulsar/pull/19138#discussion_r1070803617
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/delayed/bucket/BucketDelayedDeliveryTracker.java:
##
@@ -243,6 +263,53 @@ public synchronized boolean addMessage(long ledgerId, long
entryId, long deliver
return true;
}
+private synchronized CompletableFuture asyncMergeBucketSnapshot() {
+List values =
immutableBuckets.asMapOfRanges().values().stream().toList();
+long minNumberMessages = Long.MAX_VALUE;
+int minIndex = -1;
+for (int i = 0; i + 1 < values.size(); i++) {
+ImmutableBucket bucketL = values.get(i);
+ImmutableBucket bucketR = values.get(i + 1);
+long numberMessages = bucketL.numberBucketDelayedMessages +
bucketR.numberBucketDelayedMessages;
+if (numberMessages < minNumberMessages) {
+minNumberMessages = (int) numberMessages;
+minIndex = i;
+}
+}
+return asyncMergeBucketSnapshot(values.get(minIndex),
values.get(minIndex + 1));
+}
+
+private synchronized CompletableFuture
asyncMergeBucketSnapshot(ImmutableBucket bucketA,
+
ImmutableBucket bucketB) {
+immutableBuckets.remove(Range.closed(bucketA.startLedgerId,
bucketA.endLedgerId));
+immutableBuckets.remove(Range.closed(bucketB.startLedgerId,
bucketB.endLedgerId));
+
+CompletableFuture snapshotCreateFutureA =
+
bucketA.getSnapshotCreateFuture().orElse(CompletableFuture.completedFuture(null));
+CompletableFuture snapshotCreateFutureB =
+
bucketB.getSnapshotCreateFuture().orElse(CompletableFuture.completedFuture(null));
+
+return CompletableFuture.allOf(snapshotCreateFutureA,
snapshotCreateFutureB).thenCompose(__ -> {
+
CompletableFuture>
futureA =
+bucketA.getRemainSnapshotSegment();
+
CompletableFuture>
futureB =
+bucketB.getRemainSnapshotSegment();
+return futureA.thenCombine(futureB,
CombinedSegmentDelayedIndexQueue::wrap)
+.thenCompose(combinedDelayedIndexQueue -> {
+CompletableFuture removeAFuture =
bucketA.asyncDeleteBucketSnapshot();
+CompletableFuture removeBFuture =
bucketB.asyncDeleteBucketSnapshot();
+
+return CompletableFuture.allOf(removeAFuture,
removeBFuture).thenRun(() -> {
Review Comment:
Can we mark the old segment as deleting or merging? So that we can continue
the merge operation after the broker crashes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] codecov-commenter commented on pull request #19238: Fix flaky test CompactionReaderImplTest(wait until the consumerFuture completes)
codecov-commenter commented on PR #19238: URL: https://github.com/apache/pulsar/pull/19238#issuecomment-1383423269 # [Codecov](https://codecov.io/gh/apache/pulsar/pull/19238?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#19238](https://codecov.io/gh/apache/pulsar/pull/19238?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (636b755) into [master](https://codecov.io/gh/apache/pulsar/commit/246c2701e5c43e02e9783c82d4d107d06b019951?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (246c270) will **increase** coverage by `27.74%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/pulsar/pull/19238?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #19238 +/- ## = + Coverage 43.01% 70.76% +27.74% + Complexity10167 440 -9727 = Files 747 26 -721 Lines 72255 2254-70001 Branches 7786 245 -7541 = - Hits 31080 1595-29485 + Misses37577 486-37091 + Partials 3598 173 -3425 ``` | Flag | Coverage Δ | | |---|---|---| | unittests | `70.76% <ø> (+27.74%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/pulsar/pull/19238?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...a/org/apache/pulsar/broker/admin/v2/Functions.java](https://codecov.io/gh/apache/pulsar/pull/19238?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9hZG1pbi92Mi9GdW5jdGlvbnMuamF2YQ==) | | | | [.../client/impl/schema/generic/GenericJsonReader.java](https://codecov.io/gh/apache/pulsar/pull/19238?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL3NjaGVtYS9nZW5lcmljL0dlbmVyaWNKc29uUmVhZGVyLmphdmE=) | | | | [.../pulsar/metadata/cache/impl/MetadataCacheImpl.java](https://codecov.io/gh/apache/pulsar/pull/19238?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLW1ldGFkYXRhL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9wdWxzYXIvbWV0YWRhdGEvY2FjaGUvaW1wbC9NZXRhZGF0YUNhY2hlSW1wbC5qYXZh) | | | | [...ulsar/client/impl/schema/AvroBaseStructSchema.java](https://codecov.io/gh/apache/pulsar/pull/19238?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL3NjaGVtYS9BdnJvQmFzZVN0cnVjdFNjaGVtYS5qYXZh) | | | | [...sar/broker/protocol/ProtocolHandlerDefinition.java](https://codecov.io/gh/apache/pulsar/pull/19238?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9wcm90b2NvbC9Qcm90b2NvbEhhbmRsZXJEZWZpbml0aW9uLmphdmE=) | | | | [...lient/impl/transaction/TransactionBuilderImpl.java](https://codecov.io/gh/apache/pulsar/pull/19238?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL3RyYW5zYWN0aW9uL1RyYW5zYWN0aW9uQnVpbGRlckltcGwuamF2YQ==) | | | | [.../org/apache/pulsar/broker/admin/AdminResource.java](https://codecov.io/gh/apache/pulsar/pull/19238?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9hZG1pbi9BZG1pblJlc291cmNlLmphdmE=) | | | |
[GitHub] [pulsar] yuruguo commented on pull request #19048: [improve][client] Add unified newTableView method in PulsarClient
yuruguo commented on PR #19048: URL: https://github.com/apache/pulsar/pull/19048#issuecomment-1383391121 @poorbarcode @liangyepianzhou I have started a discussion to dev mail and see more in [here](https://lists.apache.org/thread/w6vnk6cxh36j4p8oh9czdqlr5kqhdgb7) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] yaalsn commented on pull request #19237: [fix][ci] Fix codecov reporting by configuring to wait for all builds sending coverage
yaalsn commented on PR #19237: URL: https://github.com/apache/pulsar/pull/19237#issuecomment-1383363518 The final codecov result of comparison in PR comment is not accurated because the `master` branch CI should run after every PR merged, but now it doesn't because pulsar's CI runner resource is not sufficient and it needs long time to finish. Maybe the first thing is that we need to shorten the unit test running time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] mattisonchao opened a new issue, #19239: PIP-242: Introduce `enableStrictMode` to reject creating topic with `-partition-` keyword.
mattisonchao opened a new issue, #19239: URL: https://github.com/apache/pulsar/issues/19239 ### Motivation // todo ### Goal // todo ### API Changes // todo ### Implementation // todo ### Alternatives // todo ### Anything else? // todo -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] yaalsn commented on pull request #19229: [fix][build] Fix "An error has occurred in JaCoCo report generation"
yaalsn commented on PR #19229: URL: https://github.com/apache/pulsar/pull/19229#issuecomment-1383359156 Thanks! Let's have a try. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] yaalsn commented on pull request #19237: [fix][ci] Fix codecov reporting by configuring to wait for all builds sending coverage
yaalsn commented on PR #19237: URL: https://github.com/apache/pulsar/pull/19237#issuecomment-1383356748 It would be better to add `after_n_builds`, thanks for your improvement! After every CI finishes, the codecov will update the result in the comment and its platform, so currently the final result is right. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #18959: [enhancement][broker] Introduce thread block monitoring mechnism.
github-actions[bot] commented on issue #18959: URL: https://github.com/apache/pulsar/issues/18959#issuecomment-1383354757 The issue had no activity for 30 days, mark with Stale label. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on pull request #18934: [fix][sql] Fix presto license issue
github-actions[bot] commented on PR #18934: URL: https://github.com/apache/pulsar/pull/18934#issuecomment-1383354792 The pr had no activity for 30 days, mark with Stale label. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #18962: Flaky-test: TopicsTest.cleanup
github-actions[bot] commented on issue #18962: URL: https://github.com/apache/pulsar/issues/18962#issuecomment-1383354724 The issue had no activity for 30 days, mark with Stale label. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on issue #19214: Flaky-test: CompactionReaderImplTest.test
heesung-sn commented on issue #19214: URL: https://github.com/apache/pulsar/issues/19214#issuecomment-1383242917 Sorry for this back-and-forth. I couldn't replicate the issue on my local, but we could try this fix too. Raised another PR: https://github.com/apache/pulsar/pull/19238 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] heesung-sn opened a new pull request, #19238: Fix flaky test CompactionReaderImplTest(wait until the consumerFuture completes)
heesung-sn opened a new pull request, #19238: URL: https://github.com/apache/pulsar/pull/19238 Fixes https://github.com/apache/pulsar/issues/19214 ### Motivation Fixes https://github.com/apache/pulsar/issues/19214 ### Modifications Wait until the consumer future completes before calling the issue function, `reader.getLastMessageIdAsync()` ### Verifying this change I couldn't replicate the issue in my local env. However, we better wait til the consumer init completes before calling the consumer apis. - [x] Make sure that the change passes the CI checks. ### Does this pull request potentially affect one of the following parts: *If the box was checked, please highlight the changes* - [ ] Dependencies (add or upgrade a dependency) - [ ] The public API - [ ] The schema - [ ] The default values of configurations - [ ] The threading model - [ ] The binary protocol - [ ] The REST endpoints - [ ] The admin CLI options - [ ] The metrics - [ ] Anything that affects deployment ### Documentation - [ ] `doc` - [ ] `doc-required` - [x] `doc-not-needed` - [ ] `doc-complete` ### Matching PR in forked repository PR in forked repository: https://github.com/heesung-sn/pulsar/pull/21 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on issue #19214: Flaky-test: CompactionReaderImplTest.test
heesung-sn commented on issue #19214: URL: https://github.com/apache/pulsar/issues/19214#issuecomment-1383214869 @lhotari I see. will work on this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] congbobo184 commented on issue #19169: [Bug][txn] exclusive subscription would lead to high cpu usage when do tailing-read
congbobo184 commented on issue #19169: URL: https://github.com/apache/pulsar/issues/19169#issuecomment-1383208080 oh, sure. I will make https://github.com/apache/pulsar/pull/14286 merge quickly -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] merlimat commented on a diff in pull request #19208: [improve][broker]Enable custom metadata stores
merlimat commented on code in PR #19208:
URL: https://github.com/apache/pulsar/pull/19208#discussion_r1070641969
##
pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/MetadataStoreFactoryImpl.java:
##
@@ -45,19 +62,50 @@ public static MetadataStoreExtended createExtended(String
metadataURL, MetadataS
private static MetadataStore newInstance(String metadataURL,
MetadataStoreConfig metadataStoreConfig,
boolean enableSessionWatcher)
throws MetadataStoreException {
+MetadataStoreProvider provider = findProvider(metadataURL);
+if (provider != null) {
+return provider.create(metadataURL, metadataStoreConfig,
enableSessionWatcher);
+}
+return new ZKMetadataStore(metadataURL, metadataStoreConfig,
enableSessionWatcher);
+}
+
+static void loadProviders() {
+String factoryClasses =
System.getProperty(METADATASTORE_PROVIDERS_PROPERTY);
Review Comment:
This config needs to be loaded from many places, CLI tools to make it less
practical to add the config option everywhere. We also need it in BookKeeper
process and tools, where we have cannot add Pulsar's specific configs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] congbobo184 commented on a diff in pull request #19236: [improve][monitoring]PIP-231: Add `topic_load_failed` metric
congbobo184 commented on code in PR #19236:
URL: https://github.com/apache/pulsar/pull/19236#discussion_r1070641433
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/PulsarStats.java:
##
@@ -239,6 +239,14 @@ public void recordTopicLoadTimeValue(String topic, long
topicLoadLatencyMs) {
}
}
+public void recordTopicLoadFailed(String topic) {
+try {
+brokerOperabilityMetrics.recordTopicLoadFailed();
+} catch (Exception ex) {
Review Comment:
I don't think it needs catch exception, because we don't need to catch any
RuntimeException here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] congbobo184 commented on pull request #19201: [fix] [broker] fix timeout transaction.
congbobo184 commented on PR #19201:
URL: https://github.com/apache/pulsar/pull/19201#issuecomment-1383197970
@thetumbled
https://github.com/apache/pulsar/blob/246c2701e5c43e02e9783c82d4d107d06b019951/pulsar-broker/src/main/java/org/apache/pulsar/broker/TransactionMetadataStoreService.java#L136-L137
the future `thenAccept` change the executing thread, so the problem has
never been truth fixed.
so its better to change code like this can slove this problem
```
openTransactionMetadataStore(tcId).thenAccept((store) -> {
stores.put(tcId, store);
internalPinnedExecutor.execute(() -> {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] nodece commented on a diff in pull request #19203: [improve][websocket] Add ping support
nodece commented on code in PR #19203:
URL: https://github.com/apache/pulsar/pull/19203#discussion_r1070630222
##
pulsar-broker/src/test/java/org/apache/pulsar/websocket/proxy/ProxyPingTest.java:
##
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.websocket.proxy;
+
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static
org.apache.pulsar.broker.BrokerTestUtil.spyWithClassAndConstructorArgs;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.mockito.ArgumentMatchers.anyInt;
+import static org.mockito.ArgumentMatchers.anyString;
+import static org.mockito.Mockito.doReturn;
+import java.net.URI;
+import java.util.Optional;
+import java.util.concurrent.Future;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.client.api.ProducerConsumerBase;
+import org.apache.pulsar.metadata.impl.ZKMetadataStore;
+import org.apache.pulsar.websocket.WebSocketService;
+import org.apache.pulsar.websocket.service.ProxyServer;
+import org.apache.pulsar.websocket.service.WebSocketProxyConfiguration;
+import org.apache.pulsar.websocket.service.WebSocketServiceStarter;
+import org.awaitility.Awaitility;
+import org.eclipse.jetty.websocket.api.Session;
+import org.eclipse.jetty.websocket.client.ClientUpgradeRequest;
+import org.eclipse.jetty.websocket.client.WebSocketClient;
+import org.testng.annotations.AfterMethod;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+@Test(groups = "websocket")
+@Slf4j
+public class ProxyPingTest extends ProducerConsumerBase {
+protected String methodName;
+
+private ProxyServer proxyServer;
+private WebSocketService service;
+
+private static final int TIME_TO_CHECK_BACKLOG_QUOTA = 5;
+
+@BeforeMethod
+public void setup() throws Exception {
+
conf.setBacklogQuotaCheckIntervalInSeconds(TIME_TO_CHECK_BACKLOG_QUOTA);
+
+super.internalSetup();
+super.producerBaseSetup();
+
+WebSocketProxyConfiguration config = new WebSocketProxyConfiguration();
+config.setWebServicePort(Optional.of(0));
+config.setClusterName("test");
+config.setConfigurationMetadataStoreUrl(GLOBAL_DUMMY_VALUE);
+config.setWebSocketSessionIdleTimeoutMillis(3 * 1000);
+config.setWebSocketPingDurationSeconds(2);
+service = spyWithClassAndConstructorArgs(WebSocketService.class,
config);
+doReturn(new ZKMetadataStore(mockZooKeeperGlobal)).when(service)
+.createConfigMetadataStore(anyString(), anyInt());
+proxyServer = new ProxyServer(config);
+WebSocketServiceStarter.start(proxyServer, service);
+log.info("Proxy Server Started");
+}
+
+@AfterMethod(alwaysRun = true)
+protected void cleanup() throws Exception {
+super.internalCleanup();
+if (service != null) {
+service.close();
+}
+if (proxyServer != null) {
+proxyServer.stop();
+}
+log.info("Finished Cleaning Up Test setup");
+}
+
+@Test
+public void testPing() throws Exception {
+String producerUri = "ws://localhost:" +
proxyServer.getListenPortHTTP().get() +
+"/ws/v2/producer/persistent/my-property/my-ns/my-topic1/";
+
+URI produceUri = URI.create(producerUri);
+WebSocketClient produceClient = new WebSocketClient();
+SimpleProducerSocket produceSocket = new SimpleProducerSocket();
+
+try {
+produceClient.start();
+ClientUpgradeRequest produceRequest = new ClientUpgradeRequest();
+Future producerFuture =
produceClient.connect(produceSocket, produceUri, produceRequest);
+assertThat(producerFuture).succeedsWithin(2, SECONDS);
+Session session = producerFuture.get();
+Awaitility.await().during(5, SECONDS).untilAsserted(() ->
assertThat(session.isOpen()).isTrue());
Review Comment:
`10s` is enough, and it is fast.
##
pulsar-broker/src/test/java/org/apache/pulsar/websocket/proxy/ProxyPingTest.java:
##
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ *
[GitHub] [pulsar] lhotari commented on pull request #19237: [fix][ci] Fox codecov reporting by configuring to wait for all builds sending coverage
lhotari commented on PR #19237: URL: https://github.com/apache/pulsar/pull/19237#issuecomment-1383160706 Code coverage was activated in #17382 . @yaalsn would you mind reviewing this PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] lhotari opened a new pull request, #19237: [fix][ci] Fox codecov reporting by configuring to wait for all builds sending coverage
lhotari opened a new pull request, #19237: URL: https://github.com/apache/pulsar/pull/19237 ### Motivation - currently the codecov reporting processes the uploaded coverage reports too early and thus the coverage result is invalid. - the `after_n_builds` configuration setting should be used for waiting for all builds to finish before merging the coverage reports from parallel builds. - see https://docs.codecov.com/docs/notifications#preventing-notifications-until-after-n-builds ### Modifications Set `after_n_builds` to 10. ### Documentation - [ ] `doc` - [ ] `doc-required` - [x] `doc-not-needed` - [ ] `doc-complete` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] asafm commented on pull request #19047: [improve][broker]PIP-214 Add broker level metrics statistics and expose to prometheus
asafm commented on PR #19047: URL: https://github.com/apache/pulsar/pull/19047#issuecomment-1383158946 @codelipenghui Looks good from my perspective -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-node] BewareMyPower commented on issue #284: Document minimum required libstdc++ version
BewareMyPower commented on issue #284: URL: https://github.com/apache/pulsar-client-node/issues/284#issuecomment-1383152921 > An alternative could be to not dynamically link any dependencies at all Maybe we can link statically to `libstdc++` and `libgcc_s` on Linux for Pulsar C++ client. But when building the Node Add-on, we also need to link to them statically. I think we need to fix the building script to fallback to building from source for systems with lower version libstdc++. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on a diff in pull request #19138: [feat][broker][PIP-195] Implement delayed message index bucket snapshot(merge/delete) - part8
poorbarcode commented on code in PR #19138:
URL: https://github.com/apache/pulsar/pull/19138#discussion_r1070593242
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/delayed/bucket/CombinedSegmentDelayedIndexQueue.java:
##
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.delayed.bucket;
+
+import java.util.List;
+import javax.annotation.concurrent.NotThreadSafe;
+import
org.apache.pulsar.broker.delayed.proto.DelayedMessageIndexBucketSnapshotFormat.DelayedIndex;
+import
org.apache.pulsar.broker.delayed.proto.DelayedMessageIndexBucketSnapshotFormat.SnapshotSegment;
+
+@NotThreadSafe
+public class CombinedSegmentDelayedIndexQueue implements DelayedIndexQueue {
Review Comment:
Hello @coderzc
And this comment (^_^)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on a diff in pull request #19203: [improve][websocket] Add ping support
poorbarcode commented on code in PR #19203:
URL: https://github.com/apache/pulsar/pull/19203#discussion_r1070575855
##
pulsar-broker/src/test/java/org/apache/pulsar/websocket/proxy/ProxyPingTest.java:
##
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.websocket.proxy;
+
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static
org.apache.pulsar.broker.BrokerTestUtil.spyWithClassAndConstructorArgs;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.mockito.ArgumentMatchers.anyInt;
+import static org.mockito.ArgumentMatchers.anyString;
+import static org.mockito.Mockito.doReturn;
+import java.net.URI;
+import java.util.Optional;
+import java.util.concurrent.Future;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.client.api.ProducerConsumerBase;
+import org.apache.pulsar.metadata.impl.ZKMetadataStore;
+import org.apache.pulsar.websocket.WebSocketService;
+import org.apache.pulsar.websocket.service.ProxyServer;
+import org.apache.pulsar.websocket.service.WebSocketProxyConfiguration;
+import org.apache.pulsar.websocket.service.WebSocketServiceStarter;
+import org.awaitility.Awaitility;
+import org.eclipse.jetty.websocket.api.Session;
+import org.eclipse.jetty.websocket.client.ClientUpgradeRequest;
+import org.eclipse.jetty.websocket.client.WebSocketClient;
+import org.testng.annotations.AfterMethod;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+@Test(groups = "websocket")
+@Slf4j
+public class ProxyPingTest extends ProducerConsumerBase {
+protected String methodName;
+
+private ProxyServer proxyServer;
+private WebSocketService service;
+
+private static final int TIME_TO_CHECK_BACKLOG_QUOTA = 5;
+
+@BeforeMethod
+public void setup() throws Exception {
+
conf.setBacklogQuotaCheckIntervalInSeconds(TIME_TO_CHECK_BACKLOG_QUOTA);
+
+super.internalSetup();
+super.producerBaseSetup();
+
+WebSocketProxyConfiguration config = new WebSocketProxyConfiguration();
+config.setWebServicePort(Optional.of(0));
+config.setClusterName("test");
+config.setConfigurationMetadataStoreUrl(GLOBAL_DUMMY_VALUE);
+config.setWebSocketSessionIdleTimeoutMillis(3 * 1000);
+config.setWebSocketPingDurationSeconds(2);
+service = spyWithClassAndConstructorArgs(WebSocketService.class,
config);
+doReturn(new ZKMetadataStore(mockZooKeeperGlobal)).when(service)
+.createConfigMetadataStore(anyString(), anyInt());
+proxyServer = new ProxyServer(config);
+WebSocketServiceStarter.start(proxyServer, service);
+log.info("Proxy Server Started");
+}
+
+@AfterMethod(alwaysRun = true)
+protected void cleanup() throws Exception {
+super.internalCleanup();
+if (service != null) {
+service.close();
+}
+if (proxyServer != null) {
+proxyServer.stop();
+}
+log.info("Finished Cleaning Up Test setup");
+}
+
+@Test
+public void testPing() throws Exception {
+String producerUri = "ws://localhost:" +
proxyServer.getListenPortHTTP().get() +
+"/ws/v2/producer/persistent/my-property/my-ns/my-topic1/";
+
+URI produceUri = URI.create(producerUri);
+WebSocketClient produceClient = new WebSocketClient();
+SimpleProducerSocket produceSocket = new SimpleProducerSocket();
+
+try {
+produceClient.start();
+ClientUpgradeRequest produceRequest = new ClientUpgradeRequest();
+Future producerFuture =
produceClient.connect(produceSocket, produceUri, produceRequest);
+assertThat(producerFuture).succeedsWithin(2, SECONDS);
+Session session = producerFuture.get();
+Awaitility.await().during(5, SECONDS).untilAsserted(() ->
assertThat(session.isOpen()).isTrue());
Review Comment:
I think we can set `during` to `10s`, then if someone changes the code of
keep connections by mistake, this test will tell us that he is wrong.
--
This is an automated message from the Apache Git Service.
To respond to
[GitHub] [pulsar] poorbarcode commented on a diff in pull request #19203: [improve][websocket] Add ping support
poorbarcode commented on code in PR #19203:
URL: https://github.com/apache/pulsar/pull/19203#discussion_r1070575855
##
pulsar-broker/src/test/java/org/apache/pulsar/websocket/proxy/ProxyPingTest.java:
##
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.websocket.proxy;
+
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static
org.apache.pulsar.broker.BrokerTestUtil.spyWithClassAndConstructorArgs;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.mockito.ArgumentMatchers.anyInt;
+import static org.mockito.ArgumentMatchers.anyString;
+import static org.mockito.Mockito.doReturn;
+import java.net.URI;
+import java.util.Optional;
+import java.util.concurrent.Future;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.client.api.ProducerConsumerBase;
+import org.apache.pulsar.metadata.impl.ZKMetadataStore;
+import org.apache.pulsar.websocket.WebSocketService;
+import org.apache.pulsar.websocket.service.ProxyServer;
+import org.apache.pulsar.websocket.service.WebSocketProxyConfiguration;
+import org.apache.pulsar.websocket.service.WebSocketServiceStarter;
+import org.awaitility.Awaitility;
+import org.eclipse.jetty.websocket.api.Session;
+import org.eclipse.jetty.websocket.client.ClientUpgradeRequest;
+import org.eclipse.jetty.websocket.client.WebSocketClient;
+import org.testng.annotations.AfterMethod;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+@Test(groups = "websocket")
+@Slf4j
+public class ProxyPingTest extends ProducerConsumerBase {
+protected String methodName;
+
+private ProxyServer proxyServer;
+private WebSocketService service;
+
+private static final int TIME_TO_CHECK_BACKLOG_QUOTA = 5;
+
+@BeforeMethod
+public void setup() throws Exception {
+
conf.setBacklogQuotaCheckIntervalInSeconds(TIME_TO_CHECK_BACKLOG_QUOTA);
+
+super.internalSetup();
+super.producerBaseSetup();
+
+WebSocketProxyConfiguration config = new WebSocketProxyConfiguration();
+config.setWebServicePort(Optional.of(0));
+config.setClusterName("test");
+config.setConfigurationMetadataStoreUrl(GLOBAL_DUMMY_VALUE);
+config.setWebSocketSessionIdleTimeoutMillis(3 * 1000);
+config.setWebSocketPingDurationSeconds(2);
+service = spyWithClassAndConstructorArgs(WebSocketService.class,
config);
+doReturn(new ZKMetadataStore(mockZooKeeperGlobal)).when(service)
+.createConfigMetadataStore(anyString(), anyInt());
+proxyServer = new ProxyServer(config);
+WebSocketServiceStarter.start(proxyServer, service);
+log.info("Proxy Server Started");
+}
+
+@AfterMethod(alwaysRun = true)
+protected void cleanup() throws Exception {
+super.internalCleanup();
+if (service != null) {
+service.close();
+}
+if (proxyServer != null) {
+proxyServer.stop();
+}
+log.info("Finished Cleaning Up Test setup");
+}
+
+@Test
+public void testPing() throws Exception {
+String producerUri = "ws://localhost:" +
proxyServer.getListenPortHTTP().get() +
+"/ws/v2/producer/persistent/my-property/my-ns/my-topic1/";
+
+URI produceUri = URI.create(producerUri);
+WebSocketClient produceClient = new WebSocketClient();
+SimpleProducerSocket produceSocket = new SimpleProducerSocket();
+
+try {
+produceClient.start();
+ClientUpgradeRequest produceRequest = new ClientUpgradeRequest();
+Future producerFuture =
produceClient.connect(produceSocket, produceUri, produceRequest);
+assertThat(producerFuture).succeedsWithin(2, SECONDS);
+Session session = producerFuture.get();
+Awaitility.await().during(5, SECONDS).untilAsserted(() ->
assertThat(session.isOpen()).isTrue());
Review Comment:
I think we can set `during` to `10s`, then if someone changes the code by
mistake, this test will tell us that he is wrong.
--
This is an automated message from the Apache Git Service.
To respond to the message, please
[GitHub] [pulsar] tjiuming commented on a diff in pull request #19236: [improve][monitoring]PIP-231: Add `topic_load_failed` metric
tjiuming commented on code in PR #19236:
URL: https://github.com/apache/pulsar/pull/19236#discussion_r1070571380
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/PulsarStats.java:
##
@@ -239,6 +239,14 @@ public void recordTopicLoadTimeValue(String topic, long
topicLoadLatencyMs) {
}
}
+public void recordTopicLoadFailed(String topic) {
+try {
+brokerOperabilityMetrics.recordTopicLoadFailed();
+} catch (Exception ex) {
Review Comment:
just following the old pattern:
https://github.com/apache/pulsar/blob/master/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/PulsarStats.java#L234
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] congbobo184 commented on a diff in pull request #19236: [improve][monitoring]PIP-231: Add `topic_load_failed` metric
congbobo184 commented on code in PR #19236:
URL: https://github.com/apache/pulsar/pull/19236#discussion_r1070568329
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java:
##
@@ -1176,6 +1176,10 @@ public void deleteTopicAuthenticationWithRetry(String
topic, CompletableFuture> createNonPersistentTopic(String
topic) {
CompletableFuture> topicFuture = new
CompletableFuture<>();
+topicFuture.exceptionally(t -> {
+pulsarStats.recordTopicLoadFailed(topic);
+return Optional.empty();
Review Comment:
```suggestion
return null;
```
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/PulsarStats.java:
##
@@ -239,6 +239,14 @@ public void recordTopicLoadTimeValue(String topic, long
topicLoadLatencyMs) {
}
}
+public void recordTopicLoadFailed(String topic) {
+try {
+brokerOperabilityMetrics.recordTopicLoadFailed();
+} catch (Exception ex) {
Review Comment:
why catch exception?
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java:
##
@@ -1522,6 +1526,11 @@ private void createPersistentTopic(final String topic,
boolean createIfMissing,
TopicName topicName = TopicName.get(topic);
final long topicCreateTimeMs =
TimeUnit.NANOSECONDS.toMillis(System.nanoTime());
+topicFuture.exceptionally(t -> {
+pulsarStats.recordTopicLoadFailed(topic);
+return Optional.empty();
Review Comment:
```suggestion
return null;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] mattisonchao merged pull request #19234: [improve][broker] Follow up #19230 to tighten the validation scope
mattisonchao merged PR #19234: URL: https://github.com/apache/pulsar/pull/19234 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] Technoboy- merged pull request #19182: [cleanup][broker] Remove AuthorizationProvider methods deprecated in 2.7 and 2.9
Technoboy- merged PR #19182: URL: https://github.com/apache/pulsar/pull/19182 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] tjiuming commented on pull request #19236: [improve][monitoring]PIP-231: Add `topic_load_failed` metric
tjiuming commented on PR #19236: URL: https://github.com/apache/pulsar/pull/19236#issuecomment-1383118982 Tests to be completed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] tjiuming opened a new pull request, #19236: [improve][monitoring]PIP-231: Add `topic_load_failed` metric
tjiuming opened a new pull request, #19236: URL: https://github.com/apache/pulsar/pull/19236 PIP: https://github.com/apache/pulsar/issues/18979 ### Motivation Add `topic_load_failed` metric ### Does this pull request potentially affect one of the following parts: *If the box was checked, please highlight the changes* - [ ] Dependencies (add or upgrade a dependency) - [ ] The public API - [ ] The schema - [ ] The default values of configurations - [ ] The threading model - [ ] The binary protocol - [ ] The REST endpoints - [ ] The admin CLI options - [x] The metrics - [ ] Anything that affects deployment ### Documentation - [ ] `doc` - [x] `doc-required` - [ ] `doc-not-needed` - [ ] `doc-complete` ### Matching PR in forked repository PR in forked repository: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-site] asafm commented on a diff in pull request #349: [doc][improve] Improve landing page 3 first sections
asafm commented on code in PR #349:
URL: https://github.com/apache/pulsar-site/pull/349#discussion_r1070543525
##
src/components/HomepageFeatures.js:
##
@@ -4,48 +4,48 @@ import ReactMarkdown from "react-markdown";
const FeatureList = (language) => [
{
-title: "Cloud-native",
-Svg: require("../../static/img/Technology-Solution.svg").default,
-content: "A multiple layer approach separating compute from storage to
work with cloud infrastructures and Kubernetes.",
+title: "Automatic Load Balancing",
+Svg: require("../../static/img/automatic-load-balancing.svg").default,
+content: "Add or remove nodes and let Pulsar load balance topic bundles
automatically. Hot spotted topic bundles are automatically split and evenly
distributed across the brokers.",
},
{
title: "Serverless functions",
Svg: require("../../static/img/proven-in-production.svg").default,
-content:"Write serverless functions with developer-friendly APIs to
natively process data immediately upon arrival. No need to run your own stream
processing engine.",
+content:"Write and deploy functions natively using Pulsar Functions.
Process messages using Java, Go, or Python without deploying fully-fledged
applications. Kubernetes runtime is bundled.",
},
{
-title: "Horizontally scalable",
+title: "Rapid Horizontal Scalability",
Svg: require("../../static/img/horizontally-scalable.svg").default,
-content: "Expand capacity seamlessly to hundreds of nodes.",
+content: "Scales horizontally to handle the increased load. Its unique
design and separate storage layer enable handling the sudden surge in traffic
by scaling out in seconds.",
},
{
-title: "Low latency with durability",
+title: "Low-latency messaging and streaming",
Svg: require("../../static/img/low-latency.svg").default,
-content:"Low publish latency (< 5ms) at scale with strong durability
guarantees.",
+content:"Acknowledge messages individually (RabbitMQ style) or cumulative
per partition (i.e., offset-like). Enables use cases such as distributed work
queues or order-preserving data streams at very large scales (hundreds of
nodes) and low latency (<5ms).",
},
{
-title: "Geo-replication",
+title: "Seamless Geo-Replication",
Svg: require("../../static/img/geo-replication.svg").default,
-content: "Configurable replication between data centers across multiple
geographic regions.",
+content: "Protect against complete zone outages using replication across
different geographic regions. Flexible and configurable replication strategies
across distant Pulsar Clusters. Uniquely supports automatic client failover to
healthy clusters.",
},
{
title: "Multi-tenancy",
Svg: require("../../static/img/multi-tenancy.svg").default,
-content:"Built from the ground up as a multi-tenant system. Supports
isolation, authentication, authorization, and quotas.",
+content:"Maintain one cluster for your entire organization using tenants.
Access control across data and actions using tenant policies. Isolate specific
brokers to a tenant when maximum noisy neighbor protection is needed.",
},
{
-title: "Persistent storage",
Review Comment:
Perhaps the changes proposed to previous comments lead the reader to
understand that data is persisted since it's durable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar-site] asafm commented on a diff in pull request #349: [doc][improve] Improve landing page 3 first sections
asafm commented on code in PR #349:
URL: https://github.com/apache/pulsar-site/pull/349#discussion_r1070543002
##
src/components/HomepageFeatures.js:
##
@@ -4,48 +4,48 @@ import ReactMarkdown from "react-markdown";
const FeatureList = (language) => [
{
-title: "Cloud-native",
-Svg: require("../../static/img/Technology-Solution.svg").default,
-content: "A multiple layer approach separating compute from storage to
work with cloud infrastructures and Kubernetes.",
+title: "Automatic Load Balancing",
+Svg: require("../../static/img/automatic-load-balancing.svg").default,
+content: "Add or remove nodes and let Pulsar load balance topic bundles
automatically. Hot spotted topic bundles are automatically split and evenly
distributed across the brokers.",
},
{
title: "Serverless functions",
Svg: require("../../static/img/proven-in-production.svg").default,
-content:"Write serverless functions with developer-friendly APIs to
natively process data immediately upon arrival. No need to run your own stream
processing engine.",
+content:"Write and deploy functions natively using Pulsar Functions.
Process messages using Java, Go, or Python without deploying fully-fledged
applications. Kubernetes runtime is bundled.",
},
{
-title: "Horizontally scalable",
+title: "Rapid Horizontal Scalability",
Svg: require("../../static/img/horizontally-scalable.svg").default,
-content: "Expand capacity seamlessly to hundreds of nodes.",
+content: "Scales horizontally to handle the increased load. Its unique
design and separate storage layer enable handling the sudden surge in traffic
by scaling out in seconds.",
},
{
-title: "Low latency with durability",
+title: "Low-latency messaging and streaming",
Review Comment:
Thanks for the clarification, @michaeljmarshall. How about we'll write it as
such:
**Durable low-latency messaging and streaming**
Acknowledge messages individually (RabbitMQ style) or cumulative per
partition (i.e., offset-like). Enables use cases such as distributed work
queues or order-preserving data streams at very large scales (hundreds of
nodes) and low latency (<5ms). Message durability is achieved using Bookkeeper
as a storage layer and Cloud-based tiered storage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar-site] asafm commented on pull request #349: [doc][improve] Improve landing page 3 first sections
asafm commented on PR #349: URL: https://github.com/apache/pulsar-site/pull/349#issuecomment-1383097360 > I'm not sure the best format for this discussion, but in my opinion, I think we need to discuss the motivation for changing the top level feature list @DaveDuggins. There are some fundamental changes here. I pasted the context/motivation for changing it also in the mailing list to open it up for discussion. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-site] asafm commented on pull request #349: [doc][improve] Improve landing page 3 first sections
asafm commented on PR #349: URL: https://github.com/apache/pulsar-site/pull/349#issuecomment-1383096720 > > @DaveDuggins, Can you please change the title and summary to reflect the content and intent of this PR? I think another PR you had got the description and title mixed into here by mistake. > > Great point @asafm. Please make sure to update the PR description @DaveDuggins with the intended changes and the justification for these changes. > > I also agree with @dave2wave that we need discuss many of the changes proposed in this PR on the dev mailing list. @michaeljmarshall , @DaveDuggins updated the PR with the original description that should have been there but was mixed out. I'll paste that context on the mailing list thread @dave2wave already opened. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] asafm commented on a diff in pull request #19208: [improve][broker]Enable custom metadata stores
asafm commented on code in PR #19208:
URL: https://github.com/apache/pulsar/pull/19208#discussion_r1070531001
##
pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/MetadataStoreFactoryImpl.java:
##
@@ -18,13 +18,30 @@
*/
package org.apache.pulsar.metadata.impl;
+import static
org.apache.pulsar.metadata.impl.EtcdMetadataStore.ETCD_SCHEME_IDENTIFIER;
+import static
org.apache.pulsar.metadata.impl.LocalMemoryMetadataStore.MEMORY_SCHEME_IDENTIFIER;
+import static
org.apache.pulsar.metadata.impl.RocksdbMetadataStore.ROCKSDB_SCHEME_IDENTIFIER;
+import static
org.apache.pulsar.metadata.impl.ZKMetadataStore.ZK_SCHEME_IDENTIFIER;
+import java.util.HashMap;
+import java.util.Map;
+import lombok.extern.slf4j.Slf4j;
import org.apache.pulsar.metadata.api.MetadataStore;
import org.apache.pulsar.metadata.api.MetadataStoreConfig;
import org.apache.pulsar.metadata.api.MetadataStoreException;
+import org.apache.pulsar.metadata.api.MetadataStoreProvider;
import org.apache.pulsar.metadata.api.extended.MetadataStoreExtended;
+@Slf4j
public class MetadataStoreFactoryImpl {
+public static final String METADATASTORE_PROVIDERS_PROPERTY =
"pulsar.metadatastore.providers";
+
+private static Map providers;
+
+static {
+loadProviders();
Review Comment:
This is not good for testing and, in conjunction with the comment below,
IMO, not good for production code.
In tests, you fill this static map which will never be emptied. Carrying
over leftovers from one test to the next eventually creates flaky tests and
unexpected behavior, especially over time with many people working on the same
code base IMO.
I suggest making this class stateful and instantiating it once when Pulsar
boots. This solves the issue of the tests and creates a more streamlined code.
##
pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/MetadataStoreFactoryImpl.java:
##
@@ -45,19 +62,50 @@ public static MetadataStoreExtended createExtended(String
metadataURL, MetadataS
private static MetadataStore newInstance(String metadataURL,
MetadataStoreConfig metadataStoreConfig,
boolean enableSessionWatcher)
throws MetadataStoreException {
+MetadataStoreProvider provider = findProvider(metadataURL);
+if (provider != null) {
+return provider.create(metadataURL, metadataStoreConfig,
enableSessionWatcher);
+}
+return new ZKMetadataStore(metadataURL, metadataStoreConfig,
enableSessionWatcher);
+}
+
+static void loadProviders() {
+String factoryClasses =
System.getProperty(METADATASTORE_PROVIDERS_PROPERTY);
+providers = new HashMap<>();
+providers.put(MEMORY_SCHEME_IDENTIFIER, new
DefaultMetadataStoreProvider(MEMORY_SCHEME_IDENTIFIER,
+(url, config, enableWatchers) -> new
LocalMemoryMetadataStore(url, config)));
+providers.put(ROCKSDB_SCHEME_IDENTIFIER, new
DefaultMetadataStoreProvider(ROCKSDB_SCHEME_IDENTIFIER,
+(url, config, enableWatchers) -> RocksdbMetadataStore.get(url,
config)));
+providers.put(ETCD_SCHEME_IDENTIFIER, new
DefaultMetadataStoreProvider(ETCD_SCHEME_IDENTIFIER,
+EtcdMetadataStore::new));
+providers.put(ZK_SCHEME_IDENTIFIER, new
DefaultMetadataStoreProvider(ZK_SCHEME_IDENTIFIER,
+(url, config, enableWatchers) ->
+new
ZKMetadataStore(url.substring(ZK_SCHEME_IDENTIFIER.length()), config,
enableWatchers)));
+
+if (factoryClasses == null) {
+return;
+}
+String[] classNames = factoryClasses.split(",");
+for (String className : classNames) {
+try {
+Class clazz =
+(Class)
Class.forName(className);
+MetadataStoreProvider provider =
clazz.getConstructor().newInstance();
+String scheme = provider.urlScheme();
+providers.put(scheme, provider);
+} catch (Exception e) {
+log.warn("Failed to load metadata store provider class for
name '{}'", className, e);
+}
+}
+}
-if
(metadataURL.startsWith(LocalMemoryMetadataStore.MEMORY_SCHEME_IDENTIFIER)) {
-return new LocalMemoryMetadataStore(metadataURL,
metadataStoreConfig);
-} else if
(metadataURL.startsWith(RocksdbMetadataStore.ROCKSDB_SCHEME_IDENTIFIER)) {
-return RocksdbMetadataStore.get(metadataURL, metadataStoreConfig);
-} else if
(metadataURL.startsWith(EtcdMetadataStore.ETCD_SCHEME_IDENTIFIER)) {
-return new EtcdMetadataStore(metadataURL, metadataStoreConfig,
enableSessionWatcher);
-} else if
(metadataURL.startsWith(ZKMetadataStore.ZK_SCHEME_IDENTIFIER)) {
-return new
ZKMetadataStore(metadataURL.substring(ZKMetadataStore.ZK_SCHEME_IDENTIFIER.length()),
-
[GitHub] [pulsar] BewareMyPower commented on issue #19122: no org_apache_pulsar_shade_netty_tcnative_x86_64 in java.library.path
BewareMyPower commented on issue #19122: URL: https://github.com/apache/pulsar/issues/19122#issuecomment-1383086830 No. But I think we should. The only reason that Pulsar cannot run on Windows is that RocksDB, which is depended by BookKeeper, cannot run on Windows. I have debugged before and found it might be something wrong with BK that BK does not handle the Windows file path well. So the unit tests that depend on a mocked BK can run well on Windows. I have been working on Windows for several months and the unit tests worked well. However, the client that does not depend on BK should run well on Windows. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on pull request #19147: [fix][broker] catch exception for brokerInterceptor
BewareMyPower commented on PR #19147: URL: https://github.com/apache/pulsar/pull/19147#issuecomment-1383085731 @Jason918 It's not related to whether the interceptor developers are core developers of Pulsar. It's related to the fact that there is no clear definition in Pulsar about how the broker handles an exception from the interceptor. I said "complicated" because they have to look deeper into the code when they found the interceptor threw an exception and caused an undefined behavior like this PR and its related issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codecov-commenter commented on pull request #19235: [fix][broker] Support getStats/update partitioned topic with `-partition-`
codecov-commenter commented on PR #19235: URL: https://github.com/apache/pulsar/pull/19235#issuecomment-1383068884 # [Codecov](https://codecov.io/gh/apache/pulsar/pull/19235?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#19235](https://codecov.io/gh/apache/pulsar/pull/19235?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (46cc314) into [master](https://codecov.io/gh/apache/pulsar/commit/accae9f371f87dcf0b2a2f8d4ed277b742bfdc5f?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (accae9f) will **decrease** coverage by `10.90%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/pulsar/pull/19235?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #19235 +/- ## = - Coverage 48.17% 37.27% -10.91% + Complexity 9695 1990 -7705 = Files 633 209 -424 Lines 5992214442-45480 Branches 6251 1578 -4673 = - Hits 28870 5383-23487 + Misses27947 8477-19470 + Partials 3105 582 -2523 ``` | Flag | Coverage Δ | | |---|---|---| | unittests | `37.27% <ø> (-10.91%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/pulsar/pull/19235?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...he/pulsar/broker/admin/v2/NonPersistentTopics.java](https://codecov.io/gh/apache/pulsar/pull/19235?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9hZG1pbi92Mi9Ob25QZXJzaXN0ZW50VG9waWNzLmphdmE=) | | | | [...pache/pulsar/broker/admin/v2/PersistentTopics.java](https://codecov.io/gh/apache/pulsar/pull/19235?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9hZG1pbi92Mi9QZXJzaXN0ZW50VG9waWNzLmphdmE=) | | | | [...broker/stats/prometheus/TransactionAggregator.java](https://codecov.io/gh/apache/pulsar/pull/19235?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9zdGF0cy9wcm9tZXRoZXVzL1RyYW5zYWN0aW9uQWdncmVnYXRvci5qYXZh) | | | | [...java/org/apache/pulsar/broker/admin/v3/Source.java](https://codecov.io/gh/apache/pulsar/pull/19235?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9hZG1pbi92My9Tb3VyY2UuamF2YQ==) | | | | [...schema/ProtobufNativeSchemaCompatibilityCheck.java](https://codecov.io/gh/apache/pulsar/pull/19235?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9zZXJ2aWNlL3NjaGVtYS9Qcm90b2J1Zk5hdGl2ZVNjaGVtYUNvbXBhdGliaWxpdHlDaGVjay5qYXZh) | | | | [...pulsar/proxy/extensions/ExtensionsDefinitions.java](https://codecov.io/gh/apache/pulsar/pull/19235?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLXByb3h5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9wdWxzYXIvcHJveHkvZXh0ZW5zaW9ucy9FeHRlbnNpb25zRGVmaW5pdGlvbnMuamF2YQ==) | | | | [.../org/apache/pulsar/broker/lookup/LookupResult.java](https://codecov.io/gh/apache/pulsar/pull/19235?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9sb29rdXAvTG9va3VwUmVzdWx0LmphdmE=) | | | |
[GitHub] [pulsar] Technoboy- merged pull request #19206: [branch-2.11][cherry-pick] Elasticsearch sink: Support loading config from secrets
Technoboy- merged PR #19206: URL: https://github.com/apache/pulsar/pull/19206 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] Technoboy- merged pull request #19205: [branch-2.11][cherry-pick] Debezium sources: Support loading config from secrets
Technoboy- merged PR #19205: URL: https://github.com/apache/pulsar/pull/19205 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] yuruguo opened a new pull request, #19235: [fix][broker] Support getStats/update partitioned topic with `-partition-`
yuruguo opened a new pull request, #19235:
URL: https://github.com/apache/pulsar/pull/19235
### Motivation
Same as #19230, We allow users to use the `Client API` to create the
partitioned topic which name contain `-partition-{not-number}` when they enable
partitioned type auto-creation. But we didn't get stats and update it.
### Modifications
Support getting stats and updating partitioned topics with the keyword
`-partition-{not-number}`
### Documentation
- [ ] `doc`
- [ ] `doc-required`
- [x] `doc-not-needed`
- [ ] `doc-complete`
### Matching PR in forked repository
PR in forked repository:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] Technoboy- closed pull request #19205: [branch-2.11][cherry-pick] Debezium sources: Support loading config from secrets
Technoboy- closed pull request #19205: [branch-2.11][cherry-pick] Debezium sources: Support loading config from secrets URL: https://github.com/apache/pulsar/pull/19205 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] Technoboy- closed pull request #19206: [branch-2.11][cherry-pick] Elasticsearch sink: Support loading config from secrets
Technoboy- closed pull request #19206: [branch-2.11][cherry-pick] Elasticsearch sink: Support loading config from secrets URL: https://github.com/apache/pulsar/pull/19206 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] Technoboy- merged pull request #19232: [branch-2.11][ci] Fix license check issue
Technoboy- merged PR #19232: URL: https://github.com/apache/pulsar/pull/19232 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
