[GitHub] [pulsar] coderzc commented on pull request #20176: [fix][broker] Fix `RoaringBitmap.contains` can't check value 65535
coderzc commented on PR #20176: URL: https://github.com/apache/pulsar/pull/20176#issuecomment-1521190130 > RoaringBitmap/RoaringBitmap#623 Yeah, I'm not sure if our usage is wrong -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on pull request #20176: [fix][broker] Fix `RoaringBitmap.contains` can't check value 65535
codelipenghui commented on PR #20176: URL: https://github.com/apache/pulsar/pull/20176#issuecomment-1521184920 Interesting, ConcurrentBitmapSortedLongPairSet also used the same method. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] asafm commented on a diff in pull request #19832: [fix][misc] Change PIP issue template
asafm commented on code in PR #19832: URL: https://github.com/apache/pulsar/pull/19832#discussion_r1176031180 ## .github/ISSUE_TEMPLATE/pip.md: ## @@ -0,0 +1,157 @@ +--- +name: PIP +about: Submit a Pulsar Improvement Proposal (PIP) +title: 'PIP-XYZ: ' +labels: PIP +--- + + + +# Background knowledge + + + +# Motivation + + + +# Goals + +## In Scope + + + +## Out of Scope + + + + +# High Level Design + + + +# Detailed Design + +## Design & Implementation Details + + + +## Public-facing Changes + + + +### Public API + Review Comment: @liangyepianzhou ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] asafm commented on a diff in pull request #19832: [fix][misc] Change PIP issue template
asafm commented on code in PR #19832: URL: https://github.com/apache/pulsar/pull/19832#discussion_r1176030857 ## .github/ISSUE_TEMPLATE/pip.md: ## @@ -0,0 +1,151 @@ +--- +name: PIP +about: Submit a Pulsar Improvement Proposal (PIP) +title: 'PIP-XYZ: ' +labels: PIP +--- + + + +# Background knowledge + + + +# Motivation + + + +# Goals + +## In Scope + + + +## Out of Scope + + + + +# High Level Design + + + +# Detailed Design + +## Design & Implementation Details + + + +# Background knowledge + + + +# Motivation + + + +# Goals + +## In Scope + + + +## Out of Scope + + + + +# High Level Design + + + +# Detailed Design + +## Design & Implementation Details + + + +## Public-facing Changes + + + +### Public API + +### Binary protocol + +### Configuration + +### CLI + +### Metrics + + + + +# Monitoring Review Comment: @tisonkun -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] equanz opened a new pull request, #20179: [fix][broker] Fix recentlyJoinedConsumers to address the out-of-order bug.
equanz opened a new pull request, #20179: URL: https://github.com/apache/pulsar/pull/20179 ### Motivation The Key_Shared subscription type has the following issues. 1. Key_Shared subscription has out-of-order cases because of the race condition of [the recently joined consumers feature](https://github.com/apache/pulsar/blob/35e9897742b7db4bd29349940075a819b2ad6999/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentStickyKeyDispatcherMultipleConsumers.java#L400-L408). Consider the following flow. 1. Assume that the current read position is `1:6` and the recently joined consumers is empty. 2. Called [OpReadEntry#internalReadEntriesComplete](https://github.com/apache/pulsar/blob/35e9897742b7db4bd29349940075a819b2ad6999/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpReadEntry.java#L92-L95) from thread-1. Then, the current read position is updated to `1:11` (Messages from `1:6` to `1:10` have yet to be dispatched to consumers). 3. Called [PersistentStickyKeyDispatcherMultipleConsumers#addConsumer](https://github.com/apache/pulsar/blob/35e9897742b7db4bd29349940075a819b2ad6999/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentStickyKeyDispatcherMultipleConsumers.java#L130-L131) from thread-2. Then, the new consumer is stored to recently joined consumers with read position `1:11`. 4. Called [PersistentDispatcherMultipleConsumers#trySendMessagesToConsumers](https://github.com/apache/pulsar/blob/35e9897742b7db4bd29349940075a819b2ad6999/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentStickyKeyDispatcherMultipleConsumers.java#L174) from thread-5. Then, messages from `1:6` to `1:10` are dispatched to consumers. From the recently joined consumers feature, the new consumer can receive messages from `1:6` to `1:10`. **However, it is not expected.** For example, if existing consumers have some unacked messages and disconnect, it causes out of order in some cases. 2. Key_Shared subscription has a redundant process. [The stuckConsumers feature](https://github.com/apache/pulsar/blob/35e9897742b7db4bd29349940075a819b2ad6999/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentStickyKeyDispatcherMultipleConsumers.java#L369-L375) was introduced from https://github.com/apache/pulsar/pull/7553 . However, it can't fix the issue entirely because it doesn't consider the range changes. After this commit, https://github.com/apache/pulsar/pull/10762 was introduced. It fixes the issue. ### Modifications 1. Store last sent position instead of read position to recently joined consumers. Updating read position, then dispatching messages, and adding new consumer are not exclusive. I used the last send position to get a position without any new exclusion features. 2. Keep the last sent position per key. When I introduced the last sent position, I noticed a new concern. Consider the following flow. 1. Assume that the [entries](https://github.com/apache/pulsar/blob/35e9897742b7db4bd29349940075a819b2ad6999/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentStickyKeyDispatcherMultipleConsumers.java#L174) has the following messages, - `msg-1`, key: `key-a`, position: `1:1` - `msg-2`, key: `key-a`, position: `1:2` - `msg-3`, key: `key-a`, position: `1:3` - `msg-4`, key: `key-b`, position: `1:4` - `msg-5`, key: `key-b`, position: `1:5` - `msg-6`, key: `key-b`, position: `1:6` the dispatcher has two consumers (`c1` `messagesForC` is 1, `c2` `messageForC` is 1000), and the selector will return `c1` if `key-a` and `c2` if `key-b`. 2. Send `msg-1` to `c1` and `msg-4` - `msg-6` to `c2`. - So, the current last sent position is `1:6`. - `c1` never acknowledge `msg-1`. 3. Add new consumer `c3`, and the selector will return `c3` if `key-a`. 4. Send `msg-2` - `msg-3` to `c3` because `1:2` and `1:3` are less than the last sent position, `1:6`. 5. Disconnect `c1`. 6. Send `msg-1` to `c3`. Now, `c3` receives messages without expected order about `key-a`. To avoid this issue, I introduce the last sent positions feature. 3. Remove redundant features due to the addition of the last sent positions feature. 1. Remove this condition because the last sent positions feature restricts part of messages. New consumers can receive any new messages with the message key that is not in the last sent position. https://github.com/apache/pulsar/blob/35e9897742b7db4bd29349940075a819b2ad6999/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentStickyKeyDispatcherMultipleConsumers.java#L323-L336 2. Remove this
[GitHub] [pulsar] michaeljmarshall commented on a diff in pull request #20030: [fix][broker] Fix the reason label of authentication metrics
michaeljmarshall commented on code in PR #20030:
URL: https://github.com/apache/pulsar/pull/20030#discussion_r1176006164
##
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/authentication/AuthenticationProviderToken.java:
##
@@ -158,19 +164,18 @@ public String getAuthMethodName() {
@Override
public String authenticate(AuthenticationDataSource authData) throws
AuthenticationException {
+String token;
try {
// Get Token
-String token;
token = getToken(authData);
Review Comment:
Is it possible to increment the `INVALID_AUTH_DATA` error in the `getToken`
method?
##
pulsar-broker-auth-oidc/src/main/java/org/apache/pulsar/broker/authentication/oidc/AuthenticationProviderOpenID.java:
##
@@ -89,7 +89,7 @@ public class AuthenticationProviderOpenID implements
AuthenticationProvider {
private static final String SIMPLE_NAME =
AuthenticationProviderOpenID.class.getSimpleName();
// Must match the value used by the OAuth2 Client Plugin.
-private static final String AUTH_METHOD_NAME = "token";
+private static final String AUTH_METHOD_NAME = "openid";
Review Comment:
This needs to be `token` in order to support the
`AuthenticationProviderList`. Note that the oauth2 client plugin uses `token`
as well. Also, it is valid to use the token authentication data provider in
cases where the oauth2 token is fetched for you, like the k8s service account
token.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar-site] Anonymitaet commented on a diff in pull request #537: Add producer access mode examples for Java and C++ client.
Anonymitaet commented on code in PR #537:
URL: https://github.com/apache/pulsar-site/pull/537#discussion_r1176005098
##
docs/client-libraries-producers.md:
##
@@ -798,6 +798,8 @@ This example shows how to set producer access mode.
values={[{"label":"Java","value":"Java"},{"label":"C++","value":"C++"}]}>
+ This feature is supported after Java client version 2.8.0.
Review Comment:
```suggestion
::: note
This feature is supported in Java client 2.8.0 or later versions.
:::
```
##
docs/client-libraries-producers.md:
##
@@ -809,6 +811,8 @@ This example shows how to set producer access mode.
+ This feature is supported after CPP client version 3.1.0.
Review Comment:
```suggestion
::: note
This feature is supported in C++ client 3.1.0 or later versions.
:::
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] tisonkun commented on pull request #20177: [improve][build] Upgrade jackson version to 2.15.0
tisonkun commented on PR #20177: URL: https://github.com/apache/pulsar/pull/20177#issuecomment-1521146839 After an offline discussion with @mattisonchao I notice that `jackson-dataformat-yaml` 2.14.2 still depend on snakeyaml 1.33 so this patch does fix a CVE. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-manager] saurabh21316 commented on pull request #478: Use correct path for initdb and pg_ctl
saurabh21316 commented on PR #478:
URL: https://github.com/apache/pulsar-manager/pull/478#issuecomment-1521146231
Hello Martin,
I am new to pulsar, since the v0.4.0 is not released yet, could you please
help with deploying it local with this fix or something you can point me out
to. I would really appreciate.
Here is an example of deployment
kind: Deployment
apiVersion: apps/v1
metadata:
name: pulsar-pulsar-manager
namespace: colony-events
labels:
app: pulsar
cluster: pulsar
component: pulsar-manager
release: pulsar
spec:
replicas: 1
selector:
matchLabels:
app: pulsar
component: pulsar-manager
release: pulsar
template:
metadata:
labels:
app: pulsar
cluster: pulsar
component: pulsar-manager
release: pulsar
spec:
volumes:
- name: pulsar-manager-data
emptyDir: {}
containers:
- name: pulsar-pulsar-manager
image: streamnative/pulsar-manager:v0.3.1-rc2
# image: apachepulsar/pulsar-manager:v0.3.0
ports:
- containerPort: 9527
protocol: TCP
- containerPort: 7750
protocol: TCP
envFrom:
- configMapRef:
name: pulsar-pulsar-manager
env:
- name: PULSAR_CLUSTER
value: pulsar
- name: USERNAME
valueFrom:
secretKeyRef:
name: pulsar-pulsar-manager-secret
key: PULSAR_MANAGER_ADMIN_USER
- name: PASSWORD
valueFrom:
secretKeyRef:
name: pulsar-pulsar-manager-secret
key: PULSAR_MANAGER_ADMIN_PASSWORD
- name: PULSAR_MANAGER_OPTS
value: $(PULSAR_MANAGER_OPTS) -Dlog4j2.formatMsgNoLookups=true
- name: SPRING_CONFIGURATION_FILE
value: "/pulsar-manager/pulsar-manager/application.properties"
resources:
limits:
cpu: 100m
memory: 250Mi
requests:
cpu: 100m
memory: 250Mi
volumeMounts:
- name: pulsar-manager-data
mountPath: /data
imagePullPolicy: IfNotPresent
restartPolicy: Always
terminationGracePeriodSeconds: 30
nodeSelector:
persistent-type: persistent
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] Technoboy- commented on pull request #20177: Upgrade jackson version to 2.15.0 to fix CVE-2022-1471
Technoboy- commented on PR #20177: URL: https://github.com/apache/pulsar/pull/20177#issuecomment-1521133032 > I wonder if #20085 already fixes [CVE-2022-1471](https://github.com/advisories/GHSA-mjmj-j48q-9wg2)? > > cc @Technoboy- Yes, fixed, but it's better also to accept this. because `Jackson-2.15.0` is released yesterday. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] shibd opened a new issue, #20178: [Doc] Add producer access mode examples on 2.8.x ~ 2.11.x version.
shibd opened a new issue, #20178: URL: https://github.com/apache/pulsar/issues/20178 ### Search before asking - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar. ### What issue do you find in Pulsar docs? In 2.8.x~2.11.x, There are no examples of producer access modes. ### What is your suggestion? Suggestion add place to: - Java https://user-images.githubusercontent.com/50226895/234155368-bcde125f-c768-42b1-9948-d7fdd26d2449.png;> - C++ https://user-images.githubusercontent.com/50226895/234155565-241ebc71-cf73-40af-a633-652645e02843.png;> ### Any reference? About examples, please refer this PR: https://github.com/apache/pulsar-site/pull/537 ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] mattisonchao closed pull request #20177: Upgrade jackson version to 2.15.0 to fix CVE-2022-1471
mattisonchao closed pull request #20177: Upgrade jackson version to 2.15.0 to fix CVE-2022-1471 URL: https://github.com/apache/pulsar/pull/20177 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] mattisonchao opened a new pull request, #20177: Upgrade jackson version to 2.15.0 to fix CVE-2022-1471
mattisonchao opened a new pull request, #20177: URL: https://github.com/apache/pulsar/pull/20177 ### Motivation Fix CVE https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-1471 ### Modifications - Upgrade jackson version to 2.15.0 ### Verifying this change - [x] Make sure that the change passes the CI checks. *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Does this pull request potentially affect one of the following parts: *If the box was checked, please highlight the changes* - [ ] Dependencies (add or upgrade a dependency) - [ ] The public API - [ ] The schema - [ ] The default values of configurations - [ ] The threading model - [ ] The binary protocol - [ ] The REST endpoints - [ ] The admin CLI options - [ ] The metrics - [ ] Anything that affects deployment ### Documentation - [ ] `doc` - [ ] `doc-required` - [x] `doc-not-needed` - [ ] `doc-complete` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-site] coderzc commented on a diff in pull request #534: Release 2.9.5
coderzc commented on code in PR #534: URL: https://github.com/apache/pulsar-site/pull/534#discussion_r1175973086 ## release-notes/versioned/pulsar-2.9.5.md: ## @@ -0,0 +1,123 @@ +--- +id: pulsar-2.9.5 +title: Apache Pulsar2.9.5 +sidebar_label: Apache Pulsar 2.9.5 +--- + + 2023-4-23 + +### Broker + +* [improve][broker] Omit making a copy of CommandAck when there are no broker interceptors [#18997](https://github.com/apache/pulsar/pull/18997) +* [fix][broker] Copy proto command fields into final variables in ServerCnx [#18987](https://github.com/apache/pulsar/pull/18987) +* [improve][broker] Add logs for why namespace bundle been split (#19003) +* [fix][broker] Fix deadlock in PendingAckHandleImpl (#18989) +* [fix][broker]Update interceptor handler exception (#18940) +* [improve][broker] Make Consumer#equals more effective (#18662) +* [fix][broker]add test case for deleting namespace fail when has partitioned system topic (#17338) +* [fix][broker] Fix open cursor with null-initialPosition result with earliest position (#18416) +* [improve][broker] Add ref count for sticky hash to optimize the performance of Key_Shared subscription (#19167) +* [fix][broker] Support deleting partitioned topics with the keyword -partition- (#19230) +* [improve][broker] Follow up #19230 to tighten the validation scope (#19234) +* [branch-2.9] Fix compile error caused by cherry-pick #19167. (#19241) +* [improve][websocket][branch-2.9] Add ping support (#19245) +* [fix][ml] Topics stats shows msgBacklog but there reality no backlog (#19275) +* [fix][websocket][branch-2.9] Fix webSocketPingDurationSeconds config (#19293) +* [fix] [ml] Fix the incorrect total size if use ML intercepto (#19404) +* [fix][broker] Expect msgs after server initiated CloseProducer (#19446) +* [cleanup][broker] Validate originalPrincipal earlier in ServerCnx (#19270) +* [fix][Branch-2.9] Fix bookie JVM param BOOKIE_MEM, BOOKIE_GC miss problem. #19441 +* [improve][broker] ServerCnx: go to Failed state when auth fails (#19312) +* [fix][broker][branch-2.9] Fix geo-replication admin (#19614) +* [fix] [ml] messagesConsumedCounter of NonDurableCursor was initialized incorrectly (#19355) +* [fix][ml] Fix potential NPE cause future never complete. (#19415) +* [fix] [ml] topic load fail by ledger lost (#19444) +* [fix] [broker] getLastMessageId returns a wrong batch index of last message if enabled read compacted (#18877) +* [fix][broker] Remove timestamp from broker metrics (#17419) +* [fix][broker]fix multi invocation for ledger createComplete (#18975) Review Comment: @Anonymitaet I check all occurrences and fixed them, please take a look again. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar-site] branch release-2.9.5 updated (b1eb4f87555 -> 934962471e0)
This is an automated email from the ASF dual-hosted git repository. zhaocong pushed a change to branch release-2.9.5 in repository https://gitbox.apache.org/repos/asf/pulsar-site.git from b1eb4f87555 Update release-notes/versioned/pulsar-2.9.5.md add 934962471e0 Keep consistent on the first letter of a sentence No new revisions were added by this update. Summary of changes: release-notes/versioned/client-java-2.9.5.md | 2 +- release-notes/versioned/pulsar-2.9.5.md | 14 +++--- 2 files changed, 8 insertions(+), 8 deletions(-)
[GitHub] [pulsar-site] coderzc commented on a diff in pull request #534: Release 2.9.5
coderzc commented on code in PR #534: URL: https://github.com/apache/pulsar-site/pull/534#discussion_r1175971794 ## release-notes/versioned/pulsar-2.9.5.md: ## @@ -0,0 +1,123 @@ +--- +id: pulsar-2.9.5 +title: Apache Pulsar2.9.5 +sidebar_label: Apache Pulsar 2.9.5 +--- + + 2023-4-23 + +### Broker + +* [improve][broker] Omit making a copy of CommandAck when there are no broker interceptors [#18997](https://github.com/apache/pulsar/pull/18997) +* [fix][broker] Copy proto command fields into final variables in ServerCnx [#18987](https://github.com/apache/pulsar/pull/18987) +* [improve][broker] Add logs for why namespace bundle been split (#19003) +* [fix][broker] Fix deadlock in PendingAckHandleImpl (#18989) +* [fix][broker]Update interceptor handler exception (#18940) +* [improve][broker] Make Consumer#equals more effective (#18662) +* [fix][broker]add test case for deleting namespace fail when has partitioned system topic (#17338) +* [fix][broker] Fix open cursor with null-initialPosition result with earliest position (#18416) +* [improve][broker] Add ref count for sticky hash to optimize the performance of Key_Shared subscription (#19167) +* [fix][broker] Support deleting partitioned topics with the keyword -partition- (#19230) +* [improve][broker] Follow up #19230 to tighten the validation scope (#19234) +* [branch-2.9] Fix compile error caused by cherry-pick #19167. (#19241) +* [improve][websocket][branch-2.9] Add ping support (#19245) +* [fix][ml] Topics stats shows msgBacklog but there reality no backlog (#19275) +* [fix][websocket][branch-2.9] Fix webSocketPingDurationSeconds config (#19293) +* [fix] [ml] Fix the incorrect total size if use ML intercepto (#19404) +* [fix][broker] Expect msgs after server initiated CloseProducer (#19446) +* [cleanup][broker] Validate originalPrincipal earlier in ServerCnx (#19270) +* [fix][Branch-2.9] Fix bookie JVM param BOOKIE_MEM, BOOKIE_GC miss problem. #19441 +* [improve][broker] ServerCnx: go to Failed state when auth fails (#19312) +* [fix][broker][branch-2.9] Fix geo-replication admin (#19614) +* [fix] [ml] messagesConsumedCounter of NonDurableCursor was initialized incorrectly (#19355) +* [fix][ml] Fix potential NPE cause future never complete. (#19415) +* [fix] [ml] topic load fail by ledger lost (#19444) +* [fix] [broker] getLastMessageId returns a wrong batch index of last message if enabled read compacted (#18877) +* [fix][broker] Remove timestamp from broker metrics (#17419) +* [fix][broker]fix multi invocation for ledger createComplete (#18975) Review Comment: Ok -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] coderzc opened a new pull request, #20176: [fix][broker] Fix RoaringBitmap contains can't check value 65535
coderzc opened a new pull request, #20176: URL: https://github.com/apache/pulsar/pull/20176 Master Issue: #xyz https://github.com/RoaringBitmap/RoaringBitmap/issues/623 PIP: #xyz ### Motivation Using `contains(final int x)` instead of `contains(long minimum, long supremum)`. ### Modifications ### Verifying this change - [x] Make sure that the change passes the CI checks. *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Does this pull request potentially affect one of the following parts: *If the box was checked, please highlight the changes* - [ ] Dependencies (add or upgrade a dependency) - [ ] The public API - [ ] The schema - [ ] The default values of configurations - [ ] The threading model - [ ] The binary protocol - [ ] The REST endpoints - [ ] The admin CLI options - [ ] The metrics - [ ] Anything that affects deployment ### Documentation - [ ] `doc` - [ ] `doc-required` - [x] `doc-not-needed` - [ ] `doc-complete` ### Matching PR in forked repository PR in forked repository: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-go] BewareMyPower commented on issue #1010: Seeing ten second p99 pulsar_client_consumer_processing_time_seconds_bucket values
BewareMyPower commented on issue #1010: URL: https://github.com/apache/pulsar-client-go/issues/1010#issuecomment-1521084731 It seems that we don't have any document for these prometheus metrics yet. It would be good to add some docs. And since you already found the reason of the high processing time, I think we can close this issue for now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] 3286360470 added a comment to the discussion: Pulsar-client-shaded[pulsar-client]模块生成的源码包是空的
GitHub user 3286360470 added a comment to the discussion: Pulsar-client-shaded[pulsar-client]模块生成的源码包是空的 @tisonkun thanks for your reply, your answer perfectly solved my problem! GitHub link: https://github.com/apache/pulsar/discussions/20175#discussioncomment-5714381 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar-site] Anonymitaet commented on pull request #537: Add producer access mode examples for Java and C++ client.
Anonymitaet commented on PR #537: URL: https://github.com/apache/pulsar-site/pull/537#issuecomment-1521046996 For versioned docs, how about adding the docs as below? The location might not be perfect but it's the most relevant place. Java https://user-images.githubusercontent.com/50226895/234155368-bcde125f-c768-42b1-9948-d7fdd26d2449.png;> C++ https://user-images.githubusercontent.com/50226895/234155565-241ebc71-cf73-40af-a633-652645e02843.png;> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on pull request #19464: [fix][authentication] Improve AuthenticationFilter response
github-actions[bot] commented on PR #19464: URL: https://github.com/apache/pulsar/pull/19464#issuecomment-1521045073 The pr had no activity for 30 days, mark with Stale label. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-site] shibd commented on pull request #537: Add producer access mode examples for Java and C++ client.
shibd commented on PR #537: URL: https://github.com/apache/pulsar-site/pull/537#issuecomment-1521041383 > Soft reminder: do not forget the 2nd task commented in. Hi, @Anonymitaet The structure of the current document(next) is very different from the previous document(2.11.x, 2.10.x, etc). I'm not sure where I should add it. And for the CPP client, We don't seem to manage the version of its documentation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] massakam commented on pull request #19983: [improve][broker] Log resource usage rate of brokers that need to be offloaded in ThresholdShedder
massakam commented on PR #19983: URL: https://github.com/apache/pulsar/pull/19983#issuecomment-1521026210 @eolivelli Could you please re-review? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-site] Anonymitaet commented on pull request #536: Update io-debezium-source.md
Anonymitaet commented on PR #536: URL: https://github.com/apache/pulsar-site/pull/536#issuecomment-1520948182 @liangyepianzhou Could you please review this PR from a technical perspective? TIA! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-site] Anonymitaet commented on a diff in pull request #537: Add producer access mode examples for Java and C++ client.
Anonymitaet commented on code in PR #537: URL: https://github.com/apache/pulsar-site/pull/537#discussion_r1175866970 ## docs/client-libraries-producers.md: ## @@ -784,4 +784,41 @@ This example shows how to set the `EnforceUnencrypted` encryption policy. - \ No newline at end of file + + +## Configure Access mode + +Access mode allow applications to require exclusive producer access on a topic in order to achieve a "single-writer" situation. + +More information about access mode can be found [concepts-clients](concepts-clients.md#access-mode). Review Comment: ```suggestion ``` ## docs/client-libraries-producers.md: ## @@ -784,4 +784,41 @@ This example shows how to set the `EnforceUnencrypted` encryption policy. - \ No newline at end of file + + +## Configure Access mode + +Access mode allow applications to require exclusive producer access on a topic in order to achieve a "single-writer" situation. Review Comment: ```suggestion [Access mode](concepts-clients.md#access-mode) allows applications to require exclusive producer access on a topic to achieve a "single-writer" situation. ``` ## docs/client-libraries-producers.md: ## @@ -784,4 +784,41 @@ This example shows how to set the `EnforceUnencrypted` encryption policy. - \ No newline at end of file + + +## Configure Access mode Review Comment: ```suggestion ## Configure access mode ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-go] frankjkelly commented on issue #1010: Seeing ten second p99 pulsar_client_consumer_processing_time_seconds_bucket values
frankjkelly commented on issue #1010: URL: https://github.com/apache/pulsar-client-go/issues/1010#issuecomment-1520649470 Ah ok - thanks @BewareMyPower From our side we've identified a cause of our high consumer processing time (somehow related to the choice of EC2 node in our EKS cluster strangely). Do you think we should repurpose this issue to improve the documentation of the metric or should we close this and create a new issue? Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] rdhabalia commented on pull request #20147: [feat] [broker] Add broker health check status into prometheus metrics
rdhabalia commented on PR #20147: URL: https://github.com/apache/pulsar/pull/20147#issuecomment-1520599376 @mattisonchao I don't think and I also don't see draft requirements for configuration-related changes at : https://github.com/apache/pulsar/blob/master/wiki/proposals/PIP.md also, this PR seems added an optional config to enable certain metrics which doesn't have any impact on current behavior as well. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-site] tisonkun merged pull request #538: Fix wrong examples of CPP advertised listeners.
tisonkun merged PR #538: URL: https://github.com/apache/pulsar-site/pull/538 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar-site] branch main updated: [fix][doc] CPP client advertised listeners examples (#538)
This is an automated email from the ASF dual-hosted git repository.
tison pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/pulsar-site.git
The following commit(s) were added to refs/heads/main by this push:
new e819fc14fb6 [fix][doc] CPP client advertised listeners examples (#538)
e819fc14fb6 is described below
commit e819fc14fb6c0acd4327da6d557138b1e88a37b9
Author: Baodi Shi
AuthorDate: Mon Apr 24 22:40:12 2023 +0800
[fix][doc] CPP client advertised listeners examples (#538)
---
docs/client-libraries-clients.md | 7 +++
1 file changed, 3 insertions(+), 4 deletions(-)
diff --git a/docs/client-libraries-clients.md b/docs/client-libraries-clients.md
index cb75b6ed58e..c898b13ff42 100644
--- a/docs/client-libraries-clients.md
+++ b/docs/client-libraries-clients.md
@@ -35,10 +35,9 @@ The following example creates a Python client using multiple
advertised listener
```cpp
- PulsarClient client = PulsarClient.builder()
-.serviceUrl("pulsar://:6650")
-.listenerName("external")
-.build();
+ ClientConfiguration clientConfiguration;
+ clientConfiguration.setListenerName("external");
+ Client client("pulsar://:6650", clientConfiguration);
```
[GitHub] [pulsar-site] tisonkun commented on pull request #538: Fix wrong examples of CPP advertised listeners.
tisonkun commented on PR #538: URL: https://github.com/apache/pulsar-site/pull/538#issuecomment-1520295646 Thanks for your confirmation! Merging... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-site] shibd commented on pull request #538: Fix wrong examples of CPP advertised listeners.
shibd commented on PR #538: URL: https://github.com/apache/pulsar-site/pull/538#issuecomment-1520263395 > Could you check if other maintained versions (2.10 & 2.11) are suffering similar issue? This content was added by #515, and the other version does not have this content. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] asafm commented on issue #19912: PIP-261: Restructure Getting Started section
asafm commented on issue #19912: URL: https://github.com/apache/pulsar/issues/19912#issuecomment-1520241695 I've updated the PIP according to all comments. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-site] shibd opened a new pull request, #538: Fix wrong examples of CPP advertised listeners.
shibd opened a new pull request, #538: URL: https://github.com/apache/pulsar-site/pull/538 ### Preview https://user-images.githubusercontent.com/33416836/234022727-c9ad8303-fba1-4d0e-bbda-97ae2fbc906f.png;> ### Documentation - [x] `doc` - [ ] `doc-required` - [ ] `doc-not-needed` - [ ] `doc-complete` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-site] shibd opened a new pull request, #537: Add producer access mode examples for Java and C++ client.
shibd opened a new pull request, #537: URL: https://github.com/apache/pulsar-site/pull/537 ### Preview https://user-images.githubusercontent.com/33416836/234018290-914234da-9b47-44da-8276-04678af7bb8a.png;> https://user-images.githubusercontent.com/33416836/234018308-daa53f35-714f-4caf-bc6e-21205f7a1a5e.png;> ### Documentation - [x] `doc` - [ ] `doc-required` - [ ] `doc-not-needed` - [ ] `doc-complete` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-site] Anonymitaet commented on a diff in pull request #534: Release 2.9.5
Anonymitaet commented on code in PR #534: URL: https://github.com/apache/pulsar-site/pull/534#discussion_r1175301328 ## release-notes/versioned/pulsar-2.9.5.md: ## @@ -0,0 +1,123 @@ +--- +id: pulsar-2.9.5 +title: Apache Pulsar2.9.5 +sidebar_label: Apache Pulsar 2.9.5 +--- + + 2023-4-23 + +### Broker + +* [improve][broker] Omit making a copy of CommandAck when there are no broker interceptors [#18997](https://github.com/apache/pulsar/pull/18997) +* [fix][broker] Copy proto command fields into final variables in ServerCnx [#18987](https://github.com/apache/pulsar/pull/18987) +* [improve][broker] Add logs for why namespace bundle been split (#19003) +* [fix][broker] Fix deadlock in PendingAckHandleImpl (#18989) +* [fix][broker]Update interceptor handler exception (#18940) +* [improve][broker] Make Consumer#equals more effective (#18662) +* [fix][broker]add test case for deleting namespace fail when has partitioned system topic (#17338) +* [fix][broker] Fix open cursor with null-initialPosition result with earliest position (#18416) +* [improve][broker] Add ref count for sticky hash to optimize the performance of Key_Shared subscription (#19167) +* [fix][broker] Support deleting partitioned topics with the keyword -partition- (#19230) +* [improve][broker] Follow up #19230 to tighten the validation scope (#19234) +* [branch-2.9] Fix compile error caused by cherry-pick #19167. (#19241) +* [improve][websocket][branch-2.9] Add ping support (#19245) +* [fix][ml] Topics stats shows msgBacklog but there reality no backlog (#19275) +* [fix][websocket][branch-2.9] Fix webSocketPingDurationSeconds config (#19293) +* [fix] [ml] Fix the incorrect total size if use ML intercepto (#19404) +* [fix][broker] Expect msgs after server initiated CloseProducer (#19446) +* [cleanup][broker] Validate originalPrincipal earlier in ServerCnx (#19270) +* [fix][Branch-2.9] Fix bookie JVM param BOOKIE_MEM, BOOKIE_GC miss problem. #19441 +* [improve][broker] ServerCnx: go to Failed state when auth fails (#19312) +* [fix][broker][branch-2.9] Fix geo-replication admin (#19614) +* [fix] [ml] messagesConsumedCounter of NonDurableCursor was initialized incorrectly (#19355) +* [fix][ml] Fix potential NPE cause future never complete. (#19415) +* [fix] [ml] topic load fail by ledger lost (#19444) +* [fix] [broker] getLastMessageId returns a wrong batch index of last message if enabled read compacted (#18877) +* [fix][broker] Remove timestamp from broker metrics (#17419) +* [fix][broker]fix multi invocation for ledger createComplete (#18975) Review Comment: This comment still needs to be resolved. Please keep consistent on the first letter of a sentence and check all occurrences, then ask for a review. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-cpp] Anonymitaet commented on pull request #232: Support waiting for the ACK response
Anonymitaet commented on PR #232: URL: https://github.com/apache/pulsar-client-cpp/pull/232#issuecomment-1520171751 Hi @RobertIndie, Just now, I discussed with @BewareMyPower on how we add docs for this PR and summarized the thoughts below. He suggests that you're an expert in this area and can provide tech inputs for this PR. Are you available? Thanks! Here is the doc plan: ## ✅ Pulsar feature matrix I've created ["Wait for response" in Consumer > Ack](https://docs.google.com/spreadsheets/d/1YHYTkIXR8-Ql103u-IMI18TXLlGStK8uJjDsOOA0T20/edit#gid=1784579914=91:91) and added versions for Pulsar and C++. ## ✅ Pulsar client docs 1. Create a topic "Wait for response" in [Client > Work with consumer](https://pulsar.apache.org/docs/next/client-libraries-consumers/#acknowledge-messages), and add a simple feature description (this info can be fetched from [here](https://github.com/apache/pulsar-client-cpp/pull/232/files#diff-258280272b1786eac0ecaafc163e189b87faa2a63ab740225da0b459f528358aR633-R639) and code examples (Java and C++). https://user-images.githubusercontent.com/50226895/234009958-2ae373d8-0f48-4d0c-8ff0-13f7fa905ef0.png;> 2. This doc should be added to 5 doc versions (NEXT, 2.11.x, 2.10.x, 2.9.x, 2.8.x) since this feature is available from Pulsar 2.8.x. - Original doc file of NEXT is [here](https://github.com/apache/pulsar-site/edit/main/docs/client-libraries-tableviews.md). - Original doc files of all versioned docs are [here](https://github.com/apache/pulsar-site/tree/main/versioned_docs). We can update one version first and apply the same doc changes to the rest once it gets approved. Feel free to reach out to me if you have questions. TIA! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-dotpulsar] toneill818 commented on pull request #146: Partitioned Topic Consumer Support
toneill818 commented on PR #146: URL: https://github.com/apache/pulsar-dotpulsar/pull/146#issuecomment-1520060415 @blankensteiner Go ahead and create a new release. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] asafm commented on issue #19611: PIP-249: Pulsar website redesign
asafm commented on issue #19611: URL: https://github.com/apache/pulsar/issues/19611#issuecomment-1520014420 This is issue is still being worked on -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] tisonkun edited a comment on the discussion: Pulsar-client-shaded[pulsar-client]模块生成的源码包是空的
GitHub user tisonkun edited a comment on the discussion: Pulsar-client-shaded[pulsar-client]模块生成的源码包是空的 This is a known issue https://github.com/apache/pulsar/issues/19918 and has been fixed https://github.com/apache/pulsar/pull/19956 on the master branch. IIRC 3.0 will include this fix. GitHub link: https://github.com/apache/pulsar/discussions/20175#discussioncomment-5707164 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] tisonkun edited a comment on the discussion: Pulsar-client-shaded[pulsar-client]模块生成的源码包是空的
GitHub user tisonkun edited a comment on the discussion: Pulsar-client-shaded[pulsar-client]模块生成的源码包是空的 This is a known issue https://github.com/apache/pulsar/issues/19918 and has been fixed on the master branch. IIRC 3.0 will include this fix. GitHub link: https://github.com/apache/pulsar/discussions/20175#discussioncomment-5707164 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] tisonkun edited a comment on the discussion: Pulsar-client-shaded[pulsar-client]模块生成的源码包是空的
GitHub user tisonkun edited a comment on the discussion: Pulsar-client-shaded[pulsar-client]模块生成的源码包是空的 This is a known issue https://github.com/apache/pulsar/issues/20034 and has been fixed on the master branch. IIRC 3.0 will include this fix. GitHub link: https://github.com/apache/pulsar/discussions/20175#discussioncomment-5707164 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] tisonkun added a comment to the discussion: Pulsar-client-shaded[pulsar-client]模块生成的源码包是空的
GitHub user tisonkun added a comment to the discussion: Pulsar-client-shaded[pulsar-client]模块生成的源码包是空的 This is a known issue and has been fixed on the master branch. IIRC 3.0 will include this fix. GitHub link: https://github.com/apache/pulsar/discussions/20175#discussioncomment-5707164 This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] AnonHxy commented on pull request #20152: [fix][admin] Fix examine messages if total message is zero
AnonHxy commented on PR #20152: URL: https://github.com/apache/pulsar/pull/20152#issuecomment-1519909875 > @AnonHxy hi, after check the intention of this admin interface. i think we can return a meaningful response directly without return exception to the caller. like if there is no message here, any check should return equal to latest or earliest, WDYT? Do you mean that we return an empty response, like `Response.ok().build()`? Maybe this will confuse the users I think. @lifepuzzlefun -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-site] github-actions[bot] commented on pull request #536: Update io-debezium-source.md
github-actions[bot] commented on PR #536: URL: https://github.com/apache/pulsar-site/pull/536#issuecomment-1519856278 @raymondBourges Please add the following content to your PR description and select a checkbox: ``` - [ ] `doc` - [ ] `doc-required` - [ ] `doc-not-needed` - [ ] `doc-complete` ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-site] raymondBourges opened a new pull request, #536: Update io-debezium-source.md
raymondBourges opened a new pull request, #536: URL: https://github.com/apache/pulsar-site/pull/536 Without "topicName" in YAML definition file "bin/pulsar-admin source localrun --source-config-file debezium-postgres-source-config.yaml" exits with a "Topic name cannot be null" error. Note : I just can give topic name online (without using a YAML file) within --sourceConfig parameter of bin/function-localrunner command but this command is not present in this documentation. Here, just bin/pulsar-admin is documented but it doesn't accept topic name as parameter. ### Documentation - [ ] `doc` - [ ] `doc-required` - [ ] `doc-not-needed` - [ ] `doc-complete` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch branch-3.0 updated: Revert "[fix][broker] Fix entry filter feature for the non-persistent topic (#20141)"
This is an automated email from the ASF dual-hosted git repository.
cbornet pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new e8261fcef8d Revert "[fix][broker] Fix entry filter feature for the
non-persistent topic (#20141)"
e8261fcef8d is described below
commit e8261fcef8d76061d378d8a9f3b05a5d115013fe
Author: Christophe Bornet
AuthorDate: Mon Apr 24 12:10:50 2023 +0200
Revert "[fix][broker] Fix entry filter feature for the non-persistent topic
(#20141)"
This reverts commit e27abe9e128fb71b65ffe06417574c9a7f3facbd.
---
.../service/nonpersistent/NonPersistentTopic.java | 5 ++-
.../broker/service/plugin/FilterEntryTest.java | 15 +++-
.../pulsar/broker/stats/SubscriptionStatsTest.java | 42 +++---
3 files changed, 26 insertions(+), 36 deletions(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentTopic.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentTopic.java
index 33258b06726..317b8df6b9a 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentTopic.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentTopic.java
@@ -26,7 +26,7 @@ import com.carrotsearch.hppc.ObjectObjectHashMap;
import io.netty.buffer.ByteBuf;
import io.netty.util.concurrent.FastThreadLocal;
import java.util.ArrayList;
-import java.util.Arrays;
+import java.util.Collections;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
@@ -199,8 +199,7 @@ public class NonPersistentTopic extends AbstractTopic
implements Topic, TopicPol
// entry internally retains data so, duplicateBuffer should be
release here
duplicateBuffer.release();
if (subscription.getDispatcher() != null) {
-// Dispatcher needs to call the set method to support entry
filter feature.
-
subscription.getDispatcher().sendMessages(Arrays.asList(entry));
+
subscription.getDispatcher().sendMessages(Collections.singletonList(entry));
} else {
// it happens when subscription is created but dispatcher is
not created as consumer is not added
// yet
diff --git
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/plugin/FilterEntryTest.java
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/plugin/FilterEntryTest.java
index b868858646c..4b9d91fbde2 100644
---
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/plugin/FilterEntryTest.java
+++
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/plugin/FilterEntryTest.java
@@ -51,7 +51,6 @@ import org.apache.pulsar.broker.service.AbstractTopic;
import org.apache.pulsar.broker.service.BrokerTestBase;
import org.apache.pulsar.broker.service.Dispatcher;
import org.apache.pulsar.broker.service.EntryFilterSupport;
-import org.apache.pulsar.broker.service.Subscription;
import org.apache.pulsar.broker.service.persistent.PersistentSubscription;
import org.apache.pulsar.broker.service.persistent.PersistentTopic;
import org.apache.pulsar.broker.testcontext.PulsarTestContext;
@@ -287,16 +286,10 @@ public class FilterEntryTest extends BrokerTestBase {
}
-@DataProvider(name = "topicProvider")
-public Object[][] topicProvider() {
-return new Object[][]{
-{"persistent://prop/ns-abc/topic" + UUID.randomUUID()},
-{"non-persistent://prop/ns-abc/topic" + UUID.randomUUID()},
-};
-}
-@Test(dataProvider = "topicProvider")
-public void testFilteredMsgCount(String topic) throws Throwable {
+@Test
+public void testFilteredMsgCount() throws Throwable {
+String topic = "persistent://prop/ns-abc/topic" + UUID.randomUUID();
String subName = "sub";
try (Producer producer =
pulsarClient.newProducer(Schema.STRING)
@@ -305,7 +298,7 @@ public class FilterEntryTest extends BrokerTestBase {
.subscriptionName(subName).subscribe()) {
// mock entry filters
-Subscription subscription = pulsar.getBrokerService()
+PersistentSubscription subscription = (PersistentSubscription)
pulsar.getBrokerService()
.getTopicReference(topic).get().getSubscription(subName);
Dispatcher dispatcher = subscription.getDispatcher();
Field field =
EntryFilterSupport.class.getDeclaredField("entryFilters");
diff --git
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/SubscriptionStatsTest.java
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/SubscriptionStatsTest.java
index d5e0066a86f..bf9c1d540bf 100644
---
[GitHub] [pulsar] coderzc commented on pull request #20086: [fix][broker] Ensure previous delayed index be removed from snapshotSegmentLastIndexTable & Make load operate asynchronous
coderzc commented on PR #20086: URL: https://github.com/apache/pulsar/pull/20086#issuecomment-1519841339 > This has been cherry-picked to branch-3.0 despite the code freeze. Do you feel strongly that this needs to be in the 3.0 release ? Otherwise we should revert this and delay it 3.0.1/3.1. This is a severe issue for PIP-195 so it needs into 3.0, thanks～ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] 02/04: Revert "[improve][broker] Optimize delayed metadata index bitmap (#20136)"
This is an automated email from the ASF dual-hosted git repository.
cbornet pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit f3572574b132dfb6d5a67011f7fb68096e69e1c7
Author: Christophe Bornet
AuthorDate: Mon Apr 24 11:48:03 2023 +0200
Revert "[improve][broker] Optimize delayed metadata index bitmap (#20136)"
This reverts commit ff59240165c73a9c3a3dcca20702ab44b0b18d33.
---
.../bucket/BucketDelayedDeliveryTracker.java | 3 --
.../broker/delayed/bucket/ImmutableBucket.java | 53 +-
.../broker/delayed/bucket/MutableBucket.java | 11 ++---
3 files changed, 25 insertions(+), 42 deletions(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/delayed/bucket/BucketDelayedDeliveryTracker.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/delayed/bucket/BucketDelayedDeliveryTracker.java
index b17387e276e..b4d1745e22f 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/delayed/bucket/BucketDelayedDeliveryTracker.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/delayed/bucket/BucketDelayedDeliveryTracker.java
@@ -485,9 +485,6 @@ public class BucketDelayedDeliveryTracker extends
AbstractDelayedDeliveryTracker
});
});
}
-
-// optimize bm
-
delayedIndexBitMap.values().forEach(RoaringBitmap::runOptimize);
immutableBucketDelayedIndexPair.getLeft().setDelayedIndexBitMap(delayedIndexBitMap);
afterCreateImmutableBucket(immutableBucketDelayedIndexPair, createStartTime);
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/delayed/bucket/ImmutableBucket.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/delayed/bucket/ImmutableBucket.java
index 57de5c84fcd..82e98cefa5d 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/delayed/bucket/ImmutableBucket.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/delayed/bucket/ImmutableBucket.java
@@ -20,10 +20,11 @@ package org.apache.pulsar.broker.delayed.bucket;
import static org.apache.bookkeeper.mledger.util.Futures.executeWithRetry;
import static
org.apache.pulsar.broker.delayed.bucket.BucketDelayedDeliveryTracker.NULL_LONG_PROMISE;
-import java.io.IOException;
+import com.google.protobuf.ByteString;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
+import java.util.Map;
import java.util.Optional;
import java.util.concurrent.CompletableFuture;
import java.util.function.Supplier;
@@ -36,8 +37,8 @@ import
org.apache.pulsar.broker.delayed.proto.DelayedMessageIndexBucketSnapshotF
import
org.apache.pulsar.broker.delayed.proto.DelayedMessageIndexBucketSnapshotFormat.DelayedIndex;
import
org.apache.pulsar.broker.delayed.proto.DelayedMessageIndexBucketSnapshotFormat.SnapshotSegmentMetadata;
import org.apache.pulsar.common.util.FutureUtil;
-import org.roaringbitmap.InvalidRoaringFormat;
import org.roaringbitmap.RoaringBitmap;
+import org.roaringbitmap.buffer.ImmutableRoaringBitmap;
@Slf4j
class ImmutableBucket extends Bucket {
@@ -97,7 +98,7 @@ class ImmutableBucket extends Bucket {
this.setLastSegmentEntryId(metadataList.size());
this.recoverDelayedIndexBitMapAndNumber(nextSnapshotEntryIndex, metadataList);
List firstScheduleTimestamps =
metadataList.stream().map(
-
SnapshotSegmentMetadata::getMinScheduleTimestamp).toList();
+
SnapshotSegmentMetadata::getMinScheduleTimestamp).toList();
this.setFirstScheduleTimestamps(firstScheduleTimestamps);
return nextSnapshotEntryIndex + 1;
@@ -138,37 +139,25 @@ class ImmutableBucket extends Bucket {
});
}
-/**
- * Recover delayed index bit map and message numbers.
- * @throws InvalidRoaringFormat invalid bitmap serialization format
- */
private void recoverDelayedIndexBitMapAndNumber(int startSnapshotIndex,
-
List segmentMetaList) {
-delayedIndexBitMap.clear(); // cleanup dirty bm
-final var numberMessages = new MutableLong(0);
-for (int i = startSnapshotIndex; i < segmentMetaList.size(); i++) {
-for (final var entry :
segmentMetaList.get(i).getDelayedIndexBitMapMap().entrySet()) {
-final var ledgerId = entry.getKey();
-final var bs = entry.getValue();
-final var sbm = new RoaringBitmap();
-try {
-sbm.deserialize(bs.asReadOnlyByteBuffer());
-} catch (IOException e) {
-throw new InvalidRoaringFormat(e.getMessage());
+
[pulsar] 01/04: Revert "[improve][broker] Cache LedgerHandle in BookkeeperBucketSnapshotStorage (#20117)"
This is an automated email from the ASF dual-hosted git repository.
cbornet pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 49539c20bea1d8b10b8465be5ee632c1c4bdc7ce
Author: Christophe Bornet
AuthorDate: Mon Apr 24 11:46:27 2023 +0200
Revert "[improve][broker] Cache LedgerHandle in
BookkeeperBucketSnapshotStorage (#20117)"
This reverts commit e1d63990644700bf61b3d7af1ef6d4d62145c2bb.
---
.../bucket/BookkeeperBucketSnapshotStorage.java| 52 +++---
.../BookkeeperBucketSnapshotStorageTest.java | 43 --
2 files changed, 26 insertions(+), 69 deletions(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/delayed/bucket/BookkeeperBucketSnapshotStorage.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/delayed/bucket/BookkeeperBucketSnapshotStorage.java
index 18a4c322f7b..9c30ccf1c0b 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/delayed/bucket/BookkeeperBucketSnapshotStorage.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/delayed/bucket/BookkeeperBucketSnapshotStorage.java
@@ -26,7 +26,6 @@ import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.concurrent.CompletableFuture;
-import java.util.concurrent.ConcurrentHashMap;
import javax.validation.constraints.NotNull;
import lombok.extern.slf4j.Slf4j;
import org.apache.bookkeeper.client.BKException;
@@ -49,8 +48,6 @@ public class BookkeeperBucketSnapshotStorage implements
BucketSnapshotStorage {
private final ServiceConfiguration config;
private BookKeeper bookKeeper;
-private final Map>
ledgerHandleFutureCache = new ConcurrentHashMap<>();
-
public BookkeeperBucketSnapshotStorage(PulsarService pulsar) {
this.pulsar = pulsar;
this.config = pulsar.getConfig();
@@ -69,30 +66,45 @@ public class BookkeeperBucketSnapshotStorage implements
BucketSnapshotStorage {
@Override
public CompletableFuture getBucketSnapshotMetadata(long
bucketId) {
-return getLedgerHandle(bucketId).thenCompose(ledgerHandle ->
getLedgerEntry(ledgerHandle, 0, 0)
-.thenApply(entryEnumeration ->
parseSnapshotMetadataEntry(entryEnumeration.nextElement(;
+return openLedger(bucketId).thenCompose(ledgerHandle -> {
+CompletableFuture snapshotFuture =
+getLedgerEntry(ledgerHandle, 0, 0)
+.thenApply(entryEnumeration ->
parseSnapshotMetadataEntry(entryEnumeration.nextElement()));
+
+snapshotFuture.whenComplete((__, e) -> closeLedger(ledgerHandle));
+
+return snapshotFuture;
+});
}
@Override
public CompletableFuture>
getBucketSnapshotSegment(long bucketId, long firstSegmentEntryId,
long lastSegmentEntryId) {
-return getLedgerHandle(bucketId).thenCompose(
-ledgerHandle -> getLedgerEntry(ledgerHandle,
firstSegmentEntryId, lastSegmentEntryId)
-.thenApply(this::parseSnapshotSegmentEntries));
+return openLedger(bucketId).thenCompose(ledgerHandle -> {
+CompletableFuture> parseFuture =
+getLedgerEntry(ledgerHandle, firstSegmentEntryId,
lastSegmentEntryId)
+.thenApply(this::parseSnapshotSegmentEntries);
+
+parseFuture.whenComplete((__, e) -> closeLedger(ledgerHandle));
+
+return parseFuture;
+});
}
@Override
public CompletableFuture getBucketSnapshotLength(long bucketId) {
-return getLedgerHandle(bucketId).thenCompose(
-ledgerHandle ->
CompletableFuture.completedFuture(ledgerHandle.getLength()));
+return openLedger(bucketId).thenCompose(ledgerHandle -> {
+CompletableFuture lengthFuture =
+
CompletableFuture.completedFuture(ledgerHandle.getLength());
+
+lengthFuture.whenComplete((__, e) -> closeLedger(ledgerHandle));
+
+return lengthFuture;
+});
}
@Override
public CompletableFuture deleteBucketSnapshot(long bucketId) {
-CompletableFuture ledgerHandleFuture =
ledgerHandleFutureCache.remove(bucketId);
-if (ledgerHandleFuture != null) {
-ledgerHandleFuture.whenComplete((lh, ex) -> closeLedger(lh));
-}
return deleteLedger(bucketId);
}
@@ -166,18 +178,6 @@ public class BookkeeperBucketSnapshotStorage implements
BucketSnapshotStorage {
return future;
}
-private CompletableFuture getLedgerHandle(Long ledgerId) {
-CompletableFuture ledgerHandleCompletableFuture =
-ledgerHandleFutureCache.computeIfAbsent(ledgerId, k ->
openLedger(ledgerId));
-// remove future of completed exceptionally
-ledgerHandleCompletableFuture.whenComplete((__, ex) -> {
[pulsar] branch branch-3.0 updated (89e4d46f4a5 -> 254f09b70d8)
This is an automated email from the ASF dual-hosted git repository. cbornet pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/pulsar.git from 89e4d46f4a5 [fix] [broker] Make `LeastResourceUsageWithWeight` thread safe (#20159) new 49539c20bea Revert "[improve][broker] Cache LedgerHandle in BookkeeperBucketSnapshotStorage (#20117)" new f3572574b13 Revert "[improve][broker] Optimize delayed metadata index bitmap (#20136)" new 610c5f15143 Revert "[improve][broker] Make timer execute immediately after load index (#20126)" new 254f09b70d8 Revert "[improve] [broker] Close temporary open ledger in BookkeeperBucketSnapshotStorage (#20111)" The 4 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../bucket/BookkeeperBucketSnapshotStorage.java| 33 +++--- .../bucket/BucketDelayedDeliveryTracker.java | 11 ++--- .../broker/delayed/bucket/ImmutableBucket.java | 53 +- .../broker/delayed/bucket/MutableBucket.java | 11 ++--- .../BookkeeperBucketSnapshotStorageTest.java | 43 -- 5 files changed, 36 insertions(+), 115 deletions(-)
[pulsar] 04/04: Revert "[improve] [broker] Close temporary open ledger in BookkeeperBucketSnapshotStorage (#20111)"
This is an automated email from the ASF dual-hosted git repository.
cbornet pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 254f09b70d80640095f2fe6d12ed79f046fb3241
Author: Christophe Bornet
AuthorDate: Mon Apr 24 11:50:50 2023 +0200
Revert "[improve] [broker] Close temporary open ledger in
BookkeeperBucketSnapshotStorage (#20111)"
This reverts commit 1d1a3ef864a65c995ceda4b7875ed934c2574298.
---
.../bucket/BookkeeperBucketSnapshotStorage.java| 33 +-
1 file changed, 7 insertions(+), 26 deletions(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/delayed/bucket/BookkeeperBucketSnapshotStorage.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/delayed/bucket/BookkeeperBucketSnapshotStorage.java
index 9c30ccf1c0b..e7d4f9301dd 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/delayed/bucket/BookkeeperBucketSnapshotStorage.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/delayed/bucket/BookkeeperBucketSnapshotStorage.java
@@ -66,41 +66,22 @@ public class BookkeeperBucketSnapshotStorage implements
BucketSnapshotStorage {
@Override
public CompletableFuture getBucketSnapshotMetadata(long
bucketId) {
-return openLedger(bucketId).thenCompose(ledgerHandle -> {
-CompletableFuture snapshotFuture =
-getLedgerEntry(ledgerHandle, 0, 0)
-.thenApply(entryEnumeration ->
parseSnapshotMetadataEntry(entryEnumeration.nextElement()));
-
-snapshotFuture.whenComplete((__, e) -> closeLedger(ledgerHandle));
-
-return snapshotFuture;
-});
+return openLedger(bucketId).thenCompose(
+ledgerHandle -> getLedgerEntry(ledgerHandle, 0, 0).
+thenApply(entryEnumeration ->
parseSnapshotMetadataEntry(entryEnumeration.nextElement(;
}
@Override
public CompletableFuture>
getBucketSnapshotSegment(long bucketId, long firstSegmentEntryId,
long lastSegmentEntryId) {
-return openLedger(bucketId).thenCompose(ledgerHandle -> {
-CompletableFuture> parseFuture =
-getLedgerEntry(ledgerHandle, firstSegmentEntryId,
lastSegmentEntryId)
-.thenApply(this::parseSnapshotSegmentEntries);
-
-parseFuture.whenComplete((__, e) -> closeLedger(ledgerHandle));
-
-return parseFuture;
-});
+return openLedger(bucketId).thenCompose(
+ledgerHandle -> getLedgerEntry(ledgerHandle,
firstSegmentEntryId,
+
lastSegmentEntryId).thenApply(this::parseSnapshotSegmentEntries));
}
@Override
public CompletableFuture getBucketSnapshotLength(long bucketId) {
-return openLedger(bucketId).thenCompose(ledgerHandle -> {
-CompletableFuture lengthFuture =
-
CompletableFuture.completedFuture(ledgerHandle.getLength());
-
-lengthFuture.whenComplete((__, e) -> closeLedger(ledgerHandle));
-
-return lengthFuture;
-});
+return openLedger(bucketId).thenApply(LedgerHandle::getLength);
}
@Override
[pulsar] 03/04: Revert "[improve][broker] Make timer execute immediately after load index (#20126)"
This is an automated email from the ASF dual-hosted git repository.
cbornet pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 610c5f151436557d51b56e06a97ba422553592d7
Author: Christophe Bornet
AuthorDate: Mon Apr 24 11:49:13 2023 +0200
Revert "[improve][broker] Make timer execute immediately after load index
(#20126)"
This reverts commit 49480ea558e647169e8df01bfd2e871a5386e19e.
---
.../broker/delayed/bucket/BucketDelayedDeliveryTracker.java | 8
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/delayed/bucket/BucketDelayedDeliveryTracker.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/delayed/bucket/BucketDelayedDeliveryTracker.java
index b4d1745e22f..f57248acbb7 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/delayed/bucket/BucketDelayedDeliveryTracker.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/delayed/bucket/BucketDelayedDeliveryTracker.java
@@ -541,7 +541,7 @@ public class BucketDelayedDeliveryTracker extends
AbstractDelayedDeliveryTracker
@Override
public synchronized NavigableSet getScheduledMessages(int
maxMessages) {
-if (!checkPendingLoadDone()) {
+if (!checkPendingOpDone()) {
if (log.isDebugEnabled()) {
log.debug("[{}] Skip getScheduledMessages to wait for bucket
snapshot load finish.",
dispatcher.getName());
@@ -628,11 +628,11 @@ public class BucketDelayedDeliveryTracker extends
AbstractDelayedDeliveryTracker
if (timeout != null) {
timeout.cancel();
}
-timeout = timer.newTimeout(this, 0,
TimeUnit.MILLISECONDS);
+timeout = timer.newTimeout(this, tickTimeMillis,
TimeUnit.MILLISECONDS);
}
});
-if (!checkPendingLoadDone() ||
loadFuture.isCompletedExceptionally()) {
+if (!checkPendingOpDone() ||
loadFuture.isCompletedExceptionally()) {
break;
}
}
@@ -651,7 +651,7 @@ public class BucketDelayedDeliveryTracker extends
AbstractDelayedDeliveryTracker
return positions;
}
-private synchronized boolean checkPendingLoadDone() {
+private synchronized boolean checkPendingOpDone() {
if (pendingLoad == null || pendingLoad.isDone()) {
pendingLoad = null;
return true;
[GitHub] [pulsar] cbornet commented on pull request #20086: [fix][broker] Ensure previous delayed index be removed from snapshotSegmentLastIndexTable & Make load operate asynchronous
cbornet commented on PR #20086: URL: https://github.com/apache/pulsar/pull/20086#issuecomment-1519794136 This has been cherry-picked to branch-3.0 despite the code freeze. Do you feel strongly that this needs to be in the 3.0 release ? Otherwise we should revert this and delay it 3.0.1/3.1. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch branch-2.10 updated: [branch-2.10][fix][build] Upgrade swagger version to fix CVE-2022-1471 (#20172)
This is an automated email from the ASF dual-hosted git repository. penghui pushed a commit to branch branch-2.10 in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/branch-2.10 by this push: new 73462d65a26 [branch-2.10][fix][build] Upgrade swagger version to fix CVE-2022-1471 (#20172) 73462d65a26 is described below commit 73462d65a2657424363f7737bae50e4b7da9bc6e Author: Jiwei Guo AuthorDate: Mon Apr 24 17:47:46 2023 +0800 [branch-2.10][fix][build] Upgrade swagger version to fix CVE-2022-1471 (#20172) --- distribution/server/src/assemble/LICENSE.bin.txt | 6 +++--- pom.xml | 2 +- pulsar-sql/presto-distribution/LICENSE | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/distribution/server/src/assemble/LICENSE.bin.txt b/distribution/server/src/assemble/LICENSE.bin.txt index 62e3439c97c..7ead9d790a8 100644 --- a/distribution/server/src/assemble/LICENSE.bin.txt +++ b/distribution/server/src/assemble/LICENSE.bin.txt @@ -334,9 +334,9 @@ The Apache Software License, Version 2.0 * J2ObjC Annotations -- com.google.j2objc-j2objc-annotations-1.3.jar * Netty Reactive Streams -- com.typesafe.netty-netty-reactive-streams-2.0.6.jar * Swagger -- io.swagger-swagger-annotations-1.6.2.jar -- io.swagger-swagger-core-1.6.2.jar -- io.swagger-swagger-models-1.6.2.jar +- io.swagger-swagger-annotations-1.6.10.jar +- io.swagger-swagger-core-1.6.10.jar +- io.swagger-swagger-models-1.6.10.jar * DataSketches - com.yahoo.datasketches-memory-0.8.3.jar - com.yahoo.datasketches-sketches-core-0.8.3.jar diff --git a/pom.xml b/pom.xml index 26eaa9c63d2..e1acfb6b74b 100644 --- a/pom.xml +++ b/pom.xml @@ -126,7 +126,7 @@ flexible messaging model and an intuitive client API. 1.0.2 2.14.2 0.9.11 -1.6.2 +1.6.10 8.37 1.4.13 0.5.0 diff --git a/pulsar-sql/presto-distribution/LICENSE b/pulsar-sql/presto-distribution/LICENSE index c285eff2b5d..4c59ab739e2 100644 --- a/pulsar-sql/presto-distribution/LICENSE +++ b/pulsar-sql/presto-distribution/LICENSE @@ -475,7 +475,7 @@ The Apache Software License, Version 2.0 * Apache Yetus Audience Annotations - audience-annotations-0.5.0.jar * Swagger -- swagger-annotations-1.6.2.jar +- swagger-annotations-1.6.10.jar * Perfmark - perfmark-api-0.19.0.jar * Annotations
[GitHub] [pulsar] codelipenghui merged pull request #20172: [branch-2.10][fix][build] Upgrade swagger version to fix CVE-2022-1471
codelipenghui merged PR #20172: URL: https://github.com/apache/pulsar/pull/20172 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] cbornet commented on pull request #20155: [improve][broker] Support disabling delayed bucket merging.
cbornet commented on PR #20155: URL: https://github.com/apache/pulsar/pull/20155#issuecomment-1519738507 It seems that this commit will change a default setting and so will be breaking if we do it later. So I'd be in favor of leaving it in 3.0. WDYT ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] jiangpengcheng opened a new pull request, #20174: [fix][fn] Make python install dependencies from requirements.txt
jiangpengcheng opened a new pull request, #20174: URL: https://github.com/apache/pulsar/pull/20174 Fixes #20173 Master Issue: #xyz PIP: #xyz ### Motivation Python instance should install dependencies from requirements.txt ### Modifications Make python instance install dependencies from requirements.txt ### Verifying this change - [x] Make sure that the change passes the CI checks. *(Please pick either of the following options)* - [x] This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Does this pull request potentially affect one of the following parts: *If the box was checked, please highlight the changes* - [ ] Dependencies (add or upgrade a dependency) - [ ] The public API - [ ] The schema - [ ] The default values of configurations - [ ] The threading model - [ ] The binary protocol - [ ] The REST endpoints - [ ] The admin CLI options - [ ] The metrics - [ ] Anything that affects deployment ### Documentation - [ ] `doc` - [ ] `doc-required` - [x] `doc-not-needed` - [ ] `doc-complete` ### Matching PR in forked repository PR in forked repository: https://github.com/jiangpengcheng/pulsar/pull/12 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] jiangpengcheng opened a new issue, #20173: [Bug] Python instance doesn't install dependency from requirements.txt
jiangpengcheng opened a new issue, #20173:
URL: https://github.com/apache/pulsar/issues/20173
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues)
and found nothing similar.
### Version
master, commit id: 35e9897742b7db4bd29349940075a819b2ad6999
### Minimal reproduce step
1. package a python function in zip:
```
❯ tree func
func
├── requirements.txt
└── src
└── exclamation.py
❯ cat func/requirements.txt
requests==2.28.2
❯ cat func/src/exclamation.py
import requests # just check dependencies is installed
def process(input):
return str(input) + '!'
❯ zip -r func.zip func
updating: func/ (stored 0%)
updating: func/requirements.txt (stored 0%)
updating: func/src/ (stored 0%)
updating: func/src/exclamation.py (deflated 16%)
```
2. create some required dir and file:
```
mkdir -p /pulsar/conf/python-log/ logs/functions && echo "[loggers]
keys=root
[handlers]
keys=stream_handler
[formatters]
keys=formatter
[logger_root]
level=INFO
handlers=stream_handler
[handler_stream_handler]
class=StreamHandler
level=INFO
formatter=formatter
args=(sys.stdout,)
[formatter_formatter]
format=[%(asctime)s] [%(levelname)s] %(filename)s: %(message)s
datefmt=%Y-%m-%d %H:%M:%S %z" >
/pulsar/conf/python-log/python_instance_logging.ini
```
3. run python instance directly
```
python /pulsar/instances/python-instance/python_instance_main.py --py
/pulsar/download/pulsar_functions/func.zip --logging_directory logs/functions
--logging_file exclamation-py-9e1f170e-0 --logging_config_file
/pulsar/conf/python-log/python_instance_logging.ini
--install_usercode_dependencies true --instance_id 0 --function_id
0-74a988f8-6711-4a9c-9030-a07a54eb2fc4 --function_version 0 --function_details
'{"tenant":"public","namespace":"default","name":"exclamation-py-9e1f170e","className":"exclamation","userConfig":"{}","runtime":"PYTHON","autoAck":true,"parallelism":1,"source":{"typeClassName":"[B","inputSpecs":{"persistent://public/default/test-py-input":{}},"cleanupSubscription":true},"sink":{"typeClassName":"[B","topic":"persistent://public/default/test-py-output","producerSpec":{},"forwardSourceMessageProperty":true},"resources":{"cpu":1,"ram":"1073741824"},"componentType":"FUNCTION"}'
--pulsar_serviceurl pulsar://localhost:6650 --max_buffered_tuples 100 --port
39093 --metr
ics_port 39094 --expected_healthcheck_interval -1 --cluster_name
sn-platform-pulsar --use_tls false
```
### What did you expect to see?
python instance run successfully
### What did you see instead?
[2023-04-24 09:25:45 +] [INFO] util.py: Failed to import class
exclamation from path download/pulsar_functions
[2023-04-24 09:25:45 +] [INFO] util.py: No module named 'requests'
Traceback (most recent call last):
File "/pulsar/instances/python-instance/util.py", line 40, in import_class
return import_class_from_path(from_path, full_class_name)
File "/pulsar/instances/python-instance/util.py", line 60, in
import_class_from_path
mod = importlib.import_module(class_name)
File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1014, in _gcd_import
File "", line 991, in _find_and_load
File "", line 975, in _find_and_load_unlocked
File "", line 671, in _load_unlocked
File "", line 848, in exec_module
File "", line 219, in
_call_with_frames_removed
File "/pulsar/download/pulsar_functions/exclamation/src/exclamation.py",
line 1, in
import requests # just check dependencies is installed
ModuleNotFoundError: No module named 'requests'
### Anything else?
_No response_
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar-client-go] RobertIndie commented on a diff in pull request #1003: [Flaky Test]: Fix flaky test TestMaxPendingChunkMessages()
RobertIndie commented on code in PR #1003:
URL: https://github.com/apache/pulsar-client-go/pull/1003#discussion_r1174946784
##
pulsar/message_chunking_test.go:
##
@@ -576,3 +548,27 @@ func createTestMessagePayload(size int) []byte {
}
return payload
}
+
+//nolint:all
+func sendSingleChunk(p Producer, uuid string, chunkID int, totalChunks int) {
+ msg := {
+ Payload: []byte(fmt.Sprintf("chunk-%s-%d|", uuid, chunkID)),
+ }
+ producerImpl := p.(*producer).producers[0].(*partitionProducer)
+ mm := producerImpl.genMetadata(msg, len(msg.Payload), time.Now())
+ mm.Uuid = proto.String(uuid)
+ mm.NumChunksFromMsg = proto.Int32(int32(totalChunks))
+ mm.TotalChunkMsgSize = proto.Int32(int32(len(msg.Payload)))
+ mm.ChunkId = proto.Int32(int32(chunkID))
+ producerImpl.updateMetadataSeqID(mm, msg)
+
+ producerImpl.internalSingleSend(
+ mm,
+ msg.Payload,
+ {
+ callback: func(id MessageID, producerMessage
*ProducerMessage, err error) {},
+ msg: msg,
+ },
+ uint32(internal.MaxMessageSize),
+ )
Review Comment:
We need to flush the producer after sending the message. Or handle the
callback of the `sendRequest`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar-dotpulsar] kandersen82 commented on a diff in pull request #149: Producer access mode fixes
kandersen82 commented on code in PR #149: URL: https://github.com/apache/pulsar-dotpulsar/pull/149#discussion_r1174934905 ## src/DotPulsar/Internal/Abstractions/IChannel.cs: ## @@ -23,6 +23,7 @@ public interface IChannel void ClosedByServer(); void WaitingForExclusive(); void Connected(); +void ProducerConnected(ulong topicEpoch); Review Comment: It could potentially make sense to create an IProducerChannel and IConsumerChannel. There are quite a lot of methods that are specific to one or the other. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] 02/02: [fix] [broker] Make `LeastResourceUsageWithWeight` thread safe (#20159)
This is an automated email from the ASF dual-hosted git repository.
zike pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 89e4d46f4a5585ac126a03aa298020203100ad71
Author: lifepuzzlefun
AuthorDate: Sun Apr 23 11:27:48 2023 +0800
[fix] [broker] Make `LeastResourceUsageWithWeight` thread safe (#20159)
Fix #20160
### Motivation
LeastResourceUsageWithWeight is an stateful object and current code will be
accessed by multithread.
thread 1: is execute
https://github.com/apache/pulsar/blob/2b41e4eafb1cba0e548dd90df60e8cdbb24cd490/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/strategy/LeastResourceUsageWithWeight.java#L89-L91
and thread 2: is execute
https://github.com/apache/pulsar/blob/2b41e4eafb1cba0e548dd90df60e8cdbb24cd490/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/strategy/LeastResourceUsageWithWeight.java#L147-L150
so an IndexOutOfBound occurs.
### Modifications
change the state to `ThreadLocal`
(cherry picked from commit 963260abfa142b8cf9ffe372d85d470a78c17235)
---
.../strategy/LeastResourceUsageWithWeight.java | 18 ++
1 file changed, 10 insertions(+), 8 deletions(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/strategy/LeastResourceUsageWithWeight.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/strategy/LeastResourceUsageWithWeight.java
index 902cfdaf73f..98986d84b98 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/strategy/LeastResourceUsageWithWeight.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/strategy/LeastResourceUsageWithWeight.java
@@ -23,6 +23,7 @@ import java.util.HashSet;
import java.util.Optional;
import java.util.Set;
import java.util.concurrent.ThreadLocalRandom;
+import javax.annotation.concurrent.ThreadSafe;
import lombok.extern.slf4j.Slf4j;
import org.apache.pulsar.broker.ServiceConfiguration;
import org.apache.pulsar.broker.loadbalance.extensions.LoadManagerContext;
@@ -35,14 +36,15 @@ import org.apache.pulsar.common.naming.ServiceUnitId;
* cause cluster fluctuations due to short-term load jitter.
*/
@Slf4j
+@ThreadSafe
public class LeastResourceUsageWithWeight implements BrokerSelectionStrategy {
// Maintain this list to reduce object creation.
-private final ArrayList bestBrokers;
-private final Set noLoadDataBrokers;
+private final ThreadLocal> bestBrokers;
+private final ThreadLocal> noLoadDataBrokers;
public LeastResourceUsageWithWeight() {
-this.bestBrokers = new ArrayList<>();
-this.noLoadDataBrokers = new HashSet<>();
+this.bestBrokers = ThreadLocal.withInitial(ArrayList::new);
+this.noLoadDataBrokers = ThreadLocal.withInitial(HashSet::new);
}
// A broker's max resource usage with weight using its historical load and
short-term load data with weight.
@@ -70,7 +72,6 @@ public class LeastResourceUsageWithWeight implements
BrokerSelectionStrategy {
/**
* Find a suitable broker to assign the given bundle to.
- * This method is not thread safety.
*
* @param candidates The candidates for which the bundle may be
assigned.
* @param bundleToAssign The data for the bundle to assign.
@@ -86,6 +87,9 @@ public class LeastResourceUsageWithWeight implements
BrokerSelectionStrategy {
return Optional.empty();
}
+ArrayList bestBrokers = this.bestBrokers.get();
+HashSet noLoadDataBrokers = this.noLoadDataBrokers.get();
+
bestBrokers.clear();
noLoadDataBrokers.clear();
// Maintain of list of all the best scoring brokers and then randomly
@@ -135,9 +139,7 @@ public class LeastResourceUsageWithWeight implements
BrokerSelectionStrategy {
log.info("Assign randomly as none of the brokers are
underloaded. candidatesSize:{}, "
+ "noLoadDataBrokersSize:{}", candidates.size(),
noLoadDataBrokers.size());
}
-for (String broker : candidates) {
-bestBrokers.add(broker);
-}
+bestBrokers.addAll(candidates);
}
if (debugMode) {
[pulsar] branch branch-3.0 updated (e3e1c08e513 -> 89e4d46f4a5)
This is an automated email from the ASF dual-hosted git repository. zike pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/pulsar.git from e3e1c08e513 Revert "[improve][broker] Optimization protobuf code in the bucket delayed tracker (#20158)" new 2fa28f427b8 [improve] Allow to build and push multi-arch Docker images (#19432) new 89e4d46f4a5 [fix] [broker] Make `LeastResourceUsageWithWeight` thread safe (#20159) The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .github/workflows/pulsar-ci.yaml | 1 + build/build_java_test_image.sh | 1 + build/pulsar_ci_tool.sh| 3 +- docker/pulsar-all/pom.xml | 42 +++--- docker/pulsar/pom.xml | 51 ++ pom.xml| 8 +++- .../strategy/LeastResourceUsageWithWeight.java | 18 tests/docker-images/latest-version-image/pom.xml | 6 +++ 8 files changed, 104 insertions(+), 26 deletions(-)
[pulsar] 01/02: [improve] Allow to build and push multi-arch Docker images (#19432)
This is an automated email from the ASF dual-hosted git repository.
zike pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 2fa28f427b8e5e05b4d0a233cc68f62d5bbe34d7
Author: Matteo Merli
AuthorDate: Sat Apr 22 13:28:53 2023 -0700
[improve] Allow to build and push multi-arch Docker images (#19432)
Co-authored-by: Lari Hotari
Co-authored-by: Yong Zhang
Co-authored-by: Zixuan Liu
Co-authored-by: tison
(cherry picked from commit 4190e40ab4ac448a2f94edd1c621dc1c67d9ee4b)
---
.github/workflows/pulsar-ci.yaml | 1 +
build/build_java_test_image.sh | 1 +
build/pulsar_ci_tool.sh | 3 +-
docker/pulsar-all/pom.xml| 42 +++
docker/pulsar/pom.xml| 51 +++-
pom.xml | 8 +++-
tests/docker-images/latest-version-image/pom.xml | 6 +++
7 files changed, 94 insertions(+), 18 deletions(-)
diff --git a/.github/workflows/pulsar-ci.yaml b/.github/workflows/pulsar-ci.yaml
index f7dbc755264..4d4f2902d8f 100644
--- a/.github/workflows/pulsar-ci.yaml
+++ b/.github/workflows/pulsar-ci.yaml
@@ -798,6 +798,7 @@ jobs:
# build docker image
# include building of Pulsar SQL, Connectors, Offloaders and server
distros
mvn -B -am -pl
pulsar-sql/presto-distribution,distribution/io,distribution/offloaders,distribution/server,distribution/shell,tests/docker-images/latest-version-image
install \
+ -DUBUNTU_MIRROR="${UBUNTU_MIRROR}"
-DUBUNTU_SECURITY_MIRROR="${UBUNTU_SECURITY_MIRROR}" \
-Pmain,docker -Dmaven.test.skip=true -Ddocker.squash=true \
-Dspotbugs.skip=true -Dlicense.skip=true -Dcheckstyle.skip=true
-Drat.skip=true
diff --git a/build/build_java_test_image.sh b/build/build_java_test_image.sh
index 0747e6dacb8..459bf26f98e 100755
--- a/build/build_java_test_image.sh
+++ b/build/build_java_test_image.sh
@@ -27,5 +27,6 @@ if [[ "$(docker version -f '{{.Server.Experimental}}'
2>/dev/null)" == "true" ]]
SQUASH_PARAM="-Ddocker.squash=true"
fi
mvn -am -pl tests/docker-images/java-test-image
-Pcore-modules,-main,integrationTests,docker \
+ -DUBUNTU_MIRROR="${UBUNTU_MIRROR}"
-DUBUNTU_SECURITY_MIRROR="${UBUNTU_SECURITY_MIRROR}" \
-Dmaven.test.skip=true -DskipSourceReleaseAssembly=true -Dspotbugs.skip=true
-Dlicense.skip=true $SQUASH_PARAM \
"$@" install
\ No newline at end of file
diff --git a/build/pulsar_ci_tool.sh b/build/pulsar_ci_tool.sh
index 61199eda2c5..d946edd3957 100755
--- a/build/pulsar_ci_tool.sh
+++ b/build/pulsar_ci_tool.sh
@@ -46,7 +46,8 @@ function ci_print_thread_dumps() {
# runs maven
function _ci_mvn() {
- mvn -B -ntp "$@"
+ mvn -B -ntp -DUBUNTU_MIRROR="${UBUNTU_MIRROR}"
-DUBUNTU_SECURITY_MIRROR="${UBUNTU_SECURITY_MIRROR}" \
+"$@"
}
# runs OWASP Dependency Check for all projects
diff --git a/docker/pulsar-all/pom.xml b/docker/pulsar-all/pom.xml
index b4186c8c1a9..e78eebecc7a 100644
--- a/docker/pulsar-all/pom.xml
+++ b/docker/pulsar-all/pom.xml
@@ -68,10 +68,6 @@
docker
-
-
target/apache-pulsar-io-connectors-${project.version}-bin
-
target/pulsar-offloader-distribution-${project.version}-bin.tar.gz
-
@@ -143,17 +139,25 @@
- pulsar-all
+ ${docker.organization}/pulsar-all
${project.basedir}
latest
+ ${project.version}
+
+
target/apache-pulsar-io-connectors-${project.version}-bin
+
target/pulsar-offloader-distribution-${project.version}-bin.tar.gz
+
+
+
+${docker.platforms}
+
+
- latest
- ${docker.organization}
@@ -161,5 +165,29 @@
+
+
+ docker-push
+
+
+
+io.fabric8
+docker-maven-plugin
+
+
+default
+package
+
+ build
+ tag
+ push
+
+
+
+
+
+
+
+
diff --git a/docker/pulsar/pom.xml b/docker/pulsar/pom.xml
index d83755846b4..5e849f2887e 100644
--- a/docker/pulsar/pom.xml
+++ b/docker/pulsar/pom.xml
@@ -47,15 +47,14 @@
+
+ mirror://mirrors.ubuntu.com/mirrors.txt
+
[GitHub] [pulsar-dotpulsar] kandersen82 opened a new pull request, #149: Producer access mode fixes
kandersen82 opened a new pull request, #149: URL: https://github.com/apache/pulsar-dotpulsar/pull/149 Fixes issues with Producer access modes # Description Fixes disconnect issues related to producer access modes. Also changed behavior to to align with java client. When a producer with WaitForExclusive mode has been given exclusive access it cannot go back to WaitForExclusive, and will instead be fenced. # Regression Issue introduced in unreleased code # Testing Manual testing with TCP connection killing -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] Technoboy- opened a new pull request, #20172: [branch-2.10][fix][build] Upgrade swagger version to fix CVE-2022-1471
Technoboy- opened a new pull request, #20172: URL: https://github.com/apache/pulsar/pull/20172 ### Motivation Upgrade swagger version to fix CVE-2022-1471 ### Documentation - [ ] `doc` - [ ] `doc-required` - [x] `doc-not-needed` - [ ] `doc-complete` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] gaoran10 commented on a diff in pull request #20030: [fix][broker] Fix the reason label of authentication metrics
gaoran10 commented on code in PR #20030:
URL: https://github.com/apache/pulsar/pull/20030#discussion_r1174918890
##
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/authentication/PulsarAuthenticationException.java:
##
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.authentication;
+
+import javax.naming.AuthenticationException;
+import lombok.Getter;
+
+/**
+ * PulsarAuthenticationException indicates Authentication exception in Pulsar,
it contains an error code.
+ */
+public class PulsarAuthenticationException extends AuthenticationException {
+
+public enum ErrorCode {
+UNKNOWN,
+PROVIDER_LIST_AUTH_REQUIRED,
+BASIC_INVALID_TOKEN,
+BASIC_INVALID_AUTH_DATA,
+AUTHZ_NO_CLIENT,
+AUTHZ_NO_TOKEN,
+AUTHZ_NO_PUBLIC_KEY,
+AUTHZ_DOMAIN_MISMATCH,
+AUTHZ_INVALID_TOKEN,
+TLS_NO_CERTS,
+TLS_NO_CN, // cn: common name
+TOKEN_INVALID_HEADER,
+TOKEN_NO_AUTH_DATA,
+TOKEN_EMPTY_TOKEN,
+TOKEN_INVALID_TOKEN,
+TOKEN_INVALID_AUDIENCES,
+}
Review Comment:
I removed the `PulsarAuthenticationException`, and only added some error
codes for every authentication provider, like `AuthenticationProviderOpenID `.
PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar] StevenLuMT commented on pull request #20168: [fix][broker] log update in class:LeastResourceUsageWithWeight
StevenLuMT commented on PR #20168: URL: https://github.com/apache/pulsar/pull/20168#issuecomment-1519497624 /pulsarbot run-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] thetumbled closed pull request #19662: [feat][txn] Support for idempotent commit and abort, Solution1.
thetumbled closed pull request #19662: [feat][txn] Support for idempotent commit and abort, Solution1. URL: https://github.com/apache/pulsar/pull/19662 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] eolivelli commented on a diff in pull request #20116: [improve][fn] Allow unknown fields in connectors config
eolivelli commented on code in PR #20116:
URL: https://github.com/apache/pulsar/pull/20116#discussion_r1174842575
##
pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/JavaInstanceRunnable.java:
##
@@ -870,6 +872,56 @@ private void setupInput(ContextImpl contextImpl) throws
Exception {
Thread.currentThread().setContextClassLoader(this.instanceClassLoader);
}
}
+private Map parseComponentConfig(String connectorConfigs)
throws IOException {
+return parseComponentConfig(connectorConfigs, instanceConfig,
componentClassLoader, componentType);
+}
+
+static Map parseComponentConfig(String connectorConfigs,
+InstanceConfig
instanceConfig,
+ClassLoader
componentClassLoader,
+
org.apache.pulsar.functions.proto.Function
+
.FunctionDetails.ComponentType componentType)
+throws IOException {
+final Map config = ObjectMapperFactory
+.getMapper()
+.reader()
+.forType(new TypeReference>() {})
+.readValue(connectorConfigs);
+if (instanceConfig.isIgnoreUnknownConfigFields() &&
componentClassLoader instanceof NarClassLoader) {
+final String configClass;
+if (componentType ==
org.apache.pulsar.functions.proto.Function.FunctionDetails.ComponentType.SOURCE)
{
+configClass = ConnectorUtils
+.getConnectorDefinition((NarClassLoader)
componentClassLoader).getSourceConfigClass();
+} else if (componentType ==
org.apache.pulsar.functions.proto.Function.FunctionDetails.ComponentType.SINK) {
+configClass = ConnectorUtils
+.getConnectorDefinition((NarClassLoader)
componentClassLoader).getSinkConfigClass();
+} else {
+return config;
+}
+if (configClass != null) {
+final Object configInstance =
Reflections.createInstance(configClass,
+Thread.currentThread().getContextClassLoader());
+final List allFields =
Review Comment:
I am not sure that this is correct.
ObjectMapper should follow Java Beans conventions and use getter/setters
together with public fields.
We should use some ObjectMapper utilities here in order to ensure that we
are doing the right thing
##
pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/JavaInstanceRunnable.java:
##
@@ -870,6 +872,56 @@ private void setupInput(ContextImpl contextImpl) throws
Exception {
Thread.currentThread().setContextClassLoader(this.instanceClassLoader);
}
}
+private Map parseComponentConfig(String connectorConfigs)
throws IOException {
+return parseComponentConfig(connectorConfigs, instanceConfig,
componentClassLoader, componentType);
+}
+
+static Map parseComponentConfig(String connectorConfigs,
+InstanceConfig
instanceConfig,
+ClassLoader
componentClassLoader,
+
org.apache.pulsar.functions.proto.Function
+
.FunctionDetails.ComponentType componentType)
+throws IOException {
+final Map config = ObjectMapperFactory
+.getMapper()
+.reader()
+.forType(new TypeReference>() {})
+.readValue(connectorConfigs);
+if (instanceConfig.isIgnoreUnknownConfigFields() &&
componentClassLoader instanceof NarClassLoader) {
+final String configClass;
+if (componentType ==
org.apache.pulsar.functions.proto.Function.FunctionDetails.ComponentType.SOURCE)
{
+configClass = ConnectorUtils
+.getConnectorDefinition((NarClassLoader)
componentClassLoader).getSourceConfigClass();
+} else if (componentType ==
org.apache.pulsar.functions.proto.Function.FunctionDetails.ComponentType.SINK) {
+configClass = ConnectorUtils
+.getConnectorDefinition((NarClassLoader)
componentClassLoader).getSinkConfigClass();
+} else {
+return config;
+}
+if (configClass != null) {
+final Object configInstance =
Reflections.createInstance(configClass,
+Thread.currentThread().getContextClassLoader());
+final List allFields =
+Reflections
+.getAllFields(configInstance.getClass())
+.stream()
+
[GitHub] [pulsar-client-go] Anonymitaet commented on pull request #1002: [feat][txn]Implement transactional consumer/producer API
Anonymitaet commented on PR #1002: URL: https://github.com/apache/pulsar-client-go/pull/1002#issuecomment-1519452516 Hi @liangyepianzhou thanks for introducing this great feature! As we discussed just now, for the doc side: ### ✅ Pulsar feature matrix I've ticked the three boxes (client, producer, consumer) of ["Transactions" for Go](https://docs.google.com/spreadsheets/d/1YHYTkIXR8-Ql103u-IMI18TXLlGStK8uJjDsOOA0T20/edit#gid=1784579914=19:19). ### ✅ Pulsar client docs 1. Could you please provide the tech inputs as below? Add Go code example to [Transactions > Get started ](https://pulsar.apache.org/docs/next/txn-use/). https://user-images.githubusercontent.com/50226895/233913428-365185d9-c4d8-4a32-9c32-942d7e18f859.png;> 2. This feature is available from Pulsar 2.9.5, can we add the doc to 4 doc versions (NEXT, 2.11.x, 2.10.x, 2.9.x) or just indicate what version it supports? - For 2.9.x: we can add a tip to the 2.9.x doc and indicate that this feature is available from (or in) Pulsar 2.9.5. - For 2.11.x and 2.10.x: please see the questions below. https://user-images.githubusercontent.com/50226895/233914462-8feb621f-dc57-4a41-a3a0-46eb93faeb41.png;> BTW, - Original doc file of NEXT is [here](https://github.com/apache/pulsar-site/edit/main/docs). - Original doc files of all versioned docs are [here](https://github.com/apache/pulsar-site/tree/main/versioned_docs). Feel free to reach out to me if you have questions. TIA! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-dotpulsar] blankensteiner commented on pull request #146: Partitioned Topic Consumer Support
blankensteiner commented on PR #146: URL: https://github.com/apache/pulsar-dotpulsar/pull/146#issuecomment-1519443474 Hi @toneill818 Just wanted to know if you have an ETA? (if I should create a new release and wait for this PR if you are close to finishing) :-) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
