Re: [PR] [fix][broker] Fix unload operation stuck when use ExtensibleLoadManager [pulsar]
codecov-commenter commented on PR #21332: URL: https://github.com/apache/pulsar/pull/21332#issuecomment-1754265494 ## [Codecov](https://app.codecov.io/gh/apache/pulsar/pull/21332?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report > Merging [#21332](https://app.codecov.io/gh/apache/pulsar/pull/21332?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) (965ae9c) into [master](https://app.codecov.io/gh/apache/pulsar/commit/8438e43fb02b6d8df29672c6460a2f3da924ab60?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) (8438e43) will **increase** coverage by `0.02%`. > Report is 6 commits behind head on master. > The diff coverage is `0.00%`. [](https://app.codecov.io/gh/apache/pulsar/pull/21332?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) ```diff @@ Coverage Diff @@ ## master #21332 +/- ## + Coverage 36.84% 36.87% +0.02% + Complexity12229 710 -11519 Files 1699 1699 Lines130559 132119+1560 Branches 1426414706 +442 + Hits 4810648716 +610 - Misses7612176931 +810 - Partials 6332 6472 +140 ``` | [Flag](https://app.codecov.io/gh/apache/pulsar/pull/21332/flags?src=pr=flags_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | Coverage Δ | | |---|---|---| | [inttests](https://app.codecov.io/gh/apache/pulsar/pull/21332/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `24.08% <0.00%> (-0.26%)` | :arrow_down: | | [systests](https://app.codecov.io/gh/apache/pulsar/pull/21332/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `25.02% <0.00%> (+0.29%)` | :arrow_up: | | [unittests](https://app.codecov.io/gh/apache/pulsar/pull/21332/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `31.93% <0.00%> (-0.02%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more. | [Files](https://app.codecov.io/gh/apache/pulsar/pull/21332?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | Coverage Δ | | |---|---|---| | [.../loadbalance/extensions/manager/UnloadManager.java](https://app.codecov.io/gh/apache/pulsar/pull/21332?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9sb2FkYmFsYW5jZS9leHRlbnNpb25zL21hbmFnZXIvVW5sb2FkTWFuYWdlci5qYXZh) | `0.00% <0.00%> (ø)` | | | [...xtensions/channel/ServiceUnitStateChannelImpl.java](https://app.codecov.io/gh/apache/pulsar/pull/21332?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9sb2FkYmFsYW5jZS9leHRlbnNpb25zL2NoYW5uZWwvU2VydmljZVVuaXRTdGF0ZUNoYW5uZWxJbXBsLmphdmE=) | `0.54% <0.00%> (-0.01%)` | :arrow_down: | ... and [67 files with indirect coverage changes](https://app.codecov.io/gh/apache/pulsar/pull/21332/indirect-changes?src=pr=tree-more_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][broker] Fix unload operation stuck when use ExtensibleLoadManager [pulsar]
Demogorgon314 commented on code in PR #21332:
URL: https://github.com/apache/pulsar/pull/21332#discussion_r1351283415
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##
@@ -503,13 +503,22 @@ public CompletableFuture>
getOwnerAsync(String serviceUnit) {
case Splitting -> {
return
CompletableFuture.completedFuture(Optional.of(data.sourceBroker()));
}
-case Assigning, Releasing -> {
+case Assigning -> {
return deferGetOwnerRequest(serviceUnit).whenComplete((__, e)
-> {
if (e != null) {
ownerLookUpCounters.get(state).getFailure().incrementAndGet();
}
-}).thenApply(
-broker -> broker == null ? Optional.empty() :
Optional.of(broker));
+}).thenApply(broker -> broker == null ? Optional.empty() :
Optional.of(broker));
+}
+case Releasing -> {
+if (isTransferCommand(data)) {
+return deferGetOwnerRequest(serviceUnit).whenComplete((__,
e) -> {
+if (e != null) {
+
ownerLookUpCounters.get(state).getFailure().incrementAndGet();
+}
+}).thenApply(broker -> broker == null ? Optional.empty() :
Optional.of(broker));
+}
+return CompletableFuture.completedFuture(Optional.empty());
Review Comment:
> Then, isn't your change in UnloadManager.handleEvent sufficient to return
the admin api?
No, the `unloadServiceUnit` will timeout since the current is `Releasing`
state.
You can see we are calling `closeServiceUnit` in `handleReleaseEvent`
method, but in the `closeServiceUnit`, it will check the ownership.
So when it gets ownership, this operation will be blocked since the current
state is `Releasing`.(The `Free` state message has not been sent to the channel
yet)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[I] [Bug] BacklogQuota not applied to the namespace [pulsar]
465 opened a new issue, #21337: URL: https://github.com/apache/pulsar/issues/21337 ### Search before asking - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar. ### Version pulasr 3.0.0 pulsar-resources-operator: 0.3.3 ### Minimal reproduce step 1. create a namespace 2. manage namespace by CRD ### What did you expect to see? if the request is invalid it should reject ### What did you see instead? accepts invalid(?) request and the namespace backlog quota config becomes unmodifiable ### Anything else? related: https://github.com/streamnative/pulsar-resources-operator/issues/161 this is the requester's problem that not sending valid requests, but still, the pulsar broker should validate and reject before accepting backlog_quota_map this is a valid request sent using pulsar admin CLI (3.0.0)  this is an invalid request sent from operator (0.3.3)  both logs are caputred from pulsar broker ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar-client-go] branch master updated: Fix issue where DisableReplication flag does not work (#1100)
This is an automated email from the ASF dual-hosted git repository.
nkurihar pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar-client-go.git
The following commit(s) were added to refs/heads/master by this push:
new 0bf2f9b8 Fix issue where DisableReplication flag does not work (#1100)
0bf2f9b8 is described below
commit 0bf2f9b86f4e540e912f73c65327b9dc6c4b57c9
Author: Masahiro Sakamoto
AuthorDate: Tue Oct 10 11:49:26 2023 +0900
Fix issue where DisableReplication flag does not work (#1100)
---
pulsar/producer_partition.go | 4 +-
pulsar/producer_test.go | 92
2 files changed, 94 insertions(+), 2 deletions(-)
diff --git a/pulsar/producer_partition.go b/pulsar/producer_partition.go
index 720c7df4..ebd292ec 100755
--- a/pulsar/producer_partition.go
+++ b/pulsar/producer_partition.go
@@ -551,13 +551,13 @@ func (p *partitionProducer) internalSend(request
*sendRequest) {
deliverAt = time.Now().Add(msg.DeliverAfter)
}
- mm := p.genMetadata(msg, uncompressedSize, deliverAt)
-
// set default ReplicationClusters when DisableReplication
if msg.DisableReplication {
msg.ReplicationClusters = []string{"__local__"}
}
+ mm := p.genMetadata(msg, uncompressedSize, deliverAt)
+
sendAsBatch := !p.options.DisableBatching &&
msg.ReplicationClusters == nil &&
deliverAt.UnixNano() < 0
diff --git a/pulsar/producer_test.go b/pulsar/producer_test.go
index 38fec576..29ffa780 100644
--- a/pulsar/producer_test.go
+++ b/pulsar/producer_test.go
@@ -30,7 +30,9 @@ import (
"time"
"github.com/apache/pulsar-client-go/pulsar/internal"
+ pb "github.com/apache/pulsar-client-go/pulsar/internal/pulsar_proto"
"github.com/stretchr/testify/assert"
+ "google.golang.org/protobuf/proto"
"github.com/apache/pulsar-client-go/pulsar/crypto"
plog "github.com/apache/pulsar-client-go/pulsar/log"
@@ -2298,3 +2300,93 @@ func TestFailPendingMessageWithClose(t *testing.T) {
testProducer.Close()
assert.Equal(t, 0, partitionProducerImp.pendingQueue.Size())
}
+
+type pendingQueueWrapper struct {
+ pendingQueue internal.BlockingQueue
+ writtenBuffers *[]internal.Buffer
+}
+
+func (pqw *pendingQueueWrapper) Put(item interface{}) {
+ pi := item.(*pendingItem)
+ writerIdx := pi.buffer.WriterIndex()
+ buf := internal.NewBuffer(int(writerIdx))
+ buf.Write(pi.buffer.Get(0, writerIdx))
+ *pqw.writtenBuffers = append(*pqw.writtenBuffers, buf)
+ pqw.pendingQueue.Put(item)
+}
+
+func (pqw *pendingQueueWrapper) Take() interface{} {
+ return pqw.pendingQueue.Take()
+}
+
+func (pqw *pendingQueueWrapper) Poll() interface{} {
+ return pqw.pendingQueue.Poll()
+}
+
+func (pqw *pendingQueueWrapper) CompareAndPoll(compare func(interface{}) bool)
interface{} {
+ return pqw.pendingQueue.CompareAndPoll(compare)
+}
+
+func (pqw *pendingQueueWrapper) Peek() interface{} {
+ return pqw.pendingQueue.Peek()
+}
+
+func (pqw *pendingQueueWrapper) PeekLast() interface{} {
+ return pqw.pendingQueue.PeekLast()
+}
+
+func (pqw *pendingQueueWrapper) Size() int {
+ return pqw.pendingQueue.Size()
+}
+
+func (pqw *pendingQueueWrapper) ReadableSlice() []interface{} {
+ return pqw.pendingQueue.ReadableSlice()
+}
+
+func TestDisableReplication(t *testing.T) {
+ client, err := NewClient(ClientOptions{
+ URL: serviceURL,
+ })
+ assert.NoError(t, err)
+ defer client.Close()
+
+ testProducer, err := client.CreateProducer(ProducerOptions{
+ Topic: newTopicName(),
+ DisableBatching: true,
+ })
+ assert.NoError(t, err)
+ assert.NotNil(t, testProducer)
+ defer testProducer.Close()
+
+ writtenBuffers := make([]internal.Buffer, 0)
+ pqw := {
+ pendingQueue: internal.NewBlockingQueue(1000),
+ writtenBuffers: ,
+ }
+
+ partitionProducerImp :=
testProducer.(*producer).producers[0].(*partitionProducer)
+ partitionProducerImp.pendingQueue = pqw
+
+ ID, err := testProducer.Send(context.Background(), {
+ Payload:[]byte("disable-replication"),
+ DisableReplication: true,
+ })
+ assert.NoError(t, err)
+ assert.NotNil(t, ID)
+
+ assert.Equal(t, 1, len(writtenBuffers))
+ buf := writtenBuffers[0]
+
+ buf.Skip(4)// TOTAL_SIZE
+ cmdSize := buf.ReadUint32()// CMD_SIZE
+ buf.Skip(cmdSize) // CMD
+ buf.Skip(2)// MAGIC_NUMBER
+ buf.Skip(4)// CHECKSUM
+ metadataSize := buf.ReadUint32() // METADATA_SIZE
+ metadata := buf.Read(metadataSize) // METADATA
+
+ var
Re: [PR] [fix] Fix issue where DisableReplication flag does not work [pulsar-client-go]
nkurihar merged PR #1100: URL: https://github.com/apache/pulsar-client-go/pull/1100 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][broker] Fix unload operation stuck when use ExtensibleLoadManager [pulsar]
heesung-sn commented on code in PR #21332:
URL: https://github.com/apache/pulsar/pull/21332#discussion_r1351215592
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##
@@ -503,13 +503,22 @@ public CompletableFuture>
getOwnerAsync(String serviceUnit) {
case Splitting -> {
return
CompletableFuture.completedFuture(Optional.of(data.sourceBroker()));
}
-case Assigning, Releasing -> {
+case Assigning -> {
return deferGetOwnerRequest(serviceUnit).whenComplete((__, e)
-> {
if (e != null) {
ownerLookUpCounters.get(state).getFailure().incrementAndGet();
}
-}).thenApply(
-broker -> broker == null ? Optional.empty() :
Optional.of(broker));
+}).thenApply(broker -> broker == null ? Optional.empty() :
Optional.of(broker));
+}
+case Releasing -> {
+if (isTransferCommand(data)) {
+return deferGetOwnerRequest(serviceUnit).whenComplete((__,
e) -> {
+if (e != null) {
+
ownerLookUpCounters.get(state).getFailure().incrementAndGet();
+}
+}).thenApply(broker -> broker == null ? Optional.empty() :
Optional.of(broker));
+}
+return CompletableFuture.completedFuture(Optional.empty());
Review Comment:
Then, isn't your change in UnloadManager.handleEvent sufficient to return
the admin api?
In the test env, I feel like we can configure the monitor frequency
small(like 1 sec) to fix the stuck state quicker.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [improve][client] Add a way to configure which DNS use [pulsar]
codecov-commenter commented on PR #21227: URL: https://github.com/apache/pulsar/pull/21227#issuecomment-1754218903 ## [Codecov](https://app.codecov.io/gh/apache/pulsar/pull/21227?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report > Merging [#21227](https://app.codecov.io/gh/apache/pulsar/pull/21227?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) (53f5241) into [master](https://app.codecov.io/gh/apache/pulsar/commit/8438e43fb02b6d8df29672c6460a2f3da924ab60?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) (8438e43) will **increase** coverage by `36.37%`. > Report is 6 commits behind head on master. > The diff coverage is `80.64%`. [](https://app.codecov.io/gh/apache/pulsar/pull/21227?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) ```diff @@ Coverage Diff @@ ## master #21227 +/- ## = + Coverage 36.84% 73.22% +36.37% - Complexity1222932529+20300 = Files 1699 1888 +189 Lines130559 140257 +9698 Branches 1426415445 +1181 = + Hits 48106 102703+54597 + Misses7612129457-46664 - Partials 6332 8097 +1765 ``` | [Flag](https://app.codecov.io/gh/apache/pulsar/pull/21227/flags?src=pr=flags_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | Coverage Δ | | |---|---|---| | [inttests](https://app.codecov.io/gh/apache/pulsar/pull/21227/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `24.19% <45.16%> (-0.14%)` | :arrow_down: | | [systests](https://app.codecov.io/gh/apache/pulsar/pull/21227/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `24.73% <41.93%> (+<0.01%)` | :arrow_up: | | [unittests](https://app.codecov.io/gh/apache/pulsar/pull/21227/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `72.53% <80.64%> (+40.58%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more. | [Files](https://app.codecov.io/gh/apache/pulsar/pull/21227?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | Coverage Δ | | |---|---|---| | [...va/org/apache/pulsar/broker/service/ServerCnx.java](https://app.codecov.io/gh/apache/pulsar/pull/21227?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9zZXJ2aWNlL1NlcnZlckNueC5qYXZh) | `71.76% <100.00%> (+30.24%)` | :arrow_up: | | [...n/java/org/apache/pulsar/broker/service/Topic.java](https://app.codecov.io/gh/apache/pulsar/pull/21227?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9zZXJ2aWNlL1RvcGljLmphdmE=) | `36.36% <ø> (+28.03%)` | :arrow_up: | | [...lsar/client/impl/conf/ClientConfigurationData.java](https://app.codecov.io/gh/apache/pulsar/pull/21227?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL2NvbmYvQ2xpZW50Q29uZmlndXJhdGlvbkRhdGEuamF2YQ==) | `96.69% <100.00%> (+8.45%)` | :arrow_up: | | [...g/apache/pulsar/client/impl/ClientBuilderImpl.java](https://app.codecov.io/gh/apache/pulsar/pull/21227?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL0NsaWVudEJ1aWxkZXJJbXBsLmphdmE=) | `85.98% <83.33%> (+39.62%)` | :arrow_up: | | [.../org/apache/pulsar/client/impl/ConnectionPool.java](https://app.codecov.io/gh/apache/pulsar/pull/21227?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL0Nvbm5lY3Rpb25Qb29sLmphdmE=) | `75.96% <66.66%> (+14.01%)` | :arrow_up: | |
Re: [I] [Bug]flink sink .finishConnect(..) failed: Connection refused: /127.0.0.1:6650 [pulsar]
github-actions[bot] commented on issue #21152: URL: https://github.com/apache/pulsar/issues/21152#issuecomment-1754183892 The issue had no activity for 30 days, mark with Stale label. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][broker] Fix unload operation stuck when use ExtensibleLoadManager [pulsar]
Demogorgon314 commented on code in PR #21332:
URL: https://github.com/apache/pulsar/pull/21332#discussion_r1351133299
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##
@@ -503,13 +503,22 @@ public CompletableFuture>
getOwnerAsync(String serviceUnit) {
case Splitting -> {
return
CompletableFuture.completedFuture(Optional.of(data.sourceBroker()));
}
-case Assigning, Releasing -> {
+case Assigning -> {
return deferGetOwnerRequest(serviceUnit).whenComplete((__, e)
-> {
if (e != null) {
ownerLookUpCounters.get(state).getFailure().incrementAndGet();
}
-}).thenApply(
-broker -> broker == null ? Optional.empty() :
Optional.of(broker));
+}).thenApply(broker -> broker == null ? Optional.empty() :
Optional.of(broker));
+}
+case Releasing -> {
+if (isTransferCommand(data)) {
+return deferGetOwnerRequest(serviceUnit).whenComplete((__,
e) -> {
+if (e != null) {
+
ownerLookUpCounters.get(state).getFailure().incrementAndGet();
+}
+}).thenApply(broker -> broker == null ? Optional.empty() :
Optional.of(broker));
+}
+return
CompletableFuture.completedFuture(Optional.of(data.sourceBroker()));
Review Comment:
I rechecked the `OwnershipCache` logic, and it will set the bundle state to
inactive, so we should return `empty` here. I changed back.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [fix][broker] Fix unload operation stuck when use ExtensibleLoadManager [pulsar]
heesung-sn commented on code in PR #21332:
URL: https://github.com/apache/pulsar/pull/21332#discussion_r1351128710
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##
@@ -503,13 +503,22 @@ public CompletableFuture>
getOwnerAsync(String serviceUnit) {
case Splitting -> {
return
CompletableFuture.completedFuture(Optional.of(data.sourceBroker()));
}
-case Assigning, Releasing -> {
+case Assigning -> {
return deferGetOwnerRequest(serviceUnit).whenComplete((__, e)
-> {
if (e != null) {
ownerLookUpCounters.get(state).getFailure().incrementAndGet();
}
-}).thenApply(
-broker -> broker == null ? Optional.empty() :
Optional.of(broker));
+}).thenApply(broker -> broker == null ? Optional.empty() :
Optional.of(broker));
+}
+case Releasing -> {
+if (isTransferCommand(data)) {
+return deferGetOwnerRequest(serviceUnit).whenComplete((__,
e) -> {
+if (e != null) {
+
ownerLookUpCounters.get(state).getFailure().incrementAndGet();
+}
+}).thenApply(broker -> broker == null ? Optional.empty() :
Optional.of(broker));
+}
+return
CompletableFuture.completedFuture(Optional.of(data.sourceBroker()));
Review Comment:
I am worried that this might cause some racing conditions (e.g. new pub/subs
are created after closeServiceUnit..)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [improve][offload] Avoid implementation details leak on getOffloadPolicies return value [pulsar]
tisonkun merged PR #21336: URL: https://github.com/apache/pulsar/pull/21336 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch master updated: [improve][offload] Avoid implementation details leak on getOffloadPolicies return value (#21336)
This is an automated email from the ASF dual-hosted git repository.
tison pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 9449ae41c77 [improve][offload] Avoid implementation details leak on
getOffloadPolicies return value (#21336)
9449ae41c77 is described below
commit 9449ae41c77989f509d1b6dacfa2cc1b94d1cf15
Author: tison
AuthorDate: Tue Oct 10 09:14:00 2023 +0800
[improve][offload] Avoid implementation details leak on getOffloadPolicies
return value (#21336)
Signed-off-by: tison
---
.../src/main/java/org/apache/bookkeeper/mledger/LedgerOffloader.java | 4 ++--
.../java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java | 3 +--
.../java/org/apache/bookkeeper/mledger/impl/NullLedgerOffloader.java | 4 ++--
.../offload/filesystem/impl/FileSystemManagedLedgerOffloader.java| 5 +++--
.../mledger/offload/jcloud/impl/BlobStoreManagedLedgerOffloader.java | 3 ++-
5 files changed, 10 insertions(+), 9 deletions(-)
diff --git
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/LedgerOffloader.java
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/LedgerOffloader.java
index b60ae41670d..868a8e42653 100644
---
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/LedgerOffloader.java
+++
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/LedgerOffloader.java
@@ -27,7 +27,7 @@ import org.apache.bookkeeper.client.api.ReadHandle;
import org.apache.bookkeeper.common.annotation.InterfaceAudience;
import org.apache.bookkeeper.common.annotation.InterfaceStability;
import org.apache.bookkeeper.mledger.proto.MLDataFormats;
-import org.apache.pulsar.common.policies.data.OffloadPoliciesImpl;
+import org.apache.pulsar.common.policies.data.OffloadPolicies;
/**
* Interface for offloading ledgers to long-term storage.
@@ -212,7 +212,7 @@ public interface LedgerOffloader {
*
* @return offload policies
*/
-OffloadPoliciesImpl getOffloadPolicies();
+OffloadPolicies getOffloadPolicies();
/**
* Close the resources if necessary.
diff --git
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java
index e011bf3e6d7..cc0020dcd9e 100644
---
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java
+++
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java
@@ -132,7 +132,6 @@ import
org.apache.pulsar.common.api.proto.CommandSubscribe.InitialPosition;
import org.apache.pulsar.common.policies.data.EnsemblePlacementPolicyConfig;
import org.apache.pulsar.common.policies.data.ManagedLedgerInternalStats;
import org.apache.pulsar.common.policies.data.OffloadPolicies;
-import org.apache.pulsar.common.policies.data.OffloadPoliciesImpl;
import org.apache.pulsar.common.policies.data.OffloadedReadPriority;
import org.apache.pulsar.common.policies.data.PersistentTopicInternalStats;
import org.apache.pulsar.common.protocol.Commands;
@@ -2482,7 +2481,7 @@ public class ManagedLedgerImpl implements ManagedLedger,
CreateCallback {
return;
}
-final OffloadPoliciesImpl policies =
config.getLedgerOffloader().getOffloadPolicies();
+final OffloadPolicies policies =
config.getLedgerOffloader().getOffloadPolicies();
final long offloadThresholdInBytes =
Optional.ofNullable(policies.getManagedLedgerOffloadThresholdInBytes()).orElse(-1L);
final long offloadThresholdInSeconds =
diff --git
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/NullLedgerOffloader.java
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/NullLedgerOffloader.java
index 0e5e7cf4b5b..938ceb0c7df 100644
---
a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/NullLedgerOffloader.java
+++
b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/NullLedgerOffloader.java
@@ -23,7 +23,7 @@ import java.util.UUID;
import java.util.concurrent.CompletableFuture;
import org.apache.bookkeeper.client.api.ReadHandle;
import org.apache.bookkeeper.mledger.LedgerOffloader;
-import org.apache.pulsar.common.policies.data.OffloadPoliciesImpl;
+import org.apache.pulsar.common.policies.data.OffloadPolicies;
/**
* Null implementation that throws an error on any invokation.
@@ -62,7 +62,7 @@ public class NullLedgerOffloader implements LedgerOffloader {
}
@Override
-public OffloadPoliciesImpl getOffloadPolicies() {
+public OffloadPolicies getOffloadPolicies() {
return null;
}
diff --git
a/tiered-storage/file-system/src/main/java/org/apache/bookkeeper/mledger/offload/filesystem/impl/FileSystemManagedLedgerOffloader.java
Re: [PR] [fix][broker] Fix unload operation stuck when use ExtensibleLoadManager [pulsar]
Demogorgon314 commented on code in PR #21332:
URL: https://github.com/apache/pulsar/pull/21332#discussion_r1351116375
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##
@@ -503,13 +503,22 @@ public CompletableFuture>
getOwnerAsync(String serviceUnit) {
case Splitting -> {
return
CompletableFuture.completedFuture(Optional.of(data.sourceBroker()));
}
-case Assigning, Releasing -> {
+case Assigning -> {
return deferGetOwnerRequest(serviceUnit).whenComplete((__, e)
-> {
if (e != null) {
ownerLookUpCounters.get(state).getFailure().incrementAndGet();
}
-}).thenApply(
-broker -> broker == null ? Optional.empty() :
Optional.of(broker));
+}).thenApply(broker -> broker == null ? Optional.empty() :
Optional.of(broker));
+}
+case Releasing -> {
+if (isTransferCommand(data)) {
+return deferGetOwnerRequest(serviceUnit).whenComplete((__,
e) -> {
+if (e != null) {
+
ownerLookUpCounters.get(state).getFailure().incrementAndGet();
+}
+}).thenApply(broker -> broker == null ? Optional.empty() :
Optional.of(broker));
+}
+return
CompletableFuture.completedFuture(Optional.of(data.sourceBroker()));
Review Comment:
@heesung-sn It should return the current owner when it is in the `Releasing`
state, WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [fix][broker] Fix unload operation stuck when use ExtensibleLoadManager [pulsar]
Demogorgon314 commented on code in PR #21332:
URL: https://github.com/apache/pulsar/pull/21332#discussion_r1351110853
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##
@@ -503,13 +503,22 @@ public CompletableFuture>
getOwnerAsync(String serviceUnit) {
case Splitting -> {
return
CompletableFuture.completedFuture(Optional.of(data.sourceBroker()));
}
-case Assigning, Releasing -> {
+case Assigning -> {
return deferGetOwnerRequest(serviceUnit).whenComplete((__, e)
-> {
if (e != null) {
ownerLookUpCounters.get(state).getFailure().incrementAndGet();
}
-}).thenApply(
-broker -> broker == null ? Optional.empty() :
Optional.of(broker));
+}).thenApply(broker -> broker == null ? Optional.empty() :
Optional.of(broker));
+}
+case Releasing -> {
+if (isTransferCommand(data)) {
+return deferGetOwnerRequest(serviceUnit).whenComplete((__,
e) -> {
+if (e != null) {
+
ownerLookUpCounters.get(state).getFailure().incrementAndGet();
+}
+}).thenApply(broker -> broker == null ? Optional.empty() :
Optional.of(broker));
+}
+return CompletableFuture.completedFuture(Optional.empty());
Review Comment:
The monitor will clean up this state, but we should unblock it for admin API
since we don't want the admin API timeout.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [I] [Bug] Track upstream release of curl for security purposes [pulsar-client-cpp]
merlimat commented on issue #327: URL: https://github.com/apache/pulsar-client-cpp/issues/327#issuecomment-1754122146 Thanks for heads-up. We'll create a new release once the Curl release is available. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[I] [Bug] Track upstream release of curl for security purposes [pulsar-client-cpp]
bureado opened a new issue, #327: URL: https://github.com/apache/pulsar-client-cpp/issues/327 ### Search before asking - [X] I searched in the [issues](https://github.com/apache/pulsar-client-cpp/issues) and found nothing similar. ### Version Current main ### Minimal reproduce step From a dry read of https://github.com/apache/pulsar-client-cpp/blob/5c77648d0029c4b11e0e062ab73474368ecc0e87/pkg/apk/Dockerfile#L105, a hardcoded version of `curl`, currently 7.85.0 per https://github.com/apache/pulsar-client-cpp/blob/5c77648d0029c4b11e0e062ab73474368ecc0e87/dependencies.yaml#L27, will make it into a container image used in the build process. ### What did you expect to see? I'm raising this to your attention due to https://github.com/curl/curl/discussions/12026 in case that's a release that you find necessary to track for security purposes. ### What did you see instead? It appears like using a package manager to use the latest libcurl from an OS vendor will not suffice in this scenario since https://github.com/apache/pulsar-client-cpp/blob/5c77648d0029c4b11e0e062ab73474368ecc0e87/dependencies.yaml#L27 is looked up by a helper script. ### Anything else? _No response_ ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch master updated: [fix][bk] Fix the var name for IsolationGroups (#21320)
This is an automated email from the ASF dual-hosted git repository.
houxiaoyu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new fcc123d77c3 [fix][bk] Fix the var name for IsolationGroups (#21320)
fcc123d77c3 is described below
commit fcc123d77c34c1fcd681bea57993c64eaf77dbb6
Author: houxiaoyu
AuthorDate: Tue Oct 10 06:55:35 2023 +0800
[fix][bk] Fix the var name for IsolationGroups (#21320)
---
.../bookie/rackawareness/IsolatedBookieEnsemblePlacementPolicy.java | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git
a/pulsar-broker-common/src/main/java/org/apache/pulsar/bookie/rackawareness/IsolatedBookieEnsemblePlacementPolicy.java
b/pulsar-broker-common/src/main/java/org/apache/pulsar/bookie/rackawareness/IsolatedBookieEnsemblePlacementPolicy.java
index 02ddea94874..8839e6e2d26 100644
---
a/pulsar-broker-common/src/main/java/org/apache/pulsar/bookie/rackawareness/IsolatedBookieEnsemblePlacementPolicy.java
+++
b/pulsar-broker-common/src/main/java/org/apache/pulsar/bookie/rackawareness/IsolatedBookieEnsemblePlacementPolicy.java
@@ -205,7 +205,7 @@ public class IsolatedBookieEnsemblePlacementPolicy extends
RackawareEnsemblePlac
log.debug("The bookies rack config is not available at
now.");
return excludedBookies;
}
-Set allBookies = allGroupsBookieMapping.keySet();
+Set allGroups = allGroupsBookieMapping.keySet();
int totalAvailableBookiesInPrimaryGroup = 0;
Set primaryIsolationGroup = Collections.emptySet();
Set secondaryIsolationGroup = Collections.emptySet();
@@ -214,7 +214,7 @@ public class IsolatedBookieEnsemblePlacementPolicy extends
RackawareEnsemblePlac
primaryIsolationGroup = isolationGroups.getLeft();
secondaryIsolationGroup = isolationGroups.getRight();
}
-for (String group : allBookies) {
+for (String group : allGroups) {
Set bookiesInGroup =
allGroupsBookieMapping.get(group).keySet();
if (!primaryIsolationGroup.contains(group)) {
for (String bookieAddress : bookiesInGroup) {
Re: [PR] [fix][bk] Fix the var name for IsolationGroups [pulsar]
AnonHxy merged PR #21320: URL: https://github.com/apache/pulsar/pull/21320 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch master updated: [fix][test] Fix flaky OneWayReplicatorTest.testReplicatorProducerStatInTopic (#21334)
This is an automated email from the ASF dual-hosted git repository.
tison pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 16199d2d155 [fix][test] Fix flaky
OneWayReplicatorTest.testReplicatorProducerStatInTopic (#21334)
16199d2d155 is described below
commit 16199d2d15556ee5a19044eb739d88a01b1573e5
Author: Lari Hotari
AuthorDate: Tue Oct 10 01:39:58 2023 +0300
[fix][test] Fix flaky
OneWayReplicatorTest.testReplicatorProducerStatInTopic (#21334)
---
.../java/org/apache/pulsar/broker/service/OneWayReplicatorTest.java | 6 --
1 file changed, 6 deletions(-)
diff --git
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/OneWayReplicatorTest.java
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/OneWayReplicatorTest.java
index e068cd55e90..8269f40e608 100644
---
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/OneWayReplicatorTest.java
+++
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/OneWayReplicatorTest.java
@@ -26,7 +26,6 @@ import java.util.concurrent.TimeUnit;
import org.apache.pulsar.broker.BrokerTestUtil;
import org.apache.pulsar.broker.service.persistent.PersistentTopic;
import org.apache.pulsar.client.api.Consumer;
-import org.apache.pulsar.client.api.MessageId;
import org.apache.pulsar.client.api.Producer;
import org.apache.pulsar.client.api.Schema;
import org.apache.pulsar.common.policies.data.TopicStats;
@@ -56,11 +55,6 @@ public class OneWayReplicatorTest extends
OneWayReplicatorTestBase {
final String subscribeName = "subscribe_1";
final byte[] msgValue = "test".getBytes();
-admin1.topics().createNonPartitionedTopic(topicName);
-admin2.topics().createNonPartitionedTopic(topicName);
-admin1.topics().createSubscription(topicName, subscribeName,
MessageId.earliest);
-admin2.topics().createSubscription(topicName, subscribeName,
MessageId.earliest);
-
// Verify replicator works.
Producer producer1 =
client1.newProducer().topic(topicName).create();
Consumer consumer2 =
client2.newConsumer().topic(topicName).subscriptionName(subscribeName).subscribe();
Re: [I] Flaky-test: OneWayReplicatorTest.testReplicatorProducerStatInTopic [pulsar]
tisonkun closed issue #21331: Flaky-test: OneWayReplicatorTest.testReplicatorProducerStatInTopic URL: https://github.com/apache/pulsar/issues/21331 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][test] Fix flaky OneWayReplicatorTest.testReplicatorProducerStatInTopic [pulsar]
tisonkun merged PR #21334: URL: https://github.com/apache/pulsar/pull/21334 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [PIP-264] OTel should support pushdown predicate [pulsar]
asafm commented on issue #21123: URL: https://github.com/apache/pulsar/issues/21123#issuecomment-1753679951 Still working on the PR to be approved: https://github.com/open-telemetry/opentelemetry-specification/pull/3566 Some progress have been made. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [PIP-264] Messaging metrics work in OTel for 10k topics per broker [pulsar]
asafm commented on issue #21124: URL: https://github.com/apache/pulsar/issues/21124#issuecomment-1753681393 This is an on-going issues, no need for stale label here -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][broker] Fix unload operation stuck when use ExtensibleLoadManager [pulsar]
heesung-sn commented on code in PR #21332:
URL: https://github.com/apache/pulsar/pull/21332#discussion_r1350688142
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##
@@ -503,13 +503,22 @@ public CompletableFuture>
getOwnerAsync(String serviceUnit) {
case Splitting -> {
return
CompletableFuture.completedFuture(Optional.of(data.sourceBroker()));
}
-case Assigning, Releasing -> {
+case Assigning -> {
return deferGetOwnerRequest(serviceUnit).whenComplete((__, e)
-> {
if (e != null) {
ownerLookUpCounters.get(state).getFailure().incrementAndGet();
}
-}).thenApply(
-broker -> broker == null ? Optional.empty() :
Optional.of(broker));
+}).thenApply(broker -> broker == null ? Optional.empty() :
Optional.of(broker));
+}
+case Releasing -> {
+if (isTransferCommand(data)) {
+return deferGetOwnerRequest(serviceUnit).whenComplete((__,
e) -> {
+if (e != null) {
+
ownerLookUpCounters.get(state).getFailure().incrementAndGet();
+}
+}).thenApply(broker -> broker == null ? Optional.empty() :
Optional.of(broker));
+}
+return CompletableFuture.completedFuture(Optional.empty());
Review Comment:
Also, we have the leader broker who should watch any stuck states and fix
the stuck state. Do we know why the monitor didn't kick in this case and
didin't complete the future(maybe because of the delay)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [I] Unable to import pulsar admin Packages in Go [pulsar-client-go]
jichmi commented on issue #1093: URL: https://github.com/apache/pulsar-client-go/issues/1093#issuecomment-1753453633 Look like pulsaradmin is not released at v0.11.0 https://github.com/apache/pulsar-client-go/tree/branch-0.11.0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
svn commit: r64431 - /dev/pulsar/pulsar-client-reactive-0.4.0-candidate-1/ /release/pulsar/pulsar-client-reactive-0.4.0/
Author: eolivelli Date: Mon Oct 9 17:42:20 2023 New Revision: 64431 Log: Release Reactive client for Apache Pulsar 0.4.0 Added: release/pulsar/pulsar-client-reactive-0.4.0/ - copied from r64430, dev/pulsar/pulsar-client-reactive-0.4.0-candidate-1/ Removed: dev/pulsar/pulsar-client-reactive-0.4.0-candidate-1/
[pulsar-client-reactive] annotated tag v0.4.0 updated (01bf7dc -> 7d445b8)
This is an automated email from the ASF dual-hosted git repository. cbornet pushed a change to annotated tag v0.4.0 in repository https://gitbox.apache.org/repos/asf/pulsar-client-reactive.git *** WARNING: tag v0.4.0 was modified! *** from 01bf7dc (commit) to 7d445b8 (tag) tagging 01bf7dcf9a7379e4b325c2f6d0d43ec6c451a670 (commit) replaces v0.2.0 by Christophe Bornet on Mon Oct 9 14:20:26 2023 -0300 - Log - Release v0.4.0 -BEGIN PGP SIGNATURE- iIkEABYKADEWIQSsJtR8kOYvBGFVou/hzYcUAI4EngUCZSQ2WxMcY2Jvcm5ldEBh cGFjaGUub3JnAAoJEOHNhxQAjgSeWM4BAIEUPTXg6NqKHeQYOq1xR9Mo5M/ztabm gVQs/cdRNL1NAP9VoVIlZwMSwRzlkrHgZXvszxq1KJiHeOOZ2h5mrc2QBQ== =tmnD -END PGP SIGNATURE- --- No new revisions were added by this update. Summary of changes:
Re: [PR] [fix][test] Fix flaky OneWayReplicatorTest.testReplicatorProducerStatInTopic [pulsar]
lhotari commented on PR #21334: URL: https://github.com/apache/pulsar/pull/21334#issuecomment-1753310258 @tisonkun PTAL -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][test] Fix flaky OneWayReplicatorTest.testReplicatorProducerStatInTopic [pulsar]
lhotari commented on code in PR #21334:
URL: https://github.com/apache/pulsar/pull/21334#discussion_r1350516233
##
pulsar-broker/src/test/java/org/apache/pulsar/broker/service/OneWayReplicatorTest.java:
##
@@ -57,7 +58,11 @@ public void testReplicatorProducerStatInTopic() throws
Exception {
final byte[] msgValue = "test".getBytes();
admin1.topics().createNonPartitionedTopic(topicName);
-admin2.topics().createNonPartitionedTopic(topicName);
+try {
+admin2.topics().createNonPartitionedTopic(topicName);
+} catch (PulsarAdminException.ConflictException e) {
+// Ignore
+}
Review Comment:
@tisonkun You are right that test code could be improved. I happened to
focus on addressing the flakiness without changing the test logic. I'll remove
the creation of topics and subscriptions completely since it's not needed in
the test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [fix][broker] Fix unload operation stuck when use ExtensibleLoadManager [pulsar]
heesung-sn commented on code in PR #21332:
URL: https://github.com/apache/pulsar/pull/21332#discussion_r1350499616
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##
@@ -503,13 +503,22 @@ public CompletableFuture>
getOwnerAsync(String serviceUnit) {
case Splitting -> {
return
CompletableFuture.completedFuture(Optional.of(data.sourceBroker()));
}
-case Assigning, Releasing -> {
+case Assigning -> {
return deferGetOwnerRequest(serviceUnit).whenComplete((__, e)
-> {
if (e != null) {
ownerLookUpCounters.get(state).getFailure().incrementAndGet();
}
-}).thenApply(
-broker -> broker == null ? Optional.empty() :
Optional.of(broker));
+}).thenApply(broker -> broker == null ? Optional.empty() :
Optional.of(broker));
+}
+case Releasing -> {
+if (isTransferCommand(data)) {
+return deferGetOwnerRequest(serviceUnit).whenComplete((__,
e) -> {
+if (e != null) {
+
ownerLookUpCounters.get(state).getFailure().incrementAndGet();
+}
+}).thenApply(broker -> broker == null ? Optional.empty() :
Optional.of(broker));
+}
+return CompletableFuture.completedFuture(Optional.empty());
Review Comment:
What happens if there are many lookups at this Releasing state(non-transfer)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [fix] [metadata] Fix zookeeper related flacky test [pulsar]
lhotari commented on PR #21310: URL: https://github.com/apache/pulsar/pull/21310#issuecomment-1753237010 This will help reduce flakiness a lot. Please cherry-pick also to branch-3.1 since that will be maintained for some time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][test] Fix flaky OneWayReplicatorTest.testReplicatorProducerStatInTopic [pulsar]
tisonkun commented on code in PR #21334:
URL: https://github.com/apache/pulsar/pull/21334#discussion_r1350462143
##
pulsar-broker/src/test/java/org/apache/pulsar/broker/service/OneWayReplicatorTest.java:
##
@@ -57,7 +58,11 @@ public void testReplicatorProducerStatInTopic() throws
Exception {
final byte[] msgValue = "test".getBytes();
admin1.topics().createNonPartitionedTopic(topicName);
-admin2.topics().createNonPartitionedTopic(topicName);
+try {
+admin2.topics().createNonPartitionedTopic(topicName);
+} catch (PulsarAdminException.ConflictException e) {
+// Ignore
+}
Review Comment:
Why do we need to create the same topic twice in the first place? I'm afraid
this monkey patch break the manner we want to verify, but it's more likely we
don't even need to create the topic and the subscription twice?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [fix][test] Fix flaky OneWayReplicatorTest.testReplicatorProducerStatInTopic [pulsar]
tisonkun commented on code in PR #21334:
URL: https://github.com/apache/pulsar/pull/21334#discussion_r1350462143
##
pulsar-broker/src/test/java/org/apache/pulsar/broker/service/OneWayReplicatorTest.java:
##
@@ -57,7 +58,11 @@ public void testReplicatorProducerStatInTopic() throws
Exception {
final byte[] msgValue = "test".getBytes();
admin1.topics().createNonPartitionedTopic(topicName);
-admin2.topics().createNonPartitionedTopic(topicName);
+try {
+admin2.topics().createNonPartitionedTopic(topicName);
+} catch (PulsarAdminException.ConflictException e) {
+// Ignore
+}
Review Comment:
Why do we need to create the same topic twice in the first place? I'm afraid
this monkey patch break the manner we want to verify, but it's more likely we
don't event need to create the topic and the subscription twice?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [fix] [bk-client] Fix bk client MinNumRacksPerWriteQuorum and EnforceMinNumRacksPerWriteQuorum not work problem. [pulsar]
horizonzy commented on PR #21327: URL: https://github.com/apache/pulsar/pull/21327#issuecomment-1753212143 /pulsarbot rerun-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix] [auto-recovery] [branch-2.11] Fix PulsarLedgerUnderreplicationManager notify problem. [pulsar]
horizonzy commented on PR #21318: URL: https://github.com/apache/pulsar/pull/21318#issuecomment-1753211516 /pulsarbot rerun-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch branch-3.1 updated: [fix][test] Fix flaky CompactionTest.testDispatcherMaxReadSizeBytes (#21329)
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new 8e3f68f1cba [fix][test] Fix flaky
CompactionTest.testDispatcherMaxReadSizeBytes (#21329)
8e3f68f1cba is described below
commit 8e3f68f1cba1b2a06b856926be1ddf684fd4f91a
Author: Lari Hotari
AuthorDate: Mon Oct 9 17:48:38 2023 +0300
[fix][test] Fix flaky CompactionTest.testDispatcherMaxReadSizeBytes (#21329)
(cherry picked from commit c883f50e117ff9da310c879aa048993c28ea955a)
---
.../java/org/apache/pulsar/compaction/CompactionTest.java | 13 ++---
1 file changed, 6 insertions(+), 7 deletions(-)
diff --git
a/pulsar-broker/src/test/java/org/apache/pulsar/compaction/CompactionTest.java
b/pulsar-broker/src/test/java/org/apache/pulsar/compaction/CompactionTest.java
index 4c6db644f1e..52837cbdcd5 100644
---
a/pulsar-broker/src/test/java/org/apache/pulsar/compaction/CompactionTest.java
+++
b/pulsar-broker/src/test/java/org/apache/pulsar/compaction/CompactionTest.java
@@ -1898,22 +1898,21 @@ public class CompactionTest extends
MockedPulsarServiceBaseTest {
admin.topics().unload(topicName);
-ConsumerImpl consumer = (ConsumerImpl)
client.newConsumer(Schema.BYTES)
-
.topic(topicName).readCompacted(true).receiverQueueSize(receiveQueueSize).subscriptionName(subName)
-.subscribe();
-
-PersistentTopic topic = (PersistentTopic)
pulsar.getBrokerService().getTopicReference(topicName).get();
+PersistentTopic topic =
+(PersistentTopic)
pulsar.getBrokerService().getTopic(topicName, true, Map.of()).get().get();
TopicCompactionService topicCompactionService =
Mockito.spy(topic.getTopicCompactionService());
FieldUtils.writeDeclaredField(topic, "topicCompactionService",
topicCompactionService, true);
+ConsumerImpl consumer = (ConsumerImpl)
client.newConsumer(Schema.BYTES)
+
.topic(topicName).readCompacted(true).receiverQueueSize(receiveQueueSize).subscriptionName(subName)
+.subscribe();
+
Awaitility.await().untilAsserted(() -> {
assertEquals(consumer.getStats().getMsgNumInReceiverQueue(),
1);
});
-consumer.increaseAvailablePermits(2);
-
Mockito.verify(topicCompactionService,
Mockito.times(1)).readCompactedEntries(Mockito.any(), Mockito.same(1));
consumer.close();
[PR] [improve][offload] Avoid implementation details leak on getOffloadPolicies return value [pulsar]
tisonkun opened a new pull request, #21336: URL: https://github.com/apache/pulsar/pull/21336 ### Motivation `getOffloadPolicies` should be better to return `OffloadPolicies` instead of `OffloadPoliciesImpl`. ### Modifications Change `getOffloadPolicies` return value type. This is a breaking change while the interface is marked as `Evolving`. I'd prefer to treat it as a mistake or tech debt to pay back. Implementations of `LedgerOffloader` should not be affected because Java's inheritance allows refinement. But if the application code receive the result of `getOffloadPolicies` as `OffloadPoliciesImpl`, they may now need a coercion. ### Verifying this change - [x] Make sure that the change passes the CI checks. ### Does this pull request potentially affect one of the following parts: *If the box was checked, please highlight the changes* - [ ] Dependencies (add or upgrade a dependency) - [x] The public API - [ ] The schema - [ ] The default values of configurations - [ ] The threading model - [ ] The binary protocol - [ ] The REST endpoints - [ ] The admin CLI options - [ ] The metrics - [ ] Anything that affects deployment ### Documentation - [ ] `doc` - [ ] `doc-required` - [x] `doc-not-needed` - [ ] `doc-complete` ### Matching PR in forked repository PR in forked repository: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Flaky-test: StrategicCompactionTest.testDispatcherMaxReadSizeBytes / CompactionTest testDispatcherMaxReadSizeBytes [pulsar]
lhotari closed issue #21284: Flaky-test: StrategicCompactionTest.testDispatcherMaxReadSizeBytes / CompactionTest testDispatcherMaxReadSizeBytes URL: https://github.com/apache/pulsar/issues/21284 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch master updated: [fix][test] Fix flaky CompactionTest.testDispatcherMaxReadSizeBytes (#21329)
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new c883f50e117 [fix][test] Fix flaky
CompactionTest.testDispatcherMaxReadSizeBytes (#21329)
c883f50e117 is described below
commit c883f50e117ff9da310c879aa048993c28ea955a
Author: Lari Hotari
AuthorDate: Mon Oct 9 17:48:38 2023 +0300
[fix][test] Fix flaky CompactionTest.testDispatcherMaxReadSizeBytes (#21329)
---
.../java/org/apache/pulsar/compaction/CompactionTest.java | 13 ++---
1 file changed, 6 insertions(+), 7 deletions(-)
diff --git
a/pulsar-broker/src/test/java/org/apache/pulsar/compaction/CompactionTest.java
b/pulsar-broker/src/test/java/org/apache/pulsar/compaction/CompactionTest.java
index 4c6db644f1e..52837cbdcd5 100644
---
a/pulsar-broker/src/test/java/org/apache/pulsar/compaction/CompactionTest.java
+++
b/pulsar-broker/src/test/java/org/apache/pulsar/compaction/CompactionTest.java
@@ -1898,22 +1898,21 @@ public class CompactionTest extends
MockedPulsarServiceBaseTest {
admin.topics().unload(topicName);
-ConsumerImpl consumer = (ConsumerImpl)
client.newConsumer(Schema.BYTES)
-
.topic(topicName).readCompacted(true).receiverQueueSize(receiveQueueSize).subscriptionName(subName)
-.subscribe();
-
-PersistentTopic topic = (PersistentTopic)
pulsar.getBrokerService().getTopicReference(topicName).get();
+PersistentTopic topic =
+(PersistentTopic)
pulsar.getBrokerService().getTopic(topicName, true, Map.of()).get().get();
TopicCompactionService topicCompactionService =
Mockito.spy(topic.getTopicCompactionService());
FieldUtils.writeDeclaredField(topic, "topicCompactionService",
topicCompactionService, true);
+ConsumerImpl consumer = (ConsumerImpl)
client.newConsumer(Schema.BYTES)
+
.topic(topicName).readCompacted(true).receiverQueueSize(receiveQueueSize).subscriptionName(subName)
+.subscribe();
+
Awaitility.await().untilAsserted(() -> {
assertEquals(consumer.getStats().getMsgNumInReceiverQueue(),
1);
});
-consumer.increaseAvailablePermits(2);
-
Mockito.verify(topicCompactionService,
Mockito.times(1)).readCompactedEntries(Mockito.any(), Mockito.same(1));
consumer.close();
Re: [PR] [fix][test] Fix flaky CompactionTest.testDispatcherMaxReadSizeBytes [pulsar]
lhotari merged PR #21329: URL: https://github.com/apache/pulsar/pull/21329 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][ml] Make mlOwnershipChecker asynchronous so that it doesn't block/deadlock threads [pulsar]
Demogorgon314 commented on PR #21333: URL: https://github.com/apache/pulsar/pull/21333#issuecomment-1753117697 @lhotari I think we still need #21332 because it is another issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [improve][client] Add a way to configure which DNS use [pulsar]
diegosalvi commented on PR #21227: URL: https://github.com/apache/pulsar/pull/21227#issuecomment-1753079548 Rebased again -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [improve][client] Add a way to configure which DNS use [pulsar]
lhotari commented on PR #21227: URL: https://github.com/apache/pulsar/pull/21227#issuecomment-1753058515 > Still fails for NarUnpackerTest @diegosalvi This flaky NarUnpackerTest has been fixed. You can get the fix to this PR by rebasing on or merging origin/master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch master updated: Revert "[fix][broker] Miss headersAndPayload and messageIdData in MessagePublishContext (#21245) (#21309)
This is an automated email from the ASF dual-hosted git repository.
zixuan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 66c5e1f1ae4 Revert "[fix][broker] Miss headersAndPayload and
messageIdData in MessagePublishContext (#21245) (#21309)
66c5e1f1ae4 is described below
commit 66c5e1f1ae48e49ed27a72424c372ce444217d86
Author: Penghui Li
AuthorDate: Mon Oct 9 21:48:27 2023 +0800
Revert "[fix][broker] Miss headersAndPayload and messageIdData in
MessagePublishContext (#21245) (#21309)
---
.../org/apache/pulsar/broker/service/Producer.java | 73 --
.../apache/pulsar/broker/service/ServerCnx.java| 8 ++-
.../org/apache/pulsar/broker/service/Topic.java| 9 ---
3 files changed, 30 insertions(+), 60 deletions(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Producer.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Producer.java
index 70cca8b8212..f7d2bb2dd27 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Producer.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Producer.java
@@ -46,7 +46,6 @@ import
org.apache.pulsar.broker.service.nonpersistent.NonPersistentTopic;
import org.apache.pulsar.broker.service.persistent.PersistentTopic;
import org.apache.pulsar.client.api.transaction.TxnID;
import org.apache.pulsar.common.api.proto.CommandTopicMigrated.ResourceType;
-import org.apache.pulsar.common.api.proto.MessageIdData;
import org.apache.pulsar.common.api.proto.MessageMetadata;
import org.apache.pulsar.common.api.proto.ProducerAccessMode;
import org.apache.pulsar.common.api.proto.ServerError;
@@ -187,15 +186,14 @@ public class Producer {
}
public void publishMessage(long producerId, long sequenceId, ByteBuf
headersAndPayload, long batchSize,
-boolean isChunked, boolean isMarker, MessageIdData messageIdData) {
-if (checkAndStartPublish(producerId, sequenceId, headersAndPayload,
batchSize, messageIdData)) {
-publishMessageToTopic(headersAndPayload, sequenceId, batchSize,
isChunked, isMarker, messageIdData);
+boolean isChunked, boolean isMarker, Position position) {
+if (checkAndStartPublish(producerId, sequenceId, headersAndPayload,
batchSize, position)) {
+publishMessageToTopic(headersAndPayload, sequenceId, batchSize,
isChunked, isMarker, position);
}
}
public void publishMessage(long producerId, long lowestSequenceId, long
highestSequenceId,
- ByteBuf headersAndPayload, long batchSize,
boolean isChunked, boolean isMarker,
- MessageIdData messageIdData) {
+ByteBuf headersAndPayload, long batchSize, boolean isChunked,
boolean isMarker, Position position) {
if (lowestSequenceId > highestSequenceId) {
cnx.execute(() -> {
cnx.getCommandSender().sendSendError(producerId,
highestSequenceId, ServerError.MetadataError,
@@ -204,15 +202,15 @@ public class Producer {
});
return;
}
-if (checkAndStartPublish(producerId, highestSequenceId,
headersAndPayload, batchSize, messageIdData)) {
+if (checkAndStartPublish(producerId, highestSequenceId,
headersAndPayload, batchSize, position)) {
publishMessageToTopic(headersAndPayload, lowestSequenceId,
highestSequenceId, batchSize, isChunked,
-isMarker, messageIdData);
+isMarker, position);
}
}
public boolean checkAndStartPublish(long producerId, long sequenceId,
ByteBuf headersAndPayload, long batchSize,
-MessageIdData messageIdData) {
-if (!isShadowTopic && messageIdData != null) {
+Position position) {
+if (!isShadowTopic && position != null) {
cnx.execute(() -> {
cnx.getCommandSender().sendSendError(producerId, sequenceId,
ServerError.NotAllowedError,
"Only shadow topic supports sending messages with
messageId");
@@ -220,7 +218,7 @@ public class Producer {
});
return false;
}
-if (isShadowTopic && messageIdData == null) {
+if (isShadowTopic && position == null) {
cnx.execute(() -> {
cnx.getCommandSender().sendSendError(producerId, sequenceId,
ServerError.NotAllowedError,
"Cannot send messages to a shadow topic");
@@ -269,10 +267,10 @@ public class Producer {
}
private void publishMessageToTopic(ByteBuf headersAndPayload, long
sequenceId, long batchSize, boolean isChunked,
- boolean isMarker, MessageIdData
messageIdData) {
+ boolean
Re: [PR] Revert "[fix][broker] Miss headersAndPayload and messageIdData in MessagePublishContext (#21245) [pulsar]
nodece merged PR #21309: URL: https://github.com/apache/pulsar/pull/21309 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [fix][test] Fix flaky AdminApiMaxUnackedMessagesTest.testMaxUnackedMessagesPerConsumerPriority [pulsar]
lhotari opened a new pull request, #21335: URL: https://github.com/apache/pulsar/pull/21335 Fixes #21330 ### Motivation AdminApiMaxUnackedMessagesTest.testMaxUnackedMessagesPerConsumerPriority is flaky. ### Modifications Increase receive timeout to reduce flakiness in AdminApiMaxUnackedMessagesTest. ### Documentation - [ ] `doc` - [ ] `doc-required` - [x] `doc-not-needed` - [ ] `doc-complete` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [fix][test] Fix flaky OneWayReplicatorTest.testReplicatorProducerStatInTopic [pulsar]
lhotari opened a new pull request, #21334: URL: https://github.com/apache/pulsar/pull/21334 Fixes #21331 ### Motivation OneWayReplicatorTest.testReplicatorProducerStatInTopic is flaky ### Modifications Catch PulsarAdminException.ConflictException since there's a race condition in the test code. The replicator will create the topic concurrently and that's why the test is flaky. ### Documentation - [ ] `doc` - [ ] `doc-required` - [x] `doc-not-needed` - [ ] `doc-complete` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch branch-2.8 updated: [fix][test] Fix flaky test NarUnpackerTest (#21328)
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-2.8
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-2.8 by this push:
new 6adc1205e19 [fix][test] Fix flaky test NarUnpackerTest (#21328)
6adc1205e19 is described below
commit 6adc1205e199ea72ad3879468076245a0325cee4
Author: Lari Hotari
AuthorDate: Mon Oct 9 16:26:25 2023 +0300
[fix][test] Fix flaky test NarUnpackerTest (#21328)
(cherry picked from commit e76a86e3cd1c362e9daa1c88eb8b888e6ab38ab4)
# Conflicts:
#
pulsar-common/src/test/java/org/apache/pulsar/common/nar/NarUnpackerTest.java
---
.../apache/pulsar/common/nar/NarUnpackerTest.java | 27 +++---
1 file changed, 19 insertions(+), 8 deletions(-)
diff --git
a/pulsar-common/src/test/java/org/apache/pulsar/common/nar/NarUnpackerTest.java
b/pulsar-common/src/test/java/org/apache/pulsar/common/nar/NarUnpackerTest.java
index 1d2acd0fa4e..8e7e38dea68 100644
---
a/pulsar-common/src/test/java/org/apache/pulsar/common/nar/NarUnpackerTest.java
+++
b/pulsar-common/src/test/java/org/apache/pulsar/common/nar/NarUnpackerTest.java
@@ -46,7 +46,7 @@ public class NarUnpackerTest {
public void createSampleZipFile() throws IOException {
sampleZipFile = Files.createTempFile("sample", ".zip").toFile();
try (ZipOutputStream out = new ZipOutputStream(new
FileOutputStream(sampleZipFile))) {
-for (int i = 0; i < 1; i++) {
+for (int i = 0; i < 5000; i++) {
ZipEntry e = new ZipEntry("hello" + i + ".txt");
out.putNextEntry(e);
byte[] msg = "hello world!".getBytes(StandardCharsets.UTF_8);
@@ -58,12 +58,20 @@ public class NarUnpackerTest {
}
@AfterMethod(alwaysRun = true)
-void deleteSampleZipFile() throws IOException {
-if (sampleZipFile != null) {
-sampleZipFile.delete();
+void deleteSampleZipFile() {
+if (sampleZipFile != null && sampleZipFile.exists()) {
+try {
+sampleZipFile.delete();
+} catch (Exception e) {
+log.warn("Failed to delete file {}", sampleZipFile, e);
+}
}
-if (extractDirectory != null) {
-FileUtils.deleteFile(extractDirectory, true);
+if (extractDirectory != null && extractDirectory.exists()) {
+try {
+FileUtils.deleteFile(extractDirectory, true);
+} catch (IOException e) {
+log.warn("Failed to delete directory {}", extractDirectory, e);
+}
}
}
@@ -111,7 +119,7 @@ public class NarUnpackerTest {
@Test
void shouldExtractFilesOnceInDifferentProcess() throws
InterruptedException {
-int processes = 10;
+int processes = 5;
String javaExePath = findJavaExe().getAbsolutePath();
CountDownLatch countDownLatch = new CountDownLatch(processes);
AtomicInteger exceptionCounter = new AtomicInteger();
@@ -122,7 +130,9 @@ public class NarUnpackerTest {
// fork a new process with the same classpath
Process process = new ProcessBuilder()
.command(javaExePath,
-"-Xmx64m",
+"-Xmx96m",
+"-XX:TieredStopAtLevel=1",
+"-Dlog4j2.disable.jmx=true",
"-cp",
System.getProperty("java.class.path"),
// use NarUnpackerWorker as the main class
@@ -130,6 +140,7 @@ public class NarUnpackerTest {
// pass arguments to use for testing
sampleZipFile.getAbsolutePath(),
extractDirectory.getAbsolutePath())
+.redirectErrorStream(true)
.start();
String output = IOUtils.toString(process.getInputStream(),
StandardCharsets.UTF_8);
int retval = process.waitFor();
[pulsar] branch branch-2.9 updated: [fix][test] Fix flaky test NarUnpackerTest (#21328)
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-2.9
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-2.9 by this push:
new a9a1694a202 [fix][test] Fix flaky test NarUnpackerTest (#21328)
a9a1694a202 is described below
commit a9a1694a202d342e424202acd356c3da9db4fd76
Author: Lari Hotari
AuthorDate: Mon Oct 9 16:26:25 2023 +0300
[fix][test] Fix flaky test NarUnpackerTest (#21328)
(cherry picked from commit e76a86e3cd1c362e9daa1c88eb8b888e6ab38ab4)
# Conflicts:
#
pulsar-common/src/test/java/org/apache/pulsar/common/nar/NarUnpackerTest.java
---
.../apache/pulsar/common/nar/NarUnpackerTest.java | 27 +++---
1 file changed, 19 insertions(+), 8 deletions(-)
diff --git
a/pulsar-common/src/test/java/org/apache/pulsar/common/nar/NarUnpackerTest.java
b/pulsar-common/src/test/java/org/apache/pulsar/common/nar/NarUnpackerTest.java
index 1d2acd0fa4e..8e7e38dea68 100644
---
a/pulsar-common/src/test/java/org/apache/pulsar/common/nar/NarUnpackerTest.java
+++
b/pulsar-common/src/test/java/org/apache/pulsar/common/nar/NarUnpackerTest.java
@@ -46,7 +46,7 @@ public class NarUnpackerTest {
public void createSampleZipFile() throws IOException {
sampleZipFile = Files.createTempFile("sample", ".zip").toFile();
try (ZipOutputStream out = new ZipOutputStream(new
FileOutputStream(sampleZipFile))) {
-for (int i = 0; i < 1; i++) {
+for (int i = 0; i < 5000; i++) {
ZipEntry e = new ZipEntry("hello" + i + ".txt");
out.putNextEntry(e);
byte[] msg = "hello world!".getBytes(StandardCharsets.UTF_8);
@@ -58,12 +58,20 @@ public class NarUnpackerTest {
}
@AfterMethod(alwaysRun = true)
-void deleteSampleZipFile() throws IOException {
-if (sampleZipFile != null) {
-sampleZipFile.delete();
+void deleteSampleZipFile() {
+if (sampleZipFile != null && sampleZipFile.exists()) {
+try {
+sampleZipFile.delete();
+} catch (Exception e) {
+log.warn("Failed to delete file {}", sampleZipFile, e);
+}
}
-if (extractDirectory != null) {
-FileUtils.deleteFile(extractDirectory, true);
+if (extractDirectory != null && extractDirectory.exists()) {
+try {
+FileUtils.deleteFile(extractDirectory, true);
+} catch (IOException e) {
+log.warn("Failed to delete directory {}", extractDirectory, e);
+}
}
}
@@ -111,7 +119,7 @@ public class NarUnpackerTest {
@Test
void shouldExtractFilesOnceInDifferentProcess() throws
InterruptedException {
-int processes = 10;
+int processes = 5;
String javaExePath = findJavaExe().getAbsolutePath();
CountDownLatch countDownLatch = new CountDownLatch(processes);
AtomicInteger exceptionCounter = new AtomicInteger();
@@ -122,7 +130,9 @@ public class NarUnpackerTest {
// fork a new process with the same classpath
Process process = new ProcessBuilder()
.command(javaExePath,
-"-Xmx64m",
+"-Xmx96m",
+"-XX:TieredStopAtLevel=1",
+"-Dlog4j2.disable.jmx=true",
"-cp",
System.getProperty("java.class.path"),
// use NarUnpackerWorker as the main class
@@ -130,6 +140,7 @@ public class NarUnpackerTest {
// pass arguments to use for testing
sampleZipFile.getAbsolutePath(),
extractDirectory.getAbsolutePath())
+.redirectErrorStream(true)
.start();
String output = IOUtils.toString(process.getInputStream(),
StandardCharsets.UTF_8);
int retval = process.waitFor();
[pulsar] branch branch-2.10 updated: [fix][test] Fix flaky test NarUnpackerTest (#21328)
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-2.10 by this push:
new 55878d5eb83 [fix][test] Fix flaky test NarUnpackerTest (#21328)
55878d5eb83 is described below
commit 55878d5eb83a33b2e240a94ed1d42aeefde28ffb
Author: Lari Hotari
AuthorDate: Mon Oct 9 16:26:25 2023 +0300
[fix][test] Fix flaky test NarUnpackerTest (#21328)
(cherry picked from commit e76a86e3cd1c362e9daa1c88eb8b888e6ab38ab4)
# Conflicts:
#
pulsar-common/src/test/java/org/apache/pulsar/common/nar/NarUnpackerTest.java
---
.../apache/pulsar/common/nar/NarUnpackerTest.java | 27 +++---
1 file changed, 19 insertions(+), 8 deletions(-)
diff --git
a/pulsar-common/src/test/java/org/apache/pulsar/common/nar/NarUnpackerTest.java
b/pulsar-common/src/test/java/org/apache/pulsar/common/nar/NarUnpackerTest.java
index 1d2acd0fa4e..8e7e38dea68 100644
---
a/pulsar-common/src/test/java/org/apache/pulsar/common/nar/NarUnpackerTest.java
+++
b/pulsar-common/src/test/java/org/apache/pulsar/common/nar/NarUnpackerTest.java
@@ -46,7 +46,7 @@ public class NarUnpackerTest {
public void createSampleZipFile() throws IOException {
sampleZipFile = Files.createTempFile("sample", ".zip").toFile();
try (ZipOutputStream out = new ZipOutputStream(new
FileOutputStream(sampleZipFile))) {
-for (int i = 0; i < 1; i++) {
+for (int i = 0; i < 5000; i++) {
ZipEntry e = new ZipEntry("hello" + i + ".txt");
out.putNextEntry(e);
byte[] msg = "hello world!".getBytes(StandardCharsets.UTF_8);
@@ -58,12 +58,20 @@ public class NarUnpackerTest {
}
@AfterMethod(alwaysRun = true)
-void deleteSampleZipFile() throws IOException {
-if (sampleZipFile != null) {
-sampleZipFile.delete();
+void deleteSampleZipFile() {
+if (sampleZipFile != null && sampleZipFile.exists()) {
+try {
+sampleZipFile.delete();
+} catch (Exception e) {
+log.warn("Failed to delete file {}", sampleZipFile, e);
+}
}
-if (extractDirectory != null) {
-FileUtils.deleteFile(extractDirectory, true);
+if (extractDirectory != null && extractDirectory.exists()) {
+try {
+FileUtils.deleteFile(extractDirectory, true);
+} catch (IOException e) {
+log.warn("Failed to delete directory {}", extractDirectory, e);
+}
}
}
@@ -111,7 +119,7 @@ public class NarUnpackerTest {
@Test
void shouldExtractFilesOnceInDifferentProcess() throws
InterruptedException {
-int processes = 10;
+int processes = 5;
String javaExePath = findJavaExe().getAbsolutePath();
CountDownLatch countDownLatch = new CountDownLatch(processes);
AtomicInteger exceptionCounter = new AtomicInteger();
@@ -122,7 +130,9 @@ public class NarUnpackerTest {
// fork a new process with the same classpath
Process process = new ProcessBuilder()
.command(javaExePath,
-"-Xmx64m",
+"-Xmx96m",
+"-XX:TieredStopAtLevel=1",
+"-Dlog4j2.disable.jmx=true",
"-cp",
System.getProperty("java.class.path"),
// use NarUnpackerWorker as the main class
@@ -130,6 +140,7 @@ public class NarUnpackerTest {
// pass arguments to use for testing
sampleZipFile.getAbsolutePath(),
extractDirectory.getAbsolutePath())
+.redirectErrorStream(true)
.start();
String output = IOUtils.toString(process.getInputStream(),
StandardCharsets.UTF_8);
int retval = process.waitFor();
[pulsar] branch branch-2.11 updated: [fix][test] Fix flaky test NarUnpackerTest (#21328)
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-2.11
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-2.11 by this push:
new ac7b5937787 [fix][test] Fix flaky test NarUnpackerTest (#21328)
ac7b5937787 is described below
commit ac7b5937787c6b6ab923934f00305dec5f138dfc
Author: Lari Hotari
AuthorDate: Mon Oct 9 15:51:16 2023 +0300
[fix][test] Fix flaky test NarUnpackerTest (#21328)
(cherry picked from commit e76a86e3cd1c362e9daa1c88eb8b888e6ab38ab4)
# Conflicts:
#
pulsar-common/src/test/java/org/apache/pulsar/common/nar/NarUnpackerTest.java
---
.../apache/pulsar/common/nar/NarUnpackerTest.java | 29 +++---
1 file changed, 20 insertions(+), 9 deletions(-)
diff --git
a/pulsar-common/src/test/java/org/apache/pulsar/common/nar/NarUnpackerTest.java
b/pulsar-common/src/test/java/org/apache/pulsar/common/nar/NarUnpackerTest.java
index 1f97b1b1b69..ddfdf55db40 100644
---
a/pulsar-common/src/test/java/org/apache/pulsar/common/nar/NarUnpackerTest.java
+++
b/pulsar-common/src/test/java/org/apache/pulsar/common/nar/NarUnpackerTest.java
@@ -46,7 +46,7 @@ public class NarUnpackerTest {
public void createSampleZipFile() throws IOException {
sampleZipFile = Files.createTempFile("sample", ".zip").toFile();
try (ZipOutputStream out = new ZipOutputStream(new
FileOutputStream(sampleZipFile))) {
-for (int i = 0; i < 1; i++) {
+for (int i = 0; i < 5000; i++) {
ZipEntry e = new ZipEntry("hello" + i + ".txt");
out.putNextEntry(e);
byte[] msg = "hello world!".getBytes(StandardCharsets.UTF_8);
@@ -58,12 +58,20 @@ public class NarUnpackerTest {
}
@AfterMethod(alwaysRun = true)
-void deleteSampleZipFile() throws IOException {
-if (sampleZipFile != null) {
-sampleZipFile.delete();
+void deleteSampleZipFile() {
+if (sampleZipFile != null && sampleZipFile.exists()) {
+try {
+sampleZipFile.delete();
+} catch (Exception e) {
+log.warn("Failed to delete file {}", sampleZipFile, e);
+}
}
-if (extractDirectory != null) {
-FileUtils.deleteFile(extractDirectory, true);
+if (extractDirectory != null && extractDirectory.exists()) {
+try {
+FileUtils.deleteFile(extractDirectory, true);
+} catch (IOException e) {
+log.warn("Failed to delete directory {}", extractDirectory, e);
+}
}
}
@@ -111,7 +119,7 @@ public class NarUnpackerTest {
@Test
void shouldExtractFilesOnceInDifferentProcess() throws
InterruptedException {
-int processes = 10;
+int processes = 5;
String javaExePath = findJavaExe().getAbsolutePath();
CountDownLatch countDownLatch = new CountDownLatch(processes);
AtomicInteger exceptionCounter = new AtomicInteger();
@@ -122,7 +130,9 @@ public class NarUnpackerTest {
// fork a new process with the same classpath
Process process = new ProcessBuilder()
.command(javaExePath,
-"-Xmx64m",
+"-Xmx96m",
+"-XX:TieredStopAtLevel=1",
+"-Dlog4j2.disable.jmx=true",
"-cp",
System.getProperty("java.class.path"),
// use NarUnpackerWorker as the main class
@@ -130,6 +140,7 @@ public class NarUnpackerTest {
// pass arguments to use for testing
sampleZipFile.getAbsolutePath(),
extractDirectory.getAbsolutePath())
+.redirectErrorStream(true)
.start();
String output = IOUtils.toString(process.getInputStream(),
StandardCharsets.UTF_8);
int retval = process.waitFor();
@@ -147,7 +158,7 @@ public class NarUnpackerTest {
}
}).start();
}
-assertTrue(countDownLatch.await(30, TimeUnit.SECONDS));
+assertTrue(countDownLatch.await(30, TimeUnit.SECONDS), "All processes
should finish before timeout");
assertEquals(exceptionCounter.get(), 0);
assertEquals(extractCounter.get(), 1);
}
[pulsar] branch branch-3.0 updated: [fix][test] Fix flaky test NarUnpackerTest (#21328)
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new f1f417cff4c [fix][test] Fix flaky test NarUnpackerTest (#21328)
f1f417cff4c is described below
commit f1f417cff4cb2bbd9ae1f318c9ed728b2cffd080
Author: Lari Hotari
AuthorDate: Mon Oct 9 15:51:16 2023 +0300
[fix][test] Fix flaky test NarUnpackerTest (#21328)
(cherry picked from commit e76a86e3cd1c362e9daa1c88eb8b888e6ab38ab4)
# Conflicts:
#
pulsar-common/src/test/java/org/apache/pulsar/common/nar/NarUnpackerTest.java
---
.../apache/pulsar/common/nar/NarUnpackerTest.java | 29 +++---
1 file changed, 20 insertions(+), 9 deletions(-)
diff --git
a/pulsar-common/src/test/java/org/apache/pulsar/common/nar/NarUnpackerTest.java
b/pulsar-common/src/test/java/org/apache/pulsar/common/nar/NarUnpackerTest.java
index c6c5ee180f6..f93afac2ff9 100644
---
a/pulsar-common/src/test/java/org/apache/pulsar/common/nar/NarUnpackerTest.java
+++
b/pulsar-common/src/test/java/org/apache/pulsar/common/nar/NarUnpackerTest.java
@@ -46,7 +46,7 @@ public class NarUnpackerTest {
public void createSampleZipFile() throws IOException {
sampleZipFile = Files.createTempFile("sample", ".zip").toFile();
try (ZipOutputStream out = new ZipOutputStream(new
FileOutputStream(sampleZipFile))) {
-for (int i = 0; i < 1; i++) {
+for (int i = 0; i < 5000; i++) {
ZipEntry e = new ZipEntry("hello" + i + ".txt");
out.putNextEntry(e);
byte[] msg = "hello world!".getBytes(StandardCharsets.UTF_8);
@@ -58,12 +58,20 @@ public class NarUnpackerTest {
}
@AfterMethod(alwaysRun = true)
-void deleteSampleZipFile() throws IOException {
-if (sampleZipFile != null) {
-sampleZipFile.delete();
+void deleteSampleZipFile() {
+if (sampleZipFile != null && sampleZipFile.exists()) {
+try {
+sampleZipFile.delete();
+} catch (Exception e) {
+log.warn("Failed to delete file {}", sampleZipFile, e);
+}
}
-if (extractDirectory != null) {
-FileUtils.deleteFile(extractDirectory, true);
+if (extractDirectory != null && extractDirectory.exists()) {
+try {
+FileUtils.deleteFile(extractDirectory, true);
+} catch (IOException e) {
+log.warn("Failed to delete directory {}", extractDirectory, e);
+}
}
}
@@ -111,7 +119,7 @@ public class NarUnpackerTest {
@Test
void shouldExtractFilesOnceInDifferentProcess() throws
InterruptedException {
-int processes = 10;
+int processes = 5;
String javaExePath = findJavaExe().getAbsolutePath();
CountDownLatch countDownLatch = new CountDownLatch(processes);
AtomicInteger exceptionCounter = new AtomicInteger();
@@ -122,7 +130,9 @@ public class NarUnpackerTest {
// fork a new process with the same classpath
Process process = new ProcessBuilder()
.command(javaExePath,
-"-Xmx64m",
+"-Xmx96m",
+"-XX:TieredStopAtLevel=1",
+"-Dlog4j2.disable.jmx=true",
"-cp",
System.getProperty("java.class.path"),
// use NarUnpackerWorker as the main class
@@ -130,6 +140,7 @@ public class NarUnpackerTest {
// pass arguments to use for testing
sampleZipFile.getAbsolutePath(),
extractDirectory.getAbsolutePath())
+.redirectErrorStream(true)
.start();
String output = IOUtils.toString(process.getInputStream(),
StandardCharsets.UTF_8);
int retval = process.waitFor();
@@ -147,7 +158,7 @@ public class NarUnpackerTest {
}
}).start();
}
-assertTrue(countDownLatch.await(30, TimeUnit.SECONDS));
+assertTrue(countDownLatch.await(30, TimeUnit.SECONDS), "All processes
should finish before timeout");
assertEquals(exceptionCounter.get(), 0);
assertEquals(extractCounter.get(), 1);
}
[pulsar] branch branch-3.1 updated: [fix][test] Fix flaky test NarUnpackerTest (#21328)
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new 33553cc4ba8 [fix][test] Fix flaky test NarUnpackerTest (#21328)
33553cc4ba8 is described below
commit 33553cc4ba8d55145bb4667c7386e169ac303431
Author: Lari Hotari
AuthorDate: Mon Oct 9 15:51:16 2023 +0300
[fix][test] Fix flaky test NarUnpackerTest (#21328)
(cherry picked from commit e76a86e3cd1c362e9daa1c88eb8b888e6ab38ab4)
# Conflicts:
#
pulsar-common/src/test/java/org/apache/pulsar/common/nar/NarUnpackerTest.java
---
.../apache/pulsar/common/nar/NarUnpackerTest.java | 29 +++---
1 file changed, 20 insertions(+), 9 deletions(-)
diff --git
a/pulsar-common/src/test/java/org/apache/pulsar/common/nar/NarUnpackerTest.java
b/pulsar-common/src/test/java/org/apache/pulsar/common/nar/NarUnpackerTest.java
index c6c5ee180f6..f93afac2ff9 100644
---
a/pulsar-common/src/test/java/org/apache/pulsar/common/nar/NarUnpackerTest.java
+++
b/pulsar-common/src/test/java/org/apache/pulsar/common/nar/NarUnpackerTest.java
@@ -46,7 +46,7 @@ public class NarUnpackerTest {
public void createSampleZipFile() throws IOException {
sampleZipFile = Files.createTempFile("sample", ".zip").toFile();
try (ZipOutputStream out = new ZipOutputStream(new
FileOutputStream(sampleZipFile))) {
-for (int i = 0; i < 1; i++) {
+for (int i = 0; i < 5000; i++) {
ZipEntry e = new ZipEntry("hello" + i + ".txt");
out.putNextEntry(e);
byte[] msg = "hello world!".getBytes(StandardCharsets.UTF_8);
@@ -58,12 +58,20 @@ public class NarUnpackerTest {
}
@AfterMethod(alwaysRun = true)
-void deleteSampleZipFile() throws IOException {
-if (sampleZipFile != null) {
-sampleZipFile.delete();
+void deleteSampleZipFile() {
+if (sampleZipFile != null && sampleZipFile.exists()) {
+try {
+sampleZipFile.delete();
+} catch (Exception e) {
+log.warn("Failed to delete file {}", sampleZipFile, e);
+}
}
-if (extractDirectory != null) {
-FileUtils.deleteFile(extractDirectory, true);
+if (extractDirectory != null && extractDirectory.exists()) {
+try {
+FileUtils.deleteFile(extractDirectory, true);
+} catch (IOException e) {
+log.warn("Failed to delete directory {}", extractDirectory, e);
+}
}
}
@@ -111,7 +119,7 @@ public class NarUnpackerTest {
@Test
void shouldExtractFilesOnceInDifferentProcess() throws
InterruptedException {
-int processes = 10;
+int processes = 5;
String javaExePath = findJavaExe().getAbsolutePath();
CountDownLatch countDownLatch = new CountDownLatch(processes);
AtomicInteger exceptionCounter = new AtomicInteger();
@@ -122,7 +130,9 @@ public class NarUnpackerTest {
// fork a new process with the same classpath
Process process = new ProcessBuilder()
.command(javaExePath,
-"-Xmx64m",
+"-Xmx96m",
+"-XX:TieredStopAtLevel=1",
+"-Dlog4j2.disable.jmx=true",
"-cp",
System.getProperty("java.class.path"),
// use NarUnpackerWorker as the main class
@@ -130,6 +140,7 @@ public class NarUnpackerTest {
// pass arguments to use for testing
sampleZipFile.getAbsolutePath(),
extractDirectory.getAbsolutePath())
+.redirectErrorStream(true)
.start();
String output = IOUtils.toString(process.getInputStream(),
StandardCharsets.UTF_8);
int retval = process.waitFor();
@@ -147,7 +158,7 @@ public class NarUnpackerTest {
}
}).start();
}
-assertTrue(countDownLatch.await(30, TimeUnit.SECONDS));
+assertTrue(countDownLatch.await(30, TimeUnit.SECONDS), "All processes
should finish before timeout");
assertEquals(exceptionCounter.get(), 0);
assertEquals(extractCounter.get(), 1);
}
Re: [I] Flaky-test: ExtensibleLoadManagerImplTest.initializeState [pulsar]
lhotari commented on issue #20157: URL: https://github.com/apache/pulsar/issues/20157#issuecomment-1752976235 I created #21333 which might help. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][ml] Make mlOwnershipChecker asynchronous so that it doesn't block/deadlock threads [pulsar]
lhotari commented on PR #21333: URL: https://github.com/apache/pulsar/pull/21333#issuecomment-1752969776 @Demogorgon314 I wonder if this change would help also with the deadlock that you are fixing with #21332. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [fix][ml] Make mlOwnershipChecker asynchronous so that it doesn't block/deadlock threads [pulsar]
lhotari opened a new pull request, #21333: URL: https://github.com/apache/pulsar/pull/21333 ### Motivation There's a synchronous method that is called from a thread which shouldn't be blocked. This happens in `ManagedCursorImpl.persistPositionMetaStore` method when the blocking `ledger.mlOwnershipChecker.get()` call is made. `mlOwnershipChecker` was introduced in #5604 and perhaps at that time, the method never blocked. This has changed and for example the flaky test #20157 has shown that this blocking happens in practice and causes deadlocks as reported in #21332. ### Modifications - change the existing `Supplier mlOwnershipChecker` to an asynchronous `Supplier> mlOwnershipChecker` - fix a race condition in `ManagedCursorImpl.persistPositionMetaStore` so that the callback for the top level method isn't called before the refreshing has occured. ### Documentation - [ ] `doc` - [ ] `doc-required` - [x] `doc-not-needed` - [ ] `doc-complete` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Flaky-test: NarUnpackerTest.deleteSampleZipFile [pulsar]
lhotari closed issue #21285: Flaky-test: NarUnpackerTest.deleteSampleZipFile URL: https://github.com/apache/pulsar/issues/21285 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][test] Fix flaky test NarUnpackerTest [pulsar]
lhotari merged PR #21328: URL: https://github.com/apache/pulsar/pull/21328 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch master updated: [fix][test] Fix flaky test NarUnpackerTest (#21328)
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new e76a86e3cd1 [fix][test] Fix flaky test NarUnpackerTest (#21328)
e76a86e3cd1 is described below
commit e76a86e3cd1c362e9daa1c88eb8b888e6ab38ab4
Author: Lari Hotari
AuthorDate: Mon Oct 9 15:51:16 2023 +0300
[fix][test] Fix flaky test NarUnpackerTest (#21328)
---
.../apache/pulsar/common/nar/NarUnpackerTest.java | 31 +++---
1 file changed, 21 insertions(+), 10 deletions(-)
diff --git
a/pulsar-common/src/test/java/org/apache/pulsar/common/nar/NarUnpackerTest.java
b/pulsar-common/src/test/java/org/apache/pulsar/common/nar/NarUnpackerTest.java
index b331c751dcf..a1f915c8b78 100644
---
a/pulsar-common/src/test/java/org/apache/pulsar/common/nar/NarUnpackerTest.java
+++
b/pulsar-common/src/test/java/org/apache/pulsar/common/nar/NarUnpackerTest.java
@@ -38,7 +38,7 @@ import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Test;
@Slf4j
-@Test(groups = "flaky")
+@Test
public class NarUnpackerTest {
File sampleZipFile;
File extractDirectory;
@@ -47,7 +47,7 @@ public class NarUnpackerTest {
public void createSampleZipFile() throws IOException {
sampleZipFile = Files.createTempFile("sample", ".zip").toFile();
try (ZipOutputStream out = new ZipOutputStream(new
FileOutputStream(sampleZipFile))) {
-for (int i = 0; i < 1; i++) {
+for (int i = 0; i < 5000; i++) {
ZipEntry e = new ZipEntry("hello" + i + ".txt");
out.putNextEntry(e);
byte[] msg = "hello world!".getBytes(StandardCharsets.UTF_8);
@@ -59,12 +59,20 @@ public class NarUnpackerTest {
}
@AfterMethod(alwaysRun = true)
-void deleteSampleZipFile() throws IOException {
-if (sampleZipFile != null) {
-sampleZipFile.delete();
+void deleteSampleZipFile() {
+if (sampleZipFile != null && sampleZipFile.exists()) {
+try {
+sampleZipFile.delete();
+} catch (Exception e) {
+log.warn("Failed to delete file {}", sampleZipFile, e);
+}
}
-if (extractDirectory != null) {
-FileUtils.deleteFile(extractDirectory, true);
+if (extractDirectory != null && extractDirectory.exists()) {
+try {
+FileUtils.deleteFile(extractDirectory, true);
+} catch (IOException e) {
+log.warn("Failed to delete directory {}", extractDirectory, e);
+}
}
}
@@ -112,7 +120,7 @@ public class NarUnpackerTest {
@Test
void shouldExtractFilesOnceInDifferentProcess() throws
InterruptedException {
-int processes = 10;
+int processes = 5;
String javaExePath = findJavaExe().getAbsolutePath();
CountDownLatch countDownLatch = new CountDownLatch(processes);
AtomicInteger exceptionCounter = new AtomicInteger();
@@ -123,7 +131,9 @@ public class NarUnpackerTest {
// fork a new process with the same classpath
Process process = new ProcessBuilder()
.command(javaExePath,
-"-Xmx64m",
+"-Xmx96m",
+"-XX:TieredStopAtLevel=1",
+"-Dlog4j2.disable.jmx=true",
"-cp",
System.getProperty("java.class.path"),
// use NarUnpackerWorker as the main class
@@ -131,6 +141,7 @@ public class NarUnpackerTest {
// pass arguments to use for testing
sampleZipFile.getAbsolutePath(),
extractDirectory.getAbsolutePath())
+.redirectErrorStream(true)
.start();
String output = IOUtils.toString(process.getInputStream(),
StandardCharsets.UTF_8);
int retval = process.waitFor();
@@ -148,7 +159,7 @@ public class NarUnpackerTest {
}
}).start();
}
-assertTrue(countDownLatch.await(30, TimeUnit.SECONDS));
+assertTrue(countDownLatch.await(30, TimeUnit.SECONDS), "All processes
should finish before timeout");
assertEquals(exceptionCounter.get(), 0);
assertEquals(extractCounter.get(), 1);
}
Re: [I] Flaky-test: NarUnpackerTest.shouldExtractFilesOnceInDifferentProcess [pulsar]
lhotari closed issue #21291: Flaky-test: NarUnpackerTest.shouldExtractFilesOnceInDifferentProcess URL: https://github.com/apache/pulsar/issues/21291 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][broker] Fix unload operation stuck when use ExtensibleLoadManager [pulsar]
lhotari commented on PR #21332: URL: https://github.com/apache/pulsar/pull/21332#issuecomment-1752881450 Great work @Demogorgon314! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Flaky-test: ExtensibleLoadManagerImplTest.initializeState [pulsar]
lhotari commented on issue #20157: URL: https://github.com/apache/pulsar/issues/20157#issuecomment-1752876746 This might be related: ``` 2023-10-09T10:18:19,431 - WARN - [delayer-4140-1:BaseResource$1@135] - [http://localhost:44057/admin/v2/namespaces/public/test/0x7fff_0x/unload] Failed to perform http put request: java.util.concurrent.CompletionException: org.apache.pulsar.common.util.FutureUtil$LowOverheadTimeoutException: Read timeout 2023-10-09T10:18:19,431 - ERROR - [delayer-4140-1:Namespaces@791] - [null] Failed to unload namespace public/test java.util.concurrent.CompletionException: org.apache.pulsar.client.admin.PulsarAdminException: java.util.concurrent.CompletionException: org.apache.pulsar.common.util.FutureUtil$LowOverheadTimeoutException: Read timeout at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332) ~[?:?] at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347) ~[?:?] at java.util.concurrent.CompletableFuture$BiRelay.tryFire(CompletableFuture.java:1498) ~[?:?] at java.util.concurrent.CompletableFuture$CoCompletion.tryFire(CompletableFuture.java:1219) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162) ~[?:?] at org.apache.pulsar.client.admin.internal.BaseResource$1.failed(BaseResource.java:136) ~[pulsar-client-admin-original-3.2.0-SNAPSHOT.jar:3.2.0-SNAPSHOT] at org.glassfish.jersey.client.JerseyInvocation$1.failed(JerseyInvocation.java:882) ~[jersey-client-2.34.jar:?] at org.glassfish.jersey.client.ClientRuntime.processFailure(ClientRuntime.java:247) ~[jersey-client-2.34.jar:?] at org.glassfish.jersey.client.ClientRuntime.processFailure(ClientRuntime.java:242) ~[jersey-client-2.34.jar:?] at org.glassfish.jersey.client.ClientRuntime.access$100(ClientRuntime.java:62) ~[jersey-client-2.34.jar:?] at org.glassfish.jersey.client.ClientRuntime$2.lambda$failure$1(ClientRuntime.java:178) ~[jersey-client-2.34.jar:?] at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) ~[jersey-common-2.34.jar:?] at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) ~[jersey-common-2.34.jar:?] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.34.jar:?] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.34.jar:?] at org.glassfish.jersey.internal.Errors.process(Errors.java:244) ~[jersey-common-2.34.jar:?] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:288) ~[jersey-common-2.34.jar:?] at org.glassfish.jersey.client.ClientRuntime$2.failure(ClientRuntime.java:178) ~[jersey-client-2.34.jar:?] at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$apply$1(AsyncHttpConnector.java:227) ~[pulsar-client-admin-original-3.2.0-SNAPSHOT.jar:3.2.0-SNAPSHOT] at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162) ~[?:?] at org.apache.pulsar.common.util.FutureUtil.lambda$addTimeoutHandling$9(FutureUtil.java:236) ~[pulsar-common-3.2.0-SNAPSHOT.jar:3.2.0-SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?] at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.99.Final.jar:4.1.99.Final] at java.lang.Thread.run(Thread.java:833) ~[?:?] Caused by: org.apache.pulsar.client.admin.PulsarAdminException: java.util.concurrent.CompletionException: org.apache.pulsar.common.util.FutureUtil$LowOverheadTimeoutException: Read timeout at org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:300) ~[pulsar-client-admin-original-3.2.0-SNAPSHOT.jar:3.2.0-SNAPSHOT] ... 26 more Caused by: java.util.concurrent.CompletionException: org.apache.pulsar.common.util.FutureUtil$LowOverheadTimeoutException: Read timeout at
Re: [PR] [improve][client] Add a way to configure which DNS use [pulsar]
diegosalvi commented on PR #21227: URL: https://github.com/apache/pulsar/pull/21227#issuecomment-1752873169 Still fails for NarUnpackerTest -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Flaky-test: ExtensibleLoadManagerImplTest.initializeState [pulsar]
lhotari commented on issue #20157: URL: https://github.com/apache/pulsar/issues/20157#issuecomment-1752848625 surefire logs for ExtensibleLoadManagerImplTest when the failure happened: https://gist.github.com/lhotari/b3c5519233f84c877621db372f17fa66 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [fix][broker] Fix unload operation stuck when use ExtensibleLoadManager [pulsar]
Demogorgon314 opened a new pull request, #21332: URL: https://github.com/apache/pulsar/pull/21332 ### Motivation When enabling the topic level policies in `ExtensibleLoadManagerImplTest`, the unload operation will get stuck ``` "bookkeeper-ml-scheduler-OrderedScheduler-9-0" #45 prio=5 os_prio=31 cpu=31.51ms elapsed=39.28s tid=0x000118db2600 nid=0xb303 waiting on condition [0x000176b79000] java.lang.Thread.State: TIMED_WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@17.0.3/Native Method) - parking to wait for <0x200030154288> (a java.util.concurrent.CompletableFuture$Signaller) at java.util.concurrent.locks.LockSupport.parkNanos(java.base@17.0.3/LockSupport.java:252) at java.util.concurrent.CompletableFuture$Signaller.block(java.base@17.0.3/CompletableFuture.java:1866) at java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@17.0.3/ForkJoinPool.java:3463) at java.util.concurrent.ForkJoinPool.managedBlock(java.base@17.0.3/ForkJoinPool.java:3434) at java.util.concurrent.CompletableFuture.timedGet(java.base@17.0.3/CompletableFuture.java:1939) at java.util.concurrent.CompletableFuture.get(java.base@17.0.3/CompletableFuture.java:2095) at org.apache.pulsar.broker.namespace.NamespaceService.isServiceUnitOwned(NamespaceService.java:1153) at java.lang.invoke.LambdaForm$DMH/0x0008010a1000.invokeVirtual(java.base@17.0.3/LambdaForm$DMH) at java.lang.invoke.LambdaForm$MH/0x00080157.invoke(java.base@17.0.3/LambdaForm$MH) at java.lang.invoke.LambdaForm$MH/0x000800f4a400.invoke(java.base@17.0.3/LambdaForm$MH) at java.lang.invoke.LambdaForm$MH/0x000800c14400.invokeExact_MT(java.base@17.0.3/LambdaForm$MH) at java.lang.invoke.MethodHandle.invokeWithArguments(java.base@17.0.3/MethodHandle.java:732) at org.mockito.internal.util.reflection.InstrumentationMemberAccessor$Dispatcher$ByteBuddy$ILgTSeGv.invokeWithArguments(Unknown Source) at org.mockito.internal.util.reflection.InstrumentationMemberAccessor.invoke(InstrumentationMemberAccessor.java:239) at org.mockito.internal.util.reflection.ModuleMemberAccessor.invoke(ModuleMemberAccessor.java:55) at org.mockito.internal.creation.bytebuddy.MockMethodAdvice.tryInvoke(MockMethodAdvice.java:333) at org.mockito.internal.creation.bytebuddy.MockMethodAdvice.access$500(MockMethodAdvice.java:60) at org.mockito.internal.creation.bytebuddy.MockMethodAdvice$RealMethodCall.invoke(MockMethodAdvice.java:253) at org.mockito.internal.invocation.InterceptedInvocation.callRealMethod(InterceptedInvocation.java:142) at org.mockito.internal.stubbing.answers.CallsRealMethods.answer(CallsRealMethods.java:45) at org.mockito.Answers.answer(Answers.java:99) at org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:110) at org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29) at org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:34) at org.mockito.internal.creation.bytebuddy.MockMethodInterceptor.doIntercept(MockMethodInterceptor.java:82) at org.mockito.internal.creation.bytebuddy.MockMethodAdvice.handle(MockMethodAdvice.java:151) at org.apache.pulsar.broker.namespace.NamespaceService.isServiceUnitOwned(NamespaceService.java:1153) at org.apache.pulsar.broker.service.BrokerService.isTopicNsOwnedByBroker(BrokerService.java:2141) at org.apache.pulsar.broker.service.BrokerService.lambda$createPersistentTopic$64(BrokerService.java:1751) at org.apache.pulsar.broker.service.BrokerService$$Lambda$871/0x0008013f22d0.get(Unknown Source) at org.apache.bookkeeper.mledger.impl.ManagedCursorImpl$25.operationFailed(ManagedCursorImpl.java:2691) at org.apache.bookkeeper.mledger.impl.MetaStoreImpl.lambda$asyncUpdateCursorInfo$17(MetaStoreImpl.java:271) at org.apache.bookkeeper.mledger.impl.MetaStoreImpl$$Lambda$1924/0x000801540818.run(Unknown Source) at org.apache.bookkeeper.common.util.OrderedExecutor$TimedRunnable.run(OrderedExecutor.java:201) at org.apache.bookkeeper.common.util.SingleThreadSafeScheduledExecutorService$SafeRunnable.run(SingleThreadSafeScheduledExecutorService.java:46) at java.util.concurrent.Executors$RunnableAdapter.call(java.base@17.0.3/Executors.java:539) at java.util.concurrent.FutureTask.run(java.base@17.0.3/FutureTask.java:264) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@17.0.3/ScheduledThreadPoolExecutor.java:304) at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.3/ThreadPoolExecutor.java:1136) at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.3/ThreadPoolExecutor.java:635)
[pulsar-dotpulsar] branch master updated: Stop duplicate code comments for code review. (#181)
This is an automated email from the ASF dual-hosted git repository.
blankensteiner pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar-dotpulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 197b5cc Stop duplicate code comments for code review. (#181)
197b5cc is described below
commit 197b5cc3bf442f38cf2a9f48b037a1fa39449f99
Author: entvex <1580435+ent...@users.noreply.github.com>
AuthorDate: Mon Oct 9 13:23:58 2023 +0200
Stop duplicate code comments for code review. (#181)
When reviewing a pull request, duplicated errors pollute the code view
unnecessarily.
Co-authored-by: David Jensen
---
.github/workflows/ci-unit.yaml | 17 -
1 file changed, 17 deletions(-)
diff --git a/.github/workflows/ci-unit.yaml b/.github/workflows/ci-unit.yaml
index c3c983c..0dc26f8 100644
--- a/.github/workflows/ci-unit.yaml
+++ b/.github/workflows/ci-unit.yaml
@@ -42,20 +42,3 @@ jobs:
run: |
echo 'Dotnet Version: ${{ steps.setupstep.outputs.dotnet-version }}'
dotnet test ./tests/DotPulsar.Tests/DotPulsar.Tests.csproj --logger
"trx;verbosity=detailed"
-
- doc-tests:
-runs-on: ubuntu-latest
-timeout-minutes: 120
-steps:
- - uses: actions/checkout@v3
- - name: Setup dotnet
-uses: actions/setup-dotnet@v3
-id: setupstep
-with:
- dotnet-version: '7.0.x'
- - name: Build docs with docfx
-run: |
- echo 'Dotnet Version: ${{ steps.setupstep.outputs.dotnet-version }}'
- dotnet tool update -g docfx
- dotnet build -p:TargetFramework=net7.0
- docfx docs/api/docfx.json
Re: [PR] Stop duplicate code comments for code review. [pulsar-dotpulsar]
blankensteiner merged PR #181: URL: https://github.com/apache/pulsar-dotpulsar/pull/181 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[I] Flaky-test: OneWayReplicatorTest.testReplicatorProducerStatInTopic [pulsar]
lhotari opened a new issue, #21331: URL: https://github.com/apache/pulsar/issues/21331 ### Search before asking - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar. ### Example failure https://github.com/apache/pulsar/actions/runs/6454495353/job/17523189656?pr=21328#step:10:1272 ### Exception stacktrace ``` Error: Tests run: 3, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 33.352 s <<< FAILURE! - in org.apache.pulsar.broker.service.OneWayReplicatorTest Error: org.apache.pulsar.broker.service.OneWayReplicatorTest.testReplicatorProducerStatInTopic Time elapsed: 0.674 s <<< FAILURE! org.apache.pulsar.client.admin.PulsarAdminException$ConflictException: This topic already exists at org.apache.pulsar.client.admin.PulsarAdminException.wrap(PulsarAdminException.java:252) at org.apache.pulsar.client.admin.internal.BaseResource.sync(BaseResource.java:352) at org.apache.pulsar.client.admin.internal.TopicsImpl.createNonPartitionedTopic(TopicsImpl.java:309) at org.apache.pulsar.client.admin.Topics.createNonPartitionedTopic(Topics.java:539) at org.apache.pulsar.broker.service.OneWayReplicatorTest.testReplicatorProducerStatInTopic(OneWayReplicatorTest.java:60) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139) at org.testng.internal.invokers.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:47) at org.testng.internal.invokers.InvokeMethodRunnable.call(InvokeMethodRunnable.java:76) at org.testng.internal.invokers.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) Suppressed: org.apache.pulsar.client.admin.PulsarAdminException$ConflictException: This topic already exists at org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:287) at org.apache.pulsar.client.admin.internal.BaseResource$1.failed(BaseResource.java:136) at org.glassfish.jersey.client.JerseyInvocation$1.failed(JerseyInvocation.java:882) at org.glassfish.jersey.client.JerseyInvocation$1.completed(JerseyInvocation.java:863) at org.glassfish.jersey.client.ClientRuntime.processResponse(ClientRuntime.java:229) at org.glassfish.jersey.client.ClientRuntime.access$200(ClientRuntime.java:62) at org.glassfish.jersey.client.ClientRuntime$2.lambda$response$0(ClientRuntime.java:173) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:244) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:288) at org.glassfish.jersey.client.ClientRuntime$2.response(ClientRuntime.java:173) at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$apply$1(AsyncHttpConnector.java:254) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147) at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOperation$4(AsyncHttpConnector.java:296) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) at
Re: [I] GHA: Artifacts [pulsar-dotpulsar]
entvex closed issue #182: GHA: Artifacts URL: https://github.com/apache/pulsar-dotpulsar/issues/182 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[I] Flaky-test: AdminApiMaxUnackedMessagesTest.testMaxUnackedMessagesPerConsumerPriority [pulsar]
lhotari opened a new issue, #21330: URL: https://github.com/apache/pulsar/issues/21330 ### Search before asking - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar. ### Example failure https://github.com/apache/pulsar/actions/runs/6454992455/job/17522130798?pr=21329#step:10:938 ### Exception stacktrace ``` Error: Tests run: 8, Failures: 1, Errors: 0, Skipped: 5, Time elapsed: 40.429 s <<< FAILURE! - in org.apache.pulsar.broker.admin.AdminApiMaxUnackedMessagesTest Error: org.apache.pulsar.broker.admin.AdminApiMaxUnackedMessagesTest.testMaxUnackedMessagesPerConsumerPriority Time elapsed: 7.833 s <<< FAILURE! java.lang.AssertionError: expected [3] but found [1] at org.testng.Assert.fail(Assert.java:110) at org.testng.Assert.failNotEquals(Assert.java:1577) at org.testng.Assert.assertEqualsImpl(Assert.java:149) at org.testng.Assert.assertEquals(Assert.java:131) at org.testng.Assert.assertEquals(Assert.java:1418) at org.testng.Assert.assertEquals(Assert.java:1382) at org.testng.Assert.assertEquals(Assert.java:1428) at org.apache.pulsar.broker.admin.AdminApiMaxUnackedMessagesTest.testMaxUnackedMessagesPerConsumerPriority(AdminApiMaxUnackedMessagesTest.java:192) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139) at org.testng.internal.invokers.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:47) at org.testng.internal.invokers.InvokeMethodRunnable.call(InvokeMethodRunnable.java:76) at org.testng.internal.invokers.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) ``` ### Are you willing to submit a PR? - [X] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Flaky-test: ExtensibleLoadManagerImplTest.initializeState [pulsar]
lhotari commented on issue #20157: URL: https://github.com/apache/pulsar/issues/20157#issuecomment-1752779107 yet another one: https://github.com/apache/pulsar/actions/runs/6454992455/job/17522130217?pr=21329#step:10:1169 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][test] Move NarUnpackerTest to flaky group [pulsar]
lhotari commented on PR #21316: URL: https://github.com/apache/pulsar/pull/21316#issuecomment-1752763348 btw. `quarantine` group should have been used instead of `flaky`. The `flaky` group is only handled within `pulsar-broker` module. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar-dotpulsar] branch master updated: Remove the unused artifacts from the ci-unit. (#185)
This is an automated email from the ASF dual-hosted git repository.
blankensteiner pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar-dotpulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 5def5e7 Remove the unused artifacts from the ci-unit. (#185)
5def5e7 is described below
commit 5def5e7365f901dbc5398b566b0037eb77cb61a0
Author: entvex <1580435+ent...@users.noreply.github.com>
AuthorDate: Mon Oct 9 12:20:18 2023 +0200
Remove the unused artifacts from the ci-unit. (#185)
---
.github/workflows/ci-unit.yaml | 15 ---
1 file changed, 15 deletions(-)
diff --git a/.github/workflows/ci-unit.yaml b/.github/workflows/ci-unit.yaml
index 2844db8..c3c983c 100644
--- a/.github/workflows/ci-unit.yaml
+++ b/.github/workflows/ci-unit.yaml
@@ -43,21 +43,6 @@ jobs:
echo 'Dotnet Version: ${{ steps.setupstep.outputs.dotnet-version }}'
dotnet test ./tests/DotPulsar.Tests/DotPulsar.Tests.csproj --logger
"trx;verbosity=detailed"
- - name: package artifacts
-if: failure()
-run: |
- rm -rf artifacts
- mkdir artifacts
- find . -type d -name "TestResults" -exec cp --parents -R {}
artifacts/ \;
- zip -r artifacts.zip artifacts
-
- - name: upload artifacts
-uses: actions/upload-artifact@v3
-if: failure()
-with:
- name: artifacts
- path: artifacts.zip
-
doc-tests:
runs-on: ubuntu-latest
timeout-minutes: 120
Re: [PR] Remove the unused artifacts from the ci-unit. [pulsar-dotpulsar]
blankensteiner merged PR #185: URL: https://github.com/apache/pulsar-dotpulsar/pull/185 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [fix][test] Fix flaky CompactionTest.testDispatcherMaxReadSizeBytes [pulsar]
lhotari opened a new pull request, #21329: URL: https://github.com/apache/pulsar/pull/21329 Fixes #21284 ### Motivation CompactionTest.testDispatcherMaxReadSizeBytes is extremely flaky and blocks CI very often. ### Modifications The test had a race condition in the way how the spy was injected into the PersistentTopic instance. This problem has been fixed now. ### Documentation - [ ] `doc` - [ ] `doc-required` - [x] `doc-not-needed` - [ ] `doc-complete` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][test] Fix flaky test NarUnpackerTest [pulsar]
poorbarcode commented on PR #21328: URL: https://github.com/apache/pulsar/pull/21328#issuecomment-1752615529 Since `branch-3.0` is a long-term support release, I think it also needs this fix. So I added the label `release:3.0.x` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][test] Move NarUnpackerTest to flaky group [pulsar]
lhotari commented on PR #21316: URL: https://github.com/apache/pulsar/pull/21316#issuecomment-1752601847 NarUnpackerTest is fixed by #21328 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [fix][test] Fix flaky test NarUnpackerTest [pulsar]
lhotari opened a new pull request, #21328: URL: https://github.com/apache/pulsar/pull/21328 Fixes #21291 Fixes #21285 ### Motivation NarUnpackerTest is really flaky. See the above isues that this PR fixes. When investigating the problem, I was able to reproduce the issue by running the test within Docker with constrained resources. There's a 30 second timeout in the shouldExtractFilesOnceInDifferentProcess test for the 10 different processes to complete. This takes much longer since each process uses significant amount of CPU. The fix is to reduce the CPU consumption by reducing the test data size, reducing the number of processes and tuning the JVM parameters for the forked processes. ### Modifications - reduce number of processes from 10 to 5 - reduce the number of files in the sample zip from 1 to 5000 - make the test use less CPU resources so that it will run in CI with constrained resources - make JIT compiler stop at C1 - disable JMX registration in Log4j2 which showed up in profiling - increase max heap size from 64MB to 96MB to give more head room for GC ### Documentation - [ ] `doc` - [ ] `doc-required` - [x] `doc-not-needed` - [ ] `doc-complete` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix] [bk-client] Fix bk client MinNumRacksPerWriteQuorum and EnforceMinNumRacksPerWriteQuorum not work problem. [pulsar]
horizonzy commented on PR #21327: URL: https://github.com/apache/pulsar/pull/21327#issuecomment-1752512136 /pulsarbot rerun-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][broker] fix the global topic policy config override the local topic policy config [pulsar]
chenhongSZ commented on PR #21212: URL: https://github.com/apache/pulsar/pull/21212#issuecomment-1752493029 > I'm sorry, but I don't think this change suits me. > > There are some reasons are as follows: > > 1. Apache Pulsar has the `broker`/`namespace`/`topic` level policy. The global topic policy is just a particular topic policy based on the geo-replication. We should refrain from introducing this concept into the topic entity. > 2. We should encapsulate this concept into topic policy service, which can decide to give the topic a global or local policy because the topic doesn't care if the policy is global or not. > > Therefore, I would like to let `[SystemTopicBasedTopicPoliciesService.java](https://github.com/apache/pulsar/pull/21212/files#diff-9d2948d863c111e4be6d508a1c573667a1326b98c4314e917ba9e344bb61dc27)` decide to update global or local policy to the topic to avoid `isGlobal` param everywhere. > > WDYT? /cc @codelipenghui @Technoboy- @poorbarcode I agree, It is indeed we should encapsulate this concept into topic policy service. I will make some changes as below: 1. rollback `PolicyHierarchyValue` to its previous state (use `topicValue` replace `localTopicValue` and `globalTopicValue`) 2. merge the local and global topic policies into one `TopicPolicies` at `SystemTopicBasedTopicPoliciesService` before init topic and notify topic. therefore `org.apache.pulsar.broker.service.persistent.PersistentTopic#onUpdate` method doesn't care if the policy is global or not. but the `org.apache.pulsar.common.policies.data.TopicPolicies#isGlobal` becomes meaningless in the topic context. so we should not use this variable in the topic context. Any suggestions? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][broker] fix the global topic policy config override the local topic policy config [pulsar]
chenhongSZ commented on code in PR #21212:
URL: https://github.com/apache/pulsar/pull/21212#discussion_r1349921910
##
pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/TopicPoliciesTest.java:
##
@@ -3061,6 +3084,104 @@ public void testGlobalTopicPolicies() throws Exception {
}
+@Test
+public void testInitPolicesCacheAndNotifyListeners() throws Exception {
+final String topic = testTopic + UUID.randomUUID();
+admin.topics().createNonPartitionedTopic(topic);
+pulsarClient.newProducer().topic(topic).create().close();
+
+// set up policies
+TopicName topicName = TopicName.get(topic);
+TopicPolicies localInitPolicy =
TopicPolicies.builder().maxConsumerPerTopic(10).build();
+pulsar.getTopicPoliciesService().updateTopicPoliciesAsync(topicName,
localInitPolicy).get();
+TopicPolicies globalInitPolicy =
+
TopicPolicies.builder().maxConsumerPerTopic(20).maxProducerPerTopic(30).isGlobal(true).build();
+pulsar.getTopicPoliciesService().updateTopicPoliciesAsync(topicName,
globalInitPolicy).get();
+
+// the policies cache
+SystemTopicBasedTopicPoliciesService topicPoliciesService
+= (SystemTopicBasedTopicPoliciesService)
pulsar.getTopicPoliciesService();
+
+// reload namespace to trigger init polices cache and notify listeners
+admin.namespaces().unload(myNamespace);
+
assertNull(topicPoliciesService.getPoliciesCacheInit(NamespaceName.get(myNamespace)));
+pulsarClient.newProducer().topic(topic).create().close();
+Awaitility.await().untilAsserted(
+() ->
assertEquals(topicPoliciesService.getPoliciesCacheInit(NamespaceName.get(myNamespace)).isDone()
+&&
!topicPoliciesService.getPoliciesCacheInit(NamespaceName.get(myNamespace))
+.isCompletedExceptionally(), true));
+
+// the final policies take effect in topic
+HierarchyTopicPolicies hierarchyTopicPolicies =
+
pulsar.getBrokerService().getTopics().get(topic).get().get().getHierarchyTopicPolicies();
+
+
assertEquals(topicPoliciesService.getTopicPolicies(topicName).getMaxConsumerPerTopic(),
10);
+assertEquals(topicPoliciesService.getTopicPolicies(topicName,
true).getMaxConsumerPerTopic(), 20);
+
assertEquals(hierarchyTopicPolicies.getMaxConsumerPerTopic().getGlobalTopicValue(),
20);
+
assertEquals(hierarchyTopicPolicies.getMaxConsumerPerTopic().getLocalTopicValue(),
10);
+assertEquals(hierarchyTopicPolicies.getMaxConsumerPerTopic().get(),
10);
+
+
assertEquals(topicPoliciesService.getTopicPolicies(topicName).getMaxProducerPerTopic(),
null);
+assertEquals(topicPoliciesService.getTopicPolicies(topicName,
true).getMaxProducerPerTopic(), 30);
+
assertEquals(hierarchyTopicPolicies.getMaxProducersPerTopic().getGlobalTopicValue(),
30);
+
assertEquals(hierarchyTopicPolicies.getMaxProducersPerTopic().getLocalTopicValue(),
null);
+assertEquals(hierarchyTopicPolicies.getMaxProducersPerTopic().get(),
30);
+}
+
+@Test
+public void testInitPolicesCacheAndNotifyListenersAfterCompaction() throws
Exception {
Review Comment:
I've fixed the compatibility problem and added a test to cover this change.
PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [cleanup][metadatastore] cleanup deprecated method of metadata store etcd [pulsar]
BewareMyPower commented on code in PR #21253:
URL: https://github.com/apache/pulsar/pull/21253#discussion_r1349891541
##
pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/EtcdMetadataStore.java:
##
@@ -109,9 +109,9 @@ public EtcdMetadataStore(String metadataURL,
MetadataStoreConfig conf, boolean e
try {
this.client = newEtcdClient(metadataURL, conf);
this.kv = client.getKVClient();
-this.client.getWatchClient().watch(ByteSequence.from("\0",
StandardCharsets.UTF_8),
+this.client.getWatchClient().watch(ByteSequence.from("/",
StandardCharsets.UTF_8),
Review Comment:
This modification is beyond the scope of the PR description. You need to add
a test to protect this change.
##
pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/EtcdMetadataStore.java:
##
@@ -285,7 +285,7 @@ protected void batchOperation(List ops) {
.withKeysOnly(true)
.withSortField(GetOption.SortTarget.KEY)
.withSortOrder(GetOption.SortOrder.ASCEND)
-.withPrefix(prefix)
+.isPrefix(true)
Review Comment:
Please don't change the indent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
