[GitHub] [pulsar] Samreay added a comment to the discussion: Migrating from RabbitMQ: shared consumers with new messages only on ack

2023-05-01 Thread GitBox 


GitHub user Samreay added a comment to the discussion: Migrating from RabbitMQ: 
shared consumers with new messages only on ack

Thanks for the quick response, I appreciate it!

Ah `priorityLevel` looks interesting. Is that something only available in the 
Java client? I can't see it in the [feature 
matrix](https://docs.google.com/spreadsheets/d/1YHYTkIXR8-Ql103u-IMI18TXLlGStK8uJjDsOOA0T20/edit#gid=1784579914),
 but a search through the [python 
docs](https://pulsar.apache.org/api/python/3.1.x/pulsar.Client.html#subscribe) 
doesn't show up anything about priority. Have I misunderstood something, or is 
this a feature yet to make it across?

> You can completely disable prefetching by setting the consumer receiver queue 
> to 0: each consumer will only have the message that is currently process and 
> won't block any other message.

That's also good to know, and I see the `client.subscribe` function from above 
does indeed have a `receiver_queue_size` kwarg to make things simple. I'll give 
it a whirl this afternoon and see if I can get it all working and report back!

GitHub link: 
https://github.com/apache/pulsar/discussions/20211#discussioncomment-5776889


This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] merlimat added a comment to the discussion: Migrating from RabbitMQ: shared consumers with new messages only on ack

2023-05-01 Thread GitBox 


GitHub user merlimat added a comment to the discussion: Migrating from 
RabbitMQ: shared consumers with new messages only on ack

> The first thing, which I believe is just not possible due to the nature of 
> the tech, are priority messages and priority consumers, with failover 
> subscriptions being a way to get fairly close to a priority consumer.

For messages priority, you can use separate topics for each priority.

Consumer priority can be set for shared subscriptions: 
https://pulsar.apache.org/api/client/2.11.x/org/apache/pulsar/client/api/ConsumerBuilder.html#priorityLevel(int)
 

> Is this functionality built into the [shared 
> subscription](https://pulsar.apache.org/docs/2.11.x/concepts-messaging/#shared)
>  at all, and if not, would anyone have any recommendations about how to 
> approach the problem? 

You can completely disable prefetching by setting the consumer receiver queue 
to 0: each consumer will only have the message that is currently process and 
won't block any other message.

GitHub link: 
https://github.com/apache/pulsar/discussions/20211#discussioncomment-5776848


This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] github-actions[bot] commented on pull request #19967: [improve][broker] Add haProxyProtocolEnabled more description

2023-05-01 Thread via GitHub 


github-actions[bot] commented on PR #19967:
URL: https://github.com/apache/pulsar/pull/19967#issuecomment-1530760889

   The pr had no activity for 30 days, mark with Stale label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] merlimat commented on issue #12944: ARM based docker image

2023-05-01 Thread via GitHub 


merlimat commented on issue #12944:
URL: https://github.com/apache/pulsar/issues/12944#issuecomment-1530351437

   Yes, this is going to be available in 3.0.0 which is being released this 
week.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] tyler-boyd commented on issue #12944: ARM based docker image

2023-05-01 Thread via GitHub 


tyler-boyd commented on issue #12944:
URL: https://github.com/apache/pulsar/issues/12944#issuecomment-1530310067

   I see this issue was closed (as a PR was merged), when should we expect an 
ARM image to be available on Dockerhub? We've found that running the amd64 
image through qemu is too slow to be useful for our use cases, so are very much 
looking forward to an ARM image :eyes:  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] nlu90 added a comment to the discussion: Pulsar Functions lifecycle and depolyment details.

2023-05-01 Thread GitBox 


GitHub user nlu90 added a comment to the discussion: Pulsar Functions lifecycle 
and depolyment details.

1. Running Function Worker with Pulsar Brokers is fine for testing purposes. In 
a production env, running Function Worker and Pulsar Broker separately is 
recommended so your Broker will be more stable. The ThreadRuntime will launch 
your function inside Function Worker's thread pool, and ProcessRuntime will 
launch separate JVM processes for your functions.

2.a. You can control how many instances for your function via `pulsar-admin 
functions create --parallelism `. Once submitted, it won't change during the 
runtime. 

2.b. You are responsible for making your function thread-safe and not blocking 
on the process method.

3.a. You only need to send the request once. If the same request is submitted 
multiple times, later requests will be rejected due to the `Already Exists` 
error.

3.b. Function Worker will try to shut down the instance for 10 seconds timeout 
gracefully. After 10 seconds, it will forcibly terminate the process. One thing 
to notice is that, as long as the subscription is not cleaned, the newly 
updated function instance will start from where the old instances stopped to 
continue the message processing.

GitHub link: 
https://github.com/apache/pulsar/discussions/20195#discussioncomment-5773367


This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] tisonkun commented on pull request #16901: [feat][broker] PIP-157 Bucketing topic metadata to allow more topics per namespace

2023-05-01 Thread via GitHub 


tisonkun commented on PR #16901:
URL: https://github.com/apache/pulsar/pull/16901#issuecomment-1529773618

   Thanks for your contribution @andrasbeni!
   
   It seems the community doesn't give this PR a review in time. I add this PR 
in my backlog to see if it's still relevant. Perhaps ping you for reviewing if 
so.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] andrasbeni closed pull request #16901: [feat][broker] PIP-157 Bucketing topic metadata to allow more topics per namespace

2023-05-01 Thread via GitHub 


andrasbeni closed pull request #16901: [feat][broker] PIP-157 Bucketing topic 
metadata to allow more topics per namespace
URL: https://github.com/apache/pulsar/pull/16901


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[pulsar] branch master updated: [fix][client] SchemaDefinition handle JSR310_CONVERSION_ENABLED property (#20201)

2023-05-01 Thread tison 
This is an automated email from the ASF dual-hosted git repository.

tison pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new a22700df665 [fix][client] SchemaDefinition handle 
JSR310_CONVERSION_ENABLED property (#20201)
a22700df665 is described below

commit a22700df665a9ea0a8db98334b6d4337e5e490f3
Author: tison 
AuthorDate: Mon May 1 16:47:36 2023 +0800

[fix][client] SchemaDefinition handle JSR310_CONVERSION_ENABLED property 
(#20201)

Signed-off-by: tison 
---
 .../impl/schema/SchemaDefinitionBuilderImpl.java   |  2 +-
 .../client/impl/schema/KeyValueSchemaTest.java | 34 --
 .../pulsar/functions/source/TopicSchema.java   |  2 --
 3 files changed, 33 insertions(+), 5 deletions(-)

diff --git 
a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/SchemaDefinitionBuilderImpl.java
 
b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/SchemaDefinitionBuilderImpl.java
index c50aac5c398..93869a3bb9e 100644
--- 
a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/SchemaDefinitionBuilderImpl.java
+++ 
b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/SchemaDefinitionBuilderImpl.java
@@ -129,7 +129,7 @@ public class SchemaDefinitionBuilderImpl implements 
SchemaDefinitionBuilder keyProperties = new TreeMap<>();
 keyProperties.put("foo.key1", "value");
 keyProperties.put("foo.key2", "value");
@@ -89,7 +89,6 @@ public class KeyValueSchemaTest {
 .build());
 
 Schema> keyValueSchema1 = 
Schema.KeyValue(fooSchema, barSchema);
-
 
assertEquals(keyValueSchema1.getSchemaInfo().getProperties().get("key.schema.type"),
 String.valueOf(SchemaType.AVRO));
 
assertEquals(keyValueSchema1.getSchemaInfo().getProperties().get("key.schema.properties"),
 
"{\"__alwaysAllowNull\":\"true\",\"__jsr310ConversionEnabled\":\"false\",\"foo.key1\":\"value\",\"foo.key2\":\"value\"}");
@@ -98,6 +97,37 @@ public class KeyValueSchemaTest {
 
"{\"__alwaysAllowNull\":\"true\",\"__jsr310ConversionEnabled\":\"false\",\"bar.key\":\"key\"}");
 }
 
+@Test
+public void testOverwriteSchemaDefaultProperties() {
+Map keyProperties = new TreeMap<>();
+keyProperties.put("foo.key1", "value");
+keyProperties.put("foo.key2", "value");
+keyProperties.put(SchemaDefinitionBuilderImpl.ALWAYS_ALLOW_NULL, 
"false");
+
keyProperties.put(SchemaDefinitionBuilderImpl.JSR310_CONVERSION_ENABLED, 
"true");
+
+Map valueProperties = new TreeMap<>();
+valueProperties.put("bar.key", "key");
+
+AvroSchema fooSchema = AvroSchema.of(
+SchemaDefinition.builder()
+.withPojo(Foo.class)
+.withProperties(keyProperties)
+.build());
+AvroSchema barSchema = AvroSchema.of(
+SchemaDefinition.builder()
+.withPojo(Bar.class)
+.withProperties(valueProperties)
+.build());
+
+Schema> keyValueSchema1 = 
Schema.KeyValue(fooSchema, barSchema);
+
assertEquals(keyValueSchema1.getSchemaInfo().getProperties().get("key.schema.type"),
 String.valueOf(SchemaType.AVRO));
+
assertEquals(keyValueSchema1.getSchemaInfo().getProperties().get("key.schema.properties"),
+
"{\"__alwaysAllowNull\":\"false\",\"__jsr310ConversionEnabled\":\"true\",\"foo.key1\":\"value\",\"foo.key2\":\"value\"}");
+
assertEquals(keyValueSchema1.getSchemaInfo().getProperties().get("value.schema.type"),
 String.valueOf(SchemaType.AVRO));
+
assertEquals(keyValueSchema1.getSchemaInfo().getProperties().get("value.schema.properties"),
+
"{\"__alwaysAllowNull\":\"true\",\"__jsr310ConversionEnabled\":\"false\",\"bar.key\":\"key\"}");
+}
+
 @Test
 public void testNotAllowNullAvroSchemaCreate() {
 AvroSchema fooSchema = 
AvroSchema.of(SchemaDefinition.builder().withPojo(Foo.class).withAlwaysAllowNull(false).build());
diff --git 
a/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/source/TopicSchema.java
 
b/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/source/TopicSchema.java
index 57f49fed0ca..8ed177cba3b 100644
--- 
a/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/source/TopicSchema.java
+++ 
b/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/source/TopicSchema.java
@@ -44,8 +44,6 @@ import org.apache.pulsar.functions.instance.InstanceUtils;
 @Slf4j
 public class TopicSchema {
 
-public static final String JSR_310_CONVERSION_ENABLED = 
"jsr310ConversionEnabled";
-public static final String ALWAYS_ALLOW_NULL = "alwaysAllowNull";
 private final Map> cachedSchemas = new HashMap<>();

[GitHub] [pulsar] tisonkun closed issue #20093: [BUG] Use correct property when determine if jsr310 enabled

2023-05-01 Thread via GitHub 


tisonkun closed issue #20093: [BUG] Use correct property when determine if 
jsr310 enabled
URL: https://github.com/apache/pulsar/issues/20093


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] tisonkun merged pull request #20201: [fix][client] SchemaDefinition handle JSR310_CONVERSION_ENABLED property

2023-05-01 Thread via GitHub 


tisonkun merged PR #20201:
URL: https://github.com/apache/pulsar/pull/20201


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] tisonkun commented on pull request #20201: [fix][client] SchemaDefinition handle JSR310_CONVERSION_ENABLED property

2023-05-01 Thread via GitHub 


tisonkun commented on PR #20201:
URL: https://github.com/apache/pulsar/pull/20201#issuecomment-1529490088

   Merging...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar] asafm added a comment to the discussion: Pulsar Functions lifecycle and depolyment details.

2023-05-01 Thread GitBox 


GitHub user asafm added a comment to the discussion: Pulsar Functions lifecycle 
and depolyment details.

Regarding auto-scaling - there is another scheduler (WorkerService plugin) 
which runs the function instances on k8s: https://functionmesh.io/docs/. I 
think it has auto-scale built in.

Deploy: Any SQL database for example, including NoSQL like HBase, Cassandra, if 
you set up a table, it needs to be done once. The same goes for S3 buckets. I'm 
sure there is a built-in solution in Ansible (pattern) for handling "do it 
once".




GitHub link: 
https://github.com/apache/pulsar/discussions/20195#discussioncomment-5770097


This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org

[pulsar-dotpulsar] branch master updated: Producer access mode fixes (#149)

2023-05-01 Thread blankensteiner 
This is an automated email from the ASF dual-hosted git repository.

blankensteiner pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar-dotpulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new c93a0de  Producer access mode fixes (#149)
c93a0de is described below

commit c93a0dee3a35d698376ce975b376a6dbf755f128
Author: Kristian Andersen 
AuthorDate: Mon May 1 08:51:11 2023 +0200

Producer access mode fixes (#149)

* Track topicEpoch when connection occurs after having waited to be 
exclusive

* Fix not faulting if being fenced after having become exclusive

* Fix issue with disconnect while waiting for exlusive

* Fix missing cancellation on send

* Add Fenced producer state

* Unit tests adapted to pass with new fenced status

* Review
---
 CHANGELOG.md   |  2 ++
 src/DotPulsar/Internal/Abstractions/IChannel.cs|  1 +
 .../Abstractions/IContainsProducerChannel.cs   |  2 +-
 .../Internal/Abstractions/IProducerChannel.cs  |  1 -
 src/DotPulsar/Internal/Abstractions/Process.cs |  5 +++
 src/DotPulsar/Internal/Channel.cs  |  3 ++
 src/DotPulsar/Internal/ChannelManager.cs   | 15 ++---
 .../ProducerChannelConnected.cs}   | 18 +-
 src/DotPulsar/Internal/NotReadyChannel.cs  |  2 --
 src/DotPulsar/Internal/Producer.cs | 36 +---
 src/DotPulsar/Internal/ProducerChannel.cs  |  6 +---
 src/DotPulsar/Internal/ProducerChannelFactory.cs   | 12 +--
 src/DotPulsar/Internal/ProducerProcess.cs  |  8 +++--
 src/DotPulsar/Internal/ProducerResponse.cs |  4 +--
 src/DotPulsar/Internal/SubProducer.cs  | 39 ++
 src/DotPulsar/ProducerState.cs | 17 ++
 tests/DotPulsar.Tests/ProducerTests.cs |  8 ++---
 17 files changed, 121 insertions(+), 58 deletions(-)

diff --git a/CHANGELOG.md b/CHANGELOG.md
index b6e6976..cfd3fab 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -9,10 +9,12 @@ The format is based on [Keep a 
Changelog](https://keepachangelog.com/en/1.0.0/)
 ### Added
 
 - Support `ProducerAccessMode` to prevent multiple producers on a single topic.
+- Added `Fenced` state for producers which is a final state. 
 
 ### Fixed
 
 - Fixed issue with `Send` extension methods that do include `MessageMetadata` 
in the parameter list. The issue prevents more than two messages from being 
published on namespaces where deduplication is enabled.
+- Calling `await send(...)` on a Producer did not correctly terminate with an 
exception when a send operation failed, e.g. because the producer faulted.
 
 ## [2.11.0] - 2023-03-13
 
diff --git a/src/DotPulsar/Internal/Abstractions/IChannel.cs 
b/src/DotPulsar/Internal/Abstractions/IChannel.cs
index e468a03..8269677 100644
--- a/src/DotPulsar/Internal/Abstractions/IChannel.cs
+++ b/src/DotPulsar/Internal/Abstractions/IChannel.cs
@@ -23,6 +23,7 @@ public interface IChannel
 void ClosedByServer();
 void WaitingForExclusive();
 void Connected();
+void ProducerConnected(ulong topicEpoch);
 void Deactivated();
 void Disconnected();
 void ReachedEndOfTopic();
diff --git a/src/DotPulsar/Internal/Abstractions/IContainsProducerChannel.cs 
b/src/DotPulsar/Internal/Abstractions/IContainsProducerChannel.cs
index 550be53..0558216 100644
--- a/src/DotPulsar/Internal/Abstractions/IContainsProducerChannel.cs
+++ b/src/DotPulsar/Internal/Abstractions/IContainsProducerChannel.cs
@@ -19,5 +19,5 @@ using System.Threading.Tasks;
 
 public interface IContainsProducerChannel : IContainsChannel
 {
-Task ActivateChannel(CancellationToken cancellationToken);
+Task ActivateChannel(ulong? topicEpoch, CancellationToken 
cancellationToken);
 }
diff --git a/src/DotPulsar/Internal/Abstractions/IProducerChannel.cs 
b/src/DotPulsar/Internal/Abstractions/IProducerChannel.cs
index 79e7e37..4bdbdeb 100644
--- a/src/DotPulsar/Internal/Abstractions/IProducerChannel.cs
+++ b/src/DotPulsar/Internal/Abstractions/IProducerChannel.cs
@@ -22,7 +22,6 @@ using System.Threading.Tasks;
 
 public interface IProducerChannel : IAsyncDisposable
 {
-ulong? TopicEpoch { get; }
 Task Send(MessageMetadata metadata, ReadOnlySequence payload, 
TaskCompletionSource responseTcs, CancellationToken 
cancellationToken);
 ValueTask ClosedByClient(CancellationToken cancellationToken);
 }
diff --git a/src/DotPulsar/Internal/Abstractions/Process.cs 
b/src/DotPulsar/Internal/Abstractions/Process.cs
index f6d42c5..d509bfd 100644
--- a/src/DotPulsar/Internal/Abstractions/Process.cs
+++ b/src/DotPulsar/Internal/Abstractions/Process.cs
@@ -35,6 +35,7 @@ public abstract class Process : IProcess
 }
 
 public Guid CorrelationId { get; }
+protected ulong? TopicEpoch { get; private set; }
 
 public void Start()
 => CalculateState();

[GitHub] [pulsar-dotpulsar] blankensteiner merged pull request #149: Producer access mode fixes

2023-05-01 Thread via GitHub 


blankensteiner merged PR #149:
URL: https://github.com/apache/pulsar-dotpulsar/pull/149


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-client-go] Gilthoniel closed pull request #913: [Issue 801] Properly apply the back-off policy

2023-05-01 Thread via GitHub 


Gilthoniel closed pull request #913: [Issue 801] Properly apply the back-off 
policy
URL: https://github.com/apache/pulsar-client-go/pull/913


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org