2020-03-20 09:35:39 UTC - xue: Can 2 broker clusters use 1 bookkeeper cluster
together? Or can 2 function worker clusters use 1 bookkeeper cluster together?
----
2020-03-20 12:37:29 UTC - Prashant Shandilya: Wish to use Pulsar BookKeeper as
event store, and represent each aggregate instance with topic. This way it may
create lots of topics (millions for production system). Would consume these
topics using pattern something line below -
_Pattern allTopicsInNamespace =
Pattern.compile("<persistent://public/default/.*>");_
_Consumer allTopicsConsumer = consumerBuilder_
_.topicsPattern(allTopicsInNamespace)_
_.subscribe();_
Can someone help me validating, this approach. Is it advisable to use topic per
aggregate (entity) instance, can it Pulsar handle the load ? Can consumer
handle multiple concurrent topic subscription ?
----
2020-03-20 13:32:30 UTC - Christophe Bornet: Hi. Broker unit tests are failing
for me on master. Is it a known issue or am I the only one to see this ?
----
2020-03-20 13:54:20 UTC - Chris: Here's a brief window of me consuming off the
topic. Has about 1 billion messages in backlog total, ~230m per partition
----
2020-03-20 13:54:56 UTC - Chris: And here's the trace from the consumer @Sijie
Guo
----
2020-03-20 14:15:30 UTC - Rolf Arne Corneliussen: FAILOVER SUBSCIPTION and
read-process-write pattern: For a partitioned topic, I have noticed that one
partition can be moved from one subscribing consumer to another rather
instantly - as observed by attaching a `ConsumerEventListener` to the consumer;
one consumer can get the `becameInactive` callback at the same instant as
another consumer get the `becameActive` callback. If you implement a
*read-process-write* pattern and you subscribe to the read topic in failover
mode, is there any chance you can have two processes writing to the same output
topic at the same time?
For example, if a topic/partition is deactivated on a consumer while it is
processing a batch of input messages and writing to the output topic. After a
switchover, the input message would be processed again by the new, active
consumer, but original consumer might still be processing and writing.
Or what will happen if the network link between an active consumer and the
broker (input topic) is dropped while the network link to the broker owning the
output topic is not affected? Can we guarantee that the order of the messages
on the output topic is consistent with the ordering on the input topic?
Any thoughts appreciated, also if someone could point me to the classes in the
broker that are responsible for coordination of the consumers of a topic with
failover subscription, it would be helpful.
Thanks.
----
2020-03-20 14:42:23 UTC - Christophe Bornet: It seems due to this commit :
<https://github.com/apache/pulsar/commit/9b469301e0beef765dcd2787b8383666b121d9f6>
----
2020-03-20 14:46:39 UTC - Christophe Bornet: Adding
```policies.clusterDispatchRate = Maps.newHashMap();```
in the ServerCnxTest tests fixes it
----
2020-03-20 14:58:24 UTC - Christophe Bornet: It's strange. I'm pretty sure
there's a bug here. Yet it seems to have been unnoticed for more than a month.
How can this be possible ?
----
2020-03-20 15:57:32 UTC - Chris: Should `pulsar-admin schemas delete` delete
that schema from the zookeeper? I've done some digging and found some stray zk
nodes that might be affecting further schema operations. They're not even
deleted if the namespace is deleted.
----
2020-03-20 16:28:35 UTC - Pradeesh: that works
----
2020-03-20 16:28:49 UTC - Pradeesh:
----
2020-03-20 17:12:08 UTC - Pradeesh: just keeps loading
----
2020-03-20 17:12:18 UTC - Pradeesh:
----
2020-03-20 17:35:22 UTC - lowkey: @lowkey has joined the channel
----
2020-03-20 17:46:59 UTC - Bryce Myers: @Bryce Myers has joined the channel
----
2020-03-20 17:51:27 UTC - Bryce Myers: Does the Pulsar broker and proxy support
multiple CA's (non-chained) for authentication?
----
2020-03-20 17:52:34 UTC - Matteo Merli: The CAs are being read from the
"trust-store". That can contain multiple CAs
----
2020-03-20 17:55:07 UTC - Bryce Myers: I'm pointing Bookkeeper and Zookeeper to
a truststore but for the other components I'm pointing them at a cert filepath.
Is there a way to point all components to a truststore?
----
2020-03-20 18:37:04 UTC - Sijie Guo: Are you able to `
```curl -v <http://172.27.223.166:8080/metrics/>```
from the pulsar manager instance?
----
2020-03-20 18:44:19 UTC - Pradeesh: there is no curl in the conatiner but wget
works
----
2020-03-20 18:44:23 UTC - Pradeesh: ```/pulsar-manager # wget
<http://172.27.223.166:8080/metrics/>
Connecting to 172.27.223.166:8080 (172.27.223.166:8080)
index.html 100%
|**************************************************************************************************************************************************************************************************************************************|
15154 0:00:00 ETA
/pulsar-manager #```
----
2020-03-20 18:44:38 UTC - Pradeesh: from the container
----
2020-03-20 18:47:16 UTC - Aaron Stockton: @Sijie Guo is that the intent of the
PR?
----
2020-03-20 19:04:02 UTC - Ivy Rogatko: @Ivy Rogatko has joined the channel
----
2020-03-20 19:45:29 UTC - Evan Furman: just following back up to see if we can
dig further.
```Our latest tests show that the EKS Pulsar cluster is in the range of the EC2
cluster for producing: 75K/s for EKS vs 85K/s for EC2 at this time. However,
EC2 is still the clear winner in consuming from backlog: 20K/s for EKS vs up to
200K/s for EC2.```
----
2020-03-20 19:46:44 UTC - Evan Furman: consuming from the backlog is very
variable tho — using pulsar-perf I see speeds from `18000 msg/s` to `75000
msg/s`
----
2020-03-20 21:04:22 UTC - Matt Mitchell: Thanks @David Kjerrumgaard
----
2020-03-21 02:34:56 UTC - Penghui Li: The broker does not track unacked
messages for exclusive subscription and failover subscription before. So this
PR keep the same behavior as before. For exclusive subscription and failover
subscription, users can use cumulative acknowledger. This makes calculating
unacked messages expensive.
----
2020-03-21 08:58:50 UTC - Vladimir Shchur: I've found 3 ways to define schema,
what is a recommended way?
----
2020-03-21 09:06:47 UTC - Ebere Abanonu: Looking at the Transaction Example, I
want to ask if transition can be distributed? How would a remote consumer ack
with a transaction. In the example given, transaction instance, producer, and
consumer are in a single process.
----
