2019-03-13 09:22:45 UTC - Maarten Tielemans: Looking at Prometheus stats atm.
What is the definition of "Add entry rate" (is a ~constant 500, we send 10000
msg/sec) and "Journal Queue Length" (Is currently at negative 31 million)?
----
2019-03-13 09:37:26 UTC - Sam Leung:
<https://github.com/apache/pulsar/issues/3810> :slightly_smiling_face:
----
2019-03-13 09:37:49 UTC - Sam Leung: Cool! I will try playing with that. thanks!
----
2019-03-13 10:41:14 UTC - Ben S: Hi again. I have a question about giving local
brokers higher priority. Lets say I have 100 shops around the globe. There are
brokers on AWS and elsewhere to provide global messaging between the shops. Now
assume the volume of a topic „hightraffic“ increases drastically as I enter a
shop with a device (e.g. smartphone). The topic messages shall be handled with
lowest possibly latency in case I‘m connected to the shops local network,
latency is not that much an issue when I leave the shop and the smartphone is
connected via LTE or 3G. In order to handle the latency, the shops will get an
own broker locally on site. What‘s the best practice to make sure the topic is
handled on the local low latency broker once connected to the shops network?
----
2019-03-13 11:01:09 UTC - Shivji Kumar Jha: HI, Is it possible to disable
schema compatibility check on the broker? I am looking for none. It seems only
possible values are: Full, Backward, forward
From the doc
```$ pulsar-admin namespaces set-schema-autoupdate-strategy tenant/namespace
options
Options: Compatibility level required for new schemas created via a Producer.
Possible values (Full, Backward, Forward). ```
----
2019-03-13 11:03:50 UTC - Shivji Kumar Jha: Even the enum in code suggests so:
```public enum SchemaAutoUpdateCompatibilityStrategy {
/**
* Don't allow any auto updates.
*/
AutoUpdateDisabled,
/**
* Messages written in the previous schema can be read by the new schema.
* To be backward compatible, the new schema must not add any new fields
that
* don't have default values. However, it may remove fields.
*/
Backward,
/**
* Messages written in the new schema can be read by the previous schema.
* To be forward compatible, the new schema must not remove any fields which
* don't have default values in the previous schema. However, it may add
new fields.
*/
Forward,
/**
* Backward and Forward.
*/
Full
}```
----
2019-03-13 11:55:13 UTC - Alexandre DUVAL: Hello, :slightly_smiling_face:
```kannar@pond ~/pulsar/logstash-output-pulsar/pulsar/conf/keys
(git)-[testWithPublicKey] % ../../bin/pulsar-admin topics list yo/functions
<persistent://yo/functions/assignments>
<persistent://yo/functions/coordinate>
<persistent://yo/functions/metadata>
kannar@pond ~/pulsar/logstash-output-pulsar/pulsar/conf/keys
(git)-[testWithPublicKey] % ../../bin/pulsar-admin topics list yo/logs
HTTP 500 Server Error
Reason: HTTP 500 Server Error``` I'm using the super user. I was able to list
yo/logs until I produced messages on it. But now I 500, and brokers logs with
401 ```11:50:49.628 [pulsar-web-29-8] INFO org.eclipse.jetty.server.RequestLog
- 192.168.10.11 - - [13/Mar/2019:11:50:49 +0000] "GET
/admin/v2/non-persistent/yo/logs/0xc0000000_0xffffffff HTTP/1.1" 200 50 "-"
"Jersey/2.27 (HttpUrlConnection 1.8.0_192)" 18
11:50:49.668 [pulsar-web-29-8] INFO org.eclipse.jetty.server.RequestLog -
192.168.10.11 - - [13/Mar/2019:11:50:49 +0000] "GET
/admin/v2/non-persistent/yo/logs/0x00000000_0x40000000 HTTP/1.1" 307 0 "-"
"Jersey/2.27 (HttpUrlConnection 1.8.0_192)" 16
11:50:49.669 [pulsar-web-29-6] INFO org.eclipse.jetty.server.RequestLog -
192.168.10.11 - - [13/Mar/2019:11:50:49 +0000] "GET
/admin/v2/non-persistent/yo/logs/0x80000000_0xc0000000 HTTP/1.1" 307 0 "-"
"Jersey/2.27 (HttpUrlConnection 1.8.0_192)" 12
11:50:49.673 [pulsar-web-29-7] INFO org.eclipse.jetty.server.RequestLog -
192.168.10.11 - - [13/Mar/2019:11:50:49 +0000] "GET
/admin/v2/non-persistent/yo/logs/0x40000000_0x80000000 HTTP/1.1" 307 0 "-"
"Jersey/2.27 (HttpUrlConnection 1.8.0_192)" 8
11:50:49.712 [pulsar-web-29-3] ERROR
org.apache.pulsar.broker.admin.v2.NonPersistentTopics - [admin] Failed to get
list of topics under namespace yo/logs
java.util.concurrent.ExecutionException:
org.apache.pulsar.client.admin.PulsarAdminException$NotAuthorizedException:
HTTP 401 Authentication required
``` Super user is admin as you can see in these logs.
----
2019-03-13 12:04:16 UTC - Alexandre DUVAL: Is it something related to
```kannar@pond ~/pulsar/logstash-output-pulsar/pulsar/conf/keys
(git)-[testWithPublicKey] % ../../bin/pulsar-admin namespaces policies yo/logs
[1][13:00:57 - 13/03/2019]
{
"auth_policies" : {
"namespace_auth" : {
"yo.logsapi" : [ "consume" ],
"yo.par0.logscollector" : [ "produce" ],
"proxy-to-broker" : [ "consume", "produce" ]
},
"destination_auth" : {
"<persistent://yo/logs/full>" : {
"yo.*.logscollector" : [ "produce" ]
}
},
"subscription_auth_roles" : { }
},``` destination_auth? :open_mouth:
----
2019-03-13 12:12:34 UTC - Alexandre DUVAL: And btw how to clean if wrong config
on destination_auth?
----
2019-03-13 12:30:34 UTC - Alexandre DUVAL: ```kannar@pond
~/pulsar/logstash-output-pulsar/pulsar/conf/keys (git)-[testWithPublicKey] %
../../bin/pulsar-admin topics list yo/logs
HTTP 500 Server Error
Reason: HTTP 500 Server Error
kannar@pond ~/pulsar/logstash-output-pulsar/pulsar/conf/keys
(git)-[testWithPublicKey] % ../../bin/pulsar-admin topics
list-partitioned-topics yo/logs
<persistent://yo/logs/full>``` Hum, so it still a problem on NonPersistents.
----
2019-03-13 13:06:37 UTC - Shivji Kumar Jha: @Sijie Guo
----
2019-03-13 14:44:48 UTC - Darragh: Anyone got some insight in what the journal
queue length metric entails ?
----
2019-03-13 15:51:15 UTC - FreedomJack: @FreedomJack has joined the channel
----
2019-03-13 16:09:10 UTC - Alexandre DUVAL: During broker to broker request, I
think that the bearer is not forwarded as header when a broker node receive 307
to another broker node
----
2019-03-13 16:09:16 UTC - Alexandre DUVAL: using JWT auth
----
2019-03-13 16:17:15 UTC - Alexandre DUVAL: considering this cap.
----
2019-03-13 16:17:59 UTC - Alexandre DUVAL: @Matteo Merli WDYT
----
2019-03-13 16:19:49 UTC - Matteo Merli: @Maarten Tielemans Add entry is the
rate perceived by bookies.
Clients can batch multiple messages into a single entry. That’s where the
difference comes from. If you look at the rates from broker, it will have the
correct msg/s
----
2019-03-13 16:20:22 UTC - Matteo Merli: I think the “Journal Queue Length”
counter is indeed broken.. guess it’s only decremented and never incremented..
----
2019-03-13 16:23:48 UTC - Matteo Merli: @Kev Jackson Merged, thanks!
----
2019-03-13 16:26:00 UTC - Kev Jackson: np, I want to get back into open source
on a more regular basis and Pulsar looks like a lovely project (apart from mvn
which I dislike)
----
2019-03-13 17:15:42 UTC - Maarten Tielemans: @Matteo Merli The "batch multiple
messages into a single entry", thats a producer setting correct? I believe we
are running pulsar-perf with `-b 0`, shouldn't that disable batching?
----
2019-03-13 18:36:41 UTC - Sugi Venugeethan: @Sugi Venugeethan has joined the
channel
----
2019-03-13 19:08:19 UTC - Alexandre DUVAL: ..?
----
2019-03-13 21:15:08 UTC - JAYARAM NAGARAJAN: @JAYARAM NAGARAJAN has joined the
channel
----
2019-03-13 23:18:39 UTC - JAYARAM NAGARAJAN: Hello we are following instruction
in
<https://pulsar.apache.org/docs/en/deploy-bare-metal/#deploying-a-zookeeper-cluster>
, were successful with Zookeeper and Bookies, but getting an error with
Brokers, error is below, can someone let us know what setting is needed to be
updated:
21:47:43.595 [ForkJoinPool.commonPool-worker-0] WARN
org.apache.pulsar.broker.web.PulsarWebResource - Namespace missing local
cluster name in clusters list: local_cluster=pulsar-cluster-1
ns=public/functions clusters=[standalone]
21:47:43.610 [ForkJoinPool.commonPool-worker-0] WARN
org.apache.pulsar.broker.service.ServerCnx - Failed to get Partitioned Metadata
[/127.0.0.1:38086] <persistent://public/functions/assignments>: Namespace
missing local cluster name in clusters list: local_cluster=pulsar-cluster-1
ns=public/functions clusters=[standalone]
org.apache.pulsar.broker.web.RestException: Namespace missing local cluster
name in clusters list: local_cluster=pulsar-cluster-1 ns=public/functions
clusters=[standalone]
at
org.apache.pulsar.broker.web.PulsarWebResource.lambda$4(PulsarWebResource.java:646)
~[org.apache.pulsar-pulsar-broker-2.2.1.jar:2.2.1]
----
2019-03-14 01:58:55 UTC - Sijie Guo: it seems that you enabled function worker.
but you didn’t update the `cluster` field in function_worker.yml
you need to make sure the cluster field is updated -
<https://github.com/apache/pulsar/blob/master/conf/functions_worker.yml#L55>
when you are enabling function worker
----
2019-03-14 01:59:49 UTC - Karthik Palanivelu: Or you can start the cluster with
“—no-functions-worker”
----
2019-03-14 02:12:40 UTC - Sijie Guo: @Shivji Kumar Jha I don’t think it
supports to disabling schema check. that’s one of the tasks in my todo list for
supporting multi versions of schema. I will probably get to it late this week.
but if you are interested in contributing a change for that, that would be
great :slightly_smiling_face:
----
2019-03-14 02:55:28 UTC - Shivji Kumar Jha: Sure, I will contribute a patch by
this week, thanks :slightly_smiling_face:
----
2019-03-14 03:00:18 UTC - Clive Astbury: @Clive Astbury has joined the channel
----
2019-03-14 03:18:33 UTC - pingle wang: @pingle wang has joined the channel
----
2019-03-14 06:55:17 UTC - bhagesharora: Hello everyone,
What will be the steps for run a java basic producer and consumer program
through docker
I create a jar file and place the jar file in target directory inside the
pulsar.
----
2019-03-14 07:04:02 UTC - Ali Ahmed: @bhagesharora simplest thing would be use
the pulsar python client , there are simple producer consumer examples in
github and you can just startup the container in standalone mode
----
2019-03-14 07:04:21 UTC - Ali Ahmed: python client is already installed on the
docker image.
----
