2020-08-12 11:30:12 UTC - Aaron: Any thoughts on this? Does the way Avro is
sent over Pulsar (i.e. the encoding) differ if it is sent using AUTO_PRODUCE
versus if it is sent with a producer that is of an Avro schema type?
----
2020-08-12 12:08:52 UTC - Aaron: On top of this, how does schema compatibility
work with AUTO_PRODUCE?
----
2020-08-12 12:30:44 UTC - Nikola Socec: @Nikola Socec has joined the channel
----
2020-08-12 12:38:13 UTC - Nikola Socec: Hi all,
I'm planning to set up a pulsar cluster and I'm wondering what specification
the cluster should have. It will be handling 1 MB/s of data on average, but it
will come in bursts about every 30 seconds. I'm interested specifically in the
amount of brokers, bookies and zookeepers I should use, and in the amount of
RAM the cluster should have to keep the caches from overflowing.
----
2020-08-12 12:42:13 UTC - Elina Melamed: @Elina Melamed has joined the channel
----
2020-08-12 15:36:05 UTC - Charvak Patel: @Charvak Patel has joined the channel
----
2020-08-12 15:57:13 UTC - Alexandre DUVAL: ok i thought when zero functions
worker are up the state was keep and restart node restart the functions stopped
etc
----
2020-08-12 16:08:27 UTC - Alexandre DUVAL: What can be the reason of
```Aug 12 16:03:29 yo-pulsar-c1-n1 pulsar-broker[19142]: 16:03:29.678
[pulsar-web-42-7] ERROR org.apache.pulsar.broker.web.PulsarWebResource -
Finding owner for topic <persistent://yo/accesslogsinvalid/input-par-adc-n16>
timed out
Aug 12 16:03:29 yo-pulsar-c1-n1 pulsar-broker[19142]:
java.util.concurrent.TimeoutException: null
Aug 12 16:03:29 yo-pulsar-c1-n1 pulsar-broker[19142]: at
java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1771)
~[?:1.8.0_192]
Aug 12 16:03:29 yo-pulsar-c1-n1 pulsar-broker[19142]: at
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
~[?:1.8.0_192]
Aug 12 16:03:29 yo-pulsar-c1-n1 pulsar-broker[19142]: at
org.apache.pulsar.broker.namespace.NamespaceService.getWebServiceUrl(NamespaceService.java:214)
~[org.apache.pulsar-pulsar-broker-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
Aug 12 16:03:29 yo-pulsar-c1-n1 pulsar-broker[19142]: at
org.apache.pulsar.broker.web.PulsarWebResource.validateTopicOwnership(PulsarWebResource.java:601)
~[org.apache.pulsar-pulsar-broker-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
Aug 12 16:03:29 yo-pulsar-c1-n1 pulsar-broker[19142]: at
org.apache.pulsar.broker.admin.impl.PersistentTopicsBase.internalGetStats(PersistentTopicsBase.java:894)
~[org.apache.pulsar-pulsar-broker-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
Aug 12 16:03:29 yo-pulsar-c1-n1 pulsar-broker[19142]: at
org.apache.pulsar.broker.admin.v2.PersistentTopics.getStats(PersistentTopics.java:496)
~[org.apache.pulsar-pulsar-broker-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
....```
----
2020-08-12 16:08:33 UTC - Alexandre DUVAL: (will investigate)
----
2020-08-12 16:11:36 UTC - Walter: what is the command to check bookkeeper
corresponding zookeeper?
----
2020-08-12 16:25:15 UTC - Alexandre DUVAL: @Sijie Guo not sure how to
investigate this exception, any idea?
----
2020-08-12 16:26:18 UTC - Alexandre DUVAL: 30s zk timeout is too low? seems
weird
----
2020-08-12 16:48:19 UTC - Alexandre DUVAL: ```Aug 12 16:47:45 yo-pulsar-c1-n1
pulsar-broker[19142]: 16:47:45.235 [pulsar-web-42-6] ERROR
org.apache.pulsar.broker.web.PulsarWebResource - [broker-to-broker] Failed to
check whether namespace bundle is owned
yo/accesslogsinvalid/0x00000000_0x40000000
Aug 12 16:47:45 yo-pulsar-c1-n1 pulsar-broker[19142]:
java.util.concurrent.TimeoutException: null
Aug 12 16:47:45 yo-pulsar-c1-n1 pulsar-broker[19142]: at
java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1771)
~[?:1.8.0_192]
Aug 12 16:47:45 yo-pulsar-c1-n1 pulsar-broker[19142]: at
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
~[?:1.8.0_192]
Aug 12 16:47:45 yo-pulsar-c1-n1 pulsar-broker[19142]: at
org.apache.pulsar.broker.namespace.NamespaceService.getWebServiceUrl(NamespaceService.java:224)
~[org.apache.pulsar-pulsar-broker-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
Aug 12 16:47:45 yo-pulsar-c1-n1 pulsar-broker[19142]: at
org.apache.pulsar.broker.web.PulsarWebResource.isBundleOwnedByAnyBroker(PulsarWebResource.java:499)
~[org.apache.pulsar-pulsar-broker-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
Aug 12 16:47:45 yo-pulsar-c1-n1 pulsar-broker[19142]: at
org.apache.pulsar.broker.admin.v2.NonPersistentTopics.getListFromBundle(NonPersistentTopics.java:323)
~[org.apache.pulsar-pulsar-broker-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
Aug 12 16:47:45 yo-pulsar-c1-n1 pulsar-broker[19142]: at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_192]
Aug 12 16:47:45 yo-pulsar-c1-n1 pulsar-broker[19142]: at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_192]
...```
----
2020-08-12 16:48:20 UTC - Alexandre DUVAL: related
----
2020-08-12 16:59:10 UTC - Yarden Arane: *FUNCTIONS*: logging
Hi all, when using the logger inside a Pulsar Function, I expect to see the log
message in two places:
1. the log file (location determined by `pulsar/conf/functions_worker.yaml` )
```functionRuntimeFactoryConfigs:
# location of log files for functions
logDirectory: /tmp```
for example, `/tmp/functions/public/default/function_name/function_name-0.log`
2. the log topic of the function.
The Pulsar docs recommend to use `context.getLogger` , however I don't see the
log message on the log topic when using the context's logger.
I can, however, get the log message to appear on the log topic (and log file)
if I don't use the context's logger as seen below. Any ideas why?
```import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class LoggingFunction implements Function<String, Void> {
@Override
public void apply(String input, Context context) {
Logger LOG = context.getLogger();
Logger LOG2 = LoggerFactory.getLogger(this.getClass);
<http://LOG.info|LOG.info>("This message appears in log file only")
<http://LOG2.info|LOG2.info>("This message appears in log file AND log
topic")
}
}```
----
2020-08-12 17:12:08 UTC - Alexandre DUVAL: it's limited to some topics only
:confused:
----
2020-08-12 17:12:14 UTC - Alexandre DUVAL: i restarted brokers
----
2020-08-12 17:12:17 UTC - Alexandre DUVAL: same topics have issue
----
2020-08-12 18:43:21 UTC - vikash: Hi All, I want to monitored apache
pulsar using jmx exporter for that i need to understand how we will
connect pulsar standalone with jmx Port so that i can analyse the
Cpu,thread ,processs etc ,i could not able to find any doc on this
stuff
----
2020-08-12 19:49:07 UTC - Joshua Decosta: Does anyone know how I can add a
custom configuration variable using the helm deployment to the proxy and
broker? I keep trying to simply add it within their respective configmaps but
it’s not taken once the broker/proxy is deployed
----
2020-08-12 20:20:34 UTC - Joel Cressy: Are there any projects out there
developing a web UI for tenants? Like, non-admin operations that give users
(e.g. via SSO) access to control one or more tenants on pulsar clusters.
For example, <http://kesque.io|kesque.io> has a dashboard that lets you sign up
and get a tenant on a cluster. In my scenario I'd love to deploy an interface
like this to users within my own company where they sign in through our SSO and
can provision tenants for their development teams.
----
2020-08-12 20:21:48 UTC - Joel Cressy: BTW this would be in a self-hosted
environment, so no SaaS solutions.
----
2020-08-12 20:22:34 UTC - Addison Higham: @Joshua Decosta you can add
additional entries to the helm chart with `{broker|proxy}.configData` property
in the values. If you prefix the values with `PULSAR_PREFIX_` you can add
values that aren't already in the config file
+1 : Joshua Decosta
----
2020-08-12 20:23:06 UTC - Addison Higham: have you seen
<https://github.com/apache/pulsar-manager>?
----
2020-08-12 20:23:34 UTC - Addison Higham: it has pluggable authz/authn, which
you would likely need to implement/change
----
2020-08-12 20:23:41 UTC - Addison Higham: but everything else should be there
----
2020-08-12 20:25:35 UTC - Joel Cressy: pulsar manager seems limited to cluster
operators, no?
----
2020-08-12 20:30:17 UTC - Chris Bartholomew: Joel, we have a Beta version of
the Kesque dashboard that can be self hosted in Kubernetes.
----
2020-08-12 20:32:22 UTC - Joel Cressy: Is this on github?
----
2020-08-12 20:32:51 UTC - Chris Bartholomew: No.
----
2020-08-12 20:33:10 UTC - Joel Cressy: Oh, is this a part of a product offering?
----
2020-08-12 20:33:47 UTC - Chris Bartholomew: Yes, we are making it available to
our customers on certain plans.
----
2020-08-12 20:34:51 UTC - Addison Higham: @Joel Cressy I believer it also be
usable for users as well, it is isn't super well documented at the moment and I
am not sure on all the details, but it should be possible for users to only
manage their tenants
----
2020-08-12 20:49:25 UTC - Addison Higham: @Nikola Socec 1 MB/s is not going to
require too much hardware, a single pulsar standalone instance on my laptop
(with 2 GB of heap) can easily sustain 50 MB/s. Your biggest factor is just
going to be how much fault tolerance you want, with 2+ brokers, 3 bookies, and
3 ZK nodes being the recommendation for really minimizing downtime. But each of
those containers/VMs may only need 1 CPU and a gig or two of memory.
----
2020-08-12 20:50:02 UTC - Addison Higham: @Alexandre DUVAL how many topics are
you working with?
----
2020-08-12 20:50:38 UTC - Addison Higham: but looks like it could be a ZK
issue, using the zkcli or another ZK ui can you browse the data?
----
2020-08-12 20:51:11 UTC - Alexandre DUVAL: there is currently not a lot of
topics, like ~100
----
2020-08-12 20:52:30 UTC - Addison Higham: @vikash The pulsar startup script
doesn't really support setting all the JMX stuff for you, but you easily could
add custom args for JMX. Pulsar leans pretty heavily on Prometheus and it is
configured out of the box, and that can easily give you all those metrics
----
2020-08-12 20:53:49 UTC - Addison Higham: I think this is a bug, I ran into it
a few months ago and I think I shared some details, but I am not sure they made
it into an issue. If you don't see an issue, filing one would be really
appreciated :slightly_smiling_face:
----
2020-08-12 20:54:13 UTC - Addison Higham: the fix should be fairly easy, but
the tl;dr is that the way pulsar functions creates the context isn't correct
----
2020-08-12 21:09:59 UTC - Alexandre DUVAL: @Addison Higham i can browse data
----
2020-08-12 21:10:11 UTC - Alexandre DUVAL: but im currently not having the
bundle issue anymore
----
2020-08-12 21:10:15 UTC - Alexandre DUVAL: i'm only having stuffs like
----
2020-08-12 21:10:42 UTC - Alexandre DUVAL: ```Aug 12 21:10:27
clevercloud-pulsar-c1-n1 pulsar-broker[14970]: 21:10:27.409 [pulsar-web-42-7]
ERROR org.apache.pulsar.broker.web.PulsarWebResource - Finding owner for topic
<persistent://clevercloud/accesslogs/iadvize-adc-n3> timed out
Aug 12 21:10:27 clevercloud-pulsar-c1-n1 pulsar-broker[14970]:
java.util.concurrent.TimeoutException: null
Aug 12 21:10:27 clevercloud-pulsar-c1-n1 pulsar-broker[14970]: at
java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1771)
~[?:1.8.0_192]
Aug 12 21:10:27 clevercloud-pulsar-c1-n1 pulsar-broker[14970]: at
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
~[?:1.8.0_192]
Aug 12 21:10:27 clevercloud-pulsar-c1-n1 pulsar-broker[14970]: at
org.apache.pulsar.broker.namespace.NamespaceService.getWebServiceUrl(NamespaceService.java:214)
~[org.apache.pulsar-pulsar-broker-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
Aug 12 21:10:27 clevercloud-pulsar-c1-n1 pulsar-broker[14970]: at
org.apache.pulsar.broker.web.PulsarWebResource.validateTopicOwnership(PulsarWebResource.java:601)
~[org.apache.pulsar-pulsar-broker-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
Aug 12 21:10:27 clevercloud-pulsar-c1-n1 pulsar-broker[14970]: at
org.apache.pulsar.broker.admin.impl.PersistentTopicsBase.internalGetStats(PersistentTopicsBase.java:894)
~[org.apache.pulsar-pulsar-broker-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
Aug 12 21:10:27 clevercloud-pulsar-c1-n1 pulsar-broker[14970]: at
org.apache.pulsar.broker.admin.v2.PersistentTopics.getStats(PersistentTopics.java:496)
~[org.apache.pulsar-pulsar-broker-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
Aug 12 21:10:27 clevercloud-pulsar-c1-n1 pulsar-broker[14970]: at
sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source) ~[?:?]
Aug 12 21:10:27 clevercloud-pulsar-c1-n1 pulsar-broker[14970]: at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_192]
Aug 12 21:10:27 clevercloud-pulsar-c1-n1 pulsar-broker[14970]: at
java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_192]```
----
2020-08-12 21:11:15 UTC - Yarden Arane: Thanks Addison, will do.
----
2020-08-12 21:12:32 UTC - Alexandre DUVAL: and I just restarted one broker and
everything now works fine
----
2020-08-12 21:12:41 UTC - Alexandre DUVAL: seems consistencies issue
----
2020-08-12 21:12:46 UTC - Alexandre DUVAL: weird..
----
2020-08-12 21:13:09 UTC - Alexandre DUVAL: this is not the first time it happens
----
2020-08-12 21:13:16 UTC - Alexandre DUVAL: i really don't know how to debug/fix
that
----
2020-08-12 21:15:47 UTC - Alexandre DUVAL: it seems to be problem on broker zk
cache or something which is not "fresh"
----
2020-08-12 21:16:03 UTC - Alexandre DUVAL: maybe i need tweak some broker
configuration fields
----
2020-08-12 21:16:04 UTC - Alexandre DUVAL: dunno
----
2020-08-12 21:16:29 UTC - Alexandre DUVAL: @Sijie Guo FYI^
----
2020-08-12 21:20:06 UTC - Alexandre DUVAL: what value do you have for
```# ZooKeeper cache expiry time in seconds
zooKeeperCacheExpirySeconds=300?```
----
2020-08-12 21:32:34 UTC - Evan Furman: Hi! I’m trying to use the grafana dash
here for our ec2 deployment. When running the generate_dashboards.sh script, it
prompts me for the loki datasource. Is this required to use the dashboard? We
are using prometheus.
<https://github.com/streamnative/apache-pulsar-grafana-dashboard>
----
2020-08-12 21:44:06 UTC - Sijie Guo: @Joel Cressy In Pulsar Manager 0.2.0, we
are introducing the Users concept. The goal for Pulsar Manager is to make it a
self-serve portal for the developers using Pulsar.
----
2020-08-12 21:45:00 UTC - Sijie Guo: loki datasource is not required.
heart : Evan Furman
----
2020-08-12 21:46:38 UTC - Sijie Guo: I am not sure it is a zookeeper issue. It
sounds like hitting an issue I fixed in master (which will be released in
2.6.1).
----
2020-08-12 21:48:38 UTC - Sijie Guo: check this issue
<https://github.com/apache/pulsar/pull/7319>
----
2020-08-12 21:51:05 UTC - Sijie Guo: Do you have an application to reproduce
your issue?
----
2020-08-12 22:10:07 UTC - Alexandre DUVAL: I get the error on calling /stats on
topic
----
2020-08-12 22:11:10 UTC - Sijie Guo: So it most likely stuck at loading a topic
----
2020-08-12 22:11:46 UTC - Alexandre DUVAL: yes, and a restart can rely on this?
----
2020-08-12 22:11:49 UTC - Alexandre DUVAL: (of broker)
----
2020-08-12 22:14:14 UTC - Sijie Guo: yes
----
2020-08-12 22:15:56 UTC - Sijie Guo: If you want to try that fix before the
official release is cut, you can try
<https://github.com/streamnative/pulsar/tree/v2.6.0-sn-12>
----
2020-08-12 22:49:25 UTC - Naby: @Naby has joined the channel
----
2020-08-12 23:55:36 UTC - Kirill Kosenko: Hello guys
I need to publish events with different AVRO schemas to the same topic to keep
ordering.
For example: UserCreatedEvent, UserAddressChangedEvent
Unfortunately, Pulsar doesn't support multischema topics at the moment. AFAIK,
there's work in progress.
Could someone point me on how to achieve that if it's possible.
Any help is appreciated.
Thanks
----
2020-08-13 00:15:19 UTC - Penghui Li: Hi @Kirill Kosenko Pulsar already support
multiple schemas in a topic. Maybe
<http://pulsar.apache.org/docs/en/schema-evolution-compatibility/> can help you
to understand multiple schemas.
If you want to set multiple schemas for a topic and there are no compatibility
requirements between schemas, you can set the schema compatibility policy to
`ALWAYS_COMPATIBLE`. And when you consume messages, you can use `AUTO_CONSUME`
for the consumer. So that you can get a `GenericRecord` which is a generalized
structure.
----
2020-08-13 00:40:36 UTC - Kirill Kosenko: Thank you for a quick reply. I missed
this approach
----
2020-08-13 00:52:22 UTC - Evan Furman: Thank you @Sijie Guo
----
2020-08-13 03:08:16 UTC - Alexandre DUVAL: will do it!
----
2020-08-13 03:41:15 UTC - xue: When function is created, if there are 100
concurrent functions, the CPU of the functions worker node will rise
rapidly，The functions worker cluster has 7 nodes.
----
2020-08-13 06:51:41 UTC - markg: thank you!
----
2020-08-13 06:52:20 UTC - markg: Will read though - I have not had a chance to
look here since asking the question but truly appreciate the links and response!
----
2020-08-13 07:49:39 UTC - Seun: Hi folks,
Can anyone please help me with beginner level tutorials on how to run pulsars
on GKE on google cloud. And have it listen to several microservices running on
another GKE cluster? Is it advisable to run both within the same cluster or
separate clusters? I need to get on quickly with Apache pulsar of k8s project.
All my google searches have led me to inadequate or some advance concepts and
they require enormous GKE resources for deployment.
----
2020-08-13 08:23:33 UTC - Nikola Socec: Thank you very much.
----
2020-08-13 08:53:44 UTC - Christophe Bornet: Hi, in the Java client, when
`maxPendingMessages` is reached, the calls to send() and sendAsync() will
either throw an error or block depending on `BlockIfQueueFull` . This is a push
mode. I think it would also be interesting to have a pull mode with an observer
pattern where the driver requests new messages to the application. Kind of like
the gRPC-java driver does it. This would help dealing with back-pressure when
working async and to bind the driver with reactive frameworks (rxJava, Reactor,
...). WDYT ?
----
