2020-06-04 11:00:39 UTC - Adriaan de Haan: Hi all, I am writing a batch processing system with workers processing a topic with a shared subscription. I want to slow down my application producing work if my workers fall behind - so for that I want to monitor the worker subscription backlog and scale back my producer if the backlog starts to grow. It appears that the only way to get the backlog info is through the topic stats admin api: ```String destination = "<persistent://my-tenant/my-namespace/my-topic>"; admin.persistentTopics().getStats(destination);``` Would this be the correct approach? I was wondering how heavy this api call would be and how up-to-date the stats is - is the backlog updated realtime or periodically? ---- 2020-06-04 11:17:18 UTC - alex kurtser: Hello. Does somebody collect metrics from functions by Prometheus ?
I noticed that every function instance opens random port for metrics, so it impossible to scrape it dynamically by Prometheus ---- 2020-06-04 11:18:24 UTC - alex kurtser: Is it possible to make the metrics port constant on Functions container in order to scrape the metrics by Prometheus ? ---- 2020-06-04 11:45:37 UTC - Ebere Abanonu: @Sijie Guo will there be support for Azure Blob Storage for offloader? ---- 2020-06-04 13:49:42 UTC - Ebere Abanonu: If Apache Pulsar Tier Storage uses Apache jclouds to support cloud storage, then can it be deduced that Pulsar supports Azure Blob Storage and we can set `managedLedgerOffloadDriver` = `azureblob` ? eyes : Raman Gupta ---- 2020-06-04 14:54:19 UTC - Rafael Saraiva Figueiredo: @Rafael Saraiva Figueiredo has joined the channel ---- 2020-06-04 15:57:30 UTC - Anatoliy Carmen: @Anatoliy Carmen has joined the channel ---- 2020-06-04 16:17:31 UTC - Sijie Guo: Yes. @David Kjerrumgaard has an outstanding pull request for that. +1 : Ebere Abanonu ---- 2020-06-04 16:18:47 UTC - Ebere Abanonu: What version should we be expecting it? ---- 2020-06-04 16:18:51 UTC - Sijie Guo: Are you using k8s runtime or process runtime? ---- 2020-06-04 16:19:31 UTC - Sijie Guo: yes that’s the correct api. ---- 2020-06-04 16:33:24 UTC - alex kurtser: Hi Yes, we are. ---- 2020-06-04 16:34:39 UTC - alex kurtser: Probably any annotation i should add? But again the problem is the random port. Each function or each instance exposes the metrics on different and random port ---- 2020-06-04 16:41:28 UTC - Sijie Guo: If you are running in k8s, random port is not a problem. Because promethues scrap pods based on annotations. ---- 2020-06-04 16:41:40 UTC - Sijie Guo: It can be a problem using process/thread runtime ---- 2020-06-04 16:42:05 UTC - Sijie Guo: But you can get function stats from pulsar-admin rest api. ---- 2020-06-04 16:42:43 UTC - alex kurtser: Hi ---- 2020-06-04 16:43:17 UTC - alex kurtser: I know about annotations but i still need to specify a port number in the same annotation. Am i right ? ---- 2020-06-04 16:44:38 UTC - alex kurtser: We execute the functions in diffrent pods , separately from brokers . This is what you mean when say about "proccess/thread runtime" ? ---- 2020-06-04 17:47:42 UTC - Fabien LD: Hi Pulsar people, I am running pulsar in kubernetes (so via docker images) and I am unable to find in documentation how to trigger a graceful shutdown of a broker. I am more specifically interested in doing one for brokers that are in production. I can see the default 60s timeout for graceful shutdown, so it must be supported. In the doc I also see the `pulsar-daemon stop` command (<https://pulsar.apache.org/docs/en/2.4.2/reference-cli-tools/#stop>) but in my case it cannot find the broker since it has not been started with `pulsar-daemon start`.. I can see some bits to shutdown pulsar functions or pulsar in DC/OS, but nothing in my case. Even the doc about upgrade says "stop the broker" but not telling how to do it: <https://pulsar.apache.org/docs/en/2.4.2/administration-upgrade/> Any clue or ideally pointer to doc? ---- 2020-06-04 17:57:43 UTC - Matt Mitchell: Re. topic naming… I’m working on a system that contains many microservices, each of which may need to define their own dedicated topics. Is there a recommended convention for naming topics to help organize them into related groups? For example, I was initially considering using a `/`, for example: `persistent://<tenant>/<namespace>/<microservice>/<topic-name>` but have since found that the java based `TopicName` class seems to give different results, depending on how many `/`’s proceed after `<tenant>/<namespace>`. ---- 2020-06-04 18:04:10 UTC - Konstantinos Papalias: you could potentially use namespace as your <service name> and topics underneath it +1 : Frank Kelly ---- 2020-06-04 18:05:52 UTC - Matt Mitchell: Ah thanks @Konstantinos Papalias, I should’ve mentioned that `<namespace>` is reserved for the “app”, of which there are many within a Pulsar tenant. ok_hand : Konstantinos Papalias ---- 2020-06-04 18:15:14 UTC - Fabien LD: Reading the code let me discovered that it is using java runtime shutdown hook, so `SIGTERM`, `SIGINT`, `SIGHUP` should do the trick ---- 2020-06-04 18:17:08 UTC - Fabien LD: ```18:16:19.492 [shutdown-thread-46-1] INFO org.apache.pulsar.broker.ManagedLedgerClientFactory - Closed BookKeeper client 18:16:19.492 [shutdown-thread-46-1] INFO org.apache.pulsar.broker.loadbalance.LeaderElectionService - LeaderElectionService stopped 18:16:19.564 [shutdown-thread-46-1] INFO org.apache.zookeeper.ZooKeeper - Session: 0x601a9e92b4e0001 closed 18:16:19.564 [pulsar-ordered-OrderedExecutor-2-0-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x601a9e92b4e0001 18:16:19.568 [shutdown-thread-46-1] INFO org.apache.zookeeper.ZooKeeper - Session: 0x301a9e93dc50003 closed 18:16:19.568 [pulsar-ordered-OrderedExecutor-3-0-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x301a9e93dc50003 18:16:19.583 [Thread-2] INFO org.apache.pulsar.broker.MessagingServiceShutdownHook - Completed graceful shutdown. Exiting``` With SIGINT ---- 2020-06-04 19:01:00 UTC - Adrienn Tordai: @Adrienn Tordai has joined the channel ---- 2020-06-04 19:21:20 UTC - Gary Fredericks: I'm getting one of these thrown all the way back to the client, and it's pretty frustrating that it doesn't have any additional information attached is there a reasonable change we could make to add at least the `message`, if not the class or stack trace? ---- 2020-06-04 19:21:23 UTC - Gary Fredericks: <https://github.com/apache/pulsar/blob/da0492b262b8677a7f9f6bc3adf1efebcb477734/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java#L3074> ---- 2020-06-04 19:41:34 UTC - Alexandre DUVAL: Can't find resources about bookie weight placement policy, there are documentation or recipee? ---- 2020-06-04 20:37:05 UTC - Rounak Jaggi: how can we deploy pulsar in amazon eks. I am following this <http://pulsar.apache.org/docs/en/helm-deploy/>. I am on step where it says Deploy using Helm. After that when get services for the namespace, it shows all the components but its pending. Am I missing anything? ---- 2020-06-04 20:48:20 UTC - Gary Fredericks: or just logging it, even ---- 2020-06-04 20:59:31 UTC - alex kurtser: @Sijie Guo ---- 2020-06-04 21:38:54 UTC - jujugrrr: Check your Kubernetes logs, it will tell you why it's pending (persistent volume etc..) ---- 2020-06-04 22:33:37 UTC - Sijie Guo: If you are deploying functions using kubernetes runtime, Pulsar already automatically added the metrics port to the pod annotations. ---- 2020-06-04 22:33:56 UTC - Sijie Guo: So prometheus can scrape the pods to collect the metrics. ---- 2020-06-04 23:03:07 UTC - Alexandre DUVAL: ```bin/bookkeeper shell decommissionbookie``` I just need to run this on a bookie to remove a node from a cluster? ---- 2020-06-04 23:58:40 UTC - Alexandre DUVAL: Authoritative parameter available on deletenamespace endpoint are currently not used/impl, is this parameter exposed to force the ns deletion even if topics is not empty? Maybe another way? ---- 2020-06-05 00:11:18 UTC - alex kurtser: @Sijie Guo Thank you for the answer. ---- 2020-06-05 00:12:26 UTC - alex kurtser: Unfortunately I tried but it still isn't working ---- 2020-06-05 00:14:42 UTC - alex kurtser: I remove the prometheus port from the annotations, and left only the <http://prometheus.io/scrape|prometheus.io/scrape>: "true" settings. For some reason the pod is not adding the metrics port to the pod.. Should i add some special settings to activate this feature ? Just note, we execute the functions on separate pods without brokers. Could it be a problem? ---- 2020-06-05 01:48:02 UTC - wangjialing: @wangjialing has joined the channel ---- 2020-06-05 02:29:09 UTC - Sijie Guo: > execute the functions on separate pods How do you setup functions worker? ---- 2020-06-05 03:04:36 UTC - Sunny Chan: I was using send - when I change my code using sendAsync it seems to have much higher publishing re ate. Thank you for your suggestion ---- 2020-06-05 08:13:42 UTC - Rattanjot Singh: Hi, When I am consuming my consumer is getting disconnected after consuming a single message. and giving logs such as ```Removed consumer Consumer{subscription=PersistentSubscription{topic=<persistent://public/default/drain>, name=drain}, consumerId=0, consumerName=13847, address=/10.80.124.165:50374} with pending 97 acks``` Anybody who knows what settings i might need to change in my code ---- 2020-06-05 08:17:53 UTC - charles: @charles has joined the channel ---- 2020-06-05 08:42:52 UTC - yuanbo: @yuanbo has joined the channel ----
