2019-07-03 14:07:41 UTC - Venugopal Doddi: @Sijie Guo Hi. I am going thru this link <https://pulsar.apache.org/docs/v2.0.1-incubating/deployment/Monitoring/> for monitoring the apache pulsar we have on standalone mode on OST (not K8s). But i was looking something like the data to go into influxdb but not to prometheus. Is there any community offering currently having, to monitor the pulsar cluster and other stats? ---- 2019-07-03 14:08:17 UTC - Venugopal Doddi: I do see some examples on kubernetes, but i was looking for something on cluster residing on OST. ---- 2019-07-03 14:08:30 UTC - Venugopal Doddi: any help.. that will be greatful ---- 2019-07-03 14:31:35 UTC - Guillaume Rosauro: @jia zhai @tuteng: hello guys. I am still trying Debezium connector and now I am stuck with the SMT part of Debezium (single message transform) like that : ``` transforms: "unwrap" transforms.unwrap.type: "io.debezium.transforms.UnwrapFromEnvelope" transforms.unwrap.drop.tombstones: "false" transforms.unwrap.operation.header: "true" ```
I think those values are not taken into account, or Pulsar is acting differently with the kafka-connect. I have tried to update Debezium to 0.9.5.Final but with no luck too. There is probably something I miss. Do you have any idea ? ---- 2019-07-03 16:39:04 UTC - David Kjerrumgaard: @Kim Christian Gaarder As I understand it, you have a Topic that contains messages with externally managed identifier stored in a random order. Your goal is to create an pulsar-based index that will allow you to find a starting point (msg_id) in the original topic based on the external-id, e.g f(external_id) --> msg_id in original topic? Will each external-id appear only once in the original topic? What should we do if that isn't the case? ---- 2019-07-03 16:41:09 UTC - David Kjerrumgaard: @Kim Christian Gaarder Maybe the easiest approach would be to leverage the Presto SQL engine to query the original topic to find the msg_id by external id, e.g. `select msg-id from topicA where external-id = ?` ---- 2019-07-03 16:42:41 UTC - David Kjerrumgaard: The benefits of this approach are that it doesn't require keeping an index anywhere, externally or inside the state store. Secondly, the searching is done in parallel, so it will be relatively fast. ---- 2019-07-03 17:21:19 UTC - Devin G. Bost: @Jerry Peng We're in the process of getting a mirror setup to allow us to use the Jars from <https://yahoo.bintray.com/maven> to enable us to build Pulsar from source. I haven't had a chance yet to try building it off-network, but I'm hoping the mirror will get setup shortly. I'll keep you posted. ---- 2019-07-03 18:47:02 UTC - Dineshotham Kumar Khambhammettu: @Dineshotham Kumar Khambhammettu has joined the channel ---- 2019-07-03 22:02:58 UTC - Aaron: What is the standard behavior for rewinding the cursor on consumer disconnect/reconnect? Does the bookkeeper rewind it back to the last received ack? ---- 2019-07-03 22:04:59 UTC - Matteo Merli: The broker is keeping track of the subscription position (by storing the state in BookKeeper). After a reconnection, the broker will be starting to deliver messages from the last ack. That takes into account messages that were acked out of order ---- 2019-07-03 22:42:51 UTC - Addison Higham: hrm... so turning on authz/authn for the first time (using RS512 JWTs) and getting a weird error that is putting my brokers in a crash loop ---- 2019-07-03 22:43:43 UTC - Addison Higham: it looks like something is hanging up on me, perhaps another broker? this is 2.3.2 and function workers are running inside my broker (on k8s,) ---- 2019-07-03 22:44:50 UTC - Matteo Merli: is the auth plugin (client side) configured in broker.conf? ---- 2019-07-03 22:45:44 UTC - Addison Higham: nope, and that makes sense, but I didn't see anything in the docs here: <https://pulsar.apache.org/docs/en/security-token-admin/#on-brokers> ---- 2019-07-03 22:45:57 UTC - Addison Higham: but I am assuming this is all due to teh functions worker, which does need the client auth configured ---- 2019-07-03 22:47:46 UTC - Addison Higham: @Matteo Merli is it usual just to use a superuser role for that? ---- 2019-07-03 22:47:50 UTC - Addison Higham: I imagine so? ---- 2019-07-03 22:48:09 UTC - Matteo Merli: Yes, the worker will act as super-user ---- 2019-07-03 22:49:04 UTC - Matteo Merli: Take a look at <https://github.com/apache/pulsar/blob/master/conf/broker.conf#L359> for how to configure the client credentials for broker ---- 2019-07-03 23:02:34 UTC - Addison Higham: :thinking_face: still not working, so I see the following on startup: ``` [conf/broker.conf] Applying config authenticationEnabled = true [conf/broker.conf] Applying config authenticationProviders = org.apache.pulsar.broker.authentication.AuthenticationProviderToken [conf/broker.conf] Applying config authorizationEnabled = true [conf/broker.conf] Applying config brokerClientAuthenticationParameters = file:///etc/pulsar/token/admin-token.txt [conf/broker.conf] Applying config brokerClientAuthenticationPlugin = org.apache.pulsar.client.impl.auth.AuthenticationToken [conf/broker.conf] Applying config brokerClientTlsEnabled = true [conf/broker.conf] Applying config brokerClientTrustCertsFilePath = /etc/pulsar/tls/ca.crt [conf/broker.conf] Applying config brokerServicePortTls = 6651 ... [conf/broker.conf] Applying config proxyRoles = proxy ... [conf/broker.conf] Applying config tlsCertificateFilePath = /etc/pulsar/tls/tls.crt [conf/broker.conf] Applying config tlsEnabled = true [conf/broker.conf] Applying config tlsKeyFilePath = /etc/pulsar/tls/tls.key [conf/broker.conf] Applying config tlsTrustCertsFilePath = /etc/pulsar/tls/ca.crt [conf/broker.conf] Applying config tokenPublicKey = file:///etc/pulsar/jwt/public.key ``` ---- 2019-07-03 23:02:44 UTC - Addison Higham: but am still getting the same exception above ---- 2019-07-03 23:04:48 UTC - Matteo Merli: Uhm. I think actually, that was in `conf/function_worker.yml` ---- 2019-07-03 23:04:53 UTC - Matteo Merli: ``` # the authentication plugin to be used by the pulsar client used in worker service # clientAuthenticationPlugin: # the authentication parameter to be used by the pulsar client used in worker service # clientAuthenticationParameters: ``` ---- 2019-07-03 23:05:40 UTC - Addison Higham: I was under the impression from the function worker docs that it inherits those from broker.conf? I do see those later in the log as null ---- 2019-07-03 23:06:18 UTC - Addison Higham: so if I set those in broker.conf, will they get used in function worker? just need the same keys there? ---- 2019-07-03 23:06:48 UTC - Matteo Merli: I think the worker only takes config from the yaml file, both when running embedded in broker, or when running separately ---- 2019-07-03 23:14:43 UTC - Addison Higham: ah, `PF` is used to prefix env vars to generate the yaml for functions_worker.yml ---- 2019-07-03 23:31:27 UTC - Addison Higham: okay, so some progress, confirmed it is just functions working giving me headaches, and one of the things I hard to do was switch the service and admin URLs to be their tls counterparts, which makes sense... but now I get a different error: `- [/127.0.0.1:39898] Unable to authenticate: Failed to authentication token: Illegal base64url character: ':'` ---- 2019-07-03 23:32:38 UTC - Addison Higham: I assume that is either because `clientAuthenticationParameters: "file:///etc/pulsar/token/admin-token.txt"` is only doing a base64 of that file OR that the file contains `token:<jwt>` ---- 2019-07-03 23:34:14 UTC - Matteo Merli: have you tried without the `""` quotes? ---- 2019-07-03 23:35:12 UTC - Matteo Merli: the token auth plugin will check if the params string starts with `file://` ---- 2019-07-04 02:40:02 UTC - tuteng: Pulsar did not deal with this, and these parameters were eventually handed over to debezium. ---- 2019-07-04 03:07:30 UTC - wojustme: @wojustme has joined the channel ---- 2019-07-04 04:44:39 UTC - Mahesh: Hi, I have a question regarding pulsar functions. If the input topic is regex pattern, in what subscription mode does the function run ? ---- 2019-07-04 04:47:10 UTC - Kim Christian Gaarder: @David Kjerrumgaard Your understanding is correct. Each external-id should appear only once, yes, but now that you mention it, this is not something that I can guarantee, I only assume that based on where the data comes from. ---- 2019-07-04 04:48:15 UTC - Kim Christian Gaarder: @David Kjerrumgaard This is a good idea. Does this really also work together with tiered storage? ---- 2019-07-04 04:49:05 UTC - David Kjerrumgaard: Yes it does work across tiered storage as well ---- 2019-07-04 04:49:47 UTC - Kim Christian Gaarder: nice ---- 2019-07-04 04:53:20 UTC - Kim Christian Gaarder: What is the most efficient way to peek at the latest (most recently produced) message in a topic? The question assumes that there are no existing relevant consumers or subscriptions and that we do not need or want to read all the messages in the topic, only the latest one. ---- 2019-07-04 05:33:30 UTC - tuteng: I understand that these two parameters are related to the function of Kafka compaction and have their own strategies for compaction in Pulsar. <http://pulsar.apache.org/docs/en/concepts-topic-compaction/> ---- 2019-07-04 05:40:19 UTC - shustsud: @shustsud has joined the channel ---- 2019-07-04 07:48:13 UTC - longmao: @longmao has joined the channel ----
