2020-08-14 11:02:13 UTC - Rahul Vashishth: @Addison Higham @Sijie Guo please help with bookie autoscaling? ---- 2020-08-14 11:13:39 UTC - Rahul Vashishth: i am mainly looking how the cluster storage will behave on downsizing the bookie counts. ---- 2020-08-14 11:36:38 UTC - Ravi Shah: Hi Team,
How can I implement kafka keyed partition type functionality with pulsar? ---- 2020-08-14 11:52:13 UTC - Fernando: Does anyone know if it’s possible to keep a subscription active even if there are no consumers? I keep losing my backlog when the consumer is inactive for too long ---- 2020-08-14 12:10:01 UTC - Joshua Decosta: I believe there is a backlog TTL. I would look into the configuration section for more details. Specifically the broker configs ---- 2020-08-14 12:11:21 UTC - Fernando: Do you mean backlog quota? ---- 2020-08-14 12:20:54 UTC - Joshua Decosta: i think i am referring to that, and looking at the docs now it doesn’t seem to correspond with what you’re asking about. i initially thought about that mainly cause from what i understand the backlog is what keeps track of the messages still to be consumed by clients. Which to me would say it also keeps track of the subscriptions. But i might be stretching that logic. ---- 2020-08-14 12:22:35 UTC - Roy Tarantino: The default inactivity timeout is 5 minutes for producers and consumers. With the websocket client, I reconnect the subscription after the server closes it because of inactivity. ---- 2020-08-14 12:23:39 UTC - Fernando: Is this configurable when creating the subscription? ---- 2020-08-14 12:24:34 UTC - Roy Tarantino: Depends on the client, maybe with the binary clients, I don't see it with websocket. ---- 2020-08-14 12:26:47 UTC - Roy Tarantino: websocket.conf:webSocketSessionIdleTimeoutMillis=300000 ---- 2020-08-14 12:28:01 UTC - Fernando: I’ll check how it is in python, thanks ---- 2020-08-14 12:48:34 UTC - Fernando: It doesn’t seem to have the option. Maybe it’s a bug. I’m using v2.5.0 ---- 2020-08-14 12:53:13 UTC - Roy Tarantino: api docs are sadly lacking ---- 2020-08-14 13:57:05 UTC - Julius S: @Fernando this might be a helpful overview: <https://kesque.com/understanding-pulsar-message-ttl-backlog-and-retention/> ---- 2020-08-14 13:58:23 UTC - Joshua Decosta: @Addison Higham I’m following the pattern set in TokenAuthenticationState. Is there a reason that authenticate method doesn’t return an AuthState? ---- 2020-08-14 15:15:47 UTC - Evan Furman: Has anyone else encountered this error when trying to generate the dashboards? ```apache-pulsar-grafana-dashboard [master] ./scripts/generate_dashboards.sh <http://127.0.0.1:9090> pulsar-cluster-qa Setting PULSAR_PROMETHEUS_URL from script input. PULSAR_PROMETHEUS_URL set to: <http://127.0.0.1:9090> Setting PULSAR_CLUSTER from script input. PULSAR_CLUSTER set to: pulsar-cluster-qa Using existing PULSAR_CUSTOM_PROMETHEUS environment variable. PULSAR_CUSTOM_PROMETHEUS set to: Using existing GF_LOKI_URL environment variable. GF_LOKI_URL set to: Using existing GF_LOKI_DATASOURCE_NAME environment variable. GF_LOKI_DATASOURCE_NAME set to: Generating Datasources ... Your pulsar data source is generated as ~/apache-pulsar-grafana-dashboard/target/datasources/pulsar.yml Traceback (most recent call last): File "/usr/local/bin/j2", line 548, in <module> render_file_template(file, output) File "/usr/local/bin/j2", line 476, in render_file_template output.write(template.render(env=os.environ)+'\n') UnicodeEncodeError: 'ascii' codec can't encode character u'\u2192' in position 49118: ordinal not in range(128)``` ---- 2020-08-14 15:16:07 UTC - Evan Furman: source is here <https://github.com/streamnative/apache-pulsar-grafana-dashboard> ---- 2020-08-14 15:17:09 UTC - Evan Furman: I found a fix that works: `export PYTHONIOENCODING=utf-8` +1 : Sijie Guo ---- 2020-08-14 16:16:58 UTC - Joshua Decosta: Also just an update, my newAuthState method in my custom AuthenticationProvider still isn’t getting hit. I’m currently looking at the ProxyClientCnx class and it seems like the method newConnectCommand() is getting hit. Which in turn gets the AuthenticationDataPrivider and uses the authenticate method there. I’m not using a custom AuthenticationDataProvider, I’m simply using the Token one. Do you think that could be cashing my issues? ---- 2020-08-14 17:13:28 UTC - Sijie Guo: If you want to scale down bookies, you need to run `decommissionbookie` command. You can write a script to automate the work. You can check <https://github.com/streamnative/charts/blob/master/scripts/pulsar/decommission_bookies.sh> for example. ---- 2020-08-14 17:32:14 UTC - Jim M.: pulsar consumer after is consumes a batch just hangs and doesn't pull any more ---- 2020-08-14 17:32:27 UTC - Jim M.: when I bounce the pod, it consumes more...is there a configuration I'm missing? ---- 2020-08-14 17:33:03 UTC - Addison Higham: are you acking the messages? ---- 2020-08-14 17:33:11 UTC - Jim M.: yes ---- 2020-08-14 17:38:53 UTC - Addison Higham: hrm... @Joshua Decosta do you have some code snippets you can share of your implementation? that might be the best way to solve this one ---- 2020-08-14 18:32:25 UTC - Joshua Decosta: @Addison Higham Here is my AuthenticationState class ---- 2020-08-14 18:34:35 UTC - Joshua Decosta: So for another update. I did finally see this class get called when going through the proxy. I still dont see the AuthenticationDataSource from the client in the AuthZ methods I've implemented. ---- 2020-08-14 18:52:29 UTC - Addison Higham: @Joshua Decosta okay, so one thing to keep in mind: The proxy doesn't really do *all* the auth/authz when you aren't using the Zookeeper service discovery, so that isn't totally unexpected. You do need to have authn/authz on the broker as well. That said, from what it sounds like, you aren't getting any clientAuthData in the broker? Or are you now seeing that with the fix to return your own `AuthenticationState`? ---- 2020-08-14 18:57:28 UTC - Evan Furman: It doesn’t look like metrics are enabled by default in zookeeper. What directives do we need to add to enable that? ---- 2020-08-14 18:59:53 UTC - Addison Higham: if you have a code snippet, might be easy to see if there is some issue ---- 2020-08-14 19:00:59 UTC - Addison Higham: if you use the bundled zookeeper in pulsar, it should start prometheus and be served on the `stats_server_port` ---- 2020-08-14 19:01:16 UTC - Addison Higham: (default to `8000`) ---- 2020-08-14 19:01:48 UTC - Addison Higham: @Ravi Shah Have you looked at partitioned topics? It should be as simple as using a partitioned topic and then setting a key on each message :slightly_smiling_face: ---- 2020-08-14 19:01:58 UTC - Evan Furman: ok we are explicitly setting it now because if you don’t specify, it doesn’t serve on 8000 ---- 2020-08-14 19:02:07 UTC - Evan Furman: ```ZooKeeper stats The local/global ZooKeeper server and clients that are shipped with Pulsar have been instrumented to expose detailed stats through Prometheus as well. http://$LOCAL_ZK_SERVER:8000/metrics http://$GLOBAL_ZK_SERVER:8001/metrics The default port of local ZooKeeper is 8000 and that of global ZooKeeper is 8001. These can be changed by specifying system property stats_server_port.``` ---- 2020-08-14 19:04:11 UTC - Evan Furman: ---- 2020-08-14 19:04:13 UTC - Joshua Decosta: I’m still not seeing the clientAuthData in the broker. I see the client role name, but not the authData ---- 2020-08-14 19:05:18 UTC - Joshua Decosta: Does that class look off to you? ---- 2020-08-14 19:06:13 UTC - Evan Furman: @Addison Higham still doesn’t seem to be serving metrics ---- 2020-08-14 19:06:53 UTC - Addison Higham: can you try curl/wget-ing the endpoint directly? ---- 2020-08-14 19:08:09 UTC - Evan Furman: ```[ec2-user@ip-10-3-22-14 ~]$ curl <http://10.3.22.102:8000> curl: (7) Failed to connect to 10.3.22.102 port 8000: Connection refused [ec2-user@ip-10-3-22-14 ~]$ curl <http://10.3.22.102:8000/metrics> curl: (7) Failed to connect to 10.3.22.102 port 8000: Connection refused``` ---- 2020-08-14 19:08:16 UTC - Evan Furman: zookeeper is running ---- 2020-08-14 19:15:34 UTC - Evan Furman: <https://prefetch.net/blog/2020/06/11/zookeeper-now-has-native-prometheus-metrics-support/> ---- 2020-08-14 19:16:01 UTC - Evan Furman: there is no zoo.cfg packaged with pulsar. Only a zookeeper.conf ---- 2020-08-14 19:18:26 UTC - Addison Higham: and this is the zookeeped bundled with pulsar right? How are you starting it? ---- 2020-08-14 19:18:36 UTC - Evan Furman: yep ---- 2020-08-14 19:18:50 UTC - Addison Higham: the version bundled with Pulsar is not yet on zk 3.6 ---- 2020-08-14 19:19:31 UTC - Evan Furman: yea, we are using the ansible playbook to deploy it ---- 2020-08-14 19:19:41 UTC - Evan Furman: but docs suggest it is enabled by default <https://pulsar.apache.org/docs/en/reference-metrics/> ---- 2020-08-14 19:31:42 UTC - Evan Furman: My thinking here is that it’s because we are not bundling zk 3.6.0. Is there a chance this could make it into the next release? The datadog/zk integration requires it as well. ---- 2020-08-14 19:32:32 UTC - Addison Higham: ```[Unit] Description=ZooKeeper Local After=network.target [Service] Environment=OPTS=-Dstats_server_port=2182 ExecStart=/opt/pulsar/bin/pulsar zookeeper WorkingDirectory=/opt/pulsar RestartSec=1s Restart=on-failure Type=simple [Install] WantedBy=multi-user.target``` ---- 2020-08-14 19:33:23 UTC - Addison Higham: @Evan Furman note the `stats_server_port` there in the unit file, it is getting overriden ---- 2020-08-14 19:33:37 UTC - Evan Furman: Ah ---- 2020-08-14 19:33:46 UTC - Evan Furman: so i need to add it in the unit file? ---- 2020-08-14 19:34:00 UTC - Addison Higham: or just use the port that is there, `2182` ---- 2020-08-14 19:34:08 UTC - Evan Furman: interesting… ---- 2020-08-14 19:34:11 UTC - Evan Furman: ok let me try ---- 2020-08-14 19:34:25 UTC - Addison Higham: IDK why that changes it from the default ---- 2020-08-14 19:36:26 UTC - Evan Furman: ```[root@ip-10-3-22-102 ~]# netstat -nltp | grep 2181 tcp 0 0 0.0.0.0:2181 0.0.0.0:* LISTEN 8580/java [root@ip-10-3-22-102 ~]# curl localhost:2181/metrics curl: (52) Empty reply from server``` ---- 2020-08-14 19:36:32 UTC - Evan Furman: gonna try 8000 ---- 2020-08-14 19:36:57 UTC - Addison Higham: you have 2181, it is 2182 ---- 2020-08-14 19:37:25 UTC - Evan Furman: ```[root@ip-10-3-22-102 system]# curl localhost:2182/metrics # HELP jvm_memory_bytes_used Used bytes of a given JVM memory area. # TYPE jvm_memory_bytes_used gauge jvm_memory_bytes_used{area="heap",} 2.79969792E8 jvm_memory_bytes_used{area="nonheap",} 4.4967944E7``` ---- 2020-08-14 19:37:27 UTC - Evan Furman: boom ---- 2020-08-14 19:37:28 UTC - Evan Furman: thank you ---- 2020-08-14 19:38:30 UTC - Addison Higham: np, and as far as ZK 3.6, I don't think there is anything blocking that upgrade, probably just needs someone to champion if for the next release. But the way the pulsar bundled version of zookeeper works is it adds its own prometheus integration in via a wrapper class at startup. Once Pulsar upgrades to 3.6, we will likely use the native prometheus integration ---- 2020-08-14 19:38:49 UTC - Addison Higham: zk 3.6 also has some nice performance improvements ---- 2020-08-14 19:38:57 UTC - Evan Furman: awesome ---- 2020-08-14 19:40:49 UTC - Addison Higham: is your proxy able to auth? because the proxy creds are authed first (with it's own client data) and then it checks the client ---- 2020-08-14 19:41:00 UTC - Addison Higham: so if your proxy auth fails, it won't validate the client ---- 2020-08-14 19:41:40 UTC - Joshua Decosta: Yes. ---- 2020-08-14 19:44:26 UTC - Joshua Decosta: Is there a place in the code i could double check to make sure it is happening? ---- 2020-08-14 19:44:55 UTC - Addison Higham: the logs would tell you on the broker ---- 2020-08-14 19:46:34 UTC - Addison Higham: is this HTTP (admin actions) or produce/consume? ---- 2020-08-14 19:47:37 UTC - Addison Higham: are you setting your `proxyRoles` as a `superuserRole`? ---- 2020-08-14 19:51:48 UTC - Joshua Decosta: Yes ---- 2020-08-14 19:52:02 UTC - Joshua Decosta: Produce/consume. My admin actions work fine ---- 2020-08-14 19:52:34 UTC - Joshua Decosta: They actually work correctly which is why i know the auth code works if given the correct authData. ---- 2020-08-14 19:55:02 UTC - Addison Higham: have you tried bypassing the proxy by using like a port forward (using kubectl) or like an SSH tunnel to talk directly to broker? At least then we can pin down that it is in fact the proxy causing the issue ---- 2020-08-14 19:55:57 UTC - Joshua Decosta: I can try that now ---- 2020-08-14 20:01:05 UTC - Joshua Decosta: For some reason I’m having trouble port forwarding to the proxy broker. ---- 2020-08-14 20:01:39 UTC - Joshua Decosta: It’s not throwing auth errors, it’s complaining about hostnames ---- 2020-08-14 20:03:51 UTC - Joshua Decosta: Do the pulsar images have text edit tools? I can log into the terminal and modify the client conf if that’s possible ---- 2020-08-14 20:14:20 UTC - Addison Higham: yes they do, you want to share a snippet of what you are doing? ---- 2020-08-14 20:17:17 UTC - Joshua Decosta: Sure, I did manage to get through when logged into the tookset container this is the command I used: ```bin/pulsar-client --auth-params "token:tokenhere" --auth-plugin com.company.link.pulsar.client.AuthenticationTokenIdx --url <pulsar://pulsar-broker:6650/> produce -m "hello world" <persistent://tenant/sunshine/testing> -n 1``` ---- 2020-08-14 20:17:54 UTC - Joshua Decosta: this command worked from within the pulsar-toolset and when I attempted to access a topic that I didnt have access to it said I was unauthorized. ---- 2020-08-14 20:18:06 UTC - Joshua Decosta: what are the text edit tools? I couldn't find any ---- 2020-08-14 20:18:31 UTC - Addison Higham: oh you are right, I just usually `apt update && apt install vim` ---- 2020-08-14 20:18:36 UTC - Addison Higham: because it at least has apt :slightly_smiling_face: ---- 2020-08-14 20:18:55 UTC - Joshua Decosta: oh I was misusing apt-get ---- 2020-08-14 20:19:07 UTC - Joshua Decosta: I forgot about apt install, I was being dumb and tried apt-get install ---- 2020-08-14 20:19:20 UTC - Joshua Decosta: :man-facepalming: ---- 2020-08-14 20:21:01 UTC - Joshua Decosta: is there any classes specifically you might be looking at? I've been digging into the proxyClientCnx and the ProxyConnection a bit. I see it grabbing the authenticationData from the newAuthState setup. but I can't understand where it gets used. I see in the lookUpHandler that its referenced. ---- 2020-08-14 20:22:30 UTC - Joshua Decosta: Now I haven't implemented any client side classes other than the Authentication class which is reeally just an extension of the authenticationToken class. : ```package com.company.link.pulsar.client; import org.apache.pulsar.client.impl.auth.AuthenticationToken; public class AuthenticationTokenIdx extends AuthenticationToken { @Override public String getAuthMethodName() { return "idx"; } }``` ---- 2020-08-14 20:22:40 UTC - Joshua Decosta: Do I need any client side classes for the proxy setup to work? ---- 2020-08-14 20:22:59 UTC - Joshua Decosta: I've been mostly relying on the AuthenticationToken for its setup. ---- 2020-08-14 20:23:27 UTC - Addison Higham: are you configuring the proxy to use your auth class? ---- 2020-08-14 20:23:58 UTC - Joshua Decosta: which auth class are we talking here? ---- 2020-08-14 20:24:05 UTC - Addison Higham: `brokerClientAuthenticationPlugin` ---- 2020-08-14 20:24:10 UTC - Addison Higham: what is that set at? ---- 2020-08-14 20:25:10 UTC - Joshua Decosta: ``` proxyClient: authenticationParameters: file:///pulsar/tokens/proxy/token authenticationPlugin: org.apache.pulsar.client.impl.auth.AuthenticationToken``` ---- 2020-08-14 20:25:14 UTC - Joshua Decosta: perhaps thats it? ---- 2020-08-14 20:25:39 UTC - Joshua Decosta: I wasn't sure that was needed. I've always done the custom Auth Plugin class in the client conf ---- 2020-08-14 20:26:03 UTC - Joshua Decosta: ``` brokerClientAuthenticationParameters: {{ .Values.auth.authentication.proxyClient.authenticationParameters | quote}} brokerClientAuthenticationPlugin: {{ .Values.auth.authentication.proxyClient.authenticationPlugin | quote}}``` ---- 2020-08-14 20:26:11 UTC - Joshua Decosta: from the helm chart within the proxy.configmap ---- 2020-08-14 20:26:36 UTC - Joshua Decosta: on a side note I'll be making a small pr to the helm charts for the ability to use multiple authN providers ---- 2020-08-14 20:26:47 UTC - Joshua Decosta: just trying to get this working before that PR. ---- 2020-08-14 20:27:24 UTC - Joshua Decosta: the broker/proxyClient uses the default jwt token setup ---- 2020-08-14 20:27:50 UTC - Joshua Decosta: my client though uses my custom authN class. ---- 2020-08-14 20:36:21 UTC - Addison Higham: hrm, as an experiment, could the proxy use your auth implementation as a test? it shouldn't need to, it should support different providers ---- 2020-08-14 20:39:23 UTC - Addison Higham: the other option: grab a tcpdump on your broker. It is protobuf over the wire, so it is sort of a pain to decode but not impossible ---- 2020-08-14 20:43:47 UTC - Joshua Decosta: should I also remove the additoinal provider within the authenticationProviders config on proxy? ---- 2020-08-14 20:46:27 UTC - Addison Higham: na, that should be fine ---- 2020-08-14 20:48:48 UTC - Evan Furman: @Addison Higham so i think we’re in a good place now with prometheus metrics. All are getting scraped. That said tho, when I generate the dashboards according to the instructions here, and import them into Grafana, it seems as tho they are using some dummy data? <https://github.com/streamnative/apache-pulsar-grafana-dashboard#import-dashboard-to-your-grafana-installation> ---- 2020-08-14 20:49:05 UTC - Evan Furman: ---- 2020-08-14 20:49:18 UTC - Evan Furman: we are not actively running any consumers/producers ---- 2020-08-14 20:49:31 UTC - Addison Higham: are you running functions workers? ---- 2020-08-14 20:49:49 UTC - Addison Higham: you should be able to see the names of the topics somewhere in those ---- 2020-08-14 20:49:49 UTC - Evan Furman: And some things are just crazy. Like the number of bookies ---- 2020-08-14 20:49:55 UTC - Evan Furman: We only have 4 bookies ---- 2020-08-14 20:50:04 UTC - Evan Furman: but the dashboard suggests we have waaaay more ---- 2020-08-14 20:50:24 UTC - Evan Furman: no functions workers ---- 2020-08-14 20:56:43 UTC - Evan Furman: ---- 2020-08-14 20:59:20 UTC - Joshua Decosta: so it looks like that worked the way I expected it to ---- 2020-08-14 20:59:30 UTC - Joshua Decosta: I tested it trying to access two different topics. ---- 2020-08-14 20:59:44 UTC - Joshua Decosta: what could be causing that issue? ---- 2020-08-14 21:00:11 UTC - Evan Furman: I did notice this ```efurman-mbp15 apache-pulsar-grafana-dashboard [master] cat target/dashboards/bookkeeper.json { "__inputs": [ { "name": "DS_TEST-CLUSTER", "label": "", "description": "", "type": "datasource", "pluginId": "prometheus", "pluginName": "Prometheus" } ],``` ---- 2020-08-14 21:11:03 UTC - Evan Furman: `./scripts/generate_dashboards.sh <http://127.0.0.1:9090> "pulsar-cluster-qa"` ---- 2020-08-14 21:11:25 UTC - Evan Furman: It seems the label is not getting added properly ---- 2020-08-14 21:15:40 UTC - Addison Higham: so it did work with the proxy broker client using your customized auth provider? It is probably just a not often used configuration of proxy + different auth providers that is likely a bug ---- 2020-08-14 21:18:37 UTC - Joshua Decosta: I still need to do a bit more testing but i think that is potentially what’s happening. Do you think there is a class i could look at to potentially work on that bug? ---- 2020-08-14 21:23:30 UTC - Evan Furman: I tried on mac and linux both — the `{{ PULSAR_CLUSTER }}` var is not populating. I tried using env vars too ---- 2020-08-14 21:37:11 UTC - Addison Higham: are you using the latest version? just some fixes here it looks like: <https://github.com/streamnative/apache-pulsar-grafana-dashboard/blob/master/scripts/generate_dashboards.sh> ---- 2020-08-14 21:37:55 UTC - Evan Furman: Ill double check but I pulled master ---- 2020-08-14 21:52:10 UTC - Joshua Decosta: It seems i spike too soon. I’m getting bad results with my testing currently ---- 2020-08-14 21:53:02 UTC - Joshua Decosta: I’m trying to produce on a topic I shouldn’t be able to produce on and i am able to ---- 2020-08-14 22:06:38 UTC - Joshua Decosta: I've since tested these creds within the toolset directly to the broker and the results were what I expected. ---- 2020-08-14 22:07:14 UTC - Joshua Decosta: when I was looking through the logs with the current proxy setup. ~I noticed the role being passed in was always the proxy-admin-client and not the client~ turns out the role is actually being passed ---- 2020-08-14 22:08:48 UTC - Joshua Decosta: I again think this is because the clientAuthData isn't being passed from the proxy. ---- 2020-08-14 22:16:17 UTC - Joshua Decosta: It for sure seems its the problem with the authentidationDataSource not being passed on from the proxy. ---- 2020-08-14 22:16:30 UTC - Joshua Decosta: instead its always sending the proxy token's AuthenticationDataSource. ---- 2020-08-14 22:55:18 UTC - Addison Higham: ah that is good data... so this is where the command to auth is actually being sent: <https://github.com/apache/pulsar/blob/367ce7829827e4b0853e1f5a50566192bb82bf54/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/DirectProxyHandler.java#L173> ---- 2020-08-14 23:04:43 UTC - Joshua Decosta: This is the command that is supposed to send the client auth rather than the proxy? ---- 2020-08-14 23:17:04 UTC - Addison Higham: let me take a step back a bit and provide some context that will probably be helpful: The proxy has 3 main "modes" of proxying: - admin http calls, these are really straight forward and working for you - forwarding binary lookup requests (where we don't know the broker) - forwarding binary connections (where the client already knows the broker) And a bit of explanation for those last two, because topics "belong" to a broker, the client needs to know which broker to talk to. This works in one of two ways, you either make an http call OR you can ask over the binary protocol. If you use the binary protocol though, the proxy doesn't "know" who to forward your request to. It just choose a broker. In comparison, once a producer is actually producing or a consumer is consuming, you have already done a lookup on the topic and you re-connect back to the correct broker, but this time you already know which broker to talk to and that data gets carried along That is why you see two different auth data flows in the proxy and they are subtly different ---- 2020-08-14 23:19:30 UTC - Addison Higham: if you are just using the `pulsar-client` and passing the `url`, then it can't use the HTTP lookup, so it is doing the binary lookup, so the first question would be to see if you are getting passed that ---- 2020-08-14 23:19:43 UTC - Addison Higham: turning on debug logs should help with that ---- 2020-08-14 23:21:52 UTC - Addison Higham: or another option is to try using the http lookup service, which the `pulsar-client` can do when you create client configuration file ---- 2020-08-14 23:25:54 UTC - Addison Higham: by default, it looks like `PULSAR_HOME/conf/client.conf` and looks like this: ```# Configuration for pulsar-client and pulsar-admin CLI tools # URL for Pulsar REST API (for admin operations) # For TLS: # webServiceUrl=<https://localhost:8443/> webServiceUrl=<http://localhost:8080/> # URL for Pulsar Binary Protocol (for produce and consume operations) # For TLS: # brokerServiceUrl=<pulsar+ssl://localhost:6651/> brokerServiceUrl=<pulsar://localhost:6650/> # Authentication plugin to authenticate with servers # e.g. for TLS # authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authPlugin= # Parameters passed to authentication plugin. # A comma separated list of key:value pairs. # Keys depend on the configured authPlugin. # e.g. for TLS # authParams=tlsCertFile:/path/to/client-cert.pem,tlsKeyFile:/path/to/client-key.pem authParams= # Allow TLS connections to servers whose certificate cannot be # be verified to have been signed by a trusted certificate # authority. tlsAllowInsecureConnection=false # Whether server hostname must match the common name of the certificate # the server is using. tlsEnableHostnameVerification=false # Path for the trusted TLS certificate file. # This cert is used to verify that any cert presented by a server # is signed by a certificate authority. If this verification # fails, then the cert is untrusted and the connection is dropped. tlsTrustCertsFilePath= # Enable TLS with KeyStore type configuration in broker. useKeyStoreTls=false # TLS KeyStore type configuration: JKS, PKCS12 tlsTrustStoreType=JKS # TLS TrustStore path tlsTrustStorePath= # TLS TrustStore password tlsTrustStorePassword=``` or you can put it anywhere and set `PULSAR_CLIENT_CONF` env var and point it at that file and pulsar-client will use that. What you need to do is also set your `webServiceUrl` in addition to `brokerServiceUrl` and it will use the HTTP lookup instead of binary. If that works, then we know it is just in the lookup side it is wrong ---- 2020-08-14 23:26:06 UTC - Addison Higham: which I suspect you might never be getting past the lookup ---- 2020-08-14 23:36:35 UTC - Joshua Decosta: The way I’ve been testing this is using the client.conf file in the location it expects and using bin/pulsar-client from the 2.6.0 binaries to a port forwarded proxy i have the web and broker urls set to local host and both are port forwarded to the proxy service that is in the kubernetes cluster ---- 2020-08-14 23:40:28 UTC - Joshua Decosta: Regarding the lookup. Where could i look for that flow? Is that the lookUpHandler in the Proxy? ---- 2020-08-14 23:51:11 UTC - Joshua Decosta: Idk if it matters but the error i keep seeing is a namespace permission error ---- 2020-08-14 23:52:53 UTC - Addison Higham: what is the error? ---- 2020-08-14 23:56:49 UTC - Joshua Decosta: Honestly it might be my authZ errror. I’ve shut down my computer and I’m hoping to not open it till after the weekend. I’ll look into the error and if it’s not coming from mine i will let you know ---- 2020-08-14 23:57:53 UTC - Addison Higham: heh, good idea, alright, well happy to help figure this one out, as I mentioned, this is definitely something we hope to make easier :slightly_smiling_face: ---- 2020-08-15 01:32:30 UTC - Joshua Decosta: > if you are just using the `pulsar-client` and passing the `url`, then it can't use the HTTP lookup, so it is doing the binary lookup, so the first question would be to see if you are getting passed that When you say “passing the url” are you referring to the option to use —url? Like bin/pulsar-client —url urlhere produce etc ----
