2020-08-17 11:00:55 UTC - Joshua Decosta: How can i turn on debugging? @Addison Higham ---- 2020-08-17 11:04:36 UTC - Joshua Decosta: So if i use the pulsar-client without specifying the url argument then http is enabled? ---- 2020-08-17 11:15:31 UTC - Aaron: If you are also producing these messages you can use partitioning by key (<https://pulsar.apache.org/docs/en/next/cookbooks-partitioned/>) and a custom message router that assigns a key based off one of the elements in the message ---- 2020-08-17 13:02:57 UTC - charles: Hi all,
In my performance tests, I'm reading off a topic provisioned on Docker-standalone 2.6.0 using a loop around the following (Java) code snippet: ```msg = reader.readNextAsync().get();``` This code I surrounded with a try-catch. On multiple occasions, I noticed the following exception is thrown: ```java.lang.IllegalStateException: recycled multiple times``` The exception is not thrown at exact the same time: Sometimes it's caused at the beginning of my tests, sometimes halfway. Sometimes just a few times, sometimes it goes on and on. It looks like a timing issue. Searching on the web for a solution I found the (possibly relating) issue: • *Pulsar reader throws exception java.lang.IllegalStateException: recycled multiple times* (<https://github.com/apache/pulsar/issues/7384|Github Link>) The topic looks resolved/merged/closed and I noticed the label: *release/2.6.1* However, current version on Maven Central is 2.6.0 and the current development pom.xml targets: *2.7.0-SNAPSHOT*. As I'm really eager on getting this resolved, I wonder if the *2.6.1.* version is near delivery. For my performance tests and demo's, it would really be great to use a solution through Maven Central. Any news on that? ---- 2020-08-17 13:07:07 UTC - Penghui Li: @charles v2.6.1 rc is out. After the candidate passed and the release manager complete the release jobs, you can find 2.6.1 java client at the Maven Central. ---- 2020-08-17 13:17:55 UTC - alex kurtser: Hi is it possible to upgrade the presto version to the last one *<https://prestodb.io/docs/current/release/release-0.237.html|0.237> ?* ---- 2020-08-17 13:18:10 UTC - alex kurtser: i mean in the upcoming pulsar 2.6.1 version ? ---- 2020-08-17 13:19:43 UTC - Penghui Li: We can plan it in 2.6.2 since 2.6.1 rc is out, could you please help add a github issue to track it? ---- 2020-08-17 13:24:04 UTC - alex kurtser: we just have some issues that ware resolved in the newer version f presto. The current presto version which included in pulsar is 0.206 and it seems to be very old ---- 2020-08-17 13:24:28 UTC - Raghav: is there a way to move certain partitions of a topic from one broker to another broker, so that the producers/consumers get connected to the new broker for those partitions ? ---- 2020-08-17 13:35:34 UTC - Penghui Li: You can unload the namespace bundle of that partitions, then the unloaded namespace bundle will be serve by the most idle broker. Of course, other partitions in this bundle will also move to that broker. ---- 2020-08-17 13:52:10 UTC - Joshua Decosta: If by setting the both the webService and the brokerService allows http to be used by default then I’m still getting the same error. If i need to pass a different parameter when using pulsar-client to produce a message than let me know if that’s the proxy-url or something else ---- 2020-08-17 13:54:02 UTC - Raghav: hmm but I want to distribute the load, as the bandwidth is getting saturated ---- 2020-08-17 14:24:22 UTC - charles: Hi @Penghui Li Thank you for this update! I definitely want to try out 2.6.1. I'm not aware of the duration of the delivery process details to Maven Central. You have an ETA you can share? ---- 2020-08-17 14:34:00 UTC - Greg: Hi, we are using Pulsar <http://2.6.in|2.6.0 in> kubernetes using official helm charts. We have an issue when we kill a broker, it takes almost 1 minute to switch topics to a new broker, and during that time we are not able to deliver any messages. Is there a way to reduce this failure detection time ? ---- 2020-08-17 15:19:07 UTC - Addison Higham: @Raghav what you likely want to do is split your bundles and perhaps add more partitions, that will distribute the topic among more brokers. ---- 2020-08-17 15:28:48 UTC - Raghav: yes I can try increasing the partitions and some partitions might move to the new broker that way some traffic can go the new broker. Thank you ---- 2020-08-17 16:04:42 UTC - Joshua Decosta: I also tried to enable authN/Z on the proxy alone and that also doesn’t seem to work. ---- 2020-08-17 17:54:43 UTC - Joshua Decosta: Ok, i think i understand now what you meant by http lookup. I changed the brokerUrl to “<http://pulsar-proxy:80|http://pulsar-proxy:80>” and now it seems to be working correctly. Is that what you meant by http lookup? @Addison Higham ---- 2020-08-17 17:55:42 UTC - Addison Higham: Yes ---- 2020-08-17 17:56:36 UTC - Joshua Decosta: So your hunch that it’s the binary lookup is probably correct then ---- 2020-08-17 17:58:25 UTC - Joshua Decosta: Now this only worked in the toolset container. I haven’t been able to port forward and get this to work. ---- 2020-08-17 18:16:31 UTC - Joshua Decosta: Does the http lookup not forward the proxy creds as well? I’m no longer seeing it enter my authZ methods. ---- 2020-08-17 18:32:13 UTC - Addison Higham: was out for a bit this morning, catching up now, can you show me your `PULSAR_CLIENT_CONF` file and the command you are using? ---- 2020-08-17 18:33:23 UTC - Addison Higham: both of the one that was working and the one that isn't working? ---- 2020-08-17 18:33:44 UTC - Joshua Decosta: Sure. ---- 2020-08-17 18:39:45 UTC - Joshua Decosta: working one: ```root@pulsar-toolset-0:/pulsar# cat conf/client.conf # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # <http://www.apache.org/licenses/LICENSE-2.0> # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. # # Configuration for pulsar-client and pulsar-admin CLI tools # URL for Pulsar REST API (for admin operations) # For TLS: # webServiceUrl=<https://localhost:8443/> webServiceUrl=<http://pulsar-proxy:80/> # URL for Pulsar Binary Protocol (for produce and consume operations) # For TLS: # brokerServiceUrl=<pulsar+ssl://localhost:6651/> brokerServiceUrl=<http://pulsar-proxy:80/> # Authentication plugin to authenticate with servers # e.g. for TLS # authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authPlugin=com.company.link.pulsar.client.AuthenticationTokenIdx # Parameters passed to authentication plugin. # A comma separated list of key:value pairs. # Keys depend on the configured authPlugin. # e.g. for TLS # authParams=tlsCertFile:/path/to/client-cert.pem,tlsKeyFile:/path/to/client-key.pem authParams=token:customtoken # Allow TLS connections to servers whose certificate cannot be # be verified to have been signed by a trusted certificate # authority. tlsAllowInsecureConnection=false # Whether server hostname must match the common name of the certificate # the server is using. tlsEnableHostnameVerification=false # Path for the trusted TLS certificate file. # This cert is used to verify that any cert presented by a server # is signed by a certificate authority. If this verification # fails, then the cert is untrusted and the connection is dropped. tlsTrustCertsFilePath= # Enable TLS with KeyStore type configuration in broker. useKeyStoreTls=false # TLS KeyStore type configuration: JKS, PKCS12 tlsTrustStoreType=JKS # TLS TrustStore path tlsTrustStorePath= # TLS TrustStore password tlsTrustStorePassword= root@pulsar-toolset-0:/pulsar# ``` ---- 2020-08-17 18:40:34 UTC - Joshua Decosta: the non working one's only difference was the brokerServiceUrl being `<pulsar://pulsar-broker:6650/>` ---- 2020-08-17 18:41:24 UTC - Addison Higham: weird... I am not sure how `brokerUrl` being `http` would work :confused:. The only thing you should need to set is `webServiceUrl` for you to use the http lookup instead of the binary lookup, and I expect the binary lookup is the broken part ---- 2020-08-17 18:41:44 UTC - Joshua Decosta: the code specifically looks for the brokerServiceUrl ---- 2020-08-17 18:41:50 UTC - Joshua Decosta: thats how I ended up on this ---- 2020-08-17 18:41:53 UTC - Joshua Decosta: I'll find the line ---- 2020-08-17 18:42:37 UTC - Joshua Decosta: actually, I think I misunderstood the code. It doesn't look like its doing that. ---- 2020-08-17 18:44:00 UTC - Joshua Decosta: ``` if (conf.getServiceUrl().startsWith("http")) { lookup = new HttpLookupService(conf, eventLoopGroup); } else { lookup = new BinaryProtoLookupService(this, conf.getServiceUrl(), conf.getListenerName(), conf.isUseTls(), externalExecutorProvider.getExecutor()); } ``` this is what I was looking at in `pulsarClientImpl` nd for some reason I decided to add http to the brokerServiceUrl ---- 2020-08-17 18:46:46 UTC - Joshua Decosta: I think its cause when I followed the logic in the code the logic gave me the idea that this referred to the brokerServiceUrl ---- 2020-08-17 18:47:07 UTC - Joshua Decosta: ``` if (getServiceUrl() != null && (this.getServiceUrl().startsWith("pulsar+ssl") || this.getServiceUrl().startsWith("https"))) { this.useTls = true; return true; }``` ---- 2020-08-17 18:47:38 UTC - Joshua Decosta: and I've just never seen the webServiceUrl have the `pulsar://` ---- 2020-08-17 18:50:47 UTC - Joshua Decosta: probably a bug then? ---- 2020-08-17 19:19:26 UTC - Addison Higham: I was conflating some things going between client code and proxy code, so you are correct of which field that refers too. What happens is that when you do a lookup for a topic, it is going to respond with a `pulsar://` URL and then connect that way. So you didn't need to mess with `PULSAR_CLIENT_CONF` at all. Regardless, it does seem to indicate the issue lies in the proxy when it handles binary lookup requests. ---- 2020-08-17 19:29:09 UTC - Joshua Decosta: is there a place I should start looking into to try and fix it? this is something I'll need going forward. ---- 2020-08-17 19:33:18 UTC - Addison Higham: Yeah, I am going to guess it is probably a bug, at the very least an issue with the auth API that makes it difficult to implement correctly. With where you are now, I might suggest using all this data you have collected to write a good bug report. if you could include details about the your chained custom auth provider, that it works direct against the broker, works when you use an `http` client, but fails over binary, that would be helpful... it would probably be super helpful if you could capture a tcpdump, but that is a bit involved. I was looking and it doesn't look like the debug logs would include the raw data of what is being sent, but still might be helpful. I think with the helm charts you should just be able to set `PULSAR_LOG_LEVEL: debug` as an env var and it *should* work ---- 2020-08-17 19:38:24 UTC - Joshua Decosta: Would I put the `PULSAR_LOG_LEVEL` in the configData on these components? ---- 2020-08-17 19:39:50 UTC - Addison Higham: yes, that should work ---- 2020-08-17 19:47:51 UTC - Addison Higham: hrm, actually, am trying myself and it doesn't seem to be taking effect. I think there is an issue with how the precedence is working in log4j ---- 2020-08-17 19:48:15 UTC - Joshua Decosta: I’ve never been able to get debugging working ---- 2020-08-17 19:49:30 UTC - Addison Higham: the most guaranteed way is to edit `log4j2.yaml` and change: ``` - name: "pulsar.log.root.level" value: "debug" - name: "pulsar.log.level" value: "debug"``` ---- 2020-08-17 19:49:51 UTC - Addison Higham: and in fact, if you just do that locally, your `pulsar-client` will print a fair amount of debug info ---- 2020-08-17 21:12:41 UTC - Addison Higham: (whoops, forgot to hit send on this message) we probably need to see the proxy side of things as the client likely won't be enough. In k8s when not using the helm charts, we just mounted our own config file. But that is not as easy... I am not sure of the best way to do this in the helm charts :confused: ---- 2020-08-17 21:12:54 UTC - Addison Higham: maybe manually edit the proxy deployment? ---- 2020-08-17 21:13:26 UTC - Joshua Decosta: Are you referring to enabling debug? ---- 2020-08-17 21:14:18 UTC - Addison Higham: and as far as the problem, my best guess is that it is here, or at least arguments to this class: <https://github.com/apache/pulsar/blob/dfdfdb4efa46662e7ec8cb027fdfe854d45caec5/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/ProxyClientCnx.java#L49> ---- 2020-08-17 21:14:24 UTC - Addison Higham: yes, enabling debugging ---- 2020-08-17 21:14:39 UTC - Joshua Decosta: Ok I’m currently building a custom image and overriding the conf file ---- 2020-08-17 21:14:51 UTC - Addison Higham: :thumbsup: that is a good strat as well ---- 2020-08-17 21:51:58 UTC - Joshua Decosta: I’m not sure debug is enabled in my proxies or broker. The file itself is edited the way you showed but it doesn’t seem to be printing debug ---- 2020-08-17 21:52:50 UTC - Joshua Decosta: my file: ---- 2020-08-17 21:52:50 UTC - Joshua Decosta: ```root@pulsar-proxy-0:/pulsar# cat conf/log4j2.yaml # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # <http://www.apache.org/licenses/LICENSE-2.0> # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. # Configuration: status: INFO monitorInterval: 30 name: pulsar packages: io.prometheus.client.log4j2 Properties: Property: - name: "pulsar.log.dir" value: "logs" - name: "pulsar.log.file" value: "pulsar.log" - name: "pulsar.log.appender" value: "RoutingAppender" - name: "pulsar.log.root.level" value: "debug" - name: "pulsar.log.level" value: "debug" - name: "pulsar.routing.appender.default" value: "Console" # Example: logger-filter script Scripts: ScriptFile: name: filter.js language: JavaScript path: ./conf/log4j2-scripts/filter.js charset: UTF-8 Appenders: # Console Console: name: Console target: SYSTEM_OUT PatternLayout: Pattern: "%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n" # Rolling file appender configuration RollingFile: name: RollingFile fileName: "${sys:pulsar.log.dir}/${sys:pulsar.log.file}" filePattern: "${sys:pulsar.log.dir}/${sys:pulsar.log.file}-%d{MM-dd-yyyy}-%i.log.gz" immediateFlush: false PatternLayout: Pattern: "%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n" Policies: TimeBasedTriggeringPolicy: interval: 1 modulate: true SizeBasedTriggeringPolicy: size: 1 GB # Delete file older than 30days DefaultRolloverStrategy: Delete: basePath: ${sys:pulsar.log.dir} maxDepth: 2 IfFileName: glob: "*/${sys:pulsar.log.file}*log.gz" IfLastModified: age: 30d Prometheus: name: Prometheus # Routing Routing: name: RoutingAppender Routes: pattern: "$${ctx:function}" Route: - Routing: name: InstanceRoutingAppender Routes: pattern: "$${ctx:instance}" Route: - RollingFile: name: "Rolling-${ctx:function}" fileName : "${sys:pulsar.log.dir}/functions/${ctx:function}/${ctx:functionname}-${ctx:instance}.log" filePattern : "${sys:pulsar.log.dir}/functions/${sys:pulsar.log.file}-${ctx:instance}-%d{MM-dd-yyyy}-%i.log.gz" PatternLayout: Pattern: "%d{ABSOLUTE} %level{length=5} [%thread] [instance: %X{instance}] %logger{1} - %msg%n" Policies: TimeBasedTriggeringPolicy: interval: 1 modulate: true SizeBasedTriggeringPolicy: size: "20MB" # Trigger every day at midnight that also scan # roll-over strategy that deletes older file CronTriggeringPolicy: schedule: "0 0 0 * * ?" # Delete file older than 30days DefaultRolloverStrategy: Delete: basePath: ${sys:pulsar.log.dir} maxDepth: 2 IfFileName: glob: "*/${sys:pulsar.log.file}*log.gz" IfLastModified: age: 30d - ref: "${sys:pulsar.routing.appender.default}" key: "${ctx:function}" - ref: "${sys:pulsar.routing.appender.default}" key: "${ctx:function}" Loggers: # Default root logger configuration Root: level: "${sys:pulsar.log.root.level}" additivity: true AppenderRef: - ref: "${sys:pulsar.log.appender}" level: "${sys:pulsar.log.level}" - ref: Prometheus level: info Logger: - name: org.apache.bookkeeper.bookie.BookieShell level: info additivity: false AppenderRef: - ref: Console - name: verbose level: info additivity: false AppenderRef: - ref: Console # Logger to inject filter script # - name: org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl # level: debug # additivity: false # AppenderRef: # ref: "${sys:pulsar.log.appender}" # ScriptFilter: # onMatch: ACCEPT # onMisMatch: DENY # ScriptRef: # ref: filter.js root@pulsar-proxy-0:/pulsar# ``` ---- 2020-08-17 21:55:16 UTC - Addison Higham: you don't see any debug log lines? ---- 2020-08-17 21:55:53 UTC - Joshua Decosta: Not in the proxy logs or broker. I see them when running the pulsar-client commands in toolset ---- 2020-08-17 21:56:13 UTC - Addison Higham: hrm... do you still have `PULSAR_LOG_LEVEL` set as well? ---- 2020-08-17 21:56:17 UTC - Addison Higham: to `debug` ---- 2020-08-17 21:56:23 UTC - Joshua Decosta: I don’t ---- 2020-08-17 21:56:34 UTC - Joshua Decosta: I can put that in there and see if it makes a diff ---- 2020-08-17 21:58:47 UTC - Addison Higham: you did try `PULSAR_LOG_LEVEL=debug` by itself and it didn't fix it right? ---- 2020-08-17 21:59:38 UTC - Joshua Decosta: I didn’t try it. I feel like you had said it didn’t work ---- 2020-08-17 21:59:43 UTC - Joshua Decosta: I never got around to trying it ---- 2020-08-17 22:05:47 UTC - Joshua Decosta: Ok debug looks like it’s on ---- 2020-08-17 22:05:57 UTC - Joshua Decosta: This is overwhelming amounts of info ---- 2020-08-17 22:09:19 UTC - Addison Higham: did you just set `PULSAR_LOG_LEVEL`? ---- 2020-08-17 22:09:57 UTC - Addison Higham: if so, well... weird, must be some difference between local launching and launching inside a container :shrug: ---- 2020-08-17 22:10:22 UTC - Joshua Decosta: I did just set it. ---- 2020-08-17 22:12:09 UTC - Addison Higham: and yes, it is loooots of info, but once it starts up, it should slow down a fair bit ---- 2020-08-17 22:19:28 UTC - Joshua Decosta: here is what I think is the full process from the proxy: ---- 2020-08-17 22:20:54 UTC - Joshua Decosta: I'll post the broker one as well if that helps ---- 2020-08-17 22:44:03 UTC - Joshua Decosta: It doesn't look like the broker debugs are very helpful ---- 2020-08-17 23:05:19 UTC - Addison Higham: hrm... That all looks pretty sane :confused: I expected those params to have some error... ---- 2020-08-17 23:05:59 UTC - Joshua Decosta: that client byte array is the correct token. ---- 2020-08-17 23:06:19 UTC - Joshua Decosta: I converted it to see. But for whatever reason that doesn't seem to be making it to the broker. ---- 2020-08-17 23:07:22 UTC - Joshua Decosta: what params were you looking at? ---- 2020-08-17 23:07:28 UTC - Addison Higham: can you search the broker logs for this chunk of log: ` Received Lookup from` ---- 2020-08-17 23:09:12 UTC - Addison Higham: ah! do you have this set? `authenticateOriginalAuthData` in your broker? ---- 2020-08-17 23:10:07 UTC - Joshua Decosta: for sure not ---- 2020-08-17 23:10:17 UTC - Joshua Decosta: that doesn't seem to come up in the docs ---- 2020-08-17 23:10:31 UTC - Joshua Decosta: I had seen that somewhere in the configuration classes but I didn't see it mentioned anywhere ---- 2020-08-17 23:12:01 UTC - Addison Higham: I believe this is to handle exactly this sort of case where you have two different auth providers ---- 2020-08-17 23:13:18 UTC - Addison Higham: <https://github.com/apache/pulsar/issues/6873> <- it is a relatively new feature, so it probably is not yet documented ---- 2020-08-17 23:13:37 UTC - Joshua Decosta: ```2020-08-17T22:09:59.904514618Z 22:09:59.904 [pulsar-io-25-2] INFO org.apache.pulsar.broker.service.ServerCnx - New connection from /10.1.0.100:50158 2020-08-17T22:09:59.91173649Z 22:09:59.907 [pulsar-io-25-2] DEBUG org.apache.pulsar.common.protocol.PulsarDecoder - [/10.1.0.100:50158] Received cmd CONNECT 2020-08-17T22:09:59.91175119Z 22:09:59.907 [pulsar-io-25-2] DEBUG org.apache.pulsar.broker.service.ServerCnx - Received CONNECT from /10.1.0.100:50158, auth enabled: true 2020-08-17T22:09:59.91175539Z 22:09:59.911 [pulsar-io-25-2] DEBUG org.apache.pulsar.broker.service.ServerCnx - [/10.1.0.100:50158] Client successfully authenticated with token role null and originalPrincipal null 2020-08-17T22:09:59.914125014Z 22:09:59.914 [pulsar-io-25-2] DEBUG org.apache.pulsar.common.protocol.PulsarDecoder - [/10.1.0.100:50158] Received cmd LOOKUP 2020-08-17T22:09:59.914263215Z 22:09:59.914 [pulsar-io-25-2] DEBUG org.apache.pulsar.broker.service.ServerCnx - [<persistent://uhc/sunshine/healthdata>] Received Lookup from /10.1.0.100:50158 for 0 2020-08-17T22:09:59.91576423Z 22:09:59.915 [pulsar-io-25-2] INFO com.optum.link.pulsar.broker.authorization.AuthorizationProviderIdx - canLookUpAsync: for role:proxy-admin 2020-08-17T22:09:59.915959232Z 22:09:59.915 [pulsar-io-25-2] INFO com.optum.link.pulsar.broker.authorization.AuthorizationProviderIdx - authdata:org.apache.pulsar.broker.authentication.AuthenticationDataCommand@70a437d 2020-08-17T22:09:59.916074133Z 22:09:59.916 [pulsar-io-25-2] INFO com.optum.link.pulsar.broker.authorization.AuthorizationProviderIdx - checking caProduceAsync 2020-08-17T22:09:59.916203934Z 22:09:59.916 [pulsar-io-25-2] INFO com.optum.link.pulsar.broker.authorization.AuthorizationProviderIdx - topicName:<persistent://uhc/sunshine/healthdata> 2020-08-17T22:09:59.916275035Z 22:09:59.916 [pulsar-io-25-2] INFO com.optum.link.pulsar.broker.authorization.AuthorizationProviderIdx - role:proxy-admin 2020-08-17T22:09:59.916344136Z 22:09:59.916 [pulsar-io-25-2] INFO com.optum.link.pulsar.broker.authorization.AuthorizationProviderIdx - authData:org.apache.pulsar.broker.authentication.AuthenticationDataCommand@70a437d 2020-08-17T22:09:59.916422436Z 22:09:59.916 [pulsar-io-25-2] INFO com.optum.link.pulsar.utils.TokenUtils - binary connection data 2020-08-17T22:09:59.916489537Z 22:09:59.916 [pulsar-io-25-2] INFO com.optum.link.pulsar.broker.authorization.AuthorizationProviderIdx - token:eredacted for proxy-admin 2020-08-17T22:09:59.918760359Z 22:09:59.918 [pulsar-io-25-2] INFO com.optum.link.pulsar.broker.authorization.AuthorizationProviderIdx - this is a broker token, role:proxy-admin 2020-08-17T22:09:59.918920661Z 22:09:59.918 [pulsar-io-25-2] INFO com.optum.link.pulsar.utils.TokenUtils - binary connection data 2020-08-17T22:09:59.919018662Z 22:09:59.918 [pulsar-io-25-2] INFO com.optum.link.pulsar.broker.authorization.AuthorizationProviderIdx - checking canConsume for proxy-admin with token:redacted 2020-08-17T22:09:59.921043382Z 22:09:59.920 [pulsar-io-25-2] INFO com.optum.link.pulsar.broker.authorization.AuthorizationProviderIdx - canConsume: This is a broker token 2020-08-17T22:09:59.921616288Z 22:09:59.921 [pulsar-io-25-2] INFO com.optum.link.pulsar.utils.TokenUtils - binary connection data 2020-08-17T22:09:59.921627288Z 22:09:59.921 [pulsar-io-25-2] INFO com.optum.link.pulsar.broker.authorization.AuthorizationProviderIdx - checking superuser for proxy-admin with token eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJwcm94eS1hZG1pbiJ9.165LCuseXmmR-VtENoARbsXqzHnSHOZy9kntpWaHXm8 2020-08-17T22:09:59.923942111Z 22:09:59.923 [pulsar-io-25-2] INFO com.optum.link.pulsar.utils.TokenUtils - binary connection data 2020-08-17T22:09:59.923953911Z 22:09:59.923 [pulsar-io-25-2] INFO com.optum.link.pulsar.broker.authorization.AuthorizationProviderIdx - checking superuser for pulsar-client with token redacted 2020-08-17T22:09:59.926613637Z 22:09:59.926 [Thread-16] INFO com.optum.link.pulsar.broker.authorization.AuthorizationProviderIdx - checking caProduceAsync 2020-08-17T22:09:59.926625337Z 22:09:59.926 [Thread-16] INFO com.optum.link.pulsar.broker.authorization.AuthorizationProviderIdx - topicName:<persistent://uhc/sunshine/healthdata> 2020-08-17T22:09:59.926629337Z 22:09:59.926 [Thread-16] INFO com.optum.link.pulsar.broker.authorization.AuthorizationProviderIdx - role:pulsar-client 2020-08-17T22:09:59.926632737Z 22:09:59.926 [Thread-16] INFO com.optum.link.pulsar.broker.authorization.AuthorizationProviderIdx - authData:org.apache.pulsar.broker.authentication.AuthenticationDataCommand@70a437d 2020-08-17T22:09:59.926727238Z 22:09:59.926 [Thread-16] INFO com.optum.link.pulsar.utils.TokenUtils - binary connection data 2020-08-17T22:09:59.926732938Z 22:09:59.926 [Thread-16] INFO com.optum.link.pulsar.broker.authorization.AuthorizationProviderIdx - token:redacted for pulsar-client 2020-08-17T22:09:59.928697558Z 22:09:59.928 [Thread-16] INFO com.optum.link.pulsar.broker.authorization.AuthorizationProviderIdx - this is a broker token, role:pulsar-client 2020-08-17T22:09:59.928819659Z 22:09:59.928 [pulsar-io-25-2] INFO com.optum.link.pulsar.utils.TokenUtils - binary connection data 2020-08-17T22:09:59.928827759Z 22:09:59.928 [pulsar-io-25-2] INFO com.optum.link.pulsar.broker.authorization.AuthorizationProviderIdx - checking superuser for pulsar-client with token eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJwcm94eS1hZG1pbiJ9.165LCuseXmmR-VtENoARbsXqzHnSHOZy9kntpWaHXm8 2020-08-17T22:09:59.935837429Z 22:09:59.935 [Thread-17] INFO com.optum.link.pulsar.utils.TokenUtils - binary connection data 2020-08-17T22:09:59.935880029Z 22:09:59.935 [Thread-17] INFO com.optum.link.pulsar.broker.authorization.AuthorizationProviderIdx - checking canConsume for pulsar-client with token:redacted 2020-08-17T22:09:59.938719357Z 22:09:59.938 [Thread-17] INFO com.optum.link.pulsar.broker.authorization.AuthorizationProviderIdx - canConsume: This is a broker token 2020-08-17T22:09:59.938847458Z 22:09:59.938 [pulsar-io-25-2] WARN org.apache.pulsar.broker.web.PulsarWebResource - [<persistent://uhc/sunshine/healthdata>] Role pulsar-client is not allowed to lookup topic 2020-08-17T22:09:59.939606566Z 22:09:59.939 [pulsar-io-25-2] WARN org.apache.pulsar.broker.lookup.TopicLookupBase - Failed to authorized pulsar-client on cluster <persistent://uhc/sunshine/healthdata> 2020-08-17T22:09:59.959829466Z 22:09:59.959 [pulsar-io-25-2] INFO org.apache.pulsar.broker.service.ServerCnx - Closed connection from /10.1.0.100:50158``` ---- 2020-08-17 23:15:27 UTC - Addison Higham: is that with that setting being set? ---- 2020-08-17 23:15:49 UTC - Joshua Decosta: No, this was before you just mentioned it ---- 2020-08-17 23:15:56 UTC - Joshua Decosta: does that just get set in the broker? ---- 2020-08-17 23:20:39 UTC - Addison Higham: yes ---- 2020-08-17 23:21:17 UTC - Addison Higham: you should just need to set that in the helm configData `authenticateOriginalAuthData=true` ---- 2020-08-17 23:45:06 UTC - Joshua Decosta: ok that didn't change anything. ---- 2020-08-17 23:45:20 UTC - Joshua Decosta: still the same results ---- 2020-08-17 23:48:06 UTC - Joshua Decosta: its still just passing in the proxy token instead of the client token. ---- 2020-08-17 23:48:15 UTC - Joshua Decosta: whats this line about: ```2020-08-17T23:41:31.101284484Z 23:41:31.101 [pulsar-io-25-2] DEBUG org.apache.pulsar.broker.service.ServerCnx - [/10.1.0.119:47920] Client successfully authenticated with token role null and originalPrincipal null``` ---- 2020-08-18 01:26:22 UTC - Alexandre DUVAL: great, thx @Addison Higham ---- 2020-08-18 01:26:58 UTC - Alexandre DUVAL: @Sijie Guo applied this version and didn't reproduced my issue since then. thanks :slightly_smiling_face:. ---- 2020-08-18 01:42:18 UTC - Tymm: Thanks. Im using pulsar-flink. Do you know how to perform that in pulsar-flink? Thanks. ---- 2020-08-18 04:11:01 UTC - Sijie Guo: awesome! Great to hear that! ---- 2020-08-18 06:32:08 UTC - Renkai Ge: @Renkai Ge has joined the channel ---- 2020-08-18 07:28:42 UTC - ThibaudB: @ThibaudB has joined the channel ---- 2020-08-18 08:15:28 UTC - ThibaudB: Hi, i m trying to add tiered storage to our Pulsar cluster. We would like to use the offloading mecanism on a mounted disk (not aws, google nor hdfs), did you hear about an existing local offloader ? or any alternative ? ---- 2020-08-18 08:25:20 UTC - Guillaume: Hi, Is there a way to get notified when Pulsar Java Client gets disconnected ? Thanks. ---- 2020-08-18 08:38:54 UTC - Arul: @Arul has joined the channel ----
