2020-06-08 09:48:25 UTC - Asaf Mesika: Is there any way to negativeAcknowledge a message and use `deliverAfter()` in conjuction, so I can implemented exponential backoff retry policy using that? eyes : Konstantinos Papalias ---- 2020-06-08 10:14:55 UTC - Rahul Vashishth: @Addison Higham the docs says
_`Pulsar supports mutual TLS and Athenz authentication plugins`_ can it be extended to use jwt oauth/openidconnect ? ---- 2020-06-08 10:16:22 UTC - Rahul Vashishth: @eric.olympe hey.. did you get success in securing pulsar using keycloak. i do have exact same use case. ---- 2020-06-08 10:33:13 UTC - eric.olympe: I have not tried yet. ---- 2020-06-08 10:47:25 UTC - Rahul Vashishth: @Rahul i m wondering if you were able to to use multiple roles using custom providers. A question, is the pulsar cluster roles are also mapped to some outside auth provider solution? like keycloak. ---- 2020-06-08 11:00:45 UTC - Rahul Vashishth: @Sijie Guo @Addison Higham Is it also possible to manage(CRUD) roles for topics using admin API? instead of managing roles at the namespace level. I was wondering if <https://pulsar.apache.org/docs/en/2.5.2/security-jwt/#pulsar-client|pulsar java client> can also refresh the token by itself? I was reading comments I see it might be possible to implement `AuthenticationProvider` and `AuthorizationProvider` interfaces to parse token and pass roles. ---- 2020-06-08 12:15:35 UTC - Alexandre DUVAL: @Sijie Guo ---- 2020-06-08 12:15:46 UTC - Alexandre DUVAL: @Sijie Guo ---- 2020-06-08 12:15:56 UTC - Alexandre DUVAL: @Sijie Guo ---- 2020-06-08 12:20:44 UTC - Phil Sheppard: @Phil Sheppard has joined the channel ---- 2020-06-08 12:54:44 UTC - Rahul Vashishth: Are the producer/consumer are ephemeral, or i can create it in advance for a client. I am trying to related this with a generai API gateway. The use case is such(using a custom UI) • an admin user create topics in namespace and roles for it • client's register themselv as producer or consumers. UI app will generate client_id, secret that the producer/consumer can use later for authn/authz • producer/consumer send token to cluster with the request • cluster which already has custom implementation for `AuthenticationProvider` and `AuthorizationProvider` can securly allow the consume or produce msg. Can we do CURD on producer/consumer using admin api? ---- 2020-06-08 12:57:12 UTC - Rahul Vashishth: @Addison Higham @Sijie Guo is there an easy way to calculate/estimate infra need for a pulsar culster. given we know the transaction counts, message size and number of producer consumers and topics on cluster. ---- 2020-06-08 13:35:56 UTC - Amit Pal: iiuc, wouldn't that require you to consume the message, send the ack back, enqueue the message again with `deliverAfter` set .... this should solve your usecase :thinking_face: ---- 2020-06-08 15:13:29 UTC - Alexander Ursu: Hi, I was wondering what the minimum required IAM permissions are for the `aws-s3` offload driver ---- 2020-06-08 15:33:46 UTC - Spencer: @Spencer has joined the channel ---- 2020-06-08 15:42:29 UTC - Asaf Mesika: @Amit Pal No TX support, means I may end up due to process gets killed with two resubmission of the same message, since I enqueue a new message, and got killed before I acked. Restart process, read message, enqueue a message then ack ---- 2020-06-08 16:13:12 UTC - Addison Higham: you certainly could implement oauth/oidc, I would just read the above doc ---- 2020-06-08 16:30:02 UTC - Addison Higham: I am not sure I totally follow... but pulsar-admin uses rest APIs to manage roles. Java has a pulsar-admin library and you can use them programmatically or you can use your own client just against the pulsar REST management APIs ---- 2020-06-08 16:30:12 UTC - Addison Higham: that seems reasonable what you have describe above ---- 2020-06-08 16:30:15 UTC - Addison Higham: (AFAICT) ---- 2020-06-08 17:43:53 UTC - sjmittal: @sjmittal has joined the channel ---- 2020-06-08 18:45:21 UTC - Asaf Mesika: Regarding the PR that will be released in 2.6.0. I’ve noticed this feature was implemented at the client side. This means, if you deliver a new message with retry=2 to the retry topic, and then fail, thus you don’t ack the original message. Meaning, you end up with two messages having the exact same meaning - the original message which will be redelivered since application crashed, and the new retry message. This can lead to business logic bugs, no? @Penghui Li ---- 2020-06-08 19:01:18 UTC - sb: @sb has joined the channel ---- 2020-06-08 19:05:06 UTC - sb: Hi Does <http://pulsar.apache.org/docs/en/adaptors-kafka|kafka-adaptor> support <https://pulsar.apache.org/docs/en/2.5.2/security-encryption/|message encryption>? I couldn't find any config for `CryptoKeyReader` and `EncryptionKey` ---- 2020-06-08 19:07:07 UTC - Sijie Guo: yeah. it lacks the documentation. @Enrico Olivelli can probably point you some references. ---- 2020-06-08 19:07:47 UTC - Sijie Guo: • shutdown a bookie • run the decommissionbookie command in any node that is able to connect to zookeeper. ---- 2020-06-08 19:08:10 UTC - Sijie Guo: I think it is exposed to force the ns deletion. ---- 2020-06-08 19:10:04 UTC - Sijie Guo: If you customize the provider, you can map the Realm/User in keycloak to Pulsar roles. ---- 2020-06-08 19:10:25 UTC - Enrico Olivelli: I am not sure we have docs ---- 2020-06-08 19:15:19 UTC - Sijie Guo: How did you run broker? If you are running standalone, you should modify standalone.conf. If you are running brokers in cluster, did you try to `curl -L http://<broker-ip>:8080/metrics` to see if you can find the metrics like below? ```pulsar_consumer_msg_rate_redeliver{cluster="standalone",namespace="public/default",topic="<persistent://public/default/perftopic3>",subscription="test", consumer_name="929fb",consumer_id="0"} 0.0 1591643632708 # TYPE pulsar_consumer_unacked_messages gauge pulsar_consumer_unacked_messages{cluster="standalone", namespace="public/default",topic="<persistent://public/default/perftopic3>",subscription="test", consumer_name="929fb",consumer_id="0"} 0 1591643632708 # TYPE pulsar_consumer_blocked_on_unacked_messages gauge pulsar_consumer_blocked_on_unacked_messages{cluster="standalone", namespace="public/default",topic="<persistent://public/default/perftopic3>",subscription="test", consumer_name="929fb",consumer_id="0"} 0 1591643632708 # TYPE pulsar_consumer_msg_rate_out gauge pulsar_consumer_msg_rate_out{cluster="standalone",namespace="public/default",topic="<persistent://public/default/perftopic3>",subscription="test",consumer_name="929fb", consumer_id="0"} 0.0 1591643632708``` ---- 2020-06-08 19:17:44 UTC - Enrico Olivelli: Basically it tries to place data on bookies that have more disk space available. It uses the GetBookieInfo RPC ---- 2020-06-08 19:33:21 UTC - lucas amoroso: @lucas amoroso has joined the channel ---- 2020-06-08 20:03:15 UTC - Alexandre DUVAL: but unused in the method so it doesn't force. Maybe i can contribute ot it? ---- 2020-06-08 21:21:36 UTC - Marcio Martins: Hey guys, I am running Pulsar inside an EKS cluster and am trying to setup offloading to S3, but am having issues with permissions: ```Caused by: com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException: User: arn:aws:sts::29447664:assumed-role/my-node/i-0abee06d3c85cd8ec is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::29447664/test-pulsar-offload (Service: AWSSecurityTokenService; Status Code: 403; Error Code: AccessDenied; Request ID: 45de83c0-d8d1-468b-ae7f-ec2dacab8f39)``` I am trying to use the EKS OIDC roles for service account, but can't get it to work. Anyone has any experience with this? I am trying to avoid setting the S3 permissions on every node... ---- 2020-06-08 21:22:42 UTC - Sijie Guo: Yes please ---- 2020-06-08 22:00:07 UTC - Alexandre DUVAL: Ok thx, ```21:24:52.728 [main] INFO org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Initialize rackaware ensemble placement policy @ <Bookie:192.168.10.7:0> @ /default-rack : org.apache.bookkeeper.client.TopologyAwareEnsemblePlacementPolicy$DefaultResolver. 21:24:52.729 [main] INFO org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Not weighted 21:24:52.733 [main] INFO org.apache.bookkeeper.client.BookKeeper - Weighted ledger placement is not enabled 21:24:52.772 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /default-rack/clevercloud-bookkeeper-c1-n5:3181 21:24:52.773 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /default-rack/clevercloud-bookkeeper-c1-n6:3181 21:24:52.773 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /default-rack/clevercloud-bookkeeper-c1-n3:3181 21:24:52.773 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /default-rack/clevercloud-bookkeeper-c1-n4:3181 21:24:52.773 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /default-rack/clevercloud-bookkeeper-c1-n2:3181 21:24:52.988 [main] INFO org.apache.bookkeeper.client.BookKeeperAdmin - Resetting LostBookieRecoveryDelay value: 0, to kickstart audit task 21:25:26.055 [main] INFO org.apache.bookkeeper.client.BookKeeperAdmin - Count of Ledgers which need to be rereplicated: 758 21:34:03.367 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.net.NetworkTopologyImpl - Removing a node: /default-rack/clevercloud-bookkeeper-c1-n2:3181 21:34:03.375 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /default-rack/clevercloud-bookkeeper-c1-n2:3181 21:35:30.620 [main] INFO org.apache.bookkeeper.client.BookKeeperAdmin - Count of Ledgers which need to be rereplicated: 360 21:45:32.740 [main] INFO org.apache.bookkeeper.client.BookKeeperAdmin - Count of Ledgers which need to be rereplicated: 360 21:55:34.839 [main] INFO org.apache.bookkeeper.client.BookKeeperAdmin - Count of Ledgers which need to be rereplicated: 360 22:05:13.364 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.net.NetworkTopologyImpl - Removing a node: /default-rack/clevercloud-bookkeeper-c1-n3:3181 22:05:13.384 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /default-rack/clevercloud-bookkeeper-c1-n3:3181 22:05:37.105 [main] INFO org.apache.bookkeeper.client.BookKeeperAdmin - Count of Ledgers which need to be rereplicated: 360 22:15:39.367 [main] INFO org.apache.bookkeeper.client.BookKeeperAdmin - Count of Ledgers which need to be rereplicated: 360 22:25:41.692 [main] INFO org.apache.bookkeeper.client.BookKeeperAdmin - Count of Ledgers which need to be rereplicated: 360 22:35:44.012 [main] INFO org.apache.bookkeeper.client.BookKeeperAdmin - Count of Ledgers which need to be rereplicated: 360 22:45:46.321 [main] INFO org.apache.bookkeeper.client.BookKeeperAdmin - Count of Ledgers which need to be rereplicated: 360 22:46:33.363 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.net.NetworkTopologyImpl - Removing a node: /default-rack/clevercloud-bookkeeper-c1-n3:3181 22:46:33.382 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /default-rack/clevercloud-bookkeeper-c1-n3:3181 22:55:48.403 [main] INFO org.apache.bookkeeper.client.BookKeeperAdmin - Count of Ledgers which need to be rereplicated: 360 23:05:50.513 [main] INFO org.apache.bookkeeper.client.BookKeeperAdmin - Count of Ledgers which need to be rereplicated: 360``` 360 -> 360 -> 360 is a normal behavior? ---- 2020-06-08 22:43:10 UTC - Alexandre DUVAL: i stopped one bookie, and launched decomission on it, and other nodes still try to connect, normal? ---- 2020-06-08 23:09:54 UTC - Alexandre DUVAL: @Sijie Guo ---- 2020-06-08 23:15:08 UTC - Alexandre DUVAL: i followed <https://bookkeeper.apache.org/docs/latest/admin/decomission/> ---- 2020-06-09 01:17:33 UTC - Sijie Guo: Did you have any ledgers with replication factor 1? ---- 2020-06-09 01:55:01 UTC - Alexandre DUVAL: it shouldn't but it's possible as it was the first cluster we started ---- 2020-06-09 01:55:18 UTC - Alexandre DUVAL: @Sijie Guo ---- 2020-06-09 01:56:01 UTC - Alexandre DUVAL: -meta ---- 2020-06-09 01:56:19 UTC - Alexandre DUVAL: so the ones with `ensembleSize=1` ---- 2020-06-09 01:56:52 UTC - Alexandre DUVAL: yes some are present ---- 2020-06-09 02:06:39 UTC - Alexandre DUVAL: They are probably very old ledgers. I don't how to nuke them or print their content to check them as pulsar-bookie node on which they is stopped due to decomission running. ---- 2020-06-09 03:56:32 UTC - Keli: @Keli has joined the channel ---- 2020-06-09 04:30:44 UTC - Sijie Guo: you can use `bin/bookkeeper shell recover` (the manual recover command) to dry run to figure the list of ledgers whose ensembleSize is 1. ---- 2020-06-09 05:36:37 UTC - Rahul Vashishth: can we also create producer/consumers using pulsar admin api in advance. And later a client app can only use that producer/consumer names which were created using admin api? ---- 2020-06-09 08:01:18 UTC - Huanli Meng: @Addison Higham, one more question, the default tenant and namespace is used when no tenant or namespace is specified, it is applied for all clients? or just suitable for Java client. Thanks. ---- 2020-06-09 08:50:26 UTC - Korben: @Korben has joined the channel ---- 2020-06-09 08:58:27 UTC - Korben: Hey guys There is a work in progress on Transactions support(PIP-31). I'd like to clarify if it's possible to consume one message and produce the result to one topic atomically in the latest Pulsar version? ---- 2020-06-09 09:02:40 UTC - jujugrrr: it looks like it didn't get you OIDC role but fell back to your instance role. I still have to try this as well, it's unclear from jcloud which is used by the offloading if AWS auth with WebIdentity/OIDC is supported ----
