2020-06-24 09:19:06 UTC - rani: > Can’t you update the AMI on existing bookies? No, I believe the only way to update the AMI on an ec2 instance is to re-create it. Since i’m using an ASG, the process would look as follows: 1. update launch configuration to use new AMI 2. scale up ASG 3. scale down ASG to terminate EC2 instances using old AMI
---- 2020-06-24 13:34:44 UTC - Daniel Ciocirlan: can someone help me figure this out : we are producing in 1 region and geo-replicating to 3 consuming regions, looking at replication backlog i can see the normal Producer region replication backlog to regions 1,2 and 3, but also i can see some replication from regions 3,2,1 to producer region, why is that since they only consume ? ---- 2020-06-24 14:30:51 UTC - rani: I seem to have had a breakthrough with this setup. I took your advice Sijieg, and rather than decommissioning all 3 bookkeepers simultaneously, I did so 1 at a time, and the response was much better! ---- 2020-06-24 14:53:47 UTC - Addison Higham: Geo-replication is always bi-directional, if you producer from another region, it will be replicated the other direction. Pulsar just always establishes the replication both directions even if it is never used ---- 2020-06-24 14:58:10 UTC - Abhishek Varshney: A couple of questions on ordering of messages: 1. For a partitioned topic, if the number of partitions are increased online while there are already unconsumed messages in existing partitions, is the ordering at producer end still guaranteed? This is assuming that routing mode is `RoundRobinPartition` with `key` provided and additional partitions are automatically detected by the producer as mentioned <https://pulsar.apache.org/docs/en/2.5.1/admin-api-partitioned-topics/#update|here>. I guess the answer would be *no*, because the hash computation would change and new messages on an existing key can now fall to a different partition. Please confirm. 2. Is it still possible for consumers to somehow consume with strict ordering, while the number of partitions are updated online, without disrupting operations? 3. When a message is negatively acknowledged by a `key-shared` consumer, with a retry or reconsume policy, would Pulsar not deliver subsequent messages with the same `key` until this message is successfully consumed? If not, how to handle this scenario where delivery of messages is strictly in order even when there are negative acks, without consumer keeping track of the state of consumption of a grouping key? 4. Handling scenario 3 and 1 at the same time without disruptions. ---- 2020-06-24 15:00:39 UTC - Daniel Ciocirlan: Thanks @Addison Higham ! I was aware of the bi-directional behavior but found it strange the large amounts of data being transmited from consumer regions to producer region. The backlog was larger than from producer to consumer regions ---- 2020-06-24 15:16:36 UTC - rwaweber: Hey Sijie! Thanks for the pointers, thats definitely helped me get a better picture of whats happening. Looking at the internal stats for the topic, shows that the topic has two ledgers(from the `ledgers` array, and I’m assuming that they’re _both_ open, based on `"state": "LedgerOpened"` . One of those ledgers has a size of nearly 1.1 GB and the other is completely empty. The response is also reporting a `"totalSize"` of the same size as the one ledger that is occupying space. Confusingly, the prometheus metric `pulsar_storage_size` for this topic is reporting 2.295G of space. What would you think is occupying that other 1.195G of space beyond the ledger? ---- 2020-06-24 16:52:26 UTC - Jon Cordeiro: When calling the REST API to register a function, is it possible to use a jar url that requires basic auth? e.g `<https://username>:<mailto:[email protected]|[email protected]>/......` ? When trying this I receive the following error message: ```Bad Request - {"reason":"Encountered error \"Server returned HTTP response code: 401 for URL: <JAR_URL_HERE>" when getting Function package from <JAR_URL_HERE>"}``` Hitting the same jar url in my browser prompts a download of the jar ---- 2020-06-24 16:53:24 UTC - Praveen Sannadi: @Praveen Sannadi has joined the channel ---- 2020-06-24 16:57:49 UTC - Praveen Sannadi: Hello All, ---- 2020-06-24 17:03:41 UTC - Praveen Sannadi: I am new to Pulsar and trying to explore Pulsar functions. I am trying to do CRUD operations on Pulsar functions through Admin API and functions-api. get and delete are working fine but while creating a function i am getting an exception ChannelOutputShutdownException with 2 different causes 1. Protocol wrong type for socket and 2. Broken pipe can any one know this issue already please help me I kind of stuck here. Thanks in advance. ---- 2020-06-24 17:09:16 UTC - Sijie Guo: only last ledger is open. all the other ledgers are sealed and closed. Can you paste the details of `topics stats-internal`? +1 : rwaweber ---- 2020-06-24 17:10:44 UTC - Sijie Guo: regarding `pulsar_storage_size`, do you have any other topics? you can check the raw prometheus data to see if there is any double-couting when aggegrate them together. ---- 2020-06-24 17:12:19 UTC - Praveen Sannadi: Exception in detail : - org.apache.pulsar.client.admin.PulsarAdminException: java.util.concurrent.ExecutionException: org.apache.pulsar.shade.io.netty.channel.socket.ChannelOutputShutdownException: Channel output shutdown. Caused by: <http://java.io|java.io>.IOException: Protocol wrong type for socket ---- 2020-06-24 17:13:55 UTC - Praveen Sannadi: Java Code Snippet which I am using:- ```FunctionConfig functionConfig = new FunctionConfig(); functionConfig.setTenant(tenant); functionConfig.setNamespace(namespace); functionConfig.setName("function-15"); functionConfig.setRuntime(FunctionConfig.Runtime.JAVA); functionConfig.setParallelism(1); functionConfig.setClassName("com.optum.customers.com.optum.customers.pulsar.functions.ExclamationFunction"); functionConfig.setProcessingGuarantees(FunctionConfig.ProcessingGuarantees.ATLEAST_ONCE); functionConfig.setTopicsPattern("<persistent://tenant-9/namespace-4/topic-29>"); functionConfig.setSubName("subscription97"); functionConfig.setAutoAck(true); functionConfig.setOutput("<persistent://tenant-9/namespace-4/topic-19>"); //admin.functions().deleteFunction("tenant-9", "namespace-4", "function-3"); admin.functions().getFunctions("tenant-9", "namespace-4").forEach(function -> System.out.println(function)); admin.functions().createFunction(functionConfig, "customers-0.0.1-SNAPSHOT.jar"); admin.functions().getFunctions("tenant-9", "namespace-4").forEach(function -> System.out.println(function));``` ---- 2020-06-24 17:22:59 UTC - Sijie Guo: I don’t think we support that yet. Can you file an issue for supporting that? ---- 2020-06-24 17:23:14 UTC - Jon Cordeiro: will do. Thanks :+1: ---- 2020-06-24 17:34:30 UTC - Jon Cordeiro: Issue has been created. Please let me know if any additional information is needed. Thanks! <https://github.com/apache/pulsar/issues/7354> ---- 2020-06-24 17:34:37 UTC - Abhishek Varshney: To expand more on point #3, RabbitMQ's <https://www.rabbitmq.com/semantics.html#ordering|documentation> says > Messages can be returned to the queue using AMQP methods that feature a requeue parameter (basic.recover, basic.reject and basic.nack), or due to a channel closing while holding unacknowledged messages. Any of these scenarios caused messages to be requeued at the back of the queue for RabbitMQ releases earlier than 2.7.0. From RabbitMQ release 2.7.0, messages are always held in the queue in publication order, even in the presence of requeueing or channel closure. Does Pulsar support the same in terms of maintaining the ordering? And how does it play along with <http://pulsar.apache.org/docs/en/concepts-messaging/#retry-letter-topic|retry letter topic>. ---- 2020-06-24 18:18:25 UTC - Gavin Liu: @Gavin Liu has joined the channel ---- 2020-06-24 18:26:27 UTC - Gavin Liu: Hi guys, I have a question about s3 offload, retention and backlog quota for partitioned topic, Say I have 10 partitions of a topic named `foo` if I set 100GB on s3 offload, is this limit on the topic or per partition? My experience told me it is per partition. if I set 150GB on backlog quota, is this limit on the subscription, or per partition of the subscription? If I set 200GB on retention, is the limit on the subscription, or per partition of the subscription? ---- 2020-06-24 19:08:56 UTC - b_rousseau: I have just discovered this PR <https://github.com/apache/pulsar/pull/7194> . :pray::pray: Would it make sense to also offer this component embedded in PrestoSQL default connectors (beside Hive, Kafka, PostgreSQL, MySQL, DB2, Oracle, ElasticSearch, Druid...) ? ---- 2020-06-24 19:11:51 UTC - rwaweber: Here’s a gist with the response in question: <https://gist.github.com/rwaweber/5808812e7a016fe260b4dccd6250d166> I have _one_ other topic on the cluster, though that’s only holding onto 125kb of data, I figured it wouldn’t necessarily be affecting this. On a whim, I took the value of `pulsar_storage_size` <tel:2295268076|(2,295,268,076> bytes) for that topic, and divided it by two and it’s yielding the exact same value as what’s left in the last open ledger(1,147,634,038). So it _appears_ something is causing the `pulsar_storage_size` value to be double the size of the last open ledger. Maybe this is due to a replication setting that I’m not paying attention to? ---- 2020-06-24 19:40:16 UTC - Jen Crawford: @Jen Crawford has joined the channel ---- 2020-06-24 19:48:23 UTC - Gavin Liu: Any help? I think all these 3 are limits on the partition level. ---- 2020-06-25 04:47:21 UTC - jinggang: @jinggang has joined the channel ---- 2020-06-25 04:49:08 UTC - jinggang: The golang pulsa client doesn’t support the custom authentication, right? ---- 2020-06-25 04:52:27 UTC - Matteo Merli: Not at the moment, the Auth Provider interface is kept in the `pulsar.internal.auth` package. This was done in order to not commit with a specific API until we were ready for that. ---- 2020-06-25 04:53:42 UTC - Matteo Merli: What kind of auth mechanism are you looking at? ---- 2020-06-25 04:56:07 UTC - jinggang: the authname is auth1, which is something like basic auth ---- 2020-06-25 08:56:46 UTC - Kirill Merkushev: Hello - is there any way to unterminate a topic? As I tried to replace non-partitioned persistent topic with partitioned one and with pulsarctl first terminated that, than with pulsar-admin removed and recreated it. But now producers are saying that topic was terminated (pulsar 2.5.0) ----
