2020-08-18 10:37:12 UTC - Penghui Li: Do you mean the producer or consumer disconnected from the broker? ---- 2020-08-18 10:42:00 UTC - Penghui Li: hdfs offloader is base on the FileSystem of hadoop, I think you can try to set `fs.file.impl=org.apache.hadoop.fs.LocalFileSystem` to offload data to your local disk. ---- 2020-08-18 11:30:43 UTC - Greg: Hi, is there a configuration somewhere that would allow to decrease the broker failure detection time ? ---- 2020-08-18 11:41:25 UTC - Guillaume: Yes. I'd like to get the event when producer or consumer gets disconnected from the broker. ---- 2020-08-18 11:52:48 UTC - Penghui Li: Ok, You can add a consumer event listener for listening consumer related events, but at the producer side, pulsar does not support the event listener, you can create a github issue for tracking this feature. ---- 2020-08-18 12:41:30 UTC - Guillaume: Is there any documentation on how to add a event listener for consumer ? ---- 2020-08-18 13:00:22 UTC - Arthur: Hi, my java client get disconnected from Pulsar sometimes with following error ```Close connection because received internal-server error org.apache.pulsar.client.api.PulsarClientException$LookupException: java.lang.IllegalStateException: Namespace bundle xiotnx/agent_ns/0xc0000000_0xffffffff is being unloaded 2020-08-18 11:30:47,024 WARN [pulsar-client-io-1-1] org.apache.pulsar.client.impl.ConnectionHandler: [<persistent://mytenant/agent_ns/data_up-19847c0a-0c29-4fa4-8d00-9915c054d488>] [data_subscription] Could not get connection to broker: org.apache.pulsar.client.api.PulsarClientException$LookupException: org.apache.pulsar.client.api.PulsarClientException$LookupException: java.lang.IllegalStateException: Namespace bundle mytenant/agent_ns/0xc0000000_0xffffffff is being unloaded -- Will try again in 0.19 s``` And it finally got reconnected. What does this error means? Could it be related to a high number of parallel connexion to broker? I don't find related error in pulsar broker logs for the same timestamp ---- 2020-08-18 14:28:50 UTC - Piyush: Hey everyone - is there a way to read the file header using the default file IO connector pulsar provides? <https://pulsar.apache.org/docs/en/io-file-source/#configuration> I could not find any configurations related to headers in FileIOConnector. Thanks. ---- 2020-08-18 16:24:33 UTC - Joshua Decosta: @Addison Higham any chance it has something to do with the command ```command = Commands.newLookup(topic, authoritative, requestId);``` in `LookUpProxyHandler.performLookup` ---- 2020-08-18 16:28:04 UTC - Chris: Having an issue with authorization/authentication and pulsar functions workers running separately from brokers. When I enable authn/z the function worker gets through a bit of startup, then just exits. No exception, no anything. Even attached a debugger, jdb just says it exits. Somewhat related, when the client auth parameters are provided, the fw's client doesn't use them. I see in the startup that it read the fields, but in the client initialization logs it leaves both the class name and params null. Anyone else running separate function workers with token auth? ---- 2020-08-18 17:14:01 UTC - Joshua Decosta: Just in case this is helpful. I've attached the broker config. ---- 2020-08-18 17:28:37 UTC - Sijie Guo: it means a namespace bundle is unloading. Unload can happens when load balancing. ---- 2020-08-18 17:28:59 UTC - Sijie Guo: I don’t think so. what is your requirement? ---- 2020-08-18 17:50:40 UTC - Piyush: we need to read a CSV file with headers and apply transformation on values of specific column from that CSV file. ---- 2020-08-18 19:22:23 UTC - Chris: I was able to get it to work with tls auth, but not token. Token causes that exit without any stack trace. Tls auth seems like a pain though. ---- 2020-08-18 19:23:01 UTC - Chris: And requiring authentication on the fw also makes the fw use its broker auth settings. Seems like those two shouldn't depend on each other. ---- 2020-08-18 22:17:26 UTC - Ryan Nowacoski: @Ryan Nowacoski has joined the channel ---- 2020-08-18 22:22:43 UTC - Ryan Nowacoski: Hi! What is the timeline for the 2.6.1 release? I'm looking for this fix <https://github.com/apache/pulsar/pull/5716> ---- 2020-08-18 22:29:04 UTC - Ryan Nowacoski: @xiaolong.ran, seems like you might have the answer. or @Sijie Guo maybe ---- 2020-08-18 22:32:10 UTC - Ryan Nowacoski: also is the release candidate in maven anywhere? I'd like to try it out even if it's not released yet ---- 2020-08-18 22:58:17 UTC - Sijie Guo: I think #5716 is already released in 2.6.0 ---- 2020-08-18 23:53:53 UTC - Maksym Domarev: @Maksym Domarev has joined the channel ---- 2020-08-18 23:55:07 UTC - Ryan Nowacoski: it has the 2.6.0 milestone on the PR but the ticket # wasn't included in the release notes here > <https://github.com/apache/pulsar/releases/tag/v2.6.0> ---- 2020-08-18 23:55:15 UTC - Maksym Domarev: is there any article that describes difference between message key and ordering key for shared_key consumer? I tried to google but wasn't to able to find description of this concept. ---- 2020-08-19 00:05:06 UTC - Ryan Nowacoski: actually I think you're right but it's broken or I'm using it wrong :disappointed_relieved: ---- 2020-08-19 00:05:55 UTC - Ryan Nowacoski: I'm trying to subscribe to `persistent://\w+/Test/Test1` and it fails. it also fails on `<persistent://Test/>\w+/Test1` ---- 2020-08-19 00:06:48 UTC - Ryan Nowacoski: ```java.lang.IllegalArgumentException: Invalid named entity: \w+ at org.apache.pulsar.common.naming.NamedEntity.checkName(NamedEntity.java:39) at org.apache.pulsar.common.naming.NamespaceName.validateNamespaceName(NamespaceName.java:179) at org.apache.pulsar.common.naming.NamespaceName.get(NamespaceName.java:53) at org.apache.pulsar.common.naming.TopicName.<init>(TopicName.java:151) at org.apache.pulsar.common.naming.TopicName.<init>(TopicName.java:38) at org.apache.pulsar.common.naming.TopicName$1.load(TopicName.java:63) at org.apache.pulsar.common.naming.TopicName$1.load(TopicName.java:60) at org.apache.pulsar.shade.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) at org.apache.pulsar.shade.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2276) at org.apache.pulsar.shade.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2154) at org.apache.pulsar.shade.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2044) at org.apache.pulsar.shade.com.google.common.cache.LocalCache.get(LocalCache.java:3951) at org.apache.pulsar.shade.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3973) at org.apache.pulsar.shade.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4957) at org.apache.pulsar.common.naming.TopicName.get(TopicName.java:88) at org.apache.pulsar.client.impl.PulsarClientImpl.patternTopicSubscribeAsync(PulsarClientImpl.java:403) at org.apache.pulsar.client.impl.PulsarClientImpl.subscribeAsync(PulsarClientImpl.java:333) at org.apache.pulsar.client.impl.ConsumerBuilderImpl.subscribeAsync(ConsumerBuilderImpl.java:142) at org.apache.pulsar.client.impl.ConsumerBuilderImpl.subscribe(ConsumerBuilderImpl.java:99)``` ---- 2020-08-19 00:43:21 UTC - Ryan Nowacoski: it works if I supply a list of topics `<persistent://123/Test/Test1>` and `<persistent://124/Test/Test1>` but the regex fails when validating the namespace/tenant names ---- 2020-08-19 01:12:25 UTC - Ryan Nowacoski: @Sijie Guo I created an issue here > <https://github.com/apache/pulsar/issues/7846> with full details on what is happening and why ---- 2020-08-19 01:16:49 UTC - Sijie Guo: @Ryan Nowacoski release notes only highlight a subset of important issues. You can find all the issues here at <https://github.com/apache/pulsar/issues?q=is%3Aclosed+milestone%3A2.6.0> ---- 2020-08-19 01:17:08 UTC - Sijie Guo: for issue #7846, @Penghui Li can help take a look ---- 2020-08-19 01:17:35 UTC - Ryan Nowacoski: I think I have a fairly good solution in the Proposed Solution section of the issue I created ---- 2020-08-19 01:17:53 UTC - Sijie Guo: @Ryan Nowacoski the 2.6.1 release is under voting. should be released before end of this week. ---- 2020-08-19 01:18:08 UTC - Ryan Nowacoski: I'm going to take a look at writing the change myself but any help would be appreciated. This is essential for something I'm working on currently ---- 2020-08-19 01:19:36 UTC - Ryan Nowacoski: my concern with my solution is how does Pulsar client handle automatically subscribing to new topics that are created that match the pattern. My solution might not fix that functionality ---- 2020-08-19 01:19:42 UTC - Sijie Guo: message key is used for routing messages to different partition. but by default (if no ordering key is specified), message key is also used for routing messages to consumers in key_shared subscription.
ordering key provides an alternative way to define how messages want to be dispatched to consumers. ---- 2020-08-19 01:25:44 UTC - Maksym Domarev: Thanks a lot , > You need to specify a key or orderingKey for messages Docs saying `or`, I need a strict order within a certain consumer ( among few), as far as I get, once I set up key a certain key will be routed to a certain consumer. How can I still achieve the same result but also maintain an order? Should I set in this case `key: someID` `orderingKey: 0` and increment the order? or how does that work ? ---- 2020-08-19 02:22:05 UTC - Maksym Domarev: Hi again, how does `nack` affect ordering, I saw in docks which e.g. I have local queue size of `10`, in the middle of consuming on the item `#5` error occurred , resulted to `nack`. Several questions here: 1. Will the rest of items stops , and order will maintained, if not, is there a way to enforce that behavior e.g. set queue size to `1` 2. When my nacked message retried how will it arrived just before item `#4` Is there any way to achieve this behavior 3. I assume I will use sharedKey subscription type but will exclusive be more suitable ? ---- 2020-08-19 02:53:38 UTC - Maksym Domarev: One last question, I’m aware of 1M topics limit, does non-persistent topic after few days will be completely deleted? My use case assume 1m topics every 3-4 months, but most high loaded and persistent is around 1k, the rest is just topics that lives 1-2 days. Will I have any troubles after reaching >1M of topics ? ---- 2020-08-19 03:13:49 UTC - Jennifer Huang: @Jennifer Huang set the channel topic: We’re excited to announce StreamNative Cloud, providing Apache Pulsar®-as-a-Service. Check out the news at <https://streamnative.io/blog/tech/2020-08-18-announcing-streamnative-cloud> ---- 2020-08-19 04:45:15 UTC - Vincent Wong: @Vincent Wong has joined the channel ---- 2020-08-19 05:01:51 UTC - Vincent Wong: Hi I have some issue with the pulsar anyone can help ? we have a pulsar cluster install on five nodes. the broker service on two of the nodes are down due to java OOM. then I decide to shutdown bookie, configuration-store and zookeeper and shutdown the two nodes to increase their memory after that, I start zookeeper, configuration-store, bookie and broker back. after sometime, we have errors. and the topics of some certain namespaces cannot be listed due to timeout ---- 2020-08-19 06:14:35 UTC - Maksym Domarev: I actully checked, nack will put it to the very end , it will break the order ---- 2020-08-19 08:11:45 UTC - ThibaudB: Ok i will try it thanks for your answer ----
