2020-10-27 09:27:16 UTC - Frank Kelly: Good morning . . . if a namespace is
defined as having a retention time and size of zero e.g.
```./bin/pulsar-admin --admin-url <http://platform-pulsar-broker:8080>
namespaces get-retention cogito-dialog/wav
{
"retentionTimeInMinutes" : 0,
"retentionSizeInMB" : 0
}```
How soon after message acknowledgement can we expect the message to be deleted
- is it immediate or is there some asynchronous cleanup process. Thanks in
advance! We have clients who are very sensitive to data retention.
----
2020-10-27 09:28:42 UTC - Ruian: Hi, after upgrade to pulsar 2.6.1, we
encounter `Failed to lookup topic (request timeout)` problem.
Does anyone know how to do further diagnose? The problem still exists after
bookie, broker, proxy, zookeeper restart.
And the logs are all looks normal, except this:
----
2020-10-27 09:29:02 UTC - Ruian:
----
2020-10-27 09:52:48 UTC - David Tinker: I tried this on a test cluster while
producing and consuming messages it all seems to have worked. However once I
had all brokers up and running talking to the new config store I deleted the
/admin node from the cluster zk. The I restarted each broker and the local
/admin node came back. So it looks like some cluster local stuff goes in there?
The producer and consumer continued with no issues.
----
2020-10-27 09:55:25 UTC - David Tinker: I exported the “returned” /admin nodes
----
2020-10-27 09:56:07 UTC - David Tinker: So maybe I should leave the original
cluster local /admin node alone and not delete it?
----
2020-10-27 11:29:19 UTC - Toktok Rambo: hello all, I’m running Pulsar 2.6.1
using Docker standalone (no other options, running with default settings). Even
though `pulsarclient.Ack()` is called, the messages are not removed from the
queue :thinking_face:
----
2020-10-27 11:31:40 UTC - Toktok Rambo: if I restart the app, it’s again
receiving all the previously acked messages
----
2020-10-27 11:32:29 UTC - Toktok Rambo: P.s: I’m using
<http://github.com/apache/pulsar-client-go|github.com/apache/pulsar-client-go>
v0.2.0 client
----
2020-10-27 11:43:02 UTC - Toktok Rambo: the producer and subscriber names are
unique on each run (using time stamp to make it unique)
----
2020-10-27 12:00:44 UTC - Lari Hotari: the unique subscription name for each
run is the problem. You should use the same subscription name if you wish that
the acknowledgement would have an impact.
----
2020-10-27 12:01:11 UTC - Toktok Rambo: oh, when does the Q actually remove the
messages then?
----
2020-10-27 12:02:01 UTC - Toktok Rambo: all messages are persisted forever?
----
2020-10-27 12:04:52 UTC - Toktok Rambo:
<https://pulsar.apache.org/docs/en/concepts-clients/#reader-interface>
> Whenever a new subscription is created, it is initially positioned at the
end of the topic (by default), and consumers associated with that subscription
will begin reading with the first message created afterwards.
:thinking_face:
----
2020-10-27 12:08:12 UTC - Toktok Rambo: oh I see, I have configured it to start
from earliest
----
2020-10-27 12:35:14 UTC - Lari Hotari: by default, the messages get removed
asap if there are no subscriptions or when all subscriptions have acknowledged
the message(s). This is explained in
<https://pulsar.apache.org/docs/en/cookbooks-retention-expiry/> .
btw. The documentation for configuring the retention policy was recently
updated to clarify some details, that's
<https://github.com/apache/pulsar/pull/8356> .
----
2020-10-27 13:31:56 UTC - Frank Kelly: Wow this was lucky I found this - was
just writing code that set the retention time to zero so we would just rely on
size . . . .
+1 : Lari Hotari
----
2020-10-27 13:31:59 UTC - Frank Kelly: :slightly_smiling_face: Thanks!
----
2020-10-27 15:13:48 UTC - Gilles Barbier: Hi, please can anyone from the team
tell us if we should expect stateful functions to be fixed soon or if it's not
a short-term priority? I understand it's in developer preview - but I need this
info to prioritize developments . Thx
----
2020-10-27 15:14:18 UTC - Brendan Lynch: Reading up on acknowledged messages I
am trying to do something similar. I am using pulsar for message passing and I
have a concept of memory barriers at an orchestration level. So that all jobs
released to a topic can run in parallel. Originally I was looking at ways to
reference count this and landed on using acknowledge to determine when the
topic size was zero. While this seems to work it feels like I am using
acknowledgement in an unintended way. Does anyone have some best practice
advice for reference counting items left in the queue?
----
2020-10-27 15:27:41 UTC - Lari Hotari: what issue are you referring to?
----
2020-10-27 15:32:35 UTC - Gilles Barbier: For example
<https://apache-pulsar.slack.com/archives/C5Z4T36F7/p1593169043352900> (issues
included)
----
2020-10-27 15:33:23 UTC - Gilles Barbier: I think functions are ok - the issues
seem related to stateful functions
----
2020-10-27 15:37:31 UTC - Lari Hotari: thanks. Unfortunately I don't have an
answer to your question, but perhaps someone else will be able to answer since
now the full context of your request can be understood.
----
2020-10-27 15:38:58 UTC - Gilles Barbier: Yes sorry my question was a bit vague
(as I discussed this a few weeks ago with @Sijie Guo). Thanks nevertheless for
your time
----
2020-10-27 15:40:58 UTC - Lari Hotari: nop. I guess one possibility to get it
moving would be to create a failing test case. There are already pretty good
instructions for reproducing the issue you are referring to, but anyhow a
failing test case is always part of the final fix so it would take it forward.
+1 : Gilles Barbier
----
2020-10-27 15:42:24 UTC - Gilles Barbier: I understand - it's not that easy as
those issues are not difficult to reproduce but are not deterministic. Can you
point me to some examples to help me to start?
----
2020-10-27 16:18:02 UTC - Sijie Guo: @Gilles Barbier The state is still in
developer preview for the upcoming 2.7.0 release. There are bunch of committers
has already worked on productionizing the state. But I don’t think the change
will be in 2.7.0. It will probably come after 2.7 release. Hope this give you
an idea about the timeline.
----
2020-10-27 16:18:56 UTC - Gilles Barbier: Thx @Sijie Guo
----
2020-10-27 16:23:51 UTC - Sijie Guo: What is the error did you see when you
delete `/admin` path from the local zk cluster?
----
2020-10-27 20:53:27 UTC - Bhavin: @Bhavin has joined the channel
----
2020-10-27 23:23:45 UTC - Rob Shepherd: @Rob Shepherd has joined the channel
----
2020-10-27 23:27:07 UTC - Rob Shepherd: I'm wondering what the issue or
circumstance is to have a pulsar consumer failing with a null schema? This
system was restarted from an update of application code unrelated to pulsar
handling, and has two consumers.
At first both consumers were failing but after a restart one is OK and the
other continues to fail.
I will try restarting again, but i'd like to know how the connected consumer
can encounter a null schema?
I can see the schemas using pulsar-admin, and they look exactly as they should,
and the producers are tied to the same.
----
2020-10-27 23:27:18 UTC - Rob Shepherd: > consumerIngressV1 =
pulsarClient.newConsumer(AvroSchema.of(IngressMessageV1.class))
> .subscriptionName(subscriptionName)
> .topic(topics)
> .subscribe();
> 2020-10-27 22:48:12,322 [pool-3-thread-8] INFO
o.a.p.client.impl.schema.AvroSchema - Load schema reader for version(0), schema
is : , schemaInfo: {
> "name": "",
> "schema": "",
> "type": "NONE",
> "properties": {}
> } --
> 2020-10-27 22:48:12,323 [pool-3-thread-8] ERROR
io.pethau.pl.PipelineService - Exception:
<http://org.apache.pulsar.shade.org|org.apache.pulsar.shade.org>.apache.avro.SchemaParseException:
Cannot parse <null> schema - {} - {} - {} --
>
org.apache.pulsar.shade.com.google.common.util.concurrent.UncheckedExecutionException:
<http://org.apache.pulsar.shade.org|org.apache.pulsar.shade.org>.apache.avro.SchemaParseException:
Cannot parse <null> schema
> at
org.apache.pulsar.shade.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2050)
> at
org.apache.pulsar.shade.com.google.common.cache.LocalCache.get(LocalCache.java:3951)
> at
org.apache.pulsar.shade.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3973)
> at
org.apache.pulsar.shade.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4957)
> at
org.apache.pulsar.client.impl.schema.StructSchema.decode(StructSchema.java:107)
> at
org.apache.pulsar.client.impl.MessageImpl.getValue(MessageImpl.java:293)
> at io.pethau.pl.PipelineService.lambda$run$2(PipelineService.java:103)
> at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
> at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.base/java.lang.Thread.run(Unknown Source)
> Caused by:
<http://org.apache.pulsar.shade.org|org.apache.pulsar.shade.org>.apache.avro.SchemaParseException:
Cannot parse <null> schema
> at
<http://org.apache.pulsar.shade.org|org.apache.pulsar.shade.org>.apache.avro.Schema.parse(Schema.java:1597)
> at
<http://org.apache.pulsar.shade.org|org.apache.pulsar.shade.org>.apache.avro.Schema$Parser.parse(Schema.java:1396)
> at
<http://org.apache.pulsar.shade.org|org.apache.pulsar.shade.org>.apache.avro.Schema$Parser.parse(Schema.java:1384)
> at
org.apache.pulsar.client.impl.schema.StructSchema.parseAvroSchema(StructSchema.java:182)
> at
org.apache.pulsar.client.impl.schema.AvroSchema.loadReader(AvroSchema.java:99)
> at
org.apache.pulsar.client.impl.schema.StructSchema$1.load(StructSchema.java:75)
> at
org.apache.pulsar.client.impl.schema.StructSchema$1.load(StructSchema.java:72)
> at
org.apache.pulsar.shade.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
> at
org.apache.pulsar.shade.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2276)
> at
org.apache.pulsar.shade.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2154)
> at
org.apache.pulsar.shade.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2044)
> ... 11 common frames omitted
----
2020-10-27 23:27:49 UTC - Peter Tinti: @Peter Tinti has joined the channel
----
2020-10-27 23:28:05 UTC - Rob Shepherd: Restarting didn't help. one still
failing
----
2020-10-28 00:38:50 UTC - Rob Fowler: I just looked at the inability of python
to receive an interrupt when waiting for a message. This change means it can
not be interrupted anymore: <https://github.com/apache/pulsar/pull/5706>
----
2020-10-28 00:39:10 UTC - Rob Fowler: is anyone else actually using python
still?
----
2020-10-28 01:44:40 UTC - jiangmincong: @jiangmincong has joined the channel
----
2020-10-28 03:59:18 UTC - David Tinker: I didn’t see any errors, and everything
seemed to be working fine.
+1 : Sijie Guo
----
2020-10-28 04:19:40 UTC - Addison Higham: looks like you didn't get an answer
to this. The process is asynchronous, after the ledger is completed (which
happens every few hours by default) then it will be deleted, technically the
messages are still on disk until the bookkeeper cleanup physically removes it
----
2020-10-28 08:06:49 UTC - Sankararao Routhu: Hi, is there a way to set message
expiry at message level or topic level
----
