2020-02-13 09:17:54 UTC - Abhilash Mandaliya: hello all
I am getting an exception for topic having the schema:
`Caused by: org.apache.commons.lang3.SerializationException: Failed at fetching
schema info for EMPTY`
`at
org.apache.pulsar.client.impl.schema.StructSchema.getSchemaInfoByVersion(StructSchema.java:182)
~[org.apache.pulsar-pulsar-client-original-2.5.0.jar:2.5.0]`
`at
org.apache.pulsar.client.impl.schema.generic.GenericJsonSchema.loadReader(GenericJsonSchema.java:52)
~[org.apache.pulsar-pulsar-client-original-2.5.0.jar:2.5.0]`
`at
org.apache.pulsar.client.impl.schema.StructSchema$1.load(StructSchema.java:72)
~[org.apache.pulsar-pulsar-client-original-2.5.0.jar:2.5.0]`
`at
org.apache.pulsar.client.impl.schema.StructSchema$1.load(StructSchema.java:69)
~[org.apache.pulsar-pulsar-client-original-2.5.0.jar:2.5.0]`
`at
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
~[java-instance.jar:?]`
`at
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2276)
~[java-instance.jar:?]`
`at
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2154)
~[java-instance.jar:?]`
`at
com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2044)
~[java-instance.jar:?]`
`... 11 more`
But the schema with version 0 does exist. I am getting this response while
checking via pulsar-admin:
`./pulsar-admin schemas get test-topic`
response:
`{`
`"version": 0,`
`"schemaInfo": {`
`"name": "test-topic",`
`"schema": {`
`"name": "MyClass",`
`"type": "record",`
`"namespace": "com.acme.avro",`
`"fields": [`
`{`
`"name": "name",`
`"type": "string"`
`}`
`]`
`},`
`"type": "JSON",`
`"properties": {}`
`}`
`}`
Can anyone help here?
----
2020-02-13 09:43:41 UTC - Sijie Guo: If the pulsar manager is spinning, it
usually means the pulsar manager can’t connect to your pulsar cluster. you need
to ensure the network connectivity between pulsar manager and your pulsar
cluster.
----
2020-02-13 09:46:34 UTC - Sijie Guo: a github issue with your steps led to this
error will be much better.
the error seems indicating it is attempting to read a schema of empty version.
but I have no idea how do you produce and consume messages.
----
2020-02-13 10:17:16 UTC - Eugen: For me it's only for the cluster view, not for
other views, which work fine, so there would seem to be a connection between
manager and standalone "cluster"...
----
2020-02-13 11:22:26 UTC - Miroslav Prymek: Hello, I'm experiencing a strange
problem when trying to run bookies separated from brokers (on different
machine) and also use them as a functions state storage.
options added to `bookkeeper.conf` :
```zkServers=zk-1:2181,zk-2:2181,zk-3:2181
extraServerComponents=org.apache.bookkeeper.stream.server.StreamStorageLifecycleComponent```
bookie started with:
```bin/pulsar bookie```
in the official docker image (`apachepulsar/pulsar:2.5.0`)
bookie log:
```bookie_1 | 11:17:01.151
[DLM-/stream/storage-OrderedScheduler-1-0-SendThread(localhost:2181)] INFO
org.apache.zookeeper.ClientCnxn - Opening socket connection to server
localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown
error)
bookie_1 | 11:17:01.151
[DLM-/stream/storage-OrderedScheduler-1-0-SendThread(localhost:2181)] INFO
org.apache.zookeeper.ClientCnxn - Socket error occurred:
localhost/127.0.0.1:2181: Connection refused```
It seems to me that `zkServers` option is not propagated into stream storage
service and the default (`localhost:2181`) is used. Normal Pulsar topics
storage (bookie at port 3181) works well.
Is there some option I missed? Should I force the `zkServers` option with some
environment variable? Any other possible workaround?
Thanks for any suggestion.
----
2020-02-13 11:46:45 UTC - Abhilash Mandaliya: @Sijie Guo
<https://github.com/apache/pulsar/issues/6321>
----
2020-02-13 13:42:49 UTC - Tilden: Hi All , we want to understand the
requirement/ suggestion from Apache Pulsar on the designing the storage . we
are planning to use CePH Storage from OpenStack, we want to know, what is the
recommended disk I/O parameter?
----
2020-02-13 13:58:29 UTC - Vladimir Shchur: Can't run 2.5.0 bookie in k8s with
default helm config, help is appreciated. `Error: Could not find or load main
class`
----
2020-02-13 14:02:46 UTC - Konstantinos Papalias: not sure if this helps:
<https://apache-pulsar.slack.com/archives/C5Z4T36F7/p1580739667152700>
----
2020-02-13 14:06:22 UTC - Roman Popenov: Also make sure that when you save the
config files, it’s UTF-8 encoding
----
2020-02-13 14:07:51 UTC - Roman Popenov: Otherwise, please try
``` BOOKIE_MEM: "\"-Xms128m -Xmx256m -XX:MaxDirectMemorySize=256m
-Dio.netty.leakDetectionLevel=disabled -Dio.netty.recycler.linkCapacity=1024
-XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions
-XX:+AggressiveOpts -XX:+DoEscapeAnalysis -XX:ParallelGCThreads=32
-XX:ConcGCThreads=32 -XX:G1NewSizePercent=50 -XX:+DisableExplicitGC
-XX:-ResizePLAB -XX:+ExitOnOutOfMemoryError -XX:+PerfDisableSharedMem
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime
-XX:+PrintHeapAtGC -verbosegc -XX:G1LogLevel=finest\""
BOOKIE_GC: "\"-XX:+UseG1GC -XX:MaxGCPauseMillis=10\""```
settings for bookies
----
2020-02-13 14:09:05 UTC - Roman Popenov: And I think you will need to add
`bin/apply-config-from-env.py conf/bkenv.sh` command when the container is
starting
----
2020-02-13 14:47:06 UTC - Pavel Tishkevich: @Pavel Tishkevich has joined the
channel
----
2020-02-13 14:59:31 UTC - Pavel Tishkevich: Hi All!
Are there any recommendations about number of brokers/ZK/BK instances in
cluster depending on load in terms of number of topics, in/out rate, etc?
For example: how many brokers do I need to deploy in cluster to tolerate single
broker failure w/o experiencing serious latency blip? (Considering that overall
number of topics in cluster is about 150k, publish/delivery rates is about 30k
on average)
----
2020-02-13 17:49:10 UTC - Vladimir Shchur: @Roman Popenov thank you! Adding
BOOKIE_MEM and BOOKIE_GC helped!
bananadance : Roman Popenov
----
2020-02-13 17:53:12 UTC - Sijie Guo: replied.
----
2020-02-13 18:40:42 UTC - Kenny: @Kenny has joined the channel
----
2020-02-13 22:03:50 UTC - Vimal Kumar: @Vimal Kumar has joined the channel
----
2020-02-13 23:49:31 UTC - Dzmitry Kazimirchyk: @Dzmitry Kazimirchyk has joined
the channel
----
2020-02-14 02:38:39 UTC - Eugen: I'm trying to make sense of seemingly
conflicting statements in the [load-balance
docs](<https://pulsar.apache.org/docs/en/administration-load-balance/>):
> You can "unload" a topic in Pulsar with admin operation. *Unloading means*
to close the topics, release ownership and *reassign the topics* to a new
broker, based on current load.
> [..]
> *Unloading a topic has no effect on the assignment*, but just closes and
reopens the particular topic:
So which one is right?
----
2020-02-14 02:47:58 UTC - Eugen: And one more question: How can unloaded topics
be reassigned to another bundle (without splitting bundles), if this holds:
> Topics are assigned to a particular bundle by taking the hash of the topic
name and checking in which bundle the hash falls into.
As the topic name does not change, the hash will not change, hence the bundle
should not change either. What am I missing?
----
2020-02-14 03:42:26 UTC - Sijie Guo: “unload” doesn’t change the assignment -
it basically close the topic. it can be “re-assigned” after unloaded if the
load manager redistributes the bundles for load balancing.
+1 : Eugen
----
2020-02-14 04:26:18 UTC - Aaron Zhuo: @Aaron Zhuo has joined the channel
----
2020-02-14 04:28:35 UTC - Abhilash Mandaliya: thanks. I will try to produce a
message via client code
----
2020-02-14 05:48:13 UTC - Kelvin Sajere: I experience the same thing @Eugen.
The cluster page keeps spining. But every thing else seem to work
----
2020-02-14 06:41:17 UTC - Lari Hotari: Has anyone noticed issues with
org.apache.pulsar.client.api.Reader#hasMessageAvailableAsync in Pulsar 2.5.0 ?
The behavior has changed in a specific usecase in compared to 2.4.2 . I'm using
a specified startMessageId with startMessageIdInclusive . For my usecase
hasMessageAvailableAsync always returns false in 2.5.0 . It works in 2.4.2 as
expected. I have a custom built Project Reactor adapter over the async API and
since it's not possible to cancel a CompletableFuture, I use a pattern where
before reading a message with readNextAsync, it is first checked with
hasMessageAvailableAsync that there's a message available. I don't have an
isolated test case for reproducing the failure yet.
----
