2019-09-01 13:54:44 UTC - Dylan Meissner: @Dylan Meissner has joined the channel ---- 2019-09-01 18:50:51 UTC - Poule: is it possible to upload Avro Schema using `pulsar-admin schemas upload`? ---- 2019-09-01 18:51:38 UTC - Poule: it works with simple schema like `<https://github.com/apache/pulsar/blob/master/conf/schema_example.conf>` ---- 2019-09-01 18:52:29 UTC - Poule: but with avro it ends in ``` com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize instance of `java.lang.String` out of START_OBJECT token at [Source: (File); line: 3, column: 15] (through reference chain: org.apache.pulsar.common.protocol.schema.PostSchemaPayload["schema"]) ``` ---- 2019-09-01 18:53:35 UTC - Poule: ``` { "schema": { "name": "ASchema", "type": "record", "fields": [ { "name": "comment", "type": "string" } ] }, "type": "AVRO", "properties": {} } ``` ---- 2019-09-01 18:54:45 UTC - Poule: ^^ is the output of `pulsar-admin schemas get` minus the `name` property ---- 2019-09-01 20:14:35 UTC - Poule: ok found why ---- 2019-09-01 20:19:49 UTC - David Kjerrumgaard: What was the issue? ---- 2019-09-01 20:24:00 UTC - Poule: `schema` needs to be a string and not a dict ---- 2019-09-01 20:27:16 UTC - Poule: like `"schema":"{\"type\":\"record\",\"name\":\"User\",\"name.......` ---- 2019-09-01 20:28:46 UTC - Poule: it's in the `master` doc but not in the `2.4` doc +1 : David Kjerrumgaard ---- 2019-09-01 20:39:46 UTC - David Kjerrumgaard: can you file a documentation JIRA? ---- 2019-09-01 20:44:49 UTC - Poule: I meant it's already here : <http://pulsar.apache.org/docs/en/next/schema-manage/#upload-a-schema> ---- 2019-09-01 20:45:14 UTC - David Kjerrumgaard: ok, cool... ---- 2019-09-01 20:45:51 UTC - Poule: I presume it will make the 2.4.1 doc ---- 2019-09-01 20:45:56 UTC - Poule: soon ---- 2019-09-01 21:48:00 UTC - Nicolas Ha: I have this in my kubernetes cluster - the broker was fine before, but now are stuck in a `CrashLoopBackOff` - does anyone know where I should start looking to fix it? ---- 2019-09-01 21:49:48 UTC - Chris Bartholomew: @Nicolas Ha What's in the logs for the pod? ---- 2019-09-01 21:59:42 UTC - Nicolas Ha: @Chris Bartholomew I put them there - <https://gist.github.com/nha/ee5686d808f6c37cc3173c54a7bc0c63> ---- 2019-09-01 22:00:34 UTC - Nicolas Ha: it seems like this one if the first error: ``` 19:33:57.404 [pulsar-client-io-47-1] WARN org.apache.pulsar.client.impl.ClientCnx - [id: 0xc3c7d965, L:/10.244.4.248:54874 - R:10.244.4.78/10.244.4.78:6650] Received error from server: org.apache.bookkeeper.mledger.ManagedLedgerException: Error while recovering ledger 19:33:57.406 [pulsar-client-io-47-1] ERROR org.apache.pulsar.client.impl.ProducerImpl - [<persistent://public/functions/assignments>] [null] Failed to create producer: org.apache.bookkeeper.mledger.ManagedLedgerException: Error while recovering ledger 19:33:57.407 [main] ERROR org.apache.pulsar.functions.worker.SchedulerManager - Exception while at creating producer to topic <persistent://public/functions/assignments> java.util.concurrent.ExecutionException: org.apache.pulsar.client.api.PulsarClientException$BrokerPersistenceException: org.apache.bookkeeper.mledger.ManagedLedgerException: Error while recovering ledger at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) ~[?:1.8.0_212] ``` ---- 2019-09-01 22:01:23 UTC - Nicolas Ha: `Error while recovering ledger` the ledger is bookkeeper right? So maybe it cannot reach bookkeeper? If that is the case though I am not sure how to verify that this is the issue ---- 2019-09-01 22:03:36 UTC - David Kjerrumgaard: Correct the ledgers are physically stored in BK, but the metadata is kept in ZK so I would check both. ---- 2019-09-01 22:06:38 UTC - Chris Bartholomew: @Nicolas Ha in your logs for broker-7694b8c449-vm5bx it indicates that the broker can't connect to the bookie: ```21:52:32.522 [bookkeeper-io-14-2] ERROR org.apache.bookkeeper.proto.PerChannelBookieClient - Could not connect to bookie: [id: 0x8c56b57c, L:/10.244.5.212:37914]/10.131.249.138:3181, current state CONNECTING : io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: No route to host: /10.131.249.138:3181``` This looks like some sort of networking issue in your Kubernetes cluster. ---- 2019-09-01 22:13:09 UTC - Nicolas Ha: right then it can’t connect to BK and ZK, networking issue makes sense ---- 2019-09-01 22:14:44 UTC - Chris Bartholomew: What kind of Kubernetes cluster are you using? GKE? ---- 2019-09-01 22:50:31 UTC - Nicolas Ha: well I am using the digital ocean kubernetes ` 1.14.6-do.1` ---- 2019-09-01 22:51:16 UTC - Nicolas Ha: I had some trouble with it - seems very new and they still have some bugs it seems :confused: ---- 2019-09-01 23:22:01 UTC - Nicolas Ha: Just to make sure I am digging in the right direction - is that consistent with the `MinimumReplicasUnavailable` of the kubernetes dashboard (screenshot above)? ---- 2019-09-01 23:51:42 UTC - David Kjerrumgaard: @Nicolas Ha Yes, if you only have one pod available (pulsar broker), then your other two pods that are down are most likely bookkeeper and zookeeper. Therefore your cluster won't come up until those 2 pods are available. ---- 2019-09-01 23:54:10 UTC - Nicolas Ha: The 3 pods are all the brokers in the screenshot, one being marked as available (but the logs show the same errors really and it periodically errors and restarts too) ---- 2019-09-02 01:01:59 UTC - Nicolas Ha: hm I don’t understand - I recreated the cluster from the same yaml files and all are healthy now. Will keep an eye on it - thanks Chris and David! EDIT: still running fine after an hour or so.. now if someone has good links regarding network debugging on kubernetes I’m all ears :slightly_smiling_face: +1 : Chris Bartholomew ----
