2018-01-07 18:53:40 UTC - Daniel Ferreira Jorge: Hello, I am trying to deploy the kubernetes manifests following the exact instructions on the documentation and everything goes as expected, except that "pulsar-perf produce" produces 0 messages per second... and if I go to the broker logs I see thousands of messages like: Write did not succeed to 10.142.0.19:3181, bookieIndex 1, but we have already fixed it. ---- 2018-01-07 18:55:06 UTC - Matteo Merli: it looks like all writes are failing on the bookie ---- 2018-01-07 18:55:34 UTC - Matteo Merli: or that it’s not reachable by the broker (though it’s registered as available in ZK) ---- 2018-01-07 18:56:16 UTC - Daniel Ferreira Jorge: I can ping the any bookie from any broker ---- 2018-01-07 18:56:39 UTC - Matteo Merli: bookies logs are telling anything? ---- 2018-01-07 18:56:54 UTC - Daniel Ferreira Jorge: nope... just initialized ---- 2018-01-07 18:57:43 UTC - Matteo Merli: any other error/warn message in broker log? It should say something on the reason why the write failed in the first place ---- 2018-01-07 18:57:43 UTC - Daniel Ferreira Jorge: nothing is happening ---- 2018-01-07 18:59:33 UTC - Matteo Merli: Can it be a problem of the IP that it’s being advertised by bookies?> ---- 2018-01-07 18:59:45 UTC - Matteo Merli: Are you using StatefulSet or DaemonSet? ---- 2018-01-07 18:59:59 UTC - Daniel Ferreira Jorge: daemon ---- 2018-01-07 19:00:12 UTC - Daniel Ferreira Jorge: the exact deployment from the repository ---- 2018-01-07 19:00:13 UTC - Matteo Merli: are you setting the `advertisedAddress` in bookie config? ---- 2018-01-07 19:00:17 UTC - Daniel Ferreira Jorge: absolutely nothing changed ---- 2018-01-07 19:00:20 UTC - Matteo Merli: ok ---- 2018-01-07 19:01:41 UTC - Daniel Ferreira Jorge: the advertisedAddress is status.hostIP ---- 2018-01-07 19:02:27 UTC - Daniel Ferreira Jorge: and the ip 10.142.0.19:3181 is the ip of a bookie node ---- 2018-01-07 19:02:36 UTC - Daniel Ferreira Jorge: so the broker knows where it is ---- 2018-01-07 19:02:44 UTC - Matteo Merli: ok, telnetting there from broker works? ---- 2018-01-07 19:03:13 UTC - Daniel Ferreira Jorge: pinging "bookie" works ---- 2018-01-07 19:03:49 UTC - Matteo Merli: my concern is that the Pod is not being exposed on the host network ---- 2018-01-07 19:04:22 UTC - Daniel Ferreira Jorge: I tried pinging everything from everywhere ---- 2018-01-07 19:04:24 UTC - Matteo Merli: broker might be able to “ping” but the Pod still needs to be bound on 3181 in the bookie host machine ---- 2018-01-07 19:04:38 UTC - Matteo Merli: try telnet instead of ping ---- 2018-01-07 19:04:45 UTC - Daniel Ferreira Jorge: ok ---- 2018-01-07 19:04:49 UTC - Daniel Ferreira Jorge: let me redeploy ---- 2018-01-07 19:04:56 UTC - Daniel Ferreira Jorge: I will report in 5 min ---- 2018-01-07 19:05:05 UTC - Matteo Merli: :+1: ---- 2018-01-07 19:06:03 UTC - Matteo Merli: uhm, just looking the bookie.yaml file ---- 2018-01-07 19:06:48 UTC - Matteo Merli: I think the problem is indeed the IP exposed ---- 2018-01-07 19:06:54 UTC - Matteo Merli: In this change <https://github.com/apache/incubator-pulsar/pull/764> ---- 2018-01-07 19:07:26 UTC - Matteo Merli: I had put the hostIP.. but this was missing the change to bind the bookie on `hostNetwork` ---- 2018-01-07 19:07:28 UTC - Matteo Merli: :confused: ---- 2018-01-07 19:10:34 UTC - Daniel Ferreira Jorge: so, I have to remove the advertised address? ---- 2018-01-07 19:10:55 UTC - Matteo Merli: try this: ---- 2018-01-07 19:10:55 UTC - Matteo Merli: <https://gist.github.com/merlimat/dad357c1cccde8e0b634a9639e1fcb16> ---- 2018-01-07 19:11:36 UTC - Matteo Merli: enabling `hostNetwork` tells Kubernetes to expose 3181 in the host network (rather than just bind it on the Pod IP) ---- 2018-01-07 19:12:32 UTC - Daniel Ferreira Jorge: great! trying now... will report back soon! thank you @Matteo Merli ----
