2019-04-30 16:10:13 UTC - Sébastien de Melo: Hi guys! Is it supported to have
multiple functions workers separated from the brokers?
----
2019-04-30 16:13:53 UTC - Matteo Merli: Yes
----
2019-04-30 16:14:32 UTC - Matteo Merli: You just start that as a separated
component with `bin/pulsar functions-worker`
----
2019-04-30 16:23:08 UTC - Sébastien de Melo: Thank you! That's what I did but
the stats subcommand failed with the following error:
$ pulsar-admin --admin-url <http://lucky-star-pulsar-functions-worker:6750>
functions stats --name Router
javax.ws.rs.ProcessingException:
lucky-star-pulsar-functions-worker-5988889989-s7zzm: Name or service not known
Reason: HTTP 500 Internal Server Error
command terminated with exit code 1
----
2019-04-30 16:23:22 UTC - Sébastien de Melo: I guess I have to try with a
StatefulSet
----
2019-04-30 16:27:44 UTC - Matteo Merli: Yes, it will redirect you to the worker
that is running the particular function
----
2019-04-30 16:27:58 UTC - Matteo Merli: so that advertised name needs to be
routable
----
2019-04-30 16:29:09 UTC - Poule: do Function support Python3 or just 2
----
2019-04-30 16:33:31 UTC - Poule: what means "Function in trigger function is
not ready"
----
2019-04-30 16:58:01 UTC - Sanjeev Kulkarni: Functions support ython3
----
2019-04-30 16:58:16 UTC - Sanjeev Kulkarni: That means that function is not yet
started
----
2019-04-30 17:20:59 UTC - Poule: 10 minutes is long to start
----
2019-04-30 17:21:03 UTC - Poule: :0)
----
2019-04-30 17:25:35 UTC - Poule: Do I need to start pulsar in a Python3
environment to insert python3 functions?
----
2019-04-30 17:26:13 UTC - Poule: Or all is taken care of whenever which python
version I use
----
2019-04-30 17:27:30 UTC - Ali Ahmed: @Poule the functions just the the system
python
----
2019-04-30 17:27:56 UTC - Ali Ahmed: it should point to python3 if you need to
use that.
----
2019-04-30 17:28:46 UTC - Poule: it takes python in the path I guess?
----
2019-04-30 17:42:12 UTC - Sanjeev Kulkarni: That is correct
----
2019-04-30 17:42:40 UTC - Sanjeev Kulkarni: Wrt function not starting after a
long time, check for any errors in the broker or function worker side
----
2019-04-30 17:42:57 UTC - Sanjeev Kulkarni: Are you running in threaded or
process mode?
----
2019-04-30 17:43:42 UTC - Poule: I `tail -f` the logs but nothing interesting
----
2019-04-30 17:43:51 UTC - Poule: no idea what is threaded vs process mode
----
2019-04-30 17:44:07 UTC - Poule: i just started basic `pulsar standalone`
----
2019-04-30 17:45:14 UTC - Poule: in a `pipenv shell` env
----
2019-04-30 17:47:14 UTC - Poule: so far it works in python2
----
2019-04-30 17:49:26 UTC - Sanjeev Kulkarni: It’s python right so it has to be
process mode
----
2019-04-30 17:49:46 UTC - Sanjeev Kulkarni: Do you have function logs?
----
2019-04-30 17:55:51 UTC - Poule: it does not log it fails before with
```13:54:55.548 [function-timer-thread-82-1] ERROR
org.apache.pulsar.functions.runtime.ProcessRuntime - Extracted Process death
exception
java.lang.RuntimeException:
at
org.apache.pulsar.functions.runtime.ProcessRuntime.tryExtractingDeathException(ProcessRuntime.java:378)
~[org.apache.pulsar-pulsar-functions-runtime-2.3.1.jar:2.3.1]```
----
2019-04-30 17:57:42 UTC - Poule: my guess is wrong python version, need to
investigate
----
2019-04-30 17:59:55 UTC - Devin G. Bost: We've had some concerns about how well
Pulsar might perform (in terms of availability) in Kubernetes due to the fact
that Pulsar is a stateful service. Does anyone have some wisdom they could
please share about any issues with Pulsar in Kubernetes they've experienced and
what they've done about them (and how well those solutions worked)?
----
2019-04-30 18:01:35 UTC - Poule: do you like Kubernetes?
----
2019-04-30 18:02:47 UTC - Devin G. Bost: I personally don't have enough
experience with Kubernetes to give a strong opinion about it, but my colleagues
(and those on my team) like it.
----
2019-04-30 18:03:54 UTC - Matteo Merli: @Devin G. Bost We have been running
Pulsar in Kubernetes in production for a while with very good results.
Sure, it takes a while for one to get acquainted with Kubernetes and there sure
is a learning curve (as I guess the same could be said of Pulsar…)
Though, in my view, the current status is very solid and flexible.
----
2019-04-30 18:05:54 UTC - Poule: what's the advantages of running P on K?
----
2019-04-30 18:06:38 UTC - Matteo Merli: Simplified operations and integration
with modern ecosystem.
----
2019-04-30 18:07:02 UTC - Devin G. Bost: @Matteo Merli Thanks for the feedback.
Are there any specific issues that you needed to address to keep everything
stable (such as dealing with failure states, edge cases, etc.)?
----
2019-04-30 18:07:24 UTC - Matteo Merli: There are a loot of very good tools
that are coming up with out-of-the-box K8S integrations.
----
2019-04-30 18:09:21 UTC - Matteo Merli: I’d say the biggest “hurdle” was to
“learn” kubernetes :slightly_smiling_face: and understand all the options and
how to configure them. For example the semantics of stateful sets (what’s being
preserved and what not). Or the monitoring readinessProbes and livenessProbe..
----
2019-04-30 18:10:54 UTC - Poule: merlimat you installed K by hand or through
Rancher/others
----
2019-04-30 18:11:21 UTC - Matteo Merli: Mostly EKS or GKE
----
2019-04-30 18:11:59 UTC - Poule: ok, good idea
----
2019-04-30 18:12:14 UTC - Matteo Merli: I guess that helps keeping the scary
parts far away
----
2019-04-30 18:12:25 UTC - Poule: yup!
----
2019-04-30 18:26:13 UTC - Thor Sigurjonsson: Yep. I love K8S to be sure - I
guess on our end I'm thinking we're needing a multi-site instance of pulsar --
we foresee different workloads and different latency and uptime requirements.
also -- I'm betting on doing k8s for anything but the "core" multi-site and
highest uptime requirements of that solution..
----
2019-04-30 18:27:14 UTC - Thor Sigurjonsson: I think in our world that means we
might have a "bare-metal" core cluster at each site with the high uptime
requirements.
----
2019-04-30 18:27:47 UTC - Poule: @Thor Sigurjonsson are you on EKS or GKE
----
2019-04-30 18:27:49 UTC - Thor Sigurjonsson: We're also looking to do workloads
in functions that might not play nice with that core cluster but would live
nicely in a k8s cluster.
----
2019-04-30 18:28:01 UTC - Thor Sigurjonsson: We've got one GKE cluster at the
moment
----
2019-04-30 18:28:08 UTC - Thor Sigurjonsson: but we have on-prem stuff too.
----
2019-04-30 18:28:50 UTC - Thor Sigurjonsson: we also have machine learning
workloads that we're exploring integrating somehow into pulsar functions...
+1 : Dan C, Jon Bock
----
2019-04-30 18:30:13 UTC - Thor Sigurjonsson: Our bare metal solution right now
is just Docker with host networking and local volumes mounted and all that
simply orchestrated with salt. Docker is set up also for live-restore and nicer
prod settings like that.
----
2019-04-30 18:32:51 UTC - Poule: do you like salt better than ansible
----
2019-04-30 18:44:06 UTC - Thor Sigurjonsson: Personally -- I guess so yes. I
have more experience with it -- I guess it has more "features" which make it
more usable for more than just setup of infrastructure / classic config mgmt.
----
2019-04-30 18:44:42 UTC - Thor Sigurjonsson: More reactive stuff I guess that
way.
----
2019-04-30 19:11:50 UTC - Chris Bartholomew: I run Pulsar in GKE using
pre-emptible nodes, which means my underlying nodes get destroyed and recreated
every day. For a stateful service that sounds scary, but Pulsar on k8s handles
it just fine. K8s reschedules everything, and Pulsar's architecture means it
survives this daily chaos with minimal impact.
----
2019-05-01 05:35:54 UTC - Shivji Kumar Jha: I am trying to run a broker from my
IDE (PulsarBrokerStarter.java)
I have changed the broker.conf to use standalone.
```
# Flag to control features that are meant to be used when running in standalone
mode
isRunningStandalone=true
# Name of the cluster to which this broker belongs to
clusterName=shiv-local```
how do I tell standalone broker to run its own zookeeper?
I keep getting errors saying
```Required zookeeperServers is null```
----
2019-05-01 05:36:26 UTC - jose farfan: @jose farfan has joined the channel
----
2019-05-01 05:37:35 UTC - Matt Slack: @Matt Slack has joined the channel
----
2019-05-01 05:38:59 UTC - Shivji Kumar Jha: If i run an explicit zookeeper on
my laptop and point standalone.conf to use that
```# Zookeeper quorum connection string
zookeeperServers=localhost:32770```
I still get an error saying
```Required zookeeperServers is null```
----
2019-05-01 05:49:49 UTC - Shivji Kumar Jha: Is this documented somewhere?
----
2019-05-01 06:12:52 UTC - Shivji Kumar Jha: Got it, pulsarstandalonestarter it
is it seems..
----
2019-05-01 06:12:56 UTC - Sijie Guo: @Shivji Kumar Jha not quite understand
your requirements. doesn’t bin/standalone start a standalone cluster?
----
2019-05-01 06:12:58 UTC - Shivji Kumar Jha: looking there now!
----
2019-05-01 06:13:11 UTC - Sijie Guo: if you want to start a standalone cluster
in program, use `PulsarStandaloneStarter`
----
2019-05-01 06:13:30 UTC - Shivji Kumar Jha: I want to start pulsar from IDE, to
debug bugs like the one you helped me with yesterday
----
2019-05-01 06:13:42 UTC - Shivji Kumar Jha: RIght figured that out just now
----
2019-05-01 06:14:43 UTC - Sijie Guo: i see :+1:
----
2019-05-01 06:37:36 UTC - Shivji Kumar Jha: @Sijie Guo I see the only required
field is config
This is my args
``` "--config" "/Users/shivaji.jha/workspace/pulsar/conf/standalone.conf"
"--only-broker" "true"```
But jcommander keeps failing to parse (last line below)
```
public PulsarStandaloneStarter(String[] args) throws Exception {
JCommander jcommander = new JCommander();
try {
jcommander.addObject(this);
jcommander.parse(args);
```
----
2019-05-01 07:14:46 UTC - Shivji Kumar Jha: I changed args to
“--config” “/Users/shivaji.jha/workspace/pulsar/conf/standalone.conf”
And there are other crashes now…
----
2019-05-01 07:26:12 UTC - Shivji Kumar Jha: 12:52:30.137 [main] ERROR
org.apache.pulsar.functions.worker.WorkerService - Error Starting up in worker
java.lang.RuntimeException: No JavaInstanceJar specified
at
org.apache.pulsar.functions.runtime.ProcessRuntimeFactory.<init>(ProcessRuntimeFactory.java:73)
~[classes/:?]
at
org.apache.pulsar.functions.worker.FunctionRuntimeManager.<init>(FunctionRuntimeManager.java:145)
~[classes/:?]
at
org.apache.pulsar.functions.worker.WorkerService.start(WorkerService.java:165)
[classes/:?]
at
org.apache.pulsar.broker.PulsarService.startWorkerService(PulsarService.java:1044)
[classes/:?]
at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:441)
[classes/:?]
at org.apache.pulsar.PulsarStandalone.start(PulsarStandalone.java:305)
[classes/:?]
at
org.apache.pulsar.PulsarStandaloneStarter.main(PulsarStandaloneStarter.java:102)
[classes/:?]
----
2019-05-01 07:41:57 UTC - Shivji Kumar Jha: worked it seems finally
:slightly_smiling_face:
----
