2019-11-08 09:30:49 UTC - Jasper Li: Hi everyone, I am playing with Pulsar
Functions with Python now, but my functions worker cannot load my message when
I produce it in the following way:
```
class Example(Record):
a = String()
b = Integer()
c = Boolean()
producer = client.create_producer("public/default/dev",
schema=AvroSchema(Example))
for i in range(100, 200):
producer.send(Example(a=str(i), b=i, c=True))
```
and my functions like this:
```
from pulsar import Function, SerDe
class ReadMessage(Function):
def __init__(self):
pass
def process(self, input, context):
response_msg = input + "!"
# response_msg = "".join([str(input.a), str(input.b), str(input.c)])
return response_msg
```
for the first one, it raise ```Uncaught exception in Python instance: 'ascii'
codec can't decode byte 0x84 in position 25: ordinal not in range(128)```
and for the second one, it raise
```
[ERROR]: Exception while executing user method
Traceback (most recent call last):
File "/pulsar/instances/python-instance/python_instance.py", line 242, in
actual_execution
output_object = self.function_class.process(input_object, self.contextimpl)
File "/pulsar/test.py", line 29, in process
response_msg = ",".join([str(input.a), str(input.b), str(input.c)])
AttributeError: 'str' object has no attribute 'a'
```
How can I create a Pulsar Functions to process a topic with schema? Thanks
again!!!
----
2019-11-08 11:15:34 UTC - Alexandre DUVAL: Okay thanks! I didn't know about
that.
----
2019-11-08 11:22:42 UTC - Alexandre DUVAL: But it's not supported by browsers
afaik :/.
----
2019-11-08 11:37:18 UTC - Raghavi: @Raghavi has joined the channel
----
2019-11-08 11:39:46 UTC - Raghavi: hi, is there a support for kafka like
transforms in pulsar? I saw an issue raised in github which only mentions about
builtin transforms. But for my use case, we want to create and use a custom
transform. Can you please help?
----
2019-11-08 11:50:25 UTC - Pedro Cardoso: Sounds like a use-case for pulsar
functions <https://pulsar.apache.org/docs/en/functions-overview/>
----
2019-11-08 11:51:22 UTC - Pedro Cardoso: Hello, can anyone tell me when Pulsar
functions workers should run with brokers as apposed to standalone? What are
the advantages/disadvantages?
----
2019-11-08 11:55:02 UTC - Rei Roldan: @Rei Roldan has joined the channel
----
2019-11-08 11:56:13 UTC - Alexandre DUVAL: Hi, using pulsar-manager, there is a
way to initialize database? I tried to use pg as sprng.datasource, and first
run got the following error ```Caused by: org.postgresql.util.PSQLException:
ERROR: relation "environments" does not exist
Position: 25```
----
2019-11-08 11:56:58 UTC - Sijie Guo: I think you need to create tables first.
@tuteng can help you with this.
----
2019-11-08 11:58:47 UTC - Sijie Guo: if you have a relatively smaller cluster
(e.g. at edge datacenter), you can enable function worker on brokres. so the
functions can run brokers. you don’t need additional machines. however you
don’t have good isolation since functions are sharing the cpu resource with
brokers.
using function workers in separate nodes is good for isolation.
----
2019-11-08 11:59:27 UTC - Sijie Guo: @Raghavi do you mean transforms in
connectors?
----
2019-11-08 12:01:15 UTC - Sijie Guo: it seems that you are using a string serde
to decode an avro message.
----
2019-11-08 12:01:36 UTC - Sijie Guo: you need to specify the serde when you
submit a python function. so it knows how to deserialize the messages.
----
2019-11-08 12:05:46 UTC - tuteng: if you do not use docker deployment, you can
try execute sql
<https://github.com/apache/pulsar-manager/blob/master/src/main/resources/META-INF/sql/postgresql-schema.sql>
in postgresql database @Alexandre DUVAL
----
2019-11-08 12:07:24 UTC - Alexandre DUVAL: Thanks! :slightly_smiling_face:
----
2019-11-08 12:14:48 UTC - Alexandre DUVAL: @tuteng An environnement is related
to a pulsarmanager backend? a pulsar cluster? What do you define by
environment? I see the database field name is "broker"
----
2019-11-08 12:15:04 UTC - Alexandre DUVAL: (I don't know what to put in service
url)
----
2019-11-08 12:15:14 UTC - Alexandre DUVAL: in environement creation
----
2019-11-08 12:16:09 UTC - Sijie Guo: an environment means a cluster, or a set
of geo-replicated clusters
----
2019-11-08 12:16:35 UTC - Sijie Guo: you can put the http service url of your
cluster
----
2019-11-08 12:19:04 UTC - Alexandre DUVAL: Ok, this is what i thought, but I
get the following error: `{"error":"This environment is error. Please check
it"}` without log error in frontend or backend, and the http put environement
returns status code 200 with this error as body
----
2019-11-08 12:20:39 UTC - Sijie Guo: @tuteng any ideas?
----
2019-11-08 12:22:09 UTC - Raghavi: @Pedro Cardoso looks like pulsar functions
have input and output topics. In our case, we need to transform the message
before it's being published to the topic
----
2019-11-08 12:22:52 UTC - tuteng: Please set a broker service url of 8080, for
example <http://broker-ip:8080>
----
2019-11-08 12:22:58 UTC - Raghavi: @Sijie Guo yes. Connector transforms
----
2019-11-08 12:23:42 UTC - tuteng: And can ensure that the network to this
address is connected. if the network is not connected, this exception will be
thrown.
----
2019-11-08 12:25:19 UTC - Alexandre DUVAL: We are not using the 8080 port for
your http service on our cluster
----
2019-11-08 12:25:56 UTC - tuteng: Not necessarily 8080 port
----
2019-11-08 12:26:37 UTC - tuteng: I gave an example above only, which can be
<http://broker-ip>
----
2019-11-08 12:26:48 UTC - tuteng: <http://broker-ip:port>
----
2019-11-08 12:27:18 UTC - tuteng: You can try test curl -v
<http://broker-ip:port/metrics/>
----
2019-11-08 12:27:42 UTC - Alexandre DUVAL: I'm using pulsar proxies
----
2019-11-08 12:28:25 UTC - tuteng: That is <http://proxy-ip:port>
----
2019-11-08 12:28:59 UTC - Alexandre DUVAL: yes, and got 401
----
2019-11-08 12:29:04 UTC - Alexandre DUVAL: using curl
----
2019-11-08 12:29:33 UTC - Alexandre DUVAL: curl -v
<https://c1-pulsar-clevercloud-customers.services.clever-cloud.com:2000/metrics>
----
2019-11-08 12:29:56 UTC - tuteng: your environment enable auth?
----
2019-11-08 12:30:43 UTC - Alexandre DUVAL: I just boot pulsarmanager, connected
using default creds and then it asking to configure this, but cant configure
auth
----
2019-11-08 12:31:03 UTC - Alexandre DUVAL: at this creation i got the rrror
----
2019-11-08 12:31:14 UTC - Alexandre DUVAL: it should uses authprovider: jwt
token
----
2019-11-08 12:34:04 UTC - tuteng: You can try curl -v
<https://c1-pulsar-clevercloud-customers.services.clever-cloud.com:2000/metrics/>,
got an error code of 401?
----
2019-11-08 12:34:33 UTC - tuteng: Is authentication enabled in your cluster
environment?
----
2019-11-08 12:35:38 UTC - Alexandre DUVAL: yes and yes
----
2019-11-08 12:36:19 UTC - Alexandre DUVAL: I mean the issue is more "how to
configure auth on pulsar manager when adding environment", no?
----
2019-11-08 12:36:34 UTC - tuteng: If your cluster environment has jwt
authentication enabled, you need to look at this configuration.
<https://github.com/apache/pulsar-manager/blob/master/src/README.md#enable-jwt-auth>
----
2019-11-08 12:37:26 UTC - Alexandre DUVAL: oh neat, thanks
----
2019-11-08 12:42:25 UTC - Alexandre DUVAL: it works well, thanks.
+1 : tuteng, Sijie Guo
tada : Sijie Guo
----
2019-11-08 12:53:54 UTC - Sijie Guo: @Raghavi currently it doesn’t support
transform yet. we have an outstanding issue for adding that feature. @tuteng
was trying to add one.
Can you comment in that github issue? So we can bump the priority to help you.
----
2019-11-08 12:54:43 UTC - Sijie Guo: @Igor Zubchenok I am not sure if it is the
library issue. have to check first.
----
2019-11-08 13:21:58 UTC - Raghavi: Sure thank you @Sijie Guo. Will do
----
2019-11-08 13:48:43 UTC - Vasilis Vagias: @Vasilis Vagias has joined the channel
----
2019-11-08 13:55:36 UTC - Igor Zubchenok: I got a lot of warnings in my broker
logs:
`WARN org.apache.bookkeeper.client.BookieWatcherImpl - New ensemble:
[X.X.X.X:3181, X.X.X.X:3181] is not adhering to Placement Policy.
quarantinedBookies: []`
Any clue why this happens?
----
2019-11-08 13:57:07 UTC - Alexandre DUVAL: @tuteng where is stored the default
creds pulsar/pulsar to access pulsarmanager?
----
2019-11-08 13:57:14 UTC - Alexandre DUVAL: I mean how to change them
:slightly_smiling_face:
----
2019-11-08 13:58:37 UTC - Alexandre DUVAL: another question:
`jwt.broker.public.key` is currently configured using file:///, can it be
configured using `token:<TOKEN>` as pulsar classical conf
----
2019-11-08 14:01:49 UTC - Pedro Cardoso: Does anyone have any experience with
running pulsar functions in a local minikube instance?
I'm trying to run the word count example as found in the pulsar
[codebase](<https://github.com/apache/pulsar/blob/v2.4.1/pulsar-functions/java-examples/src/main/java/org/apache/pulsar/functions/api/examples/WordCountFunction.java>):
This the what I'm trying to do in the terminal
```
experiments/pulsar/apache-pulsar-2.4.1
$ bin/pulsar-admin functions create \
--jar
/home/pedro.cardoso/dev/pulsar/pulsar-functions/java-examples/target/pulsar-functions-api-examples.jar
\
--name wordcount \
--classname org.apache.pulsar.functions.api.examples.WordCountFunction \
--inputs <persistent://public/default/in> \
--output <persistent://public/default/out>
"Created successfully"
experiments/pulsar/apache-pulsar-2.4.1
$ bin/pulsar-admin functions trigger \
--tenant public \
--namespace default \
--name wordcount \
--trigger-value "hello world"
Request Timed Out
Reason: Request Timed Out
```
Can anyone see anything wrong? I'm using pulsar version 2.4.1, deployed on a
minikube instance using the default values of the kubernetes helm-based
deployment for local testing.
----
2019-11-08 14:03:30 UTC - tuteng: first question: Currently, pulsar-manager has
an administrator's account and password in the configuration file
<https://github.com/apache/pulsar-manager/blob/master/src/README.md#supported-configurations-of-backend>.
You can modify the configuration or specify it when starting the service
through the command line add parameters --pulsar-manager.account=admin
--pulsar-manager.password=password.
----
2019-11-08 14:04:15 UTC - Alexandre DUVAL: Ok sorry I didn't check the backend
readme ~~.
----
2019-11-08 14:09:36 UTC - Pedro Cardoso: nvm, I got the LoggingFunction example
to work
<https://github.com/apache/pulsar/blob/master/pulsar-functions/java-examples/src/main/java/org/apache/pulsar/functions/api/examples/LoggingFunction.java>
----
2019-11-08 14:09:37 UTC - tuteng: second question: Two options,
jwt.broker.public.key and jwt.broker.private.key, are used to manage token. If
you only as an administrator to operate cluster, you can configure only one
option backend.jwt.token. Reference
<https://github.com/apache/pulsar-manager/pull/205>
----
2019-11-08 14:39:25 UTC - Alexandre DUVAL: ok
----
2019-11-08 15:43:59 UTC - Sijie Guo: it is *WARN*. that’s a check in bk 4.9.2
----
2019-11-08 16:33:03 UTC - Nicolas Ha: The pulsar stats from the broker.. Do
they rely on Prometheus?
<https://pulsar.apache.org/docs/v1.20.0-incubating/admin/Stats/> in particular
the backlog
----
2019-11-08 17:31:17 UTC - Pierre Zemb: Are you asking for the 1.20 version? In
newest version yes, backlog is exposed through Prometheus but if I recall
correctly, you have a set a Boolean in the configuration
<https://pulsar.apache.org/docs/en/reference-metrics/>
----
2019-11-08 17:31:57 UTC - Pierre Zemb: > Subscription metrics are only
exposed when exposeTopicLevelMetricsInPrometheus is set to true.
----
2019-11-08 18:40:31 UTC - Luke Lu:
<https://github.com/apache/pulsar/issues/5598>
----
2019-11-08 18:43:18 UTC - Alexandre DUVAL: Yup, I'm working with waxzce :smile:.
----
2019-11-08 18:55:31 UTC - Isaiah Rairdon: @Isaiah Rairdon has joined the channel
----
2019-11-08 19:13:28 UTC - GC: Hi All, I am trying to execute pulsar-admin
commands in broker. I am getting *Reason: This operation requires super-user
access*. The Issue related to any configs? can some one suggest any conf
changes..
----
2019-11-08 19:41:27 UTC - Alexandre DUVAL: Hi, your broker.conf should define
superUserRoles with the prinicpal of the user you are using to auth.
----
2019-11-08 22:25:46 UTC - Nicolas Ha: Right I didn’t know there were changes in
that area between versions. Does that mean these
<https://pulsar.apache.org/docs/en/administration-stats/> do not need
prometheus then, but are to be deprecated?
----
2019-11-08 22:29:46 UTC - Nicolas Ha: I guess I don’t get the differences
between two links really, and if there are different use cases for using
<https://pulsar.apache.org/docs/en/reference-metrics/>
vs <https://pulsar.apache.org/docs/en/administration-stats/>
----
2019-11-09 03:38:29 UTC - cheranthian muthuvinayagam: @cheranthian
muthuvinayagam has joined the channel
----
2019-11-09 06:10:02 UTC - Naveen Kumar: @Naveen Kumar has joined the channel
----
2019-11-09 07:23:56 UTC - Naveen Kumar: We need to do transformation while
using the debezium connector. After CDC using debezium, before publishing to
the topic, we need to replace existing column / add new column.
- suppose retail_outlet_id column with value 1 as always, we need to replace
the retail_outlet_id with some other value based on the database (which will be
configured during pulsar sources creation)
We could be able to help with PR / contributions (discussion / guidance needed)
----
