2019-11-08 09:30:49 UTC - Jasper Li: Hi everyone, I am playing with Pulsar 
Functions with Python now, but my functions worker cannot load my message when 
I produce it in the following way:
class Example(Record):
    a = String()
    b = Integer()
    c = Boolean()
producer = client.create_producer("public/default/dev", 
for i in range(100, 200):
    producer.send(Example(a=str(i), b=i, c=True))

and my functions like this:
from pulsar import Function, SerDe

class ReadMessage(Function):
    def __init__(self):

    def process(self, input, context):
        response_msg = input + "!"
        # response_msg = "".join([str(input.a), str(input.b), str(input.c)])
        return response_msg
for the first one, it raise ```Uncaught exception in Python instance: 'ascii' 
codec can't decode byte 0x84 in position 25: ordinal not in range(128)```
and for the second one, it raise
[ERROR]: Exception while executing user method
Traceback (most recent call last):
  File "/pulsar/instances/python-instance/python_instance.py", line 242, in 
    output_object = self.function_class.process(input_object, self.contextimpl)
  File "/pulsar/test.py", line 29, in process
    response_msg = ",".join([str(input.a), str(input.b), str(input.c)])
AttributeError: 'str' object has no attribute 'a'

How can I create a Pulsar Functions to process a topic with schema? Thanks 
2019-11-08 11:15:34 UTC - Alexandre DUVAL: Okay thanks! I didn't know about 
2019-11-08 11:22:42 UTC - Alexandre DUVAL: But it's not supported by browsers 
afaik :/.
2019-11-08 11:37:18 UTC - Raghavi: @Raghavi has joined the channel
2019-11-08 11:39:46 UTC - Raghavi: hi, is there a support for kafka like 
transforms in pulsar? I saw an issue raised in github which only mentions about 
builtin transforms. But for my use case, we want to create and use a custom 
transform. Can you please help?
2019-11-08 11:50:25 UTC - Pedro Cardoso: Sounds like a use-case for pulsar 
functions <https://pulsar.apache.org/docs/en/functions-overview/>
2019-11-08 11:51:22 UTC - Pedro Cardoso: Hello, can anyone tell me when Pulsar 
functions workers should run with brokers as apposed to standalone? What are 
the advantages/disadvantages?
2019-11-08 11:55:02 UTC - Rei Roldan: @Rei Roldan has joined the channel
2019-11-08 11:56:13 UTC - Alexandre DUVAL: Hi, using pulsar-manager, there is a 
way to initialize database? I tried to use pg as sprng.datasource, and first 
run got the following error ```Caused by: org.postgresql.util.PSQLException: 
ERROR: relation "environments" does not exist
  Position: 25```
2019-11-08 11:56:58 UTC - Sijie Guo: I think you need to create tables first. 
@tuteng can help you with this.
2019-11-08 11:58:47 UTC - Sijie Guo: if you have a relatively smaller cluster 
(e.g. at edge datacenter), you can enable function worker on brokres. so the 
functions can run brokers. you  don’t need additional machines. however you 
don’t have good isolation since functions are sharing the cpu resource with 

using function workers in separate nodes is good for isolation.
2019-11-08 11:59:27 UTC - Sijie Guo: @Raghavi do you mean transforms in 
2019-11-08 12:01:15 UTC - Sijie Guo: it seems that you are using a string serde 
to decode an avro message.
2019-11-08 12:01:36 UTC - Sijie Guo: you need to specify the serde when you 
submit a python function. so it knows how to deserialize the messages.
2019-11-08 12:05:46 UTC - tuteng: if you do not use docker deployment, you can 
try execute sql 
 in postgresql database @Alexandre DUVAL
2019-11-08 12:07:24 UTC - Alexandre DUVAL: Thanks! :slightly_smiling_face:
2019-11-08 12:14:48 UTC - Alexandre DUVAL: @tuteng An environnement is related 
to a pulsarmanager backend? a pulsar cluster? What do you define by 
environment? I see the database field name is "broker"
2019-11-08 12:15:04 UTC - Alexandre DUVAL: (I don't know what to put in service 
2019-11-08 12:15:14 UTC - Alexandre DUVAL: in environement creation
2019-11-08 12:16:09 UTC - Sijie Guo: an environment means a cluster, or a set 
of geo-replicated clusters
2019-11-08 12:16:35 UTC - Sijie Guo: you can put the http service url of your 
2019-11-08 12:19:04 UTC - Alexandre DUVAL: Ok, this is what i thought, but I 
get the following error: `{"error":"This environment is error. Please check 
it"}` without log error in frontend or backend, and the http put environement 
returns status code 200 with this error as body
2019-11-08 12:20:39 UTC - Sijie Guo: @tuteng any ideas?
2019-11-08 12:22:09 UTC - Raghavi: @Pedro Cardoso  looks like pulsar functions 
have input and output topics. In our case, we need to transform the message 
before it's being published to the topic
2019-11-08 12:22:52 UTC - tuteng: Please set a broker service url of 8080, for 
example <http://broker-ip:8080>
2019-11-08 12:22:58 UTC - Raghavi: @Sijie Guo  yes. Connector transforms
2019-11-08 12:23:42 UTC - tuteng: And can ensure that the network to this 
address is connected. if the network is not connected, this exception will be 
2019-11-08 12:25:19 UTC - Alexandre DUVAL: We are not using the 8080 port for 
your http service on our cluster
2019-11-08 12:25:56 UTC - tuteng: Not necessarily 8080 port
2019-11-08 12:26:37 UTC - tuteng: I gave an example above only, which can be 
2019-11-08 12:26:48 UTC - tuteng: <http://broker-ip:port>
2019-11-08 12:27:18 UTC - tuteng: You can try test curl -v 
2019-11-08 12:27:42 UTC - Alexandre DUVAL: I'm using pulsar proxies
2019-11-08 12:28:25 UTC - tuteng: That is <http://proxy-ip:port>
2019-11-08 12:28:59 UTC - Alexandre DUVAL: yes, and got 401
2019-11-08 12:29:04 UTC - Alexandre DUVAL: using curl
2019-11-08 12:29:33 UTC - Alexandre DUVAL: curl -v 
2019-11-08 12:29:56 UTC - tuteng: your environment enable auth?
2019-11-08 12:30:43 UTC - Alexandre DUVAL: I just boot pulsarmanager, connected 
using default creds and then it asking to configure this, but cant configure 
2019-11-08 12:31:03 UTC - Alexandre DUVAL: at this creation i got the rrror
2019-11-08 12:31:14 UTC - Alexandre DUVAL: it should uses authprovider: jwt 
2019-11-08 12:34:04 UTC - tuteng: You can try curl -v 
 got an error code of 401?
2019-11-08 12:34:33 UTC - tuteng: Is authentication enabled in your cluster 
2019-11-08 12:35:38 UTC - Alexandre DUVAL: yes and yes
2019-11-08 12:36:19 UTC - Alexandre DUVAL: I mean the issue is more "how to 
configure auth on pulsar manager when adding environment", no?
2019-11-08 12:36:34 UTC - tuteng: If your cluster environment has jwt 
authentication enabled, you need to look at this configuration. 
2019-11-08 12:37:26 UTC - Alexandre DUVAL: oh neat, thanks
2019-11-08 12:42:25 UTC - Alexandre DUVAL: it works well, thanks.
+1 : tuteng, Sijie Guo
tada : Sijie Guo
2019-11-08 12:53:54 UTC - Sijie Guo: @Raghavi currently it doesn’t support 
transform yet. we have an outstanding issue for adding that feature. @tuteng 
was trying to add one.

Can you comment in that github issue? So we can bump the priority to help you.
2019-11-08 12:54:43 UTC - Sijie Guo: @Igor Zubchenok I am not sure if it is the 
library issue. have to check first.
2019-11-08 13:21:58 UTC - Raghavi: Sure thank you @Sijie Guo. Will do
2019-11-08 13:48:43 UTC - Vasilis Vagias: @Vasilis Vagias has joined the channel
2019-11-08 13:55:36 UTC - Igor Zubchenok: I got a lot of warnings in my broker 
`WARN  org.apache.bookkeeper.client.BookieWatcherImpl - New ensemble: 
[X.X.X.X:3181, X.X.X.X:3181] is not adhering to Placement Policy. 
quarantinedBookies: []`
Any clue why this happens?
2019-11-08 13:57:07 UTC - Alexandre DUVAL: @tuteng where is stored the default 
creds pulsar/pulsar to access pulsarmanager?
2019-11-08 13:57:14 UTC - Alexandre DUVAL: I mean how to change them 
2019-11-08 13:58:37 UTC - Alexandre DUVAL: another question: 
`jwt.broker.public.key` is currently configured using file:///, can it be 
configured using `token:&lt;TOKEN&gt;` as pulsar classical conf
2019-11-08 14:01:49 UTC - Pedro Cardoso: Does anyone have any experience with 
running pulsar functions in a local minikube instance?
I'm trying to run the word count example as found in the pulsar 

This the what I'm trying to do in the terminal
$ bin/pulsar-admin functions create \               
  --name wordcount \
  --classname org.apache.pulsar.functions.api.examples.WordCountFunction \
  --inputs <persistent://public/default/in> \
  --output <persistent://public/default/out>
"Created successfully"

$ bin/pulsar-admin functions trigger \              
  --tenant public \                                                             
  --namespace default \
  --name wordcount \                                                      
  --trigger-value "hello world"
Request Timed Out

Reason: Request Timed Out


Can anyone see anything wrong? I'm using pulsar version 2.4.1, deployed on a 
minikube instance using the default values of the kubernetes helm-based 
deployment for local testing.
2019-11-08 14:03:30 UTC - tuteng: first question: Currently, pulsar-manager has 
an administrator's account and password in the configuration file 
 You can modify the configuration or specify it when starting the service 
through the command line add parameters --pulsar-manager.account=admin 
2019-11-08 14:04:15 UTC - Alexandre DUVAL: Ok sorry I didn't check the backend 
readme ~~.
2019-11-08 14:09:36 UTC - Pedro Cardoso: nvm, I got the LoggingFunction example 
to work 
2019-11-08 14:09:37 UTC - tuteng: second question: Two options, 
jwt.broker.public.key and jwt.broker.private.key, are used to manage token. If 
you only as an administrator to operate cluster, you can configure only one 
option backend.jwt.token. Reference 
2019-11-08 14:39:25 UTC - Alexandre DUVAL: ok
2019-11-08 15:43:59 UTC - Sijie Guo: it is  *WARN*. that’s a check in bk 4.9.2
2019-11-08 16:33:03 UTC - Nicolas Ha: The pulsar stats from the broker.. Do 
they rely on Prometheus? 
<https://pulsar.apache.org/docs/v1.20.0-incubating/admin/Stats/> in particular 
the backlog
2019-11-08 17:31:17 UTC - Pierre Zemb: Are you asking for the 1.20 version? In 
newest version yes, backlog is exposed through Prometheus but if I recall 
correctly, you have a set a Boolean in the configuration 
2019-11-08 17:31:57 UTC - Pierre Zemb: &gt; Subscription metrics are only 
exposed when exposeTopicLevelMetricsInPrometheus is set to true.
2019-11-08 18:40:31 UTC - Luke Lu: 
2019-11-08 18:43:18 UTC - Alexandre DUVAL: Yup, I'm working with waxzce :smile:.
2019-11-08 18:55:31 UTC - Isaiah Rairdon: @Isaiah Rairdon has joined the channel
2019-11-08 19:13:28 UTC - GC: Hi All, I am trying to execute pulsar-admin 
commands in broker. I am getting *Reason: This operation requires super-user 
access*.  The Issue related to any configs? can some one suggest any conf 
2019-11-08 19:41:27 UTC - Alexandre DUVAL: Hi, your broker.conf should define 
superUserRoles with the prinicpal of the user you are using to auth.
2019-11-08 22:25:46 UTC - Nicolas Ha: Right I didn’t know there were changes in 
that area between versions. Does that mean these 
<https://pulsar.apache.org/docs/en/administration-stats/> do not need 
prometheus then, but are to be deprecated?
2019-11-08 22:29:46 UTC - Nicolas Ha: I guess I don’t get the differences 
between two links really, and if there are different use cases for using

vs <https://pulsar.apache.org/docs/en/administration-stats/>
2019-11-09 03:38:29 UTC - cheranthian muthuvinayagam: @cheranthian 
muthuvinayagam has joined the channel
2019-11-09 06:10:02 UTC - Naveen Kumar: @Naveen Kumar has joined the channel
2019-11-09 07:23:56 UTC - Naveen Kumar: We need to do transformation while 
using the debezium connector. After CDC using debezium, before publishing to 
the topic, we need to replace existing column / add new column.

- suppose retail_outlet_id column with value 1 as always, we need to replace 
the retail_outlet_id with some other value based on the database (which will be 
configured during pulsar sources creation)

We could be able to help with PR / contributions (discussion / guidance needed)

Reply via email to