2019-11-08 09:30:49 UTC - Jasper Li: Hi everyone, I am playing with Pulsar 
Functions with Python now, but my functions worker cannot load my message when 
I produce it in the following way:
```
class Example(Record):
    a = String()
    b = Integer()
    c = Boolean()
producer = client.create_producer("public/default/dev", 
schema=AvroSchema(Example))
for i in range(100, 200):
    producer.send(Example(a=str(i), b=i, c=True))
```

and my functions like this:
```
from pulsar import Function, SerDe

class ReadMessage(Function):
    def __init__(self):
        pass

    def process(self, input, context):
        response_msg = input + "!"
        # response_msg = "".join([str(input.a), str(input.b), str(input.c)])
        return response_msg
```
for the first one, it raise ```Uncaught exception in Python instance: 'ascii' 
codec can't decode byte 0x84 in position 25: ordinal not in range(128)```
and for the second one, it raise
```
[ERROR]: Exception while executing user method
Traceback (most recent call last):
  File "/pulsar/instances/python-instance/python_instance.py", line 242, in 
actual_execution
    output_object = self.function_class.process(input_object, self.contextimpl)
  File "/pulsar/test.py", line 29, in process
    response_msg = ",".join([str(input.a), str(input.b), str(input.c)])
AttributeError: 'str' object has no attribute 'a'
```

How can I create a Pulsar Functions to process a topic with schema? Thanks 
again!!!
----
2019-11-08 11:15:34 UTC - Alexandre DUVAL: Okay thanks! I didn't know about 
that.
----
2019-11-08 11:22:42 UTC - Alexandre DUVAL: But it's not supported by browsers 
afaik :/.
----
2019-11-08 11:37:18 UTC - Raghavi: @Raghavi has joined the channel
----
2019-11-08 11:39:46 UTC - Raghavi: hi, is there a support for kafka like 
transforms in pulsar? I saw an issue raised in github which only mentions about 
builtin transforms. But for my use case, we want to create and use a custom 
transform. Can you please help?
----
2019-11-08 11:50:25 UTC - Pedro Cardoso: Sounds like a use-case for pulsar 
functions <https://pulsar.apache.org/docs/en/functions-overview/>
----
2019-11-08 11:51:22 UTC - Pedro Cardoso: Hello, can anyone tell me when Pulsar 
functions workers should run with brokers as apposed to standalone? What are 
the advantages/disadvantages?
----
2019-11-08 11:55:02 UTC - Rei Roldan: @Rei Roldan has joined the channel
----
2019-11-08 11:56:13 UTC - Alexandre DUVAL: Hi, using pulsar-manager, there is a 
way to initialize database? I tried to use pg as sprng.datasource, and first 
run got the following error ```Caused by: org.postgresql.util.PSQLException: 
ERROR: relation "environments" does not exist
  Position: 25```
----
2019-11-08 11:56:58 UTC - Sijie Guo: I think you need to create tables first. 
@tuteng can help you with this.
----
2019-11-08 11:58:47 UTC - Sijie Guo: if you have a relatively smaller cluster 
(e.g. at edge datacenter), you can enable function worker on brokres. so the 
functions can run brokers. you  don’t need additional machines. however you 
don’t have good isolation since functions are sharing the cpu resource with 
brokers.

using function workers in separate nodes is good for isolation.
----
2019-11-08 11:59:27 UTC - Sijie Guo: @Raghavi do you mean transforms in 
connectors?
----
2019-11-08 12:01:15 UTC - Sijie Guo: it seems that you are using a string serde 
to decode an avro message.
----
2019-11-08 12:01:36 UTC - Sijie Guo: you need to specify the serde when you 
submit a python function. so it knows how to deserialize the messages.
----
2019-11-08 12:05:46 UTC - tuteng: if you do not use docker deployment, you can 
try execute sql 
<https://github.com/apache/pulsar-manager/blob/master/src/main/resources/META-INF/sql/postgresql-schema.sql>
 in postgresql database @Alexandre DUVAL
----
2019-11-08 12:07:24 UTC - Alexandre DUVAL: Thanks! :slightly_smiling_face:
----
2019-11-08 12:14:48 UTC - Alexandre DUVAL: @tuteng An environnement is related 
to a pulsarmanager backend? a pulsar cluster? What do you define by 
environment? I see the database field name is "broker"
----
2019-11-08 12:15:04 UTC - Alexandre DUVAL: (I don't know what to put in service 
url)
----
2019-11-08 12:15:14 UTC - Alexandre DUVAL: in environement creation
----
2019-11-08 12:16:09 UTC - Sijie Guo: an environment means a cluster, or a set 
of geo-replicated clusters
----
2019-11-08 12:16:35 UTC - Sijie Guo: you can put the http service url of your 
cluster
----
2019-11-08 12:19:04 UTC - Alexandre DUVAL: Ok, this is what i thought, but I 
get the following error: `{"error":"This environment is error. Please check 
it"}` without log error in frontend or backend, and the http put environement 
returns status code 200 with this error as body
----
2019-11-08 12:20:39 UTC - Sijie Guo: @tuteng any ideas?
----
2019-11-08 12:22:09 UTC - Raghavi: @Pedro Cardoso  looks like pulsar functions 
have input and output topics. In our case, we need to transform the message 
before it's being published to the topic
----
2019-11-08 12:22:52 UTC - tuteng: Please set a broker service url of 8080, for 
example <http://broker-ip:8080>
----
2019-11-08 12:22:58 UTC - Raghavi: @Sijie Guo  yes. Connector transforms
----
2019-11-08 12:23:42 UTC - tuteng: And can ensure that the network to this 
address is connected. if the network is not connected, this exception will be 
thrown.
----
2019-11-08 12:25:19 UTC - Alexandre DUVAL: We are not using the 8080 port for 
your http service on our cluster
----
2019-11-08 12:25:56 UTC - tuteng: Not necessarily 8080 port
----
2019-11-08 12:26:37 UTC - tuteng: I gave an example above only, which can be 
<http://broker-ip>
----
2019-11-08 12:26:48 UTC - tuteng: <http://broker-ip:port>
----
2019-11-08 12:27:18 UTC - tuteng: You can try test curl -v 
<http://broker-ip:port/metrics/>
----
2019-11-08 12:27:42 UTC - Alexandre DUVAL: I'm using pulsar proxies
----
2019-11-08 12:28:25 UTC - tuteng: That is <http://proxy-ip:port>
----
2019-11-08 12:28:59 UTC - Alexandre DUVAL: yes, and got 401
----
2019-11-08 12:29:04 UTC - Alexandre DUVAL: using curl
----
2019-11-08 12:29:33 UTC - Alexandre DUVAL: curl -v 
<https://c1-pulsar-clevercloud-customers.services.clever-cloud.com:2000/metrics>
----
2019-11-08 12:29:56 UTC - tuteng: your environment enable auth?
----
2019-11-08 12:30:43 UTC - Alexandre DUVAL: I just boot pulsarmanager, connected 
using default creds and then it asking to configure this, but cant configure 
auth
----
2019-11-08 12:31:03 UTC - Alexandre DUVAL: at this creation i got the rrror
----
2019-11-08 12:31:14 UTC - Alexandre DUVAL: it should uses authprovider: jwt 
token
----
2019-11-08 12:34:04 UTC - tuteng: You can try curl -v 
<https://c1-pulsar-clevercloud-customers.services.clever-cloud.com:2000/metrics/>,
 got an error code of 401?
----
2019-11-08 12:34:33 UTC - tuteng: Is authentication enabled in your cluster 
environment?
----
2019-11-08 12:35:38 UTC - Alexandre DUVAL: yes and yes
----
2019-11-08 12:36:19 UTC - Alexandre DUVAL: I mean the issue is more "how to 
configure auth on pulsar manager when adding environment", no?
----
2019-11-08 12:36:34 UTC - tuteng: If your cluster environment has jwt 
authentication enabled, you need to look at this configuration. 
<https://github.com/apache/pulsar-manager/blob/master/src/README.md#enable-jwt-auth>
----
2019-11-08 12:37:26 UTC - Alexandre DUVAL: oh neat, thanks
----
2019-11-08 12:42:25 UTC - Alexandre DUVAL: it works well, thanks.
+1 : tuteng, Sijie Guo
tada : Sijie Guo
----
2019-11-08 12:53:54 UTC - Sijie Guo: @Raghavi currently it doesn’t support 
transform yet. we have an outstanding issue for adding that feature. @tuteng 
was trying to add one.

Can you comment in that github issue? So we can bump the priority to help you.
----
2019-11-08 12:54:43 UTC - Sijie Guo: @Igor Zubchenok I am not sure if it is the 
library issue. have to check first.
----
2019-11-08 13:21:58 UTC - Raghavi: Sure thank you @Sijie Guo. Will do
----
2019-11-08 13:48:43 UTC - Vasilis Vagias: @Vasilis Vagias has joined the channel
----
2019-11-08 13:55:36 UTC - Igor Zubchenok: I got a lot of warnings in my broker 
logs:
`WARN  org.apache.bookkeeper.client.BookieWatcherImpl - New ensemble: 
[X.X.X.X:3181, X.X.X.X:3181] is not adhering to Placement Policy. 
quarantinedBookies: []`
Any clue why this happens?
----
2019-11-08 13:57:07 UTC - Alexandre DUVAL: @tuteng where is stored the default 
creds pulsar/pulsar to access pulsarmanager?
----
2019-11-08 13:57:14 UTC - Alexandre DUVAL: I mean how to change them 
:slightly_smiling_face:
----
2019-11-08 13:58:37 UTC - Alexandre DUVAL: another question: 
`jwt.broker.public.key` is currently configured using file:///, can it be 
configured using `token:&lt;TOKEN&gt;` as pulsar classical conf
----
2019-11-08 14:01:49 UTC - Pedro Cardoso: Does anyone have any experience with 
running pulsar functions in a local minikube instance?
I'm trying to run the word count example as found in the pulsar 
[codebase](<https://github.com/apache/pulsar/blob/v2.4.1/pulsar-functions/java-examples/src/main/java/org/apache/pulsar/functions/api/examples/WordCountFunction.java>):

This the what I'm trying to do in the terminal
```
experiments/pulsar/apache-pulsar-2.4.1                                          
                                                                                
                                                                                
                                                
$ bin/pulsar-admin functions create \               
  --jar 
/home/pedro.cardoso/dev/pulsar/pulsar-functions/java-examples/target/pulsar-functions-api-examples.jar
 \
  --name wordcount \
  --classname org.apache.pulsar.functions.api.examples.WordCountFunction \
  --inputs <persistent://public/default/in> \
  --output <persistent://public/default/out>
"Created successfully"

experiments/pulsar/apache-pulsar-2.4.1                                          
                                                                                
                                                                                
                                           
$ bin/pulsar-admin functions trigger \              
  --tenant public \                                                             
                                
  --namespace default \
  --name wordcount \                                                      
  --trigger-value "hello world"
Request Timed Out

Reason: Request Timed Out

```

Can anyone see anything wrong? I'm using pulsar version 2.4.1, deployed on a 
minikube instance using the default values of the kubernetes helm-based 
deployment for local testing.
----
2019-11-08 14:03:30 UTC - tuteng: first question: Currently, pulsar-manager has 
an administrator's account and password in the configuration file 
<https://github.com/apache/pulsar-manager/blob/master/src/README.md#supported-configurations-of-backend>.
 You can modify the configuration or specify it when starting the service 
through the command line add parameters --pulsar-manager.account=admin 
--pulsar-manager.password=password.
----
2019-11-08 14:04:15 UTC - Alexandre DUVAL: Ok sorry I didn't check the backend 
readme ~~.
----
2019-11-08 14:09:36 UTC - Pedro Cardoso: nvm, I got the LoggingFunction example 
to work 
<https://github.com/apache/pulsar/blob/master/pulsar-functions/java-examples/src/main/java/org/apache/pulsar/functions/api/examples/LoggingFunction.java>
----
2019-11-08 14:09:37 UTC - tuteng: second question: Two options, 
jwt.broker.public.key and jwt.broker.private.key, are used to manage token. If 
you only as an administrator to operate cluster, you can configure only one 
option backend.jwt.token. Reference 
<https://github.com/apache/pulsar-manager/pull/205>
----
2019-11-08 14:39:25 UTC - Alexandre DUVAL: ok
----
2019-11-08 15:43:59 UTC - Sijie Guo: it is  *WARN*. that’s a check in bk 4.9.2
----
2019-11-08 16:33:03 UTC - Nicolas Ha: The pulsar stats from the broker.. Do 
they rely on Prometheus? 
<https://pulsar.apache.org/docs/v1.20.0-incubating/admin/Stats/> in particular 
the backlog
----
2019-11-08 17:31:17 UTC - Pierre Zemb: Are you asking for the 1.20 version? In 
newest version yes, backlog is exposed through Prometheus but if I recall 
correctly, you have a set a Boolean in the configuration 
<https://pulsar.apache.org/docs/en/reference-metrics/>
----
2019-11-08 17:31:57 UTC - Pierre Zemb: &gt; Subscription metrics are only 
exposed when exposeTopicLevelMetricsInPrometheus is set to true.
----
2019-11-08 18:40:31 UTC - Luke Lu: 
<https://github.com/apache/pulsar/issues/5598>
----
2019-11-08 18:43:18 UTC - Alexandre DUVAL: Yup, I'm working with waxzce :smile:.
----
2019-11-08 18:55:31 UTC - Isaiah Rairdon: @Isaiah Rairdon has joined the channel
----
2019-11-08 19:13:28 UTC - GC: Hi All, I am trying to execute pulsar-admin 
commands in broker. I am getting *Reason: This operation requires super-user 
access*.  The Issue related to any configs? can some one suggest any conf 
changes..
----
2019-11-08 19:41:27 UTC - Alexandre DUVAL: Hi, your broker.conf should define 
superUserRoles with the prinicpal of the user you are using to auth.
----
2019-11-08 22:25:46 UTC - Nicolas Ha: Right I didn’t know there were changes in 
that area between versions. Does that mean these 
<https://pulsar.apache.org/docs/en/administration-stats/> do not need 
prometheus then, but are to be deprecated?
----
2019-11-08 22:29:46 UTC - Nicolas Ha: I guess I don’t get the differences 
between two links really, and if there are different use cases for using

<https://pulsar.apache.org/docs/en/reference-metrics/>
vs <https://pulsar.apache.org/docs/en/administration-stats/>
----
2019-11-09 03:38:29 UTC - cheranthian muthuvinayagam: @cheranthian 
muthuvinayagam has joined the channel
----
2019-11-09 06:10:02 UTC - Naveen Kumar: @Naveen Kumar has joined the channel
----
2019-11-09 07:23:56 UTC - Naveen Kumar: We need to do transformation while 
using the debezium connector. After CDC using debezium, before publishing to 
the topic, we need to replace existing column / add new column.

- suppose retail_outlet_id column with value 1 as always, we need to replace 
the retail_outlet_id with some other value based on the database (which will be 
configured during pulsar sources creation)

We could be able to help with PR / contributions (discussion / guidance needed)
----

Reply via email to