Statefun with no Protobuf ingress and egress

2022-02-14 Thread mrAlexTFB
Hi,

I have a very simple schema where one python statefun application reads
from a kafka topic and writes in another kafka topic, those topics are
produced and consumed with another python script as it is done in the Python
Flink Walkthrough
,
Is there a way to read and write in those topics a plain string (as a JSON)
and not to use Protobuf?

More concrete:
I'm trying to use statefun as a solution for not finding some libraries in
JAVA or SCALA that exists in python, then, I'm trying to combine embedded
Flink applications with statefun applications using with docker a master
and worker with the embedded applications with JobManager and TaskManager,
All the embedded applications communicate using JSON, now that I want to
use an statefun Application in between, is there a way to communicate using
JSON and not protobuf?

Thanks in advance.


Problem with kafka with key=None using pyhton-kafka module

2022-02-10 Thread mrAlexTFB
Hello,

I am following the example in Python Walkthrough
,
I downloaded the zip file with the project skeleton. I'm having a problem
when changing the key attribute in the function producer.send to none.
From:

def produce():
if len(sys.argv) == 2:
delay_seconds = int(sys.argv[1])
else:
delay_seconds = 1
producer = KafkaProducer(bootstrap_servers=[KAFKA_BROKER])
for request in random_requests():
key = request.name.encode('utf-8')
val = request.SerializeToString()
producer.send(topic='names', key=key, value=val)
producer.flush()
time.sleep(delay_seconds)

To:

def produce():
if len(sys.argv) == 2:
delay_seconds = int(sys.argv[1])
else:
delay_seconds = 1
producer = KafkaProducer(bootstrap_servers=[KAFKA_BROKER])
for request in random_requests():
key = request.name.encode('utf-8')
val = request.SerializeToString()
producer.send(topic='names', key=None, value=val)
producer.flush()
time.sleep(delay_seconds)

After doing this the consumer is not displaying anything.

I modified python code so the message arrived is printed and it is not
being printed here, I suppose that the problem could be a bad configuration
in module.yaml file?

I understand that by putting key=None the topic partition will be chosen
randomly, that was the behaviour that I was aiming for as I do not need any
 ordering in the messages.

Do I need any additional configuration in this walkthrough to achieve this?

Thank you very much in advance.