Hi all,
I have IoT time series data in Kafka and reading it in static dataframe as:
df = spark.read\
.option("zookeeper.connect", "localhost:2181")\
.option("kafka.bootstrap.servers", "localhost:9092")\
.option("subscribe", "test_topic")\
.option("failOnDataLoss", "false")\
.option("startingOffsets", "earliest")\
.select(from_json(col("value").cast("string"), schema).alias("stream"))\

I want to know how data is distributed over multiple executors. I want the
data to be distributed on the basis of dev_id, each executor gets all data
from one dev_id. Later I group by dev_id and run @pandas_udf on each group.
Please note that it is a static dataframe not streaming dataframe as
@pandas_udf don't support streaming dataframe.
Arbab Khalil
Software Design Engineer

Reply via email to