Thanks, Mich for acknowledging.
Yes, I am providing the checkpoint path. I omitted it here in the code
snippet.
I believe this is due to spark version 3.1.x, this config is there only in
versions greater than 3.2.x
On Thu, Apr 27, 2023 at 9:26 PM Mich Talebzadeh
wrote:
> Is this all of your wr
Is this all of your writeStream?
df.writeStream()
.foreachBatch(new KafkaS3PipelineImplementation(applicationId, appConfig))
.start()
.awaitTermination();
What happened to the checkpoint location?
option('checkpointLocation', checkpoint_path).
example
checkpoint_path = "file:///ss
Hi Team,
I am using Spark Streaming to read from Kafka and write to S3.
Version: 3.1.2
Scala Version: 2.12
Spark Kafka connector: spark-sql-kafka-0-10_2.12
Dataset df =
spark
.readStream()
.format("kafka")
.options(appConfig.getKafka().getConf())
.load()
