Consider there is a spark query(A) which is dependent on Kafka topics t1 and t2.
After running this query in the streaming mode, a checkpoint(C1) directory for the query gets created with offsets and sources directories. Now I add a third topic(t3) on which the query is dependent. Now if I restart spark with the same checkpoint C1, Spark crashes as expected, as it could not find the entry for the third topic(t3). So just as part of a hack, I tried to add the topic t3 to the checkpoint manually to the sources and offset directories of the query in the checkpoint. But spark still crashed. Whats the correct way to solve this problem? How to handle such upgrade paths in structured streaming? -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
