Hi Priya and Vincent
Thank you for your reply!
It looks the new feature is implemented only in the latest version.
But I'm using Spark 2.3.0 so, in my understanding, I need to stop and
reload apps.
Thanks
On 2018/12/19 9:09, vincent gromakowski wrote:
I totally missed this new feature.
I totally missed this new feature. Thanks for the pointer
Le mar. 18 déc. 2018 à 21:18, Priya Matpadi a écrit :
> Changes in streaming query that allow or disallow recovery from checkpoint
> is clearly provided in
>
Changes in streaming query that allow or disallow recovery from checkpoint
is clearly provided in
https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#recovery-semantics-after-changes-in-a-streaming-query
.
On Tue, Dec 18, 2018 at 9:45 AM vincent gromakowski <
Checkpointing is only used for failure recovery not for app upgrades. You
need to manually code the unload/load and save it to a persistent store
Le mar. 18 déc. 2018 à 17:29, Priya Matpadi a écrit :
> Using checkpointing for graceful updates is my understanding as well,
> based on the writeup
Using checkpointing for graceful updates is my understanding as well, based
on the writeup in
https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#recovering-from-failures-with-checkpointing,
and some prototyping. Have you faced any missed events?
On Mon, Dec 17, 2018
Hi
Now I'm trying to update my structured streaming application.
But I have no idea how to update it gracefully.
Should I stop it, replace a jar file then restart it?
In my understanding, in that case, all the state will be recovered if I
use checkpoints.
Is this correct?
Thank you,
--
