RE: Savepoint API challenged with large savepoints

2022-03-10 Thread Schwalbe Matthias
From: Chesnay Schepler Sent: Donnerstag, 10. März 2022 10:47 To: Schwalbe Matthias ; user@flink.apache.org Subject: Re: Savepoint API challenged with large savepoints That all sounds very interesting; I'd go ahead with creating tickets. On 08/03/2022 13:43, Schwalbe Matthias wrote: Dear Flink Team,

Re: Savepoint API challenged with large savepoints

2022-03-10 Thread Chesnay Schepler
That all sounds very interesting; I'd go ahead with creating tickets. On 08/03/2022 13:43, Schwalbe Matthias wrote: Dear Flink Team, In the last weeks I was faced with a large savepoint (around 40GiB) that contained lots of obsolete data points and overwhelmed our infrastructure (i.e.

Savepoint API challenged with large savepoints

2022-03-08 Thread Schwalbe Matthias
Dear Flink Team, In the last weeks I was faced with a large savepoint (around 40GiB) that contained lots of obsolete data points and overwhelmed our infrastructure (i.e. failed to load/restart). We could not afford to lose the state, hence I spent the time to transcode the savepoint into