Hi Alexey,
I'm using StatefulSet for JM exactly as you describe (Deployment for TM
is just fine). The main advantage is that you don't need distributed
storage for JM fault tolerance, because you can use persistent volume
mount (provided your cloud provider provides it as fault tolerant
volume). You still need zookeeper (AFAIK), but that might be a feature
request, as it is not actually necessary in this case. If this could be
incorporated in the mentioned Flink Operators, that would be great. :-)
Best,
Jan
On 8/14/20 5:09 AM, Yang Wang wrote:
Hi Alexey,
Actually, StatefulSets could also be used to start the JobManager and
TaskManager.
So why do we suggest to use Deployment in the Flink documentation?
* StatefulSets requires the user to have persistent volume in the K8s
cluster. However, it is not always true,
especially for the unmanaged(self-build) K8s cluster.
* Flink uses Zookeeper and distributed storage(S3, GFS, etc.) to
process the fault tolerance. If you start multiple
JobManagers, the leader election and leader retrieval will be done
via Zookeeper. Also the meta information will
be stored in the Zookeeper. So it is unnecessary to use StatefulSet
to do more things.
* The local data of JobManager and TaskManager is ephemeral. It could
be discarded after crashed.
Best,
Yang
Arvid Heise <ar...@ververica.com <mailto:ar...@ververica.com>>
于2020年8月13日周四 下午4:38写道：
Hi Alexey,
I don't see any issue in using stateful sets immediately.
I'd recommend using one of the K8s operators or Ververica's
community edition [1] though if you start with a new setup as they
may solve even more issues that you might experience in the future.
[1] https://www.ververica.com/getting-started
On Mon, Aug 10, 2020 at 11:22 PM Alexey Trenikhun <yen...@msn.com
<mailto:yen...@msn.com>> wrote:
Hello,
Flink documentation suggests to use Deployments to deploy JM
and TM for kubernetes job cluster. Is any known potential
issues with using StatefulSets instead, seems StatefullSet
provides uniqueness for JM during upgrade/rollback, while with
Deployments could be multiple JM pods (e.g.1 terminating and 1
running)
Thanks,
Alexey
--
Arvid Heise| Senior Java Developer
<https://www.ververica.com/>
Follow us @VervericaData
--
Join Flink Forward <https://flink-forward.org/>- The Apache
FlinkConference
Stream Processing | Event Driven | Real Time
--
Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
--
Ververica GmbHRegistered at Amtsgericht Charlottenburg: HRB 158244
BManaging Directors: Timothy Alexander Steinert, Yip Park Tung
Jason, Ji (Toni) Cheng
