Re: Flink support for multiple data centers

2018-10-04 Thread Andrey Zagrebin
Hi Olga,

At the moment Flink does not have any embedded support for multi region 
deployment or failover.
You can try to automate it, for example, using savepoints [1] and backing them 
up in multiple data centres.
This way you could restore failed job in other regions from the latest 
savepoint.
Of course, job input/output should be also available in multiple data centres.

Flink does not have any embedded encryption mechanism for state.
The state value is just de-serialisable byte array for Flink.
For example, you can implement custom TypeSerializer [2] or wrap existing one 
with encryption.

[1] 
https://ci.apache.org/projects/flink/flink-docs-master/ops/state/savepoints.html
[2] 
https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/state/custom_serialization.html

Best,
Andrey

> On 2 Oct 2018, at 02:57, Olga Luganska  wrote:
> 
> Hello,
> 
> Does Flink support multiple data center implementation and failover 
> procedures in case one of the data centers goes down?
> 
> Another question I have is about data encryption. If  application state which 
> needs to be checkpointed contains data elements which are considered to be a 
> personally identifiable information, or maybe some credit card information, 
> do you provide any encryption mechanisms to make sure that this data will be 
> secured?
> 
> Thank you very much,
> Olga 



Flink support for multiple data centers

2018-10-01 Thread Olga Luganska
Hello,

Does Flink support multiple data center implementation and failover procedures 
in case one of the data centers goes down?

Another question I have is about data encryption. If  application state which 
needs to be checkpointed contains data elements which are considered to be a 
personally identifiable information, or maybe some credit card information, do 
you provide any encryption mechanisms to make sure that this data will be 
secured?

Thank you very much,
Olga