Hi,
Thanks for your inputs. It kind of makes sense to use a container
orchestrator to plough through networking under the hood.
How do you tackle security?
I don't see a way to authorize users for job management. I understand
few orchestrators provide name space isolation and security policies on
these. How do this work if the flink cluster is standalone on AWS ?
Best Regards
CVP
On Fri, Mar 24, 2017 at 8:49 AM, Philippe Caparroy <
philippe.capar...@orange.fr> wrote:
> Hi,
>
> If I can give my 2 cents.
>
> One simple solution to your problem is using weave (
> https://www.weave.works/) a Docker network plugin.
>
> We’ve been working for more then year with dockerized
> (Flink+zookeeper+Yarn+spark+Kafka+hadoop+elasticsearch ) cluster using
> weave.
>
> Design your docker container so that you can set the cluster size on
> startup (number of task manager stand job managers should be a docker arg).
>
> Weave will act as a switch with dns server embedded. Your containers will
> only have to be configured with hosts names such as :
> flink.taskmanager-1.weave.local, link.taskmanager-2.weave.local,
> flink.jobmanager-1.weave.local, and so on …
>
> with flink Yarn it’s even simpler, but you have to dockerize a Yarn
> cluster.
>
> It works perfectly on bare metal machines and in the cloud (digital-ocean,
> aws,…).
>
>
>
> Le 24 mars 2017 à 08:50, Chakravarthy varaga <chakravarth...@gmail.com> a
> écrit :
>
> Hi,
>
> I request someone to help here.
>
> Best Regards
> CVP
>
> On Thu, Mar 23, 2017 at 10:13 PM, Chakravarthy varaga <
> chakravarth...@gmail.com> wrote:
>
>> I'm looking forward to hearing some updates on this...
>>
>> Any help here is highly appreciated !!
>>
>> On Thu, Mar 23, 2017 at 4:20 PM, Chakravarthy varaga <
>> chakravarth...@gmail.com> wrote:
>>
>>> Hi Team,
>>>
>>> We are doing a PoC to deploy Flink cluster on AWS. All runtime
>>> components will be dockerized.
>>>
>>> I have few questions in relation to discover & security:
>>>
>>> 1. How does Job Manager discover task managers? Do they talk to
>>> over TCP ?
>>>
>>> 2. If the runtime components TM, JM are containerized how are the
>>> IPs resolved dynamically? Basically do I have to configure the JM with the
>>> hostnames of the TMs. If so, if the TMs are on ephemeral IPs and on restart
>>> of TM how does the job manager know the TM's (IP/Host). Before I go into
>>> DNS and subnets, I'd like to understand how they disvoer & talk to each
>>> other !
>>>
>>> 3. I went through some Flink materials on the web on security
>>> precisely on kerebros. However how do I ensure that user level
>>> authentication is applied on job management. For ex., only certain users
>>> are allowed to start/stop jobs ? This question is in relation to if flink
>>> is deployed as standalone-cluster
>>>
>>> Thanks & Regards
>>> CVP
>>>
>>
>>
>
>
