Thanks for the update Gabor. I'll take a look and respond in the document. Cheers, Till
On Wed, Jun 9, 2021 at 12:59 PM Gabor Somogyi <gabor.g.somo...@gmail.com> wrote: > Hi Till, > > Your proxy suggestion has been considered in-depth and updated the FLIP > accordingly. > We've considered 2 proxy implementation (Nginx and Squid) but according to > our analysis and testing it's not suitable for the mentioned use-cases. > Please take a look at the rejected alternatives for detailed explanation. > > Thanks for your time in advance! > > BR, > G > > > On Fri, Jun 4, 2021 at 3:31 PM Till Rohrmann <trohrm...@apache.org> wrote: > >> As I've said I am not a security expert and that's why I have to ask for >> clarification, Gabor. You are saying that if we configure a truststore for >> the REST endpoint with a single trusted certificate which has been >> generated by the operator of the Flink cluster, then the attacker can >> generate a new certificate, sign it and then talk to the Flink cluster if >> he has access to the node on which the REST endpoint runs? My understanding >> was that you need the corresponding private key which in my proposed setup >> would be under the control of the operator as well (e.g. stored in a >> keystore on the same machine but guarded by some secret). That way (if I am >> not mistaken), only the entity which has access to the keystore is able to >> talk to the Flink cluster. >> >> Maybe we are also getting our wires crossed here and are talking about >> different things. >> >> Thanks for listing the pros and cons of Kerberos. Concerning what other >> authentication mechanisms are used in the industry, I am not 100% sure. >> >> Cheers, >> Till >> >> On Fri, Jun 4, 2021 at 11:09 AM Gabor Somogyi <gabor.g.somo...@gmail.com> >> wrote: >> >>> > I did not mean for the user to sign its own certificates but for the >>> operator of the cluster. Once the user request hits the proxy, it should no >>> longer be under his control. I think I do not fully understand yet why this >>> would not work. >>> I said it's not solving the authentication problem over any proxy. Even >>> if the operator is signing the certificate one can have access to an >>> internal node. >>> Such case anybody can craft certificates which is accepted by the >>> server. When it's accepted a bad guy can cancel jobs causing huge impacts. >>> >>> > Also, I am missing a bit the comparison of Kerberos to other >>> authentication mechanisms and why they were rejected in favour of Kerberos. >>> PROS: >>> * Since it's not depending on cloud provider and/or k8s or bare-metal >>> etc. deployment it's the biggest plus >>> * Centralized with tools and no need to write tons of tools around >>> * There are clients/tools on almost all OS-es and several languages >>> * Super huge users are using it for years in production w/o huge issues >>> * Provides cross-realm trust possibility amongst other features >>> * Several open source components using it which could increase >>> compatibility >>> >>> CONS: >>> * Not everybody using kerberos >>> * It would increase the code footprint but this is true for many >>> features (as a side note I'm here to maintain it) >>> >>> Feel free to add your points because it only represents a single >>> viewpoint. >>> Also if you have any better option for strong authentication please >>> share it and we can consider the pros/cons here. >>> >>> BR, >>> G >>> >>> >>> On Fri, Jun 4, 2021 at 10:32 AM Till Rohrmann <trohrm...@apache.org> >>> wrote: >>> >>>> I did not mean for the user to sign its own certificates but for the >>>> operator of the cluster. Once the user request hits the proxy, it should no >>>> longer be under his control. I think I do not fully understand yet why this >>>> would not work. >>>> >>>> What I would like to avoid is to add more complexity into Flink if >>>> there is an easy solution which fulfills the requirements. That's why I >>>> would like to exercise thoroughly through the different alternatives. Also, >>>> I am missing a bit the comparison of Kerberos to other authentication >>>> mechanisms and why they were rejected in favour of Kerberos. >>>> >>>> Cheers, >>>> Till >>>> >>>> On Fri, Jun 4, 2021 at 10:26 AM Gyula Fóra <gyf...@apache.org> wrote: >>>> >>>>> Hi! >>>>> >>>>> I think there might be possible alternatives but it seems Kerberos on >>>>> the rest endpoint ticks all the right boxes and provides a super clean and >>>>> simple solution for strong authentication. >>>>> >>>>> I wouldn’t even consider sidecar proxies etc if we can solve it in >>>>> such a simple way as proposed by G. >>>>> >>>>> Cheers >>>>> Gyula >>>>> >>>>> On Fri, 4 Jun 2021 at 10:03, Till Rohrmann <trohrm...@apache.org> >>>>> wrote: >>>>> >>>>>> I am not saying that we shouldn't add a strong authentication >>>>>> mechanism if there are good reasons for it. I primarily would like to >>>>>> understand the context a bit better in order to give qualified feedback >>>>>> and >>>>>> come to a good decision. In order to do this, I have the feeling that we >>>>>> haven't fully considered all available options which are on the table, >>>>>> tbh. >>>>>> >>>>>> Does the problem of certificate expiry also apply for self-signed >>>>>> certificates? If yes, then this should then also be a problem for the >>>>>> internal encryption of Flink's communication. If not, then one could use >>>>>> self-signed certificates with a longer validity to solve the mentioned >>>>>> issue. >>>>>> >>>>>> I think you can set up Flink in such a way that you don't have to >>>>>> handle all the different certificates. For example, you could deploy >>>>>> Flink >>>>>> with a "sidecar proxy" which is responsible for the authentication using >>>>>> an >>>>>> arbitrary method (e.g. Kerberos) and then bind the REST endpoint to a >>>>>> local >>>>>> network interface. That way, the REST endpoint would only be available >>>>>> through the sidecar proxy. Additionally, one could enable SSL for this >>>>>> communication. Would this be a solution for the problem? >>>>>> >>>>>> Cheers, >>>>>> Till >>>>>> >>>>>> On Thu, Jun 3, 2021 at 10:46 PM Márton Balassi < >>>>>> balassi.mar...@gmail.com> wrote: >>>>>> >>>>>>> That is an interesting idea, Till. >>>>>>> >>>>>>> The main issue with it is that TLS certificates have an expiration >>>>>>> time, usually they get approved for a couple years. Forcing our users to >>>>>>> restart jobs to reprovision TLS certificates would be weird when we >>>>>>> could >>>>>>> just implement a single proper strong authentication mechanism instead >>>>>>> in a >>>>>>> couple hundred lines of code. :-) >>>>>>> >>>>>>> In many cases it is also impractical to go the TLS mutual route, >>>>>>> because the Flink Dashboard can end up on any node in the k8s/Yarn >>>>>>> cluster >>>>>>> which means that we need a certificate per node (due to the mutual >>>>>>> auth), >>>>>>> but if we also want to protect the private key of these from users >>>>>>> accidentally or intentionally leaking them then we need this per user. >>>>>>> As >>>>>>> in we end up managing user*machine number certificates and having to >>>>>>> renew >>>>>>> them periodically, which albeit automatable is unfortunately not yet >>>>>>> automated in all large organizations. >>>>>>> >>>>>>> I fully agree that TLS certificate mutual authentication has its >>>>>>> nice properties, especially at very large (multiple thousand node) >>>>>>> clusters >>>>>>> - but it has its own challenges too. Thanks for bringing it up. >>>>>>> >>>>>>> Happy to have this added to the rejected alternative list so that we >>>>>>> have the full picture documented. >>>>>>> >>>>>>> On Thu, Jun 3, 2021 at 5:52 PM Till Rohrmann <trohrm...@apache.org> >>>>>>> wrote: >>>>>>> >>>>>>>> I guess the idea would then be to let the proxy do the >>>>>>>> authentication job and only forward the request via an SSL mutually >>>>>>>> encrypted connection to the Flink cluster. Would this be possible? The >>>>>>>> beauty of this setup is in my opinion that this setup should work with >>>>>>>> all >>>>>>>> kinds of authentication mechanisms. >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Till >>>>>>>> >>>>>>>> On Thu, Jun 3, 2021 at 3:12 PM Gabor Somogyi < >>>>>>>> gabor.g.somo...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Thanks for giving options to fulfil the need. >>>>>>>>> >>>>>>>>> Users are looking for a solution where users can be identified on >>>>>>>>> the whole cluster and restrict access to resources/actions. >>>>>>>>> A good example for such an action is cancelling other users >>>>>>>>> running jobs. >>>>>>>>> >>>>>>>>> * SSL does provide mutual authentication but when authentication >>>>>>>>> passed there is no user based on restrictions can be made. >>>>>>>>> * The less problematic part is that generating/maintaining short >>>>>>>>> time valid certificates would be a hard (that's the reason KDC like >>>>>>>>> servers >>>>>>>>> exist). >>>>>>>>> Having long time valid certificates would widen the attack surface >>>>>>>>> but since the first concern is there this is just a cosmetic issue. >>>>>>>>> >>>>>>>>> All in all using TLS certificates is not sufficient in these >>>>>>>>> environments unfortunately. >>>>>>>>> >>>>>>>>> BR, >>>>>>>>> G >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Jun 3, 2021 at 12:49 PM Till Rohrmann < >>>>>>>>> trohrm...@apache.org> wrote: >>>>>>>>> >>>>>>>>>> Thanks for the information Gabor. If it is about securing the >>>>>>>>>> communication between the REST client and the REST server, then Flink >>>>>>>>>> already supports enabling mutual SSL authentication [1]. Would this >>>>>>>>>> be >>>>>>>>>> enough to secure the communication and to pass an audit? >>>>>>>>>> >>>>>>>>>> [1] >>>>>>>>>> https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/security/security-ssl/#external--rest-connectivity >>>>>>>>>> >>>>>>>>>> Cheers, >>>>>>>>>> Till >>>>>>>>>> >>>>>>>>>> On Thu, Jun 3, 2021 at 10:33 AM Gabor Somogyi < >>>>>>>>>> gabor.g.somo...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Till, >>>>>>>>>>> >>>>>>>>>>> Since I'm working in security area 10+ years let me share my >>>>>>>>>>> thought. >>>>>>>>>>> I would like to emphasise there are experts better than me but I >>>>>>>>>>> have some >>>>>>>>>>> basics. >>>>>>>>>>> The discussion is open and not trying to tell alone things... >>>>>>>>>>> >>>>>>>>>>> > I mean if an attacker can get access to one of the machines, >>>>>>>>>>> then it >>>>>>>>>>> should also be possible to obtain the right Kerberos token. >>>>>>>>>>> Not necessarily. For example if one gets access to a specific >>>>>>>>>>> user's >>>>>>>>>>> credentials then it's not possible to compromise other user's >>>>>>>>>>> jobs, data, >>>>>>>>>>> etc... >>>>>>>>>>> Security is like an onion, the more layers has been added the >>>>>>>>>>> more time an >>>>>>>>>>> attacker needs to proceed. >>>>>>>>>>> At the end of the day if one is in, then most probably can find >>>>>>>>>>> the way but >>>>>>>>>>> this time is normally enough to sysadmins or security experts to >>>>>>>>>>> close down the system and minimize the damage. >>>>>>>>>>> >>>>>>>>>>> The other thing is that all tokens has a timeout and if the >>>>>>>>>>> token is >>>>>>>>>>> invalid then the attacker can't proceed further. >>>>>>>>>>> >>>>>>>>>>> > Is Kerberos also the standard authentication protocol for >>>>>>>>>>> Kubernetes >>>>>>>>>>> deployments? >>>>>>>>>>> Kerberos is an industry standard which is cloud/deployment >>>>>>>>>>> agnostic and it >>>>>>>>>>> can be used in any deployments including k8s. >>>>>>>>>>> The main intention is to use kerberos in k8s deployments too >>>>>>>>>>> since we're >>>>>>>>>>> going this direction as well. >>>>>>>>>>> Please see how Spark does this: >>>>>>>>>>> >>>>>>>>>>> https://spark.apache.org/docs/latest/security.html#secure-interaction-with-kubernetes >>>>>>>>>>> >>>>>>>>>>> Last but not least the most important reason to add at least one >>>>>>>>>>> strong >>>>>>>>>>> authentication is that we have users who has >>>>>>>>>>> hard requirements on this. They're doing security audits and if >>>>>>>>>>> they fail >>>>>>>>>>> then it's deal breaking. >>>>>>>>>>> That is why we have added kerberos at the first place. >>>>>>>>>>> Unfortunately we >>>>>>>>>>> can't name them in this public list, however >>>>>>>>>>> the customers who specifically asked for this were mainly in the >>>>>>>>>>> banking >>>>>>>>>>> and telco sector. >>>>>>>>>>> >>>>>>>>>>> BR, >>>>>>>>>>> G >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Thu, Jun 3, 2021 at 9:20 AM Till Rohrmann < >>>>>>>>>>> trohrm...@apache.org> wrote: >>>>>>>>>>> >>>>>>>>>>> > Thanks for updating the document Márton. Why is it that banks >>>>>>>>>>> will >>>>>>>>>>> > consider it more secure if Flink comes with Kerberos >>>>>>>>>>> authentication >>>>>>>>>>> > (assuming a properly secured setup)? I mean if an attacker can >>>>>>>>>>> get access >>>>>>>>>>> > to one of the machines, then it should also be possible to >>>>>>>>>>> obtain the right >>>>>>>>>>> > Kerberos token. >>>>>>>>>>> > >>>>>>>>>>> > I am not an authentication expert and that's why I wanted to >>>>>>>>>>> ask what are >>>>>>>>>>> > other authentication protocols other than Kerberos? Why did we >>>>>>>>>>> select >>>>>>>>>>> > Kerberos and not any other authentication protocol? Maybe you >>>>>>>>>>> can list the >>>>>>>>>>> > pros and cons for the different protocols. Is Kerberos also >>>>>>>>>>> the standard >>>>>>>>>>> > authentication protocol for Kubernetes deployments? If not, >>>>>>>>>>> what would be >>>>>>>>>>> > the answer when deploying on K8s? >>>>>>>>>>> > >>>>>>>>>>> > Cheers, >>>>>>>>>>> > Till >>>>>>>>>>> > >>>>>>>>>>> > On Wed, Jun 2, 2021 at 12:07 PM Gabor Somogyi < >>>>>>>>>>> gabor.g.somo...@gmail.com> >>>>>>>>>>> > wrote: >>>>>>>>>>> > >>>>>>>>>>> >> Hi team, >>>>>>>>>>> >> >>>>>>>>>>> >> Happy to be here and hope I can provide quality additions in >>>>>>>>>>> the future. >>>>>>>>>>> >> >>>>>>>>>>> >> Thank you all for helpful the suggestions! >>>>>>>>>>> >> Considering them the FLIP has been modified and the work >>>>>>>>>>> continues on the >>>>>>>>>>> >> already existing Jira. >>>>>>>>>>> >> >>>>>>>>>>> >> BR, >>>>>>>>>>> >> G >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> >> On Wed, Jun 2, 2021 at 11:23 AM Márton Balassi < >>>>>>>>>>> balassi.mar...@gmail.com> >>>>>>>>>>> >> wrote: >>>>>>>>>>> >> >>>>>>>>>>> >>> Thanks, Chesney - I totally missed that. Answered on the >>>>>>>>>>> ticket too, let >>>>>>>>>>> >>> us continue there then. >>>>>>>>>>> >>> >>>>>>>>>>> >>> Till, I agree that we should keep this codepath as slim as >>>>>>>>>>> possible. It >>>>>>>>>>> >>> is an important design decision that we aim to keep the list >>>>>>>>>>> of >>>>>>>>>>> >>> authentication protocols to a minimum. We believe that this >>>>>>>>>>> should not be a >>>>>>>>>>> >>> primary concern of Flink and a trusted proxy service (for >>>>>>>>>>> example Apache >>>>>>>>>>> >>> Knox) should be used to enable a multitude of enduser >>>>>>>>>>> authentication >>>>>>>>>>> >>> mechanisms. The bare minimum of authentication mechanisms to >>>>>>>>>>> support >>>>>>>>>>> >>> consequently consist of a single strong authentication >>>>>>>>>>> protocol for which >>>>>>>>>>> >>> Kerberos is the enterprise solution and HTTP Basic primary >>>>>>>>>>> for development >>>>>>>>>>> >>> and light-weight scenarios. >>>>>>>>>>> >>> >>>>>>>>>>> >>> Added the above wording to G's doc. >>>>>>>>>>> >>> >>>>>>>>>>> >>> >>>>>>>>>>> https://docs.google.com/document/d/1NMPeJ9H0G49TGy3AzTVVJVKmYC0okwOtqLTSPnGqzHw/edit >>>>>>>>>>> >>> >>>>>>>>>>> >>> >>>>>>>>>>> >>> >>>>>>>>>>> >>> On Tue, Jun 1, 2021 at 11:47 AM Chesnay Schepler < >>>>>>>>>>> ches...@apache.org> >>>>>>>>>>> >>> wrote: >>>>>>>>>>> >>> >>>>>>>>>>> >>>> There's a related effort: >>>>>>>>>>> >>>> https://issues.apache.org/jira/browse/FLINK-21108 >>>>>>>>>>> >>>> >>>>>>>>>>> >>>> On 6/1/2021 10:14 AM, Till Rohrmann wrote: >>>>>>>>>>> >>>> > Hi Gabor, welcome to the Flink community! >>>>>>>>>>> >>>> > >>>>>>>>>>> >>>> > Thanks for sharing this proposal with the community >>>>>>>>>>> Márton. In >>>>>>>>>>> >>>> general, I >>>>>>>>>>> >>>> > agree that authentication is missing and that this is >>>>>>>>>>> required for >>>>>>>>>>> >>>> using >>>>>>>>>>> >>>> > Flink within an enterprise. The thing I am wondering is >>>>>>>>>>> whether this >>>>>>>>>>> >>>> > feature strictly needs to be implemented inside of Flink >>>>>>>>>>> or whether a >>>>>>>>>>> >>>> proxy >>>>>>>>>>> >>>> > setup could do the job? Have you considered this option? >>>>>>>>>>> If yes, then >>>>>>>>>>> >>>> it >>>>>>>>>>> >>>> > would be good to list it under the point of rejected >>>>>>>>>>> alternatives. >>>>>>>>>>> >>>> > >>>>>>>>>>> >>>> > I do see the benefit of implementing this feature inside >>>>>>>>>>> of Flink if >>>>>>>>>>> >>>> many >>>>>>>>>>> >>>> > users need it. If not, then it might be easier for the >>>>>>>>>>> project to not >>>>>>>>>>> >>>> > increase the surface area since it makes the overall >>>>>>>>>>> maintenance >>>>>>>>>>> >>>> harder. >>>>>>>>>>> >>>> > >>>>>>>>>>> >>>> > Cheers, >>>>>>>>>>> >>>> > Till >>>>>>>>>>> >>>> > >>>>>>>>>>> >>>> > On Mon, May 31, 2021 at 4:57 PM Márton Balassi < >>>>>>>>>>> mbala...@apache.org> >>>>>>>>>>> >>>> wrote: >>>>>>>>>>> >>>> > >>>>>>>>>>> >>>> >> Hi team, >>>>>>>>>>> >>>> >> >>>>>>>>>>> >>>> >> Firstly I would like to introduce Gabor or G [1] for >>>>>>>>>>> short to the >>>>>>>>>>> >>>> >> community, he is a Spark committer who has recently >>>>>>>>>>> transitioned to >>>>>>>>>>> >>>> the >>>>>>>>>>> >>>> >> Flink Engineering team at Cloudera and is looking >>>>>>>>>>> forward to >>>>>>>>>>> >>>> contributing >>>>>>>>>>> >>>> >> to Apache Flink. Previously G primarily focused on Spark >>>>>>>>>>> Streaming >>>>>>>>>>> >>>> and >>>>>>>>>>> >>>> >> security. >>>>>>>>>>> >>>> >> >>>>>>>>>>> >>>> >> Based on requests from our customers G has implemented >>>>>>>>>>> Kerberos and >>>>>>>>>>> >>>> HTTP >>>>>>>>>>> >>>> >> Basic Authentication for the Flink Dashboard and >>>>>>>>>>> HistoryServer. >>>>>>>>>>> >>>> Previously >>>>>>>>>>> >>>> >> lacked an authentication story. >>>>>>>>>>> >>>> >> >>>>>>>>>>> >>>> >> We are looking to contribute this functionality back to >>>>>>>>>>> the >>>>>>>>>>> >>>> community, we >>>>>>>>>>> >>>> >> believe that given Flink's maturity there should be a >>>>>>>>>>> common code >>>>>>>>>>> >>>> solution >>>>>>>>>>> >>>> >> for this general pattern. >>>>>>>>>>> >>>> >> >>>>>>>>>>> >>>> >> We are looking forward to your feedback on G's design. >>>>>>>>>>> [2] >>>>>>>>>>> >>>> >> >>>>>>>>>>> >>>> >> [1] http://gaborsomogyi.com/ >>>>>>>>>>> >>>> >> [2] >>>>>>>>>>> >>>> >> >>>>>>>>>>> >>>> >> >>>>>>>>>>> >>>> >>>>>>>>>>> https://docs.google.com/document/d/1NMPeJ9H0G49TGy3AzTVVJVKmYC0okwOtqLTSPnGqzHw/edit >>>>>>>>>>> >>>> >> >>>>>>>>>>> >>>> >>>>>>>>>>> >>>> >>>>>>>>>>> >>>>>>>>>>
