[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16116980#comment-16116980 ] ASF GitHub Bot commented on FLINK-3930: --- Github user EronWright commented on the issue: https://github.com/apache/flink/pull/2425 Note to future self: to generate a self-signed certificate, use `CertAndKeyGen` and see [OPENDJ-2247](https://bugster.forgerock.org/jira/browse/OPENDJ-2247). > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security cookie > - Implement data transfer authentication > - Secure the web dashboard -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930108#comment-15930108 ] ASF GitHub Bot commented on FLINK-3930: --- Github user vijikarthi closed the pull request at: https://github.com/apache/flink/pull/2425 > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security cookie > - Implement data transfer authentication > - Secure the web dashboard -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928528#comment-15928528 ] ASF GitHub Bot commented on FLINK-3930: --- Github user vijikarthi commented on the issue: https://github.com/apache/flink/pull/2425 @StephanEwen It's absolutely fine with me and I will cancel this PR. > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security cookie > - Implement data transfer authentication > - Secure the web dashboard -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[
https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927451#comment-15927451
]
ASF GitHub Bot commented on FLINK-3930:
---
Github user WangTaoTheTonic commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r106335560
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/CookieHandler.java
---
@@ -0,0 +1,130 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network.netty;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.channel.Channel;
+import io.netty.channel.ChannelHandlerContext;
+import io.netty.channel.ChannelInboundHandlerAdapter;
+import io.netty.handler.codec.MessageToMessageDecoder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+public class CookieHandler {
+
+ public static class ClientCookieHandler extends
ChannelInboundHandlerAdapter {
+
+ private final Logger LOG =
LoggerFactory.getLogger(ClientCookieHandler.class);
+
+ private final String secureCookie;
+
+ final Charset DEFAULT_CHARSET = Charset.forName("utf-8");
+
+ public ClientCookieHandler(String secureCookie) {
+ this.secureCookie = secureCookie;
+ }
+
+ @Override
+ public void channelActive(ChannelHandlerContext ctx) throws
Exception {
+ super.channelActive(ctx);
+ LOG.debug("In channelActive method of
ClientCookieHandler");
+
+ if(this.secureCookie != null &&
this.secureCookie.length() != 0) {
+ LOG.debug("In channelActive method of
ClientCookieHandler -> sending secure cookie");
+ final ByteBuf buffer = Unpooled.buffer(4 +
this.secureCookie.getBytes(DEFAULT_CHARSET).length);
+
buffer.writeInt(secureCookie.getBytes(DEFAULT_CHARSET).length);
+
buffer.writeBytes(secureCookie.getBytes(DEFAULT_CHARSET));
+ ctx.writeAndFlush(buffer);
+ }
+ }
+ }
+
+ public static class ServerCookieDecoder extends
MessageToMessageDecoder {
+
+ private final String secureCookie;
+
+ private final List channelList = new ArrayList<>();
+
+ private final Charset DEFAULT_CHARSET =
Charset.forName("utf-8");
+
+ private final Logger LOG =
LoggerFactory.getLogger(ServerCookieDecoder.class);
+
+ public ServerCookieDecoder(String secureCookie) {
+ this.secureCookie = secureCookie;
+ }
+
+ /**
+* Decode from one message to an other. This method will be
called for each written message that can be handled
+* by this encoder.
+*
+* @param ctx the {@link ChannelHandlerContext} which this
{@link MessageToMessageDecoder} belongs to
+* @param msg the message to decode to an other one
+* @param out the {@link List} to which decoded messages should
be added
+* @throws Exception is thrown if an error accour
+*/
+ @Override
+ protected void decode(ChannelHandlerContext ctx, ByteBuf msg,
List out) throws Exception {
+
+ LOG.debug("ChannelHandlerContext name: {}, channel:
{}", ctx.name(), ctx.channel());
+
+ if(secureCookie == null || secureCookie.length() == 0) {
+ LOG.debug("Not validating secure cookie since
the server configuration is not enabled to use cookie");
+
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[
https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927446#comment-15927446
]
ASF GitHub Bot commented on FLINK-3930:
---
Github user WangTaoTheTonic commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r106335331
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/CookieHandler.java
---
@@ -0,0 +1,130 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network.netty;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.channel.Channel;
+import io.netty.channel.ChannelHandlerContext;
+import io.netty.channel.ChannelInboundHandlerAdapter;
+import io.netty.handler.codec.MessageToMessageDecoder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+public class CookieHandler {
+
+ public static class ClientCookieHandler extends
ChannelInboundHandlerAdapter {
+
+ private final Logger LOG =
LoggerFactory.getLogger(ClientCookieHandler.class);
+
+ private final String secureCookie;
+
+ final Charset DEFAULT_CHARSET = Charset.forName("utf-8");
+
+ public ClientCookieHandler(String secureCookie) {
+ this.secureCookie = secureCookie;
+ }
+
+ @Override
+ public void channelActive(ChannelHandlerContext ctx) throws
Exception {
+ super.channelActive(ctx);
+ LOG.debug("In channelActive method of
ClientCookieHandler");
+
+ if(this.secureCookie != null &&
this.secureCookie.length() != 0) {
+ LOG.debug("In channelActive method of
ClientCookieHandler -> sending secure cookie");
+ final ByteBuf buffer = Unpooled.buffer(4 +
this.secureCookie.getBytes(DEFAULT_CHARSET).length);
+
buffer.writeInt(secureCookie.getBytes(DEFAULT_CHARSET).length);
+
buffer.writeBytes(secureCookie.getBytes(DEFAULT_CHARSET));
+ ctx.writeAndFlush(buffer);
+ }
+ }
+ }
+
+ public static class ServerCookieDecoder extends
MessageToMessageDecoder {
+
+ private final String secureCookie;
+
+ private final List channelList = new ArrayList<>();
--- End diff --
Is it better to use `Set` instead of a `List` here? As it is mainly used
for lookup.
> Implement Service-Level Authorization
> -
>
> Key: FLINK-3930
> URL: https://issues.apache.org/jira/browse/FLINK-3930
> Project: Flink
> Issue Type: New Feature
> Components: Security
>Reporter: Eron Wright
>Assignee: Vijay Srinivasaraghavan
> Labels: security
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
> design doc._
> Service-level authorization is the initial authorization mechanism to ensure
> clients (or servers) connecting to the Flink cluster are authorized to do so.
> The purpose is to prevent a cluster from being used by an unauthorized
> user, whether to execute jobs, disrupt cluster functionality, or gain access
> to secrets stored within the cluster.
> Implement service-level authorization as described in the design doc.
> - Introduce a shared secret cookie
> - Enable Akka security cookie
> - Implement data transfer authentication
> - Secure the web dashboard
--
This message was sent by Atlassian JIRA
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15926045#comment-15926045 ] ASF GitHub Bot commented on FLINK-3930: --- Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/2425 @vijikarthi I hope you are okay with exploring that option - this is not saying that this pull request is not a good solution, but whenever we have to maintain less code it makes things easier. > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security cookie > - Implement data transfer authentication > - Secure the web dashboard -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15926044#comment-15926044 ] ASF GitHub Bot commented on FLINK-3930: --- Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/2425 Thanks Eron, that makes a lot of sense. My first thought would be: Let's add SSL mutual authentication. That seems desirable anyways and we would not need another mechanism (shared secret). Do you know if newer versions of Akka support this mutual auth? We may be able to upgrade if we drop Java 7, or we could see if there is a lightweight way to patch this into flakka. > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security cookie > - Implement data transfer authentication > - Secure the web dashboard -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15924834#comment-15924834 ] ASF GitHub Bot commented on FLINK-3930: --- Github user EronWright commented on the issue: https://github.com/apache/flink/pull/2425 @StephanEwen keep in mind that Flink's current SSL support in Flink doesn't achieve _mutual authentication_ - there's no client certificate there.With SSL enabled, an untrusted client can launch jobs in your Flink cluster and thus gain access to the Kerberos credential associated with the cluster. SSL mutual authentication is a good alternative to a shared secret, but at the time we were limited to built-in Akka functionality (which doesn't include mutual auth). Given the "flakka" fork that's now in place, a pure SSL solution might now be possible (I haven't thought it through completely). The fact remains that, today, _all the secrets known to a Flink job are exposed to everyone who can connect to the cluster's endpoint_. It would be nice to construct a holistic plan that worked out how the Web UI would support authentication and also incorporated FLIP-6. Both YARN and Mesos interpose a web proxy for the UI with its own limitations, notably no support for SSL mutual auth. > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security cookie > - Implement data transfer authentication > - Secure the web dashboard -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15924477#comment-15924477 ] ASF GitHub Bot commented on FLINK-3930: --- Github user vijikarthi commented on the issue: https://github.com/apache/flink/pull/2425 @StephanEwen The shared secret serves can be considered as an additional security extension on top of TLS integration, thus it designates only an authorized identity to execute actions on a running cluster. > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security cookie > - Implement data transfer authentication > - Secure the web dashboard -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15924235#comment-15924235 ] ASF GitHub Bot commented on FLINK-3930: --- Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/2425 Sorry for chiming in a bit late here with this more fundamental question. I would like to understand from a security architecture, what additional security this shared secret gives us: - If there is no encryption, then this shared secret is not very secure, as it can be sniffed from the network - When there is encryption, isn't the current assumption that all parties have access to the server-side certificate? Would that already be a form of shared secret, meaning that certificate-based authentication as part of the SSL handshake already covers the mechanism of a shared secret? > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security cookie > - Implement data transfer authentication > - Secure the web dashboard -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15906765#comment-15906765 ] ASF GitHub Bot commented on FLINK-3930: --- Github user Rucongzhang commented on the issue: https://github.com/apache/flink/pull/2425 @vijikarthi ,when you will push this issue to the master? I can help you, if you need any help. Thanks! > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security cookie > - Implement data transfer authentication > - Secure the web dashboard -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675235#comment-15675235 ] ASF GitHub Bot commented on FLINK-3930: --- Github user vijikarthi commented on a diff in the pull request: https://github.com/apache/flink/pull/2425#discussion_r88576086 --- Diff: docs/internals/flink_security.md --- @@ -84,4 +86,79 @@ Security implementation details are based on https://github.com/apache/ ## Token Renewal -UGI and Kafka/ZK login module implementations takes care of auto-renewing the tickets upon reaching expiry and no further action is needed on the part of Flink. \ No newline at end of file +UGI and Kafka/ZK login module implementations takes care of auto-renewing the tickets upon reaching expiry and no further action is needed on the part of Flink. + +# Authorization Support + +Service-level authorization is the initial authorization mechanism to ensure clients (or servers) connecting to the Flink cluster are authorized to do so. The purpose is to prevent a cluster from being used by an unauthorized user, whether to execute jobs, disrupt cluster functionality, or gain access to secrets stored within the cluster. + +The primary goal is to secure the following components by introducing a shared secret mechanism to control the authorization. When security is enabled, the configured shared secret will be used as the basis to validate all the incoming/outgoing request. + +- Akka Endpoints + +- Flink Web Module + +- Blob Service + +- Task Manager/Netty data transfer communication + +## Security Configurations + +Secure cookie configuration can be supplied by adding below configuration elements to Flink configuration file: + +- `security.enabled`: A boolean value (true|false) indicating security is enabled or not. + +- `security.cookie` : Secure cookie value to be used for authorization + +Once a cluster is configured to run with secure cookie option, any request to the cluster will be validated for the existence of secure cookie. + +## Standalone Mode: + +In standalone mode of deployment, if security is enabled then it is mandatory to provide the secure cookie configuration in the Flink configuration file. A missing cookie configuration will flag an error. + +## Yarn Mode: + +In Yarn mode of deployment, secure cookie can be provided in multiple ways. + +- Flink configuration + +- As command line argument (-k or --cookie) to Yarn session CLI + +- Auto generated if not supplied through Flink configuration or Yarn session CLI argument + +The secure cookie will be made available as container environment variable for the application containers (JM/TM) to make use of it. + +On the client machine from where the Yarn session CLI is used to create the Flink application, the application specific secure cookie will be persisted in an INI file format in the user home directory. Any subsequent access to the Flink cluster using Yarn Session CLI (by passing the application ID) will automatically include appropriate secure cookie associated with the application ID to communicate with the cluster. + +Since the secure cookie is persisted in the user home directory, it is safe enough to consider that it can be accessed only by the user who created the cluster. --- End diff -- standard linux file permission - 664 > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security cookie > - Implement data transfer authentication > - Secure the web dashboard -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15652272#comment-15652272 ] ASF GitHub Bot commented on FLINK-3930: --- Github user vijikarthi commented on the issue: https://github.com/apache/flink/pull/2425 @StephanEwen @mxm - Could you please review the proposed change and let me know if you are okay with it. > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security cookie > - Implement data transfer authentication > - Secure the web dashboard -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[
https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15642713#comment-15642713
]
ASF GitHub Bot commented on FLINK-3930:
---
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2425
>
The cookie is added to every single message/buffer that is transferred.
That is too much - securing the integrity of the stream is responsibility of
the encryption layer. The cookie should be added to requests messages that
establish connections only.
I will change the code to address cookie handling right after the SSL
handshake using a new handler and drop the cookie passing for every messages.
The handler will be added to the pipeline of both `NettyClient` and
`NettyServer`. Client will send the cookie when the channel becomes active and
the server will validate and keep track of the clients that are authorized.
Here is the pseudo-code for Client and Server handlers. Please take a look
and let me know if you are okay with this approach and I will modify the code.
---
public static class ClientCookieHandler extends
ChannelInboundHandlerAdapter {
private final String secureCookie;
final Charset DEFAULT_CHARSET = Charset.forName("utf-8");
public ClientCookieHandler(String secureCookie) {
this.secureCookie = secureCookie;
}
@Override
public void channelActive(ChannelHandlerContext ctx) throws
Exception {
super.channelActive(ctx);
if(this.secureCookie != null &&
this.secureCookie.length() != 0) {
final ByteBuf buffer = Unpooled.buffer(4 +
this.secureCookie.getBytes(DEFAULT_CHARSET).length);
buffer.writeInt(secureCookie.getBytes(DEFAULT_CHARSET).length);
buffer.writeBytes(secureCookie.getBytes(DEFAULT_CHARSET));
ctx.writeAndFlush(buffer);
}
}
}
public static class ServerCookieDecoder extends
MessageToMessageDecoder {
private final String secureCookie;
private final List channelList = new ArrayList<>();
private final Charset DEFAULT_CHARSET =
Charset.forName("utf-8");
public ServerCookieDecoder(String secureCookie) {
this.secureCookie = secureCookie;
}
@Override
protected void decode(ChannelHandlerContext ctx, ByteBuf msg,
List out) throws Exception {
if(secureCookie == null || secureCookie.length() == 0) {
return;
}
if(channelList.contains(ctx.channel())) {
return;
}
//read cookie based on the cookie length passed
int cookieLength = msg.readInt();
if(cookieLength !=
secureCookie.getBytes(DEFAULT_CHARSET).length) {
String message = "Cookie length does not match
with source cookie. Invalid secure cookie passed.";
throw new IllegalStateException(message);
}
//read only if cookie length is greater than zero
if(cookieLength > 0) {
final byte[] buffer = new
byte[secureCookie.getBytes(DEFAULT_CHARSET).length];
msg.readBytes(buffer, 0, cookieLength);
if(!Arrays.equals(secureCookie.getBytes(DEFAULT_CHARSET), buffer)) {
LOG.error("Secure cookie from the
client is not matching with the server's identity");
throw new
IllegalStateException("Invalid secure cookie passed.");
}
LOG.info("Secure cookie validation passed");
channelList.add(ctx.channel());
}
}
}
---
> Implement Service-Level Authorization
> -
>
> Key: FLINK-3930
> URL: https://issues.apache.org/jira/browse/FLINK-3930
> Project: Flink
> Issue Type: New Feature
> Components: Security
>Reporter: Eron Wright
>Assignee: Vijay Srinivasaraghavan
> Labels: security
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> _This issue is part of a series of improvements
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15636639#comment-15636639 ] ASF GitHub Bot commented on FLINK-3930: --- Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/2425 The Netty logic needs some improvements: - The cookie is added to every single message/buffer that is transferred. That is too much - securing the integrity of the stream is responsibility of the encryption layer. The cookie should be added to requests messages that establish connections only. - Charset lookups and cookie to bytes encoding happens for every buffer, rather than once in an initialization step. - The String to byte conversion is not consistent. Sometimes it uses the default platform encoding, sometimes "UTF-8". > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security cookie > - Implement data transfer authentication > - Secure the web dashboard -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15636613#comment-15636613 ] ASF GitHub Bot commented on FLINK-3930: --- Github user StephanEwen commented on a diff in the pull request: https://github.com/apache/flink/pull/2425#discussion_r86564819 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyMessage.java --- @@ -57,24 +61,37 @@ // constructor in order to work with the generic deserializer. // - static final int HEADER_LENGTH = 4 + 4 + 1; // frame length (4), magic number (4), msg ID (1) + static final int HEADER_LENGTH = 4 + 4 + 4 + 1; // frame length (4), magic number (4), Cookie (4), msg ID (1) --- End diff -- Looks like this is the cookie length > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security cookie > - Implement data transfer authentication > - Secure the web dashboard -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15636444#comment-15636444 ] ASF GitHub Bot commented on FLINK-3930: --- Github user StephanEwen commented on a diff in the pull request: https://github.com/apache/flink/pull/2425#discussion_r86548851 --- Diff: docs/setup/yarn_setup.md --- @@ -134,6 +140,14 @@ Flink on YARN will only start all requested containers if enough resources are a some account also for the number of vcores. By default, the number of vcores is equal to the processing slots (`-s`) argument. The `yarn.containers.vcores` allows overwriting the number of vcores with a custom value. +### Service Authorization using Secure Cookie + +If service authorization for the cluster components (Akka, Blob Service, Web UI) is enabled, you could pass the secure cookie value as command line argument (-k or --cookie) instead of hardcoding the value in Flink configuration file. --- End diff -- I would link to the main security docs from here. A crucial thing to point out here is that when users use this with YARN sessions, all jobs running in that session will use the same cookie. The cookie is a "per-cluster" or "per-processes" parameter. Please add that for proper security between jobs, jobs should be submitted individually, not via a Flink Yarn Session. > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security cookie > - Implement data transfer authentication > - Secure the web dashboard -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15636447#comment-15636447 ] ASF GitHub Bot commented on FLINK-3930: --- Github user StephanEwen commented on a diff in the pull request: https://github.com/apache/flink/pull/2425#discussion_r86544195 --- Diff: docs/internals/flink_security.md --- @@ -84,4 +86,79 @@ Security implementation details are based on https://github.com/apache/ ## Token Renewal -UGI and Kafka/ZK login module implementations takes care of auto-renewing the tickets upon reaching expiry and no further action is needed on the part of Flink. \ No newline at end of file +UGI and Kafka/ZK login module implementations takes care of auto-renewing the tickets upon reaching expiry and no further action is needed on the part of Flink. + +# Authorization Support + +Service-level authorization is the initial authorization mechanism to ensure clients (or servers) connecting to the Flink cluster are authorized to do so. The purpose is to prevent a cluster from being used by an unauthorized user, whether to execute jobs, disrupt cluster functionality, or gain access to secrets stored within the cluster. + +The primary goal is to secure the following components by introducing a shared secret mechanism to control the authorization. When security is enabled, the configured shared secret will be used as the basis to validate all the incoming/outgoing request. + +- Akka Endpoints + +- Flink Web Module + +- Blob Service + +- Task Manager/Netty data transfer communication + +## Security Configurations + +Secure cookie configuration can be supplied by adding below configuration elements to Flink configuration file: + +- `security.enabled`: A boolean value (true|false) indicating security is enabled or not. + +- `security.cookie` : Secure cookie value to be used for authorization + +Once a cluster is configured to run with secure cookie option, any request to the cluster will be validated for the existence of secure cookie. + +## Standalone Mode: + +In standalone mode of deployment, if security is enabled then it is mandatory to provide the secure cookie configuration in the Flink configuration file. A missing cookie configuration will flag an error. + +## Yarn Mode: + +In Yarn mode of deployment, secure cookie can be provided in multiple ways. + +- Flink configuration + +- As command line argument (-k or --cookie) to Yarn session CLI + +- Auto generated if not supplied through Flink configuration or Yarn session CLI argument + +The secure cookie will be made available as container environment variable for the application containers (JM/TM) to make use of it. + +On the client machine from where the Yarn session CLI is used to create the Flink application, the application specific secure cookie will be persisted in an INI file format in the user home directory. Any subsequent access to the Flink cluster using Yarn Session CLI (by passing the application ID) will automatically include appropriate secure cookie associated with the application ID to communicate with the cluster. + +Since the secure cookie is persisted in the user home directory, it is safe enough to consider that it can be accessed only by the user who created the cluster. + +### Akka endpoints --- End diff -- Can we mark the sections following from here as "Notes on the Implementation"? Users should not get confused about that they need to do anything there. We may even want to factor them out into a separate document later > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15636443#comment-15636443 ] ASF GitHub Bot commented on FLINK-3930: --- Github user StephanEwen commented on a diff in the pull request: https://github.com/apache/flink/pull/2425#discussion_r86540326 --- Diff: docs/internals/flink_security.md --- @@ -84,4 +86,79 @@ Security implementation details are based on https://github.com/apache/ ## Token Renewal -UGI and Kafka/ZK login module implementations takes care of auto-renewing the tickets upon reaching expiry and no further action is needed on the part of Flink. \ No newline at end of file +UGI and Kafka/ZK login module implementations takes care of auto-renewing the tickets upon reaching expiry and no further action is needed on the part of Flink. + +# Authorization Support + +Service-level authorization is the initial authorization mechanism to ensure clients (or servers) connecting to the Flink cluster are authorized to do so. The purpose is to prevent a cluster from being used by an unauthorized user, whether to execute jobs, disrupt cluster functionality, or gain access to secrets stored within the cluster. + +The primary goal is to secure the following components by introducing a shared secret mechanism to control the authorization. When security is enabled, the configured shared secret will be used as the basis to validate all the incoming/outgoing request. --- End diff -- This section first talks about a shared secret then about a cookie. It would be good to say somewhere that the cookie is the shared secret. > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security cookie > - Implement data transfer authentication > - Secure the web dashboard -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15636442#comment-15636442 ] ASF GitHub Bot commented on FLINK-3930: --- Github user StephanEwen commented on a diff in the pull request: https://github.com/apache/flink/pull/2425#discussion_r86546406 --- Diff: docs/setup/config.md --- @@ -125,6 +125,28 @@ Kerberos ticket renewal is abstracted and automatically handled by the Hadoop/Ka For Kafka and ZK, process-wide JAAS config will be created using the provided security credentials and the Kerberos authentication will be handled by Kafka/ZK login handlers. +### Secure Cookie Authentication + --- End diff -- How about keeping this brief ans instead linking to the proper docs. ``` Flink supports configuring a *secure cookie* (a shared secret) to secure Flink processes. The secure cookie is used to authorize all access to and between Flink Processes. For more details, see [link to docs] ``` > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security cookie > - Implement data transfer authentication > - Secure the web dashboard -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[
https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15636439#comment-15636439
]
ASF GitHub Bot commented on FLINK-3930:
---
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r86550433
--- Diff:
flink-core/src/main/java/org/apache/flink/configuration/GlobalConfiguration.java
---
@@ -144,8 +144,15 @@ private static Configuration loadYAMLResource(File
file) {
continue;
}
- LOG.debug("Loading configuration
property: {}, {}", key, value);
config.setString(key, value);
+
+ //to prevent logging the secure cookie
+
if(key.equals(ConfigConstants.SECURITY_COOKIE) && value != null) {
+ value = "**";
--- End diff --
Can all suppressed keys have a common prefix?
This should also be guarded by a test ;-)
> Implement Service-Level Authorization
> -
>
> Key: FLINK-3930
> URL: https://issues.apache.org/jira/browse/FLINK-3930
> Project: Flink
> Issue Type: New Feature
> Components: Security
>Reporter: Eron Wright
>Assignee: Vijay Srinivasaraghavan
> Labels: security
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
> design doc._
> Service-level authorization is the initial authorization mechanism to ensure
> clients (or servers) connecting to the Flink cluster are authorized to do so.
> The purpose is to prevent a cluster from being used by an unauthorized
> user, whether to execute jobs, disrupt cluster functionality, or gain access
> to secrets stored within the cluster.
> Implement service-level authorization as described in the design doc.
> - Introduce a shared secret cookie
> - Enable Akka security cookie
> - Implement data transfer authentication
> - Secure the web dashboard
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15636448#comment-15636448 ] ASF GitHub Bot commented on FLINK-3930: --- Github user StephanEwen commented on a diff in the pull request: https://github.com/apache/flink/pull/2425#discussion_r86542654 --- Diff: docs/internals/flink_security.md --- @@ -84,4 +86,79 @@ Security implementation details are based on https://github.com/apache/ ## Token Renewal -UGI and Kafka/ZK login module implementations takes care of auto-renewing the tickets upon reaching expiry and no further action is needed on the part of Flink. \ No newline at end of file +UGI and Kafka/ZK login module implementations takes care of auto-renewing the tickets upon reaching expiry and no further action is needed on the part of Flink. + +# Authorization Support + +Service-level authorization is the initial authorization mechanism to ensure clients (or servers) connecting to the Flink cluster are authorized to do so. The purpose is to prevent a cluster from being used by an unauthorized user, whether to execute jobs, disrupt cluster functionality, or gain access to secrets stored within the cluster. + +The primary goal is to secure the following components by introducing a shared secret mechanism to control the authorization. When security is enabled, the configured shared secret will be used as the basis to validate all the incoming/outgoing request. + +- Akka Endpoints + +- Flink Web Module + +- Blob Service + +- Task Manager/Netty data transfer communication + +## Security Configurations + +Secure cookie configuration can be supplied by adding below configuration elements to Flink configuration file: + +- `security.enabled`: A boolean value (true|false) indicating security is enabled or not. + +- `security.cookie` : Secure cookie value to be used for authorization + +Once a cluster is configured to run with secure cookie option, any request to the cluster will be validated for the existence of secure cookie. + +## Standalone Mode: + +In standalone mode of deployment, if security is enabled then it is mandatory to provide the secure cookie configuration in the Flink configuration file. A missing cookie configuration will flag an error. + +## Yarn Mode: + +In Yarn mode of deployment, secure cookie can be provided in multiple ways. + +- Flink configuration + +- As command line argument (-k or --cookie) to Yarn session CLI + +- Auto generated if not supplied through Flink configuration or Yarn session CLI argument + +The secure cookie will be made available as container environment variable for the application containers (JM/TM) to make use of it. + +On the client machine from where the Yarn session CLI is used to create the Flink application, the application specific secure cookie will be persisted in an INI file format in the user home directory. Any subsequent access to the Flink cluster using Yarn Session CLI (by passing the application ID) will automatically include appropriate secure cookie associated with the application ID to communicate with the cluster. --- End diff -- Will this only happen if the cookie is auto generated? > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security cookie > - Implement data transfer authentication > - Secure the web dashboard -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15636438#comment-15636438 ] ASF GitHub Bot commented on FLINK-3930: --- Github user StephanEwen commented on a diff in the pull request: https://github.com/apache/flink/pull/2425#discussion_r86543115 --- Diff: docs/internals/flink_security.md --- @@ -84,4 +86,79 @@ Security implementation details are based on https://github.com/apache/ ## Token Renewal -UGI and Kafka/ZK login module implementations takes care of auto-renewing the tickets upon reaching expiry and no further action is needed on the part of Flink. \ No newline at end of file +UGI and Kafka/ZK login module implementations takes care of auto-renewing the tickets upon reaching expiry and no further action is needed on the part of Flink. + +# Authorization Support --- End diff -- How about elaborating a bit on the interplay between the Authorization and Encryption support? For example, it is important to know that secrets go plain text over every wire unless encryption is enabled. > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security cookie > - Implement data transfer authentication > - Secure the web dashboard -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15636440#comment-15636440 ] ASF GitHub Bot commented on FLINK-3930: --- Github user StephanEwen commented on a diff in the pull request: https://github.com/apache/flink/pull/2425#discussion_r86550641 --- Diff: flink-dist/src/main/resources/flink-conf.yaml --- @@ -173,3 +174,13 @@ jobmanager.web.port: 8081 # Override below configuration to provide custom ZK service name if configured # # zookeeper.sasl.service-name: zookeeper + +#== +# Service Authorization Configuration (optional configuration) --- End diff -- Can we group all security related parts together? > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security cookie > - Implement data transfer authentication > - Secure the web dashboard -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15636441#comment-15636441 ] ASF GitHub Bot commented on FLINK-3930: --- Github user StephanEwen commented on a diff in the pull request: https://github.com/apache/flink/pull/2425#discussion_r86543731 --- Diff: docs/internals/flink_security.md --- @@ -28,14 +28,16 @@ This document briefly describes how Flink security works in the context of vario and the connectors that participates in Flink Job execution stage. This documentation can be helpful for both administrators and developers --- End diff -- This document should probably start with an introduction to the different aspects of security: - Authentication (Flink authenticates at other services) - Authorization (No one unauthorized access the Flink cluster) - Encryption (no sniffing off data and credentials) Authorization alone is probably only meaningful to prevent "accidental mixups", all meaningfully secure setups would need Authorization and Encryption > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security cookie > - Implement data transfer authentication > - Secure the web dashboard -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15636445#comment-15636445 ] ASF GitHub Bot commented on FLINK-3930: --- Github user StephanEwen commented on a diff in the pull request: https://github.com/apache/flink/pull/2425#discussion_r86544336 --- Diff: docs/setup/cli.md --- @@ -217,6 +217,8 @@ Action "run" compiles and runs a program. java.net.URLClassLoader}. -d,--detached If present, runs the job in detached mode + -k,--cookie Secure cookie to +authenticate --- End diff -- Agree > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security cookie > - Implement data transfer authentication > - Secure the web dashboard -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15636437#comment-15636437 ] ASF GitHub Bot commented on FLINK-3930: --- Github user StephanEwen commented on a diff in the pull request: https://github.com/apache/flink/pull/2425#discussion_r86540839 --- Diff: docs/internals/flink_security.md --- @@ -84,4 +86,79 @@ Security implementation details are based on https://github.com/apache/ ## Token Renewal -UGI and Kafka/ZK login module implementations takes care of auto-renewing the tickets upon reaching expiry and no further action is needed on the part of Flink. \ No newline at end of file +UGI and Kafka/ZK login module implementations takes care of auto-renewing the tickets upon reaching expiry and no further action is needed on the part of Flink. + +# Authorization Support + +Service-level authorization is the initial authorization mechanism to ensure clients (or servers) connecting to the Flink cluster are authorized to do so. The purpose is to prevent a cluster from being used by an unauthorized user, whether to execute jobs, disrupt cluster functionality, or gain access to secrets stored within the cluster. + +The primary goal is to secure the following components by introducing a shared secret mechanism to control the authorization. When security is enabled, the configured shared secret will be used as the basis to validate all the incoming/outgoing request. + +- Akka Endpoints --- End diff -- How about describing these parts by their role? I do not expect users to generally know that Flink uses Akka for distributed coordination. How about - Coordination / RPC communication between JobManager, ResourceManager, and TaskManager *(via Akka)* - Flink Web Module - File distribution, like JAR files, etc *(BLOB Service)* - Data exchange between TaskManagers *(via Netty)* > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security cookie > - Implement data transfer authentication > - Secure the web dashboard -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15636450#comment-15636450 ] ASF GitHub Bot commented on FLINK-3930: --- Github user StephanEwen commented on a diff in the pull request: https://github.com/apache/flink/pull/2425#discussion_r86540198 --- Diff: docs/internals/flink_security.md --- @@ -84,4 +86,79 @@ Security implementation details are based on https://github.com/apache/ ## Token Renewal -UGI and Kafka/ZK login module implementations takes care of auto-renewing the tickets upon reaching expiry and no further action is needed on the part of Flink. \ No newline at end of file +UGI and Kafka/ZK login module implementations takes care of auto-renewing the tickets upon reaching expiry and no further action is needed on the part of Flink. --- End diff -- What is `UGI`? Can we spell this out? > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security cookie > - Implement data transfer authentication > - Secure the web dashboard -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15636446#comment-15636446 ] ASF GitHub Bot commented on FLINK-3930: --- Github user StephanEwen commented on a diff in the pull request: https://github.com/apache/flink/pull/2425#discussion_r86549853 --- Diff: flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java --- @@ -715,6 +715,15 @@ /** Flag to enable/disable hostname verification for the ssl connections */ public static final String SECURITY_SSL_VERIFY_HOSTNAME = "security.ssl.verify-hostname"; + // Secure Cookie Authentication --- + + /** Flag that specify whether service authentication is enabled or not **/ + public static final String SECURITY_ENABLED = "security.enabled"; --- End diff -- Can we add these options directly via `ConfogOptions` similar to that: https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/configuration/HighAvailabilityOptions.java Maybe start a new class, `SecurityOptions`. > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security cookie > - Implement data transfer authentication > - Secure the web dashboard -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15636436#comment-15636436 ] ASF GitHub Bot commented on FLINK-3930: --- Github user StephanEwen commented on a diff in the pull request: https://github.com/apache/flink/pull/2425#discussion_r86542840 --- Diff: docs/internals/flink_security.md --- @@ -84,4 +86,79 @@ Security implementation details are based on https://github.com/apache/ ## Token Renewal -UGI and Kafka/ZK login module implementations takes care of auto-renewing the tickets upon reaching expiry and no further action is needed on the part of Flink. \ No newline at end of file +UGI and Kafka/ZK login module implementations takes care of auto-renewing the tickets upon reaching expiry and no further action is needed on the part of Flink. + +# Authorization Support + +Service-level authorization is the initial authorization mechanism to ensure clients (or servers) connecting to the Flink cluster are authorized to do so. The purpose is to prevent a cluster from being used by an unauthorized user, whether to execute jobs, disrupt cluster functionality, or gain access to secrets stored within the cluster. + +The primary goal is to secure the following components by introducing a shared secret mechanism to control the authorization. When security is enabled, the configured shared secret will be used as the basis to validate all the incoming/outgoing request. + +- Akka Endpoints + +- Flink Web Module + +- Blob Service + +- Task Manager/Netty data transfer communication + +## Security Configurations + +Secure cookie configuration can be supplied by adding below configuration elements to Flink configuration file: + +- `security.enabled`: A boolean value (true|false) indicating security is enabled or not. + +- `security.cookie` : Secure cookie value to be used for authorization + +Once a cluster is configured to run with secure cookie option, any request to the cluster will be validated for the existence of secure cookie. + +## Standalone Mode: + +In standalone mode of deployment, if security is enabled then it is mandatory to provide the secure cookie configuration in the Flink configuration file. A missing cookie configuration will flag an error. + +## Yarn Mode: + +In Yarn mode of deployment, secure cookie can be provided in multiple ways. + +- Flink configuration + +- As command line argument (-k or --cookie) to Yarn session CLI + +- Auto generated if not supplied through Flink configuration or Yarn session CLI argument + +The secure cookie will be made available as container environment variable for the application containers (JM/TM) to make use of it. + +On the client machine from where the Yarn session CLI is used to create the Flink application, the application specific secure cookie will be persisted in an INI file format in the user home directory. Any subsequent access to the Flink cluster using Yarn Session CLI (by passing the application ID) will automatically include appropriate secure cookie associated with the application ID to communicate with the cluster. + +Since the secure cookie is persisted in the user home directory, it is safe enough to consider that it can be accessed only by the user who created the cluster. --- End diff -- I think this should be a bigger warning, as it shows a fundamental assumption that users should be aware of. With what access permission settings is the ini file persisted? > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security cookie > - Implement data transfer authentication > - Secure the web dashboard
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15636449#comment-15636449 ] ASF GitHub Bot commented on FLINK-3930: --- Github user StephanEwen commented on a diff in the pull request: https://github.com/apache/flink/pull/2425#discussion_r86546515 --- Diff: docs/setup/yarn_setup.md --- @@ -101,13 +101,19 @@ Usage: Optional -D Dynamic properties -d,--detached Start detached + -id,--applicationIdAttach to running YARN session + -j,--jar Path to Flink jar file -jm,--jobManagerMemory Memory for JobManager Container [in MB] - -nm,--name Set a custom name for the application on YARN + -k,--cookieSecure cookie to authenticate --- End diff -- Why not? Is it not possible to manually specify a cookie in YARN? > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security cookie > - Implement data transfer authentication > - Secure the web dashboard -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15635973#comment-15635973 ] ASF GitHub Bot commented on FLINK-3930: --- Github user mxm commented on the issue: https://github.com/apache/flink/pull/2425 Thank you for the changes. I wonder, could we remove the cookie header completely for Netty or the BlobServer in case the authorization is turned off? The Netty protocol has a `MAGIC_NUMBER` which is checked when decoding the message. We could use a different "magic number" to check whether we use the normal or the cookie-based Netty protocol. This would eliminate all the overhead of the cookie transmission. Furthermore, we should strip the cookie from the message once we have verified it is correct. > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security cookie > - Implement data transfer authentication > - Secure the web dashboard -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634525#comment-15634525 ] ASF GitHub Bot commented on FLINK-3930: --- Github user vijikarthi commented on the issue: https://github.com/apache/flink/pull/2425 @mxm - Sorry that I have missed to address some of your comments. Attached patch that includes Netty code null precondition validation and fixes the Blob service cookie length issue. Please take a look and see if they are okay? Thanks for your patience. > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security cookie > - Implement data transfer authentication > - Secure the web dashboard -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[
https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15632707#comment-15632707
]
ASF GitHub Bot commented on FLINK-3930:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r86340045
--- Diff:
flink-yarn/src/main/java/org/apache/flink/yarn/cli/FlinkYarnSessionCli.java ---
@@ -788,75 +719,125 @@ private void logAndSysout(String message) {
}
}
- public static File getYarnPropertiesLocation(Configuration conf) {
- String defaultPropertiesFileLocation =
System.getProperty("java.io.tmpdir");
- String currentUser = System.getProperty("user.name");
- String propertiesFileLocation =
-
conf.getString(ConfigConstants.YARN_PROPERTIES_FILE_LOCATION,
defaultPropertiesFileLocation);
-
- return new File(propertiesFileLocation, YARN_PROPERTIES_FILE +
currentUser);
+ public static File getYarnPropertiesLocation() {
+ String path = System.getProperty("user.home") + File.separator
+ YARN_APP_INI;
+ File stateFile;
+ try {
+ stateFile = new File(path);
+ if(!stateFile.exists()) {
+ stateFile.createNewFile();
+ }
+ } catch(IOException e) {
+ throw new RuntimeException(e);
+ }
+ return stateFile;
}
- public static void persistAppState(String appId, String cookie) {
- if(appId == null || cookie == null) {
- return;
+ public static void persistAppState(YarnAppState appState) {
+
+ final String appId = appState.getApplicationId();
+ final String parallelism = appState.getParallelism();
+ final String dynaProps = appState.getDynamicProperties();
+ final String cookie = appState.getCookie();
+
+ if(appId == null) {
+ throw new RuntimeException("Missing application ID from
Yarn application state");
}
- String path = System.getProperty("user.home") + File.separator
+ fileName;
- LOG.debug("Going to persist cookie for the appID: {} in {} ",
appId, path);
+
+ String path = getYarnPropertiesLocation().getAbsolutePath();
+
+ LOG.debug("Going to persist Yarn application state: {} in {}",
appState,path);
+
try {
- File f = new File(path);
- if(!f.exists()) {
- f.createNewFile();
- }
HierarchicalINIConfiguration config = new
HierarchicalINIConfiguration(path);
+
SubnodeConfiguration subNode = config.getSection(appId);
- if (subNode.containsKey(cookieKey)) {
- String errorMessage = "Secure Cookie is already
found in "+ path + " for the appID: "+ appId;
- LOG.error(errorMessage);
- throw new RuntimeException(errorMessage);
+ if(!subNode.isEmpty()) {
+ throw new RuntimeException("Application with ID
" + appId + "already exists");
}
- subNode.addProperty(cookieKey, cookie);
+
+ subNode.addProperty(YARN_PROPERTIES_PARALLELISM,
parallelism);
+
subNode.addProperty(YARN_PROPERTIES_DYNAMIC_PROPERTIES_STRING, dynaProps);
+ subNode.addProperty(YARN_PROPERTIES_SECURE_COOKIE,
cookie);
+
+ //update latest entry section with the most recent APP
Id
+ config.clearTree(YARN_LATEST_ENTRY_SECTION_NAME);
+ SubnodeConfiguration activeAppSection =
config.getSection(YARN_LATEST_ENTRY_SECTION_NAME);
+ activeAppSection.addProperty(YARN_APPLICATION_ID_KEY,
appId);
+
config.save();
- LOG.debug("Persisted cookie for the appID: {}", appId);
+ LOG.debug("Persisted Yarn App state: {}", appState);
} catch(Exception e) {
- LOG.error("Exception occurred while persisting app
state for app id: {}", appId, e);
throw new RuntimeException(e);
}
}
- public static String getAppSecureCookie(String appId) {
+ public static YarnAppState retrieveMostRecentYarnApp() {
+ String path = getYarnPropertiesLocation().getAbsolutePath();
+ LOG.debug("Going to fetch app state from {}", path);
+
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[
https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15632481#comment-15632481
]
ASF GitHub Bot commented on FLINK-3930:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r86324769
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyMessage.java
---
@@ -54,24 +58,36 @@
// constructor in order to work with the generic deserializer.
//
- static final int HEADER_LENGTH = 4 + 4 + 1; // frame length (4), magic
number (4), msg ID (1)
+ static final int HEADER_LENGTH = 4 + 4 + 4 + 1; // frame length (4),
magic number (4), Cookie (4), msg ID (1)
static final int MAGIC_NUMBER = 0xBADC0FFE;
+ static final String NO_SECURE_COOKIE = "";
+
abstract ByteBuf write(ByteBufAllocator allocator) throws Exception;
abstract void readFrom(ByteBuf buffer) throws Exception;
//
- private static ByteBuf allocateBuffer(ByteBufAllocator allocator, byte
id) {
- return allocateBuffer(allocator, id, 0);
+ private static ByteBuf allocateBuffer(ByteBufAllocator allocator, byte
id, String secureCookie) {
+ return allocateBuffer(allocator, id, 0, secureCookie);
}
- private static ByteBuf allocateBuffer(ByteBufAllocator allocator, byte
id, int length) {
+ private static ByteBuf allocateBuffer(ByteBufAllocator allocator, byte
id, int length, String secureCookie) {
+ final Charset DEFAULT_CHARSET = Charset.forName("utf-8");
+ secureCookie = secureCookie == null ? "": secureCookie;
--- End diff --
Should be removed in favor of never passing a null value here.
> Implement Service-Level Authorization
> -
>
> Key: FLINK-3930
> URL: https://issues.apache.org/jira/browse/FLINK-3930
> Project: Flink
> Issue Type: New Feature
> Components: Security
>Reporter: Eron Wright
>Assignee: Vijay Srinivasaraghavan
> Labels: security
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
> design doc._
> Service-level authorization is the initial authorization mechanism to ensure
> clients (or servers) connecting to the Flink cluster are authorized to do so.
> The purpose is to prevent a cluster from being used by an unauthorized
> user, whether to execute jobs, disrupt cluster functionality, or gain access
> to secrets stored within the cluster.
> Implement service-level authorization as described in the design doc.
> - Introduce a shared secret cookie
> - Enable Akka security cookie
> - Implement data transfer authentication
> - Secure the web dashboard
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[
https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15632475#comment-15632475
]
ASF GitHub Bot commented on FLINK-3930:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r86323404
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobServerConnection.java
---
@@ -101,6 +102,14 @@ public void run() {
final byte[] buffer = new byte[BUFFER_SIZE];
while (true) {
+
+ int keyLength = inputStream.read();
--- End diff --
Here the cookie length is limited to one byte.
> Implement Service-Level Authorization
> -
>
> Key: FLINK-3930
> URL: https://issues.apache.org/jira/browse/FLINK-3930
> Project: Flink
> Issue Type: New Feature
> Components: Security
>Reporter: Eron Wright
>Assignee: Vijay Srinivasaraghavan
> Labels: security
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
> design doc._
> Service-level authorization is the initial authorization mechanism to ensure
> clients (or servers) connecting to the Flink cluster are authorized to do so.
> The purpose is to prevent a cluster from being used by an unauthorized
> user, whether to execute jobs, disrupt cluster functionality, or gain access
> to secrets stored within the cluster.
> Implement service-level authorization as described in the design doc.
> - Introduce a shared secret cookie
> - Enable Akka security cookie
> - Implement data transfer authentication
> - Secure the web dashboard
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[
https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15632474#comment-15632474
]
ASF GitHub Bot commented on FLINK-3930:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r86321320
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyMessage.java
---
@@ -64,26 +64,25 @@
static final String NO_SECURE_COOKIE = "";
- static final Charset DEFAULT_CHARSET = Charset.forName("utf-8");
-
abstract ByteBuf write(ByteBufAllocator allocator) throws Exception;
abstract void readFrom(ByteBuf buffer) throws Exception;
//
private static ByteBuf allocateBuffer(ByteBufAllocator allocator, byte
id, String secureCookie) {
- return allocateBuffer(allocator, id, secureCookie, 0);
+ return allocateBuffer(allocator, id, 0, secureCookie);
}
- private static ByteBuf allocateBuffer(ByteBufAllocator allocator, byte
id, String secureCookie, int length) {
+ private static ByteBuf allocateBuffer(ByteBufAllocator allocator, byte
id, int length, String secureCookie) {
+ final Charset DEFAULT_CHARSET = Charset.forName("utf-8");
secureCookie = secureCookie == null ? "": secureCookie;
--- End diff --
Still think we should never pass a null cookie. Then the check wouldn't be
necessary.
> Implement Service-Level Authorization
> -
>
> Key: FLINK-3930
> URL: https://issues.apache.org/jira/browse/FLINK-3930
> Project: Flink
> Issue Type: New Feature
> Components: Security
>Reporter: Eron Wright
>Assignee: Vijay Srinivasaraghavan
> Labels: security
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
> design doc._
> Service-level authorization is the initial authorization mechanism to ensure
> clients (or servers) connecting to the Flink cluster are authorized to do so.
> The purpose is to prevent a cluster from being used by an unauthorized
> user, whether to execute jobs, disrupt cluster functionality, or gain access
> to secrets stored within the cluster.
> Implement service-level authorization as described in the design doc.
> - Introduce a shared secret cookie
> - Enable Akka security cookie
> - Implement data transfer authentication
> - Secure the web dashboard
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[
https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15632476#comment-15632476
]
ASF GitHub Bot commented on FLINK-3930:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r86322286
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyMessage.java
---
@@ -120,6 +136,15 @@ static LengthFieldBasedFrameDecoder
createFrameLengthDecoder() {
@ChannelHandler.Sharable
static class NettyMessageDecoder extends
MessageToMessageDecoder {
+ private static final Logger LOG =
LoggerFactory.getLogger(NettyMessageDecoder.class);
+
+ final byte[] secureCookie;
+
+ public NettyMessageDecoder(String secureCookie) {
+ secureCookie = secureCookie == null ? "": secureCookie;
--- End diff --
Should be removed.
> Implement Service-Level Authorization
> -
>
> Key: FLINK-3930
> URL: https://issues.apache.org/jira/browse/FLINK-3930
> Project: Flink
> Issue Type: New Feature
> Components: Security
>Reporter: Eron Wright
>Assignee: Vijay Srinivasaraghavan
> Labels: security
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
> design doc._
> Service-level authorization is the initial authorization mechanism to ensure
> clients (or servers) connecting to the Flink cluster are authorized to do so.
> The purpose is to prevent a cluster from being used by an unauthorized
> user, whether to execute jobs, disrupt cluster functionality, or gain access
> to secrets stored within the cluster.
> Implement service-level authorization as described in the design doc.
> - Introduce a shared secret cookie
> - Enable Akka security cookie
> - Implement data transfer authentication
> - Secure the web dashboard
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[
https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15632473#comment-15632473
]
ASF GitHub Bot commented on FLINK-3930:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r86321008
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyMessage.java
---
@@ -64,26 +64,25 @@
static final String NO_SECURE_COOKIE = "";
- static final Charset DEFAULT_CHARSET = Charset.forName("utf-8");
-
abstract ByteBuf write(ByteBufAllocator allocator) throws Exception;
abstract void readFrom(ByteBuf buffer) throws Exception;
//
private static ByteBuf allocateBuffer(ByteBufAllocator allocator, byte
id, String secureCookie) {
- return allocateBuffer(allocator, id, secureCookie, 0);
+ return allocateBuffer(allocator, id, 0, secureCookie);
}
- private static ByteBuf allocateBuffer(ByteBufAllocator allocator, byte
id, String secureCookie, int length) {
+ private static ByteBuf allocateBuffer(ByteBufAllocator allocator, byte
id, int length, String secureCookie) {
+ final Charset DEFAULT_CHARSET = Charset.forName("utf-8");
secureCookie = secureCookie == null ? "": secureCookie;
--- End diff --
Simpler:
```java
if (secureCookie == null) {
secureCookie = NO_SECURE_COOKIE;
}
```
> Implement Service-Level Authorization
> -
>
> Key: FLINK-3930
> URL: https://issues.apache.org/jira/browse/FLINK-3930
> Project: Flink
> Issue Type: New Feature
> Components: Security
>Reporter: Eron Wright
>Assignee: Vijay Srinivasaraghavan
> Labels: security
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
> design doc._
> Service-level authorization is the initial authorization mechanism to ensure
> clients (or servers) connecting to the Flink cluster are authorized to do so.
> The purpose is to prevent a cluster from being used by an unauthorized
> user, whether to execute jobs, disrupt cluster functionality, or gain access
> to secrets stored within the cluster.
> Implement service-level authorization as described in the design doc.
> - Introduce a shared secret cookie
> - Enable Akka security cookie
> - Implement data transfer authentication
> - Secure the web dashboard
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15620793#comment-15620793 ] ASF GitHub Bot commented on FLINK-3930: --- Github user vijikarthi commented on the issue: https://github.com/apache/flink/pull/2425 Addressed multiple application support/Yarn configuration file changes as part of FLINK-4950 patch. > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security cookie > - Implement data transfer authentication > - Secure the web dashboard -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15611368#comment-15611368 ] ASF GitHub Bot commented on FLINK-3930: --- Github user mxm commented on the issue: https://github.com/apache/flink/pull/2425 CC @uce to check out the network layer changes. This is a very sensitive and performance critical part of Flink. We should be very sure nothing breaks it with the changes. @vijikarthi Please have a look at the null checks in the network code. I would replace them with `checkNotNull` and never pass any null values in there. It would be desirable that turned off security doesn't have any overhead with the security support built in. > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security cookie > - Implement data transfer authentication > - Secure the web dashboard -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15609088#comment-15609088 ] ASF GitHub Bot commented on FLINK-3930: --- Github user mxm commented on a diff in the pull request: https://github.com/apache/flink/pull/2425#discussion_r85175840 --- Diff: flink-yarn/src/main/java/org/apache/flink/yarn/cli/FlinkYarnSessionCli.java --- @@ -108,6 +111,11 @@ private final Options ALL_OPTIONS; + private static final String fileName = "yarn-app.ini"; + private static final String cookieKey = "secureCookie"; --- End diff -- I think the ini file format is actually fine. Could also be JSON but I don't mind. To not break backwards-compatibility, I think we have to keep the behavior to use the last-used application id in case none is supplied. We could have an extra config entry for that. > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security cookie > - Implement data transfer authentication > - Secure the web dashboard -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15608966#comment-15608966 ] ASF GitHub Bot commented on FLINK-3930: --- Github user vijikarthi commented on a diff in the pull request: https://github.com/apache/flink/pull/2425#discussion_r85167083 --- Diff: flink-yarn/src/main/java/org/apache/flink/yarn/cli/FlinkYarnSessionCli.java --- @@ -108,6 +111,11 @@ private final Options ALL_OPTIONS; + private static final String fileName = "yarn-app.ini"; + private static final String cookieKey = "secureCookie"; --- End diff -- Yes, I will make the change. - Do you object to retain the ini file format and port the current properties file implementation to INI format (to persist multiple application states)? - Per current implementation (retrieveCluster), the CLI code fetches the application ID from properties file if not supplied through CLI argument. When we support multiple application state, then we expect application ID to be supplied always since there could be more than one application ID and the default functionality will go away. Do you concur? > If we really need to provide backward compatibility support, then we could return the application ID from the INI file should there be just only instance persisted? If more than one application ID exists, then we throw an error indicating "Application ID" needs to be supplied as CLI argument. Please let me know how you want me to approach and I will make the changes accordingly. > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security cookie > - Implement data transfer authentication > - Secure the web dashboard -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15607973#comment-15607973 ] ASF GitHub Bot commented on FLINK-3930: --- Github user mxm commented on the issue: https://github.com/apache/flink/pull/2425 @vijikarthi I haven't forgotten about your PR. Thanks for the feedback. I'll get back to you today. > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security cookie > - Implement data transfer authentication > - Secure the web dashboard -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603349#comment-15603349 ] ASF GitHub Bot commented on FLINK-3930: --- Github user vijikarthi commented on the issue: https://github.com/apache/flink/pull/2425 @mxm - Please take a look when you get a chance? > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security cookie > - Implement data transfer authentication > - Secure the web dashboard -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15590228#comment-15590228 ] ASF GitHub Bot commented on FLINK-3930: --- Github user vijikarthi commented on the issue: https://github.com/apache/flink/pull/2425 Thanks @mxm for the review. I will incorporate your feedback and attach the patch. > When security is enabled, encryption should also be turned on by default. Otherwise we will transmit plain-text passwords over the wire. Yes it makes sense. I will make a conditional check and throw an error if encryption is not enabled when security is enabled? > Should there be too modes for network transmission, 1) with cookie, one without? Do we need 32 bits for the cookie length? We should be precise about the maximum length. I saw it is set to 1024 in other places. Yes, max cookie length validation is 1024. I will change the code where the buffer length was allocated to a high value, instead it will use the byte array length read from the message. > Should we really always send the cookie for every message? The security document mentions in T2-3 that we only want to authorize upon the first connection. Yes, we took the approach to pass secure cookie for every message to keep minimal changes to the current design > Why do we transmit the cookie to the client? This seems like a major security concern. The client should always provide the cookie. edit: I see this has been specified in the document in T2-4. Still, I wonder if it would make sense to simply add this now because the workaround to fetch the cookie from the JobManager doesn't look appealing. Good catch. I forgot to revert the code after the merge and it is not required. Will fix it. > You added a Yarn specific cookie option which should be part of the general options instead. It is added since secure cookie can be supplied when using both Yarn session CLI as well as Flink CLI. I have provided detailed explanation in one of the comments. > You've introduce a new config file to persist the app state. We already have the Yarn properties file for that. I have provided explanation on why we need new config file in one of the comments. Please take a look. > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security cookie > - Implement data transfer authentication > - Secure the web dashboard -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[
https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15589986#comment-15589986
]
ASF GitHub Bot commented on FLINK-3930:
---
Github user vijikarthi commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r84175964
--- Diff:
flink-yarn/src/main/java/org/apache/flink/yarn/cli/FlinkYarnSessionCli.java ---
@@ -442,8 +453,10 @@ public static void runInteractiveCli(YarnClusterClient
yarnCluster, boolean read
case "quit":
case "stop":
yarnCluster.shutdownCluster();
+ if
(yarnCluster.hasBeenShutdown()) {
+
removeAppState(applicationId);
--- End diff --
Will move the logic to shutdown handler code
> Implement Service-Level Authorization
> -
>
> Key: FLINK-3930
> URL: https://issues.apache.org/jira/browse/FLINK-3930
> Project: Flink
> Issue Type: New Feature
> Components: Security
>Reporter: Eron Wright
>Assignee: Vijay Srinivasaraghavan
> Labels: security
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
> design doc._
> Service-level authorization is the initial authorization mechanism to ensure
> clients (or servers) connecting to the Flink cluster are authorized to do so.
> The purpose is to prevent a cluster from being used by an unauthorized
> user, whether to execute jobs, disrupt cluster functionality, or gain access
> to secrets stored within the cluster.
> Implement service-level authorization as described in the design doc.
> - Introduce a shared secret cookie
> - Enable Akka security cookie
> - Implement data transfer authentication
> - Secure the web dashboard
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15589968#comment-15589968 ] ASF GitHub Bot commented on FLINK-3930: --- Github user vijikarthi commented on a diff in the pull request: https://github.com/apache/flink/pull/2425#discussion_r84174244 --- Diff: flink-yarn/src/main/java/org/apache/flink/yarn/cli/FlinkYarnSessionCli.java --- @@ -108,6 +111,11 @@ private final Options ALL_OPTIONS; + private static final String fileName = "yarn-app.ini"; + private static final String cookieKey = "secureCookie"; --- End diff -- I agree it is not manageable to have multiple files but there are two main reasons for introducing this new file. - Yarn properties file is stored in /tmp location which is accessible to all the users. We want to store the secure cookie in user home location to prevent cookie leak - Current implementation of Yarn properties file does not take multiple applications (Yarn application ID) in to account which is resolved using the ini file implementation. > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security cookie > - Implement data transfer authentication > - Secure the web dashboard -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[
https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15589905#comment-15589905
]
ASF GitHub Bot commented on FLINK-3930:
---
Github user vijikarthi commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r84170399
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyMessage.java
---
@@ -112,9 +129,9 @@ public void write(ChannelHandlerContext ctx, Object
msg, ChannelPromise promise)
// Create the frame length decoder here as it depends on the
encoder
//
- //
+--+--++++
- // | FRAME LENGTH (4) | MAGIC NUMBER (4) | ID (1) || CUSTOM
MESSAGE |
- //
+--+--++++
+ //
+--+--+++++
+ // | FRAME LENGTH (4) | MAGIC NUMBER (4) | COOKIE (4) | ID (1)
|| CUSTOM MESSAGE |
+ //
+--+--+++++
static LengthFieldBasedFrameDecoder createFrameLengthDecoder() {
return new
LengthFieldBasedFrameDecoder(Integer.MAX_VALUE, 0, 4, -4, 4);
--- End diff --
We don't strip the cookie and hence there is no change to adjust the decoder
> Implement Service-Level Authorization
> -
>
> Key: FLINK-3930
> URL: https://issues.apache.org/jira/browse/FLINK-3930
> Project: Flink
> Issue Type: New Feature
> Components: Security
>Reporter: Eron Wright
>Assignee: Vijay Srinivasaraghavan
> Labels: security
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
> design doc._
> Service-level authorization is the initial authorization mechanism to ensure
> clients (or servers) connecting to the Flink cluster are authorized to do so.
> The purpose is to prevent a cluster from being used by an unauthorized
> user, whether to execute jobs, disrupt cluster functionality, or gain access
> to secrets stored within the cluster.
> Implement service-level authorization as described in the design doc.
> - Introduce a shared secret cookie
> - Enable Akka security cookie
> - Implement data transfer authentication
> - Secure the web dashboard
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[
https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15589888#comment-15589888
]
ASF GitHub Bot commented on FLINK-3930:
---
Github user vijikarthi commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r84169656
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyMessage.java
---
@@ -57,24 +61,37 @@
// constructor in order to work with the generic deserializer.
//
- static final int HEADER_LENGTH = 4 + 4 + 1; // frame length (4), magic
number (4), msg ID (1)
+ static final int HEADER_LENGTH = 4 + 4 + 4 + 1; // frame length (4),
magic number (4), Cookie (4), msg ID (1)
static final int MAGIC_NUMBER = 0xBADC0FFE;
+ static final int BUFFER_SIZE = 65536;
+
+ static final Charset DEFAULT_CHARSET = Charset.forName("utf-8");
+
abstract ByteBuf write(ByteBufAllocator allocator) throws Exception;
abstract void readFrom(ByteBuf buffer) throws Exception;
//
- private static ByteBuf allocateBuffer(ByteBufAllocator allocator, byte
id) {
- return allocateBuffer(allocator, id, 0);
+ private static ByteBuf allocateBuffer(ByteBufAllocator allocator, byte
id, String secureCookie) {
+ return allocateBuffer(allocator, id, secureCookie, 0);
}
- private static ByteBuf allocateBuffer(ByteBufAllocator allocator, byte
id, int length) {
+ private static ByteBuf allocateBuffer(ByteBufAllocator allocator, byte
id, String secureCookie, int length) {
+ secureCookie = (secureCookie == null || secureCookie.length()
== 0) ? "": secureCookie;
+ length+=secureCookie.getBytes().length;
final ByteBuf buffer = length != 0 ?
allocator.directBuffer(HEADER_LENGTH + length) : allocator.directBuffer();
buffer.writeInt(HEADER_LENGTH + length);
buffer.writeInt(MAGIC_NUMBER);
+
+ buffer.writeInt(secureCookie.length());
--- End diff --
Good catch. Will change it.
> Implement Service-Level Authorization
> -
>
> Key: FLINK-3930
> URL: https://issues.apache.org/jira/browse/FLINK-3930
> Project: Flink
> Issue Type: New Feature
> Components: Security
>Reporter: Eron Wright
>Assignee: Vijay Srinivasaraghavan
> Labels: security
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
> design doc._
> Service-level authorization is the initial authorization mechanism to ensure
> clients (or servers) connecting to the Flink cluster are authorized to do so.
> The purpose is to prevent a cluster from being used by an unauthorized
> user, whether to execute jobs, disrupt cluster functionality, or gain access
> to secrets stored within the cluster.
> Implement service-level authorization as described in the design doc.
> - Introduce a shared secret cookie
> - Enable Akka security cookie
> - Implement data transfer authentication
> - Secure the web dashboard
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[
https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588388#comment-15588388
]
ASF GitHub Bot commented on FLINK-3930:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r84029858
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/execution/librarycache/BlobLibraryCacheManager.java
---
@@ -199,6 +199,11 @@ public int getBlobServerPort() {
return blobService.getPort();
}
+ public String getSecureCookie() {
+ return blobService.getSecureCookie() == null
+ ? "": blobService.getSecureCookie();
--- End diff --
I feel like this logic should be delegated to the blobService. Also, this
might be clearer:
```java
String secureCookie = blobService.getSecureCookie();
return secureCookie != null ? secureCookie : ""
```
> Implement Service-Level Authorization
> -
>
> Key: FLINK-3930
> URL: https://issues.apache.org/jira/browse/FLINK-3930
> Project: Flink
> Issue Type: New Feature
> Components: Security
>Reporter: Eron Wright
>Assignee: Vijay Srinivasaraghavan
> Labels: security
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
> design doc._
> Service-level authorization is the initial authorization mechanism to ensure
> clients (or servers) connecting to the Flink cluster are authorized to do so.
> The purpose is to prevent a cluster from being used by an unauthorized
> user, whether to execute jobs, disrupt cluster functionality, or gain access
> to secrets stored within the cluster.
> Implement service-level authorization as described in the design doc.
> - Introduce a shared secret cookie
> - Enable Akka security cookie
> - Implement data transfer authentication
> - Secure the web dashboard
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588405#comment-15588405 ] ASF GitHub Bot commented on FLINK-3930: --- Github user mxm commented on a diff in the pull request: https://github.com/apache/flink/pull/2425#discussion_r84039872 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/blob/BlobServerDeleteTest.java --- @@ -18,6 +18,7 @@ package org.apache.flink.runtime.blob; +import org.apache.flink.configuration.ConfigConstants; --- End diff -- Unused import > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security cookie > - Implement data transfer authentication > - Secure the web dashboard -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[
https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588394#comment-15588394
]
ASF GitHub Bot commented on FLINK-3930:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r84033606
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyMessage.java
---
@@ -57,24 +61,37 @@
// constructor in order to work with the generic deserializer.
//
- static final int HEADER_LENGTH = 4 + 4 + 1; // frame length (4), magic
number (4), msg ID (1)
+ static final int HEADER_LENGTH = 4 + 4 + 4 + 1; // frame length (4),
magic number (4), Cookie (4), msg ID (1)
static final int MAGIC_NUMBER = 0xBADC0FFE;
+ static final int BUFFER_SIZE = 65536;
+
+ static final Charset DEFAULT_CHARSET = Charset.forName("utf-8");
+
abstract ByteBuf write(ByteBufAllocator allocator) throws Exception;
abstract void readFrom(ByteBuf buffer) throws Exception;
//
- private static ByteBuf allocateBuffer(ByteBufAllocator allocator, byte
id) {
- return allocateBuffer(allocator, id, 0);
+ private static ByteBuf allocateBuffer(ByteBufAllocator allocator, byte
id, String secureCookie) {
+ return allocateBuffer(allocator, id, secureCookie, 0);
}
- private static ByteBuf allocateBuffer(ByteBufAllocator allocator, byte
id, int length) {
+ private static ByteBuf allocateBuffer(ByteBufAllocator allocator, byte
id, String secureCookie, int length) {
+ secureCookie = (secureCookie == null || secureCookie.length()
== 0) ? "": secureCookie;
+ length+=secureCookie.getBytes().length;
final ByteBuf buffer = length != 0 ?
allocator.directBuffer(HEADER_LENGTH + length) : allocator.directBuffer();
buffer.writeInt(HEADER_LENGTH + length);
buffer.writeInt(MAGIC_NUMBER);
+
+ buffer.writeInt(secureCookie.length());
--- End diff --
Here you're writing `cookie.length()` but early you write calculate with
`secureCookie.getBytes().length`. I think this causes issues with non ASCII
chars.
> Implement Service-Level Authorization
> -
>
> Key: FLINK-3930
> URL: https://issues.apache.org/jira/browse/FLINK-3930
> Project: Flink
> Issue Type: New Feature
> Components: Security
>Reporter: Eron Wright
>Assignee: Vijay Srinivasaraghavan
> Labels: security
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
> design doc._
> Service-level authorization is the initial authorization mechanism to ensure
> clients (or servers) connecting to the Flink cluster are authorized to do so.
> The purpose is to prevent a cluster from being used by an unauthorized
> user, whether to execute jobs, disrupt cluster functionality, or gain access
> to secrets stored within the cluster.
> Implement service-level authorization as described in the design doc.
> - Introduce a shared secret cookie
> - Enable Akka security cookie
> - Implement data transfer authentication
> - Secure the web dashboard
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588380#comment-15588380 ] ASF GitHub Bot commented on FLINK-3930: --- Github user mxm commented on a diff in the pull request: https://github.com/apache/flink/pull/2425#discussion_r84028789 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobServerProtocol.java --- @@ -53,6 +53,9 @@ /** Internal code to identify a reference via jobId as the key */ static final byte JOB_ID_SCOPE = 2; + /** The maximum length of secure cookie. */ + static final int MAX_LENGTH_SECURE_COOKIE = 1024; --- End diff -- Shouldn't this be tied to the buffer size which limits the cookie length? > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security cookie > - Implement data transfer authentication > - Secure the web dashboard -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588386#comment-15588386 ] ASF GitHub Bot commented on FLINK-3930: --- Github user mxm commented on a diff in the pull request: https://github.com/apache/flink/pull/2425#discussion_r84029066 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobUtils.java --- @@ -393,6 +399,24 @@ static void copyFromRecoveryPath(String recoveryPath, File localBlobFile) throws } /** +* Utility method to validate secure cookie from Flink configuration instance +* @throws --- End diff -- Throw description is missing. > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security cookie > - Implement data transfer authentication > - Secure the web dashboard -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[
https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588403#comment-15588403
]
ASF GitHub Bot commented on FLINK-3930:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r84034258
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyMessage.java
---
@@ -131,6 +157,31 @@ protected void decode(ChannelHandlerContext ctx,
ByteBuf msg, List out)
throw new IllegalStateException("Network stream
corrupted: received incorrect magic number.");
}
+ //read cookie based on the cookie length passed
+ int cookieLength = msg.readInt();
+
+ LOG.debug("Cookie Length Read: {}, Source Cookie
Length: {}", cookieLength, secureCookie.length);
+
+ if(cookieLength != secureCookie.length) {
+ String message = "Cookie length does not match
with source cookie. Invalid secure cookie passed.";
+ LOG.error(message);
+ throw new IllegalStateException(message);
--- End diff --
Exceptions are always logged. No need to do the logging here.
> Implement Service-Level Authorization
> -
>
> Key: FLINK-3930
> URL: https://issues.apache.org/jira/browse/FLINK-3930
> Project: Flink
> Issue Type: New Feature
> Components: Security
>Reporter: Eron Wright
>Assignee: Vijay Srinivasaraghavan
> Labels: security
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
> design doc._
> Service-level authorization is the initial authorization mechanism to ensure
> clients (or servers) connecting to the Flink cluster are authorized to do so.
> The purpose is to prevent a cluster from being used by an unauthorized
> user, whether to execute jobs, disrupt cluster functionality, or gain access
> to secrets stored within the cluster.
> Implement service-level authorization as described in the design doc.
> - Introduce a shared secret cookie
> - Enable Akka security cookie
> - Implement data transfer authentication
> - Secure the web dashboard
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588418#comment-15588418 ] ASF GitHub Bot commented on FLINK-3930: --- Github user mxm commented on a diff in the pull request: https://github.com/apache/flink/pull/2425#discussion_r84039893 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/blob/BlobServerPutTest.java --- @@ -18,6 +18,7 @@ package org.apache.flink.runtime.blob; +import org.apache.flink.configuration.ConfigConstants; --- End diff -- Unused import > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security cookie > - Implement data transfer authentication > - Secure the web dashboard -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588370#comment-15588370 ] ASF GitHub Bot commented on FLINK-3930: --- Github user mxm commented on a diff in the pull request: https://github.com/apache/flink/pull/2425#discussion_r84020950 --- Diff: docs/setup/cli.md --- @@ -239,6 +241,7 @@ Action "run" compiles and runs a program. Zookeeper sub-paths for high availability mode Options for yarn-cluster mode: + -k,--cookie Secure cookie to authenticate --- End diff -- Should be removed. > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security cookie > - Implement data transfer authentication > - Secure the web dashboard -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[
https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588387#comment-15588387
]
ASF GitHub Bot commented on FLINK-3930:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r84030215
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyConfig.java
---
@@ -91,6 +94,14 @@ public NettyConfig(
this.config = checkNotNull(config);
+ boolean security =
config.getBoolean(ConfigConstants.SECURITY_ENABLED, false);
+ this.secureCookie =
config.getString(ConfigConstants.SECURITY_COOKIE, "");
+
+ if(security && this.secureCookie == "") {
--- End diff --
Strings have to be compared with the `equals(String other)` method.
> Implement Service-Level Authorization
> -
>
> Key: FLINK-3930
> URL: https://issues.apache.org/jira/browse/FLINK-3930
> Project: Flink
> Issue Type: New Feature
> Components: Security
>Reporter: Eron Wright
>Assignee: Vijay Srinivasaraghavan
> Labels: security
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
> design doc._
> Service-level authorization is the initial authorization mechanism to ensure
> clients (or servers) connecting to the Flink cluster are authorized to do so.
> The purpose is to prevent a cluster from being used by an unauthorized
> user, whether to execute jobs, disrupt cluster functionality, or gain access
> to secrets stored within the cluster.
> Implement service-level authorization as described in the design doc.
> - Introduce a shared secret cookie
> - Enable Akka security cookie
> - Implement data transfer authentication
> - Secure the web dashboard
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588374#comment-15588374 ] ASF GitHub Bot commented on FLINK-3930: --- Github user mxm commented on a diff in the pull request: https://github.com/apache/flink/pull/2425#discussion_r84024718 --- Diff: flink-dist/src/main/resources/flink-conf.yaml --- @@ -172,4 +173,14 @@ jobmanager.web.port: 8081 # Override below configuration to provide custom ZK service name if configured # -# zookeeper.sasl.service-name: zookeeper \ No newline at end of file +# zookeeper.sasl.service-name: zookeeper + +#== +# Service Authorization Configuration (optional configuration) +#== + +# Flag to enable/disable service level authorization (disabled by default) +#security.enabled: false + +#secure random cookie to be used (auto-generated or static if value supplied) +#security.cookie: foo --- End diff -- Could we change `foo` to something like `notsecure` or `changeme`? > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security cookie > - Implement data transfer authentication > - Secure the web dashboard -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[
https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588402#comment-15588402
]
ASF GitHub Bot commented on FLINK-3930:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r84034051
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyMessage.java
---
@@ -123,6 +140,15 @@ static LengthFieldBasedFrameDecoder
createFrameLengthDecoder() {
@ChannelHandler.Sharable
static class NettyMessageDecoder extends
MessageToMessageDecoder {
+ private static final Logger LOG =
LoggerFactory.getLogger(NettyMessageDecoder.class);
+
+ final byte[] secureCookie;
+
+ public NettyMessageDecoder(String secureCookie) {
+ secureCookie = (secureCookie == null ||
secureCookie.length() == 0) ? "": secureCookie;
--- End diff --
String should be checked for non null.
> Implement Service-Level Authorization
> -
>
> Key: FLINK-3930
> URL: https://issues.apache.org/jira/browse/FLINK-3930
> Project: Flink
> Issue Type: New Feature
> Components: Security
>Reporter: Eron Wright
>Assignee: Vijay Srinivasaraghavan
> Labels: security
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
> design doc._
> Service-level authorization is the initial authorization mechanism to ensure
> clients (or servers) connecting to the Flink cluster are authorized to do so.
> The purpose is to prevent a cluster from being used by an unauthorized
> user, whether to execute jobs, disrupt cluster functionality, or gain access
> to secrets stored within the cluster.
> Implement service-level authorization as described in the design doc.
> - Introduce a shared secret cookie
> - Enable Akka security cookie
> - Implement data transfer authentication
> - Secure the web dashboard
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588415#comment-15588415 ] ASF GitHub Bot commented on FLINK-3930: --- Github user mxm commented on a diff in the pull request: https://github.com/apache/flink/pull/2425#discussion_r84041338 --- Diff: flink-yarn/src/main/java/org/apache/flink/yarn/cli/FlinkYarnSessionCli.java --- @@ -108,6 +111,11 @@ private final Options ALL_OPTIONS; + private static final String fileName = "yarn-app.ini"; + private static final String cookieKey = "secureCookie"; + + private final Option SECURE_COOKIE_OPTION; --- End diff -- This option can be retrieved from the main options. No need for yarn specific option. Please remove. > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security cookie > - Implement data transfer authentication > - Secure the web dashboard -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[
https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588411#comment-15588411
]
ASF GitHub Bot commented on FLINK-3930:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r84036664
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/PartitionRequestProtocol.java
---
@@ -30,16 +30,29 @@
private final NettyMessageEncoder messageEncoder = new
NettyMessageEncoder();
- private final NettyMessage.NettyMessageDecoder messageDecoder = new
NettyMessage.NettyMessageDecoder();
+ private final NettyMessage.NettyMessageDecoder serverMessageDecoder;
+
+ private final NettyMessage.NettyMessageDecoder clientMessageDecoder;
private final ResultPartitionProvider partitionProvider;
private final TaskEventDispatcher taskEventDispatcher;
private final NetworkBufferPool networkbufferPool;
+ private final String secureCookie;
- PartitionRequestProtocol(ResultPartitionProvider partitionProvider,
TaskEventDispatcher taskEventDispatcher, NetworkBufferPool networkbufferPool) {
+ PartitionRequestProtocol(ResultPartitionProvider partitionProvider,
TaskEventDispatcher taskEventDispatcher,
+ NetworkBufferPool
networkbufferPool, String secureCookie) {
this.partitionProvider = partitionProvider;
this.taskEventDispatcher = taskEventDispatcher;
this.networkbufferPool = networkbufferPool;
+ this.secureCookie = (secureCookie == null ||
secureCookie.length() == 0) ? "": secureCookie;
+
+ serverMessageDecoder = new
NettyMessage.NettyMessageDecoder(secureCookie);
+
+ /*
+* Client decoder does not validate the secure cookie from
server since
+* the server protocol does not transmit the secure cookie on
the wire
+*/
+ clientMessageDecoder = new
NettyMessage.NettyMessageDecoder(null);
--- End diff --
But it still sends an empty cookie?
> Implement Service-Level Authorization
> -
>
> Key: FLINK-3930
> URL: https://issues.apache.org/jira/browse/FLINK-3930
> Project: Flink
> Issue Type: New Feature
> Components: Security
>Reporter: Eron Wright
>Assignee: Vijay Srinivasaraghavan
> Labels: security
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
> design doc._
> Service-level authorization is the initial authorization mechanism to ensure
> clients (or servers) connecting to the Flink cluster are authorized to do so.
> The purpose is to prevent a cluster from being used by an unauthorized
> user, whether to execute jobs, disrupt cluster functionality, or gain access
> to secrets stored within the cluster.
> Implement service-level authorization as described in the design doc.
> - Introduce a shared secret cookie
> - Enable Akka security cookie
> - Implement data transfer authentication
> - Secure the web dashboard
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588366#comment-15588366 ] ASF GitHub Bot commented on FLINK-3930: --- Github user mxm commented on a diff in the pull request: https://github.com/apache/flink/pull/2425#discussion_r84024166 --- Diff: flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java --- @@ -871,6 +871,19 @@ @Deprecated public static final String ZOOKEEPER_MAX_RETRY_ATTEMPTS = "recovery.zookeeper.client.max-retry-attempts"; + // Secure Cookie Authentication --- + + /** Flag that specify whether service authentication is enabled or not **/ + public static final String SECURITY_ENABLED = "security.enabled"; --- End diff -- When security is enabled, encryption should also be turned on. It probably makes sense to disable encryption for debugging purposes but please make sure it is enabled by default. Please see `SECURITY_SSL_ENABLED`. > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security cookie > - Implement data transfer authentication > - Secure the web dashboard -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[
https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588406#comment-15588406
]
ASF GitHub Bot commented on FLINK-3930:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r84042013
--- Diff:
flink-yarn/src/main/java/org/apache/flink/yarn/cli/FlinkYarnSessionCli.java ---
@@ -703,6 +796,93 @@ public static File
getYarnPropertiesLocation(Configuration conf) {
return new File(propertiesFileLocation, YARN_PROPERTIES_FILE +
currentUser);
}
+ public static void persistAppState(String appId, String cookie) {
--- End diff --
Please use the Yarn properties file for that.
> Implement Service-Level Authorization
> -
>
> Key: FLINK-3930
> URL: https://issues.apache.org/jira/browse/FLINK-3930
> Project: Flink
> Issue Type: New Feature
> Components: Security
>Reporter: Eron Wright
>Assignee: Vijay Srinivasaraghavan
> Labels: security
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
> design doc._
> Service-level authorization is the initial authorization mechanism to ensure
> clients (or servers) connecting to the Flink cluster are authorized to do so.
> The purpose is to prevent a cluster from being used by an unauthorized
> user, whether to execute jobs, disrupt cluster functionality, or gain access
> to secrets stored within the cluster.
> Implement service-level authorization as described in the design doc.
> - Introduce a shared secret cookie
> - Enable Akka security cookie
> - Implement data transfer authentication
> - Secure the web dashboard
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588385#comment-15588385 ] ASF GitHub Bot commented on FLINK-3930: --- Github user mxm commented on a diff in the pull request: https://github.com/apache/flink/pull/2425#discussion_r84032194 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyMessage.java --- @@ -57,24 +61,37 @@ // constructor in order to work with the generic deserializer. // - static final int HEADER_LENGTH = 4 + 4 + 1; // frame length (4), magic number (4), msg ID (1) + static final int HEADER_LENGTH = 4 + 4 + 4 + 1; // frame length (4), magic number (4), Cookie (4), msg ID (1) --- End diff -- 4 bytes for the cookie or cookie **length**? How is this in line with the size restrictions assumed in other places? > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security cookie > - Implement data transfer authentication > - Secure the web dashboard -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[
https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588400#comment-15588400
]
ASF GitHub Bot commented on FLINK-3930:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r84035347
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyMessage.java
---
@@ -424,13 +481,17 @@ public String toString() {
ResultPartitionID partitionId;
+ String secureCookie = "";
+
public TaskEventRequest() {
}
- TaskEventRequest(TaskEvent event, ResultPartitionID
partitionId, InputChannelID receiverId) {
+ TaskEventRequest(TaskEvent event, ResultPartitionID
partitionId, InputChannelID receiverId,
+ String secureCookie) {
this.event = event;
this.receiverId = receiverId;
this.partitionId = partitionId;
+ this.secureCookie = (secureCookie == null ||
secureCookie.length() == 0) ? "": secureCookie;
--- End diff --
Please pass a proper non-null secureCookie instead.
> Implement Service-Level Authorization
> -
>
> Key: FLINK-3930
> URL: https://issues.apache.org/jira/browse/FLINK-3930
> Project: Flink
> Issue Type: New Feature
> Components: Security
>Reporter: Eron Wright
>Assignee: Vijay Srinivasaraghavan
> Labels: security
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
> design doc._
> Service-level authorization is the initial authorization mechanism to ensure
> clients (or servers) connecting to the Flink cluster are authorized to do so.
> The purpose is to prevent a cluster from being used by an unauthorized
> user, whether to execute jobs, disrupt cluster functionality, or gain access
> to secrets stored within the cluster.
> Implement service-level authorization as described in the design doc.
> - Introduce a shared secret cookie
> - Enable Akka security cookie
> - Implement data transfer authentication
> - Secure the web dashboard
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588371#comment-15588371 ] ASF GitHub Bot commented on FLINK-3930: --- Github user mxm commented on a diff in the pull request: https://github.com/apache/flink/pull/2425#discussion_r84024238 --- Diff: flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java --- @@ -871,6 +871,19 @@ @Deprecated public static final String ZOOKEEPER_MAX_RETRY_ATTEMPTS = "recovery.zookeeper.client.max-retry-attempts"; + // Secure Cookie Authentication --- --- End diff -- Could we move all secure related settings to the same section in ConfigConstants? > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security cookie > - Implement data transfer authentication > - Secure the web dashboard -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[
https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588381#comment-15588381
]
ASF GitHub Bot commented on FLINK-3930:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r84029158
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobUtils.java ---
@@ -393,6 +399,24 @@ static void copyFromRecoveryPath(String recoveryPath,
File localBlobFile) throws
}
/**
+* Utility method to validate secure cookie from Flink configuration
instance
+* @throws
+*/
+ public static String validateAndGetSecureCookie(Configuration
configuration) {
+ String secureCookie = null;
+ if(configuration.getBoolean(ConfigConstants.SECURITY_ENABLED,
DEFAULT_SECURITY_ENABLED) == true) {
+ secureCookie =
configuration.getString(ConfigConstants.SECURITY_COOKIE, null);
+ if(secureCookie == null) {
+ String message = "Missing " +
ConfigConstants.SECURITY_COOKIE +
+ " configuration in Flink
configuration file";
+ LOG.error(message);
+ throw new RuntimeException(message);
--- End diff --
- Exceptions are logged anyways.
- This should be `IllegalConfigurationException`
> Implement Service-Level Authorization
> -
>
> Key: FLINK-3930
> URL: https://issues.apache.org/jira/browse/FLINK-3930
> Project: Flink
> Issue Type: New Feature
> Components: Security
>Reporter: Eron Wright
>Assignee: Vijay Srinivasaraghavan
> Labels: security
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
> design doc._
> Service-level authorization is the initial authorization mechanism to ensure
> clients (or servers) connecting to the Flink cluster are authorized to do so.
> The purpose is to prevent a cluster from being used by an unauthorized
> user, whether to execute jobs, disrupt cluster functionality, or gain access
> to secrets stored within the cluster.
> Implement service-level authorization as described in the design doc.
> - Introduce a shared secret cookie
> - Enable Akka security cookie
> - Implement data transfer authentication
> - Secure the web dashboard
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[
https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588422#comment-15588422
]
ASF GitHub Bot commented on FLINK-3930:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r84035560
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyMessage.java
---
@@ -527,13 +591,17 @@ void readFrom(ByteBuf buffer) throws Exception {
static class CloseRequest extends NettyMessage {
private static final byte ID = 5;
+ String secureCookie = "";
+
+ public CloseRequest() {}
- public CloseRequest() {
+ public CloseRequest(String secureCookie) {
+ this.secureCookie = (secureCookie == null) ? "":
secureCookie;
--- End diff --
Please no checks like this. You can use
`Preconditions.checkNotNull(secureCookie)`.
> Implement Service-Level Authorization
> -
>
> Key: FLINK-3930
> URL: https://issues.apache.org/jira/browse/FLINK-3930
> Project: Flink
> Issue Type: New Feature
> Components: Security
>Reporter: Eron Wright
>Assignee: Vijay Srinivasaraghavan
> Labels: security
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
> design doc._
> Service-level authorization is the initial authorization mechanism to ensure
> clients (or servers) connecting to the Flink cluster are authorized to do so.
> The purpose is to prevent a cluster from being used by an unauthorized
> user, whether to execute jobs, disrupt cluster functionality, or gain access
> to secrets stored within the cluster.
> Implement service-level authorization as described in the design doc.
> - Introduce a shared secret cookie
> - Enable Akka security cookie
> - Implement data transfer authentication
> - Secure the web dashboard
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[
https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588384#comment-15588384
]
ASF GitHub Bot commented on FLINK-3930:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r84030447
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobClient.java ---
@@ -88,6 +93,19 @@
*/
public BlobClient(InetSocketAddress serverAddress, Configuration
clientConfig) throws IOException {
+ this.socket = new Socket();
+
+ if(clientConfig != null) {
+ boolean securityEnabled =
clientConfig.getBoolean(ConfigConstants.SECURITY_ENABLED,
+
ConfigConstants.DEFAULT_SECURITY_ENABLED);
+
+ this.secureCookie =
clientConfig.getString(ConfigConstants.SECURITY_COOKIE, "");
+
+ if (securityEnabled && this.secureCookie == "") {
--- End diff --
Strings have to be compared with the `equals(String other)` method.
> Implement Service-Level Authorization
> -
>
> Key: FLINK-3930
> URL: https://issues.apache.org/jira/browse/FLINK-3930
> Project: Flink
> Issue Type: New Feature
> Components: Security
>Reporter: Eron Wright
>Assignee: Vijay Srinivasaraghavan
> Labels: security
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
> design doc._
> Service-level authorization is the initial authorization mechanism to ensure
> clients (or servers) connecting to the Flink cluster are authorized to do so.
> The purpose is to prevent a cluster from being used by an unauthorized
> user, whether to execute jobs, disrupt cluster functionality, or gain access
> to secrets stored within the cluster.
> Implement service-level authorization as described in the design doc.
> - Introduce a shared secret cookie
> - Enable Akka security cookie
> - Implement data transfer authentication
> - Secure the web dashboard
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[
https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588373#comment-15588373
]
ASF GitHub Bot commented on FLINK-3930:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r84025149
--- Diff:
flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/HttpRequestHandler.java
---
@@ -99,7 +110,43 @@ public void channelRead0(ChannelHandlerContext ctx,
HttpObject msg) {
currentDecoder.destroy();
currentDecoder = null;
}
-
+
+ if(secureCookie != null) {
--- End diff --
I agree with Robert, we need a to check the config whether security is
enabled. This requires passing the config here.
> Implement Service-Level Authorization
> -
>
> Key: FLINK-3930
> URL: https://issues.apache.org/jira/browse/FLINK-3930
> Project: Flink
> Issue Type: New Feature
> Components: Security
>Reporter: Eron Wright
>Assignee: Vijay Srinivasaraghavan
> Labels: security
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
> design doc._
> Service-level authorization is the initial authorization mechanism to ensure
> clients (or servers) connecting to the Flink cluster are authorized to do so.
> The purpose is to prevent a cluster from being used by an unauthorized
> user, whether to execute jobs, disrupt cluster functionality, or gain access
> to secrets stored within the cluster.
> Implement service-level authorization as described in the design doc.
> - Introduce a shared secret cookie
> - Enable Akka security cookie
> - Implement data transfer authentication
> - Secure the web dashboard
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[
https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588378#comment-15588378
]
ASF GitHub Bot commented on FLINK-3930:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r84027778
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobClient.java ---
@@ -725,7 +755,21 @@ else if (response != RETURN_OKAY) {
Object msg =
JobManagerMessages.getRequestBlobManagerPort();
Future futureBlobPort = jobManager.ask(msg,
askTimeout);
+ Object secureCookieMsg =
JobManagerMessages.getRequestBlobManagerSecureCookie();
+ Future futureSecureCookie =
jobManager.ask(secureCookieMsg, askTimeout);
+
try {
+ String secureCookie = null;
+
+ Object cookie =
Await.result(futureSecureCookie, askTimeout);
+ if(cookie instanceof String) {
+ secureCookie = (String) cookie;
+ }
--- End diff --
We are transferring the cookie here from the JobManager? That should never
be the case. The client has to provide the cookie, otherwise a client must not
be able to communicate with the JobManager.
> Implement Service-Level Authorization
> -
>
> Key: FLINK-3930
> URL: https://issues.apache.org/jira/browse/FLINK-3930
> Project: Flink
> Issue Type: New Feature
> Components: Security
>Reporter: Eron Wright
>Assignee: Vijay Srinivasaraghavan
> Labels: security
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
> design doc._
> Service-level authorization is the initial authorization mechanism to ensure
> clients (or servers) connecting to the Flink cluster are authorized to do so.
> The purpose is to prevent a cluster from being used by an unauthorized
> user, whether to execute jobs, disrupt cluster functionality, or gain access
> to secrets stored within the cluster.
> Implement service-level authorization as described in the design doc.
> - Introduce a shared secret cookie
> - Enable Akka security cookie
> - Implement data transfer authentication
> - Secure the web dashboard
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588420#comment-15588420 ] ASF GitHub Bot commented on FLINK-3930: --- Github user mxm commented on a diff in the pull request: https://github.com/apache/flink/pull/2425#discussion_r84039884 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/blob/BlobServerGetTest.java --- @@ -18,6 +18,7 @@ package org.apache.flink.runtime.blob; +import org.apache.flink.configuration.ConfigConstants; --- End diff -- Unused import > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security cookie > - Implement data transfer authentication > - Secure the web dashboard -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[
https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588419#comment-15588419
]
ASF GitHub Bot commented on FLINK-3930:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r84036090
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/PartitionRequestProtocol.java
---
@@ -30,16 +30,29 @@
private final NettyMessageEncoder messageEncoder = new
NettyMessageEncoder();
- private final NettyMessage.NettyMessageDecoder messageDecoder = new
NettyMessage.NettyMessageDecoder();
+ private final NettyMessage.NettyMessageDecoder serverMessageDecoder;
+
+ private final NettyMessage.NettyMessageDecoder clientMessageDecoder;
private final ResultPartitionProvider partitionProvider;
private final TaskEventDispatcher taskEventDispatcher;
private final NetworkBufferPool networkbufferPool;
+ private final String secureCookie;
- PartitionRequestProtocol(ResultPartitionProvider partitionProvider,
TaskEventDispatcher taskEventDispatcher, NetworkBufferPool networkbufferPool) {
+ PartitionRequestProtocol(ResultPartitionProvider partitionProvider,
TaskEventDispatcher taskEventDispatcher,
+ NetworkBufferPool
networkbufferPool, String secureCookie) {
this.partitionProvider = partitionProvider;
this.taskEventDispatcher = taskEventDispatcher;
this.networkbufferPool = networkbufferPool;
+ this.secureCookie = (secureCookie == null ||
secureCookie.length() == 0) ? "": secureCookie;
--- End diff --
See above comments about this.
> Implement Service-Level Authorization
> -
>
> Key: FLINK-3930
> URL: https://issues.apache.org/jira/browse/FLINK-3930
> Project: Flink
> Issue Type: New Feature
> Components: Security
>Reporter: Eron Wright
>Assignee: Vijay Srinivasaraghavan
> Labels: security
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
> design doc._
> Service-level authorization is the initial authorization mechanism to ensure
> clients (or servers) connecting to the Flink cluster are authorized to do so.
> The purpose is to prevent a cluster from being used by an unauthorized
> user, whether to execute jobs, disrupt cluster functionality, or gain access
> to secrets stored within the cluster.
> Implement service-level authorization as described in the design doc.
> - Introduce a shared secret cookie
> - Enable Akka security cookie
> - Implement data transfer authentication
> - Secure the web dashboard
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[
https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588376#comment-15588376
]
ASF GitHub Bot commented on FLINK-3930:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r84028705
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobServerConnection.java
---
@@ -463,6 +482,39 @@ private static String readKey(byte[] buf, InputStream
inputStream) throws IOExce
return new String(buf, 0, keyLength, BlobUtils.DEFAULT_CHARSET);
}
+
+ /**
+* Reads secure cookie from the given input stream.
+*
+* @param inputStream
+*the input stream to read the secure cookie from
+* @param keyLength
+*buffer length to read
+* @return
+* @throws IOException
+* thrown if an I/O error occurs while reading the secure
cookie data from the input stream
+*/
+
+ private void validateSecureCookie(InputStream inputStream, int
keyLength) throws IOException {
+
+ if (keyLength > MAX_LENGTH_SECURE_COOKIE) {
+ throw new IOException("Unexpected secure cookie length
" + keyLength);
+ }
+
+ final byte[] buffer = new byte[BUFFER_SIZE];
+
+ readFully(inputStream, buffer, 0, keyLength, "SecureCookie");
+
+ final String cookie = new String(buffer, 0, keyLength,
BlobUtils.DEFAULT_CHARSET);
+
+ if(blobServer.isSecurityEnabled()) {
+ if(StringUtils.isBlank(cookie) ||
!cookie.equals(blobServer.getSecureCookie())) {
+ LOG.error("Missing valid secure cookie");
+ throw new IOException("Missing valid secure
cookie");
--- End diff --
Exceptions will be logged anyways, you can remove the logging.
> Implement Service-Level Authorization
> -
>
> Key: FLINK-3930
> URL: https://issues.apache.org/jira/browse/FLINK-3930
> Project: Flink
> Issue Type: New Feature
> Components: Security
>Reporter: Eron Wright
>Assignee: Vijay Srinivasaraghavan
> Labels: security
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
> design doc._
> Service-level authorization is the initial authorization mechanism to ensure
> clients (or servers) connecting to the Flink cluster are authorized to do so.
> The purpose is to prevent a cluster from being used by an unauthorized
> user, whether to execute jobs, disrupt cluster functionality, or gain access
> to secrets stored within the cluster.
> Implement service-level authorization as described in the design doc.
> - Introduce a shared secret cookie
> - Enable Akka security cookie
> - Implement data transfer authentication
> - Secure the web dashboard
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[
https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588416#comment-15588416
]
ASF GitHub Bot commented on FLINK-3930:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r84039201
--- Diff:
flink-runtime/src/main/scala/org/apache/flink/runtime/akka/AkkaUtils.scala ---
@@ -389,7 +396,22 @@ object AkkaUtils {
""
}
-ConfigFactory.parseString(configString + hostnameConfigString +
sslConfigString).resolve()
+val cookieConfigString = if(securityEnabled){
+ s"""
+ |akka {
+ | remote {
+ |require-cookie = $requireCookie
+ |secure-cookie = "$secureCookie"
+ | }
+ |}
+ """.stripMargin
+}else{
--- End diff --
No need for the else block. We set set `require-cookie` to `off` in case it
shouldn't be used
> Implement Service-Level Authorization
> -
>
> Key: FLINK-3930
> URL: https://issues.apache.org/jira/browse/FLINK-3930
> Project: Flink
> Issue Type: New Feature
> Components: Security
>Reporter: Eron Wright
>Assignee: Vijay Srinivasaraghavan
> Labels: security
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
> design doc._
> Service-level authorization is the initial authorization mechanism to ensure
> clients (or servers) connecting to the Flink cluster are authorized to do so.
> The purpose is to prevent a cluster from being used by an unauthorized
> user, whether to execute jobs, disrupt cluster functionality, or gain access
> to secrets stored within the cluster.
> Implement service-level authorization as described in the design doc.
> - Introduce a shared secret cookie
> - Enable Akka security cookie
> - Implement data transfer authentication
> - Secure the web dashboard
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[
https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588379#comment-15588379
]
ASF GitHub Bot commented on FLINK-3930:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r84028453
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobServerConnection.java
---
@@ -463,6 +482,39 @@ private static String readKey(byte[] buf, InputStream
inputStream) throws IOExce
return new String(buf, 0, keyLength, BlobUtils.DEFAULT_CHARSET);
}
+
+ /**
+* Reads secure cookie from the given input stream.
+*
+* @param inputStream
+*the input stream to read the secure cookie from
+* @param keyLength
+*buffer length to read
+* @return
+* @throws IOException
+* thrown if an I/O error occurs while reading the secure
cookie data from the input stream
+*/
+
+ private void validateSecureCookie(InputStream inputStream, int
keyLength) throws IOException {
+
+ if (keyLength > MAX_LENGTH_SECURE_COOKIE) {
+ throw new IOException("Unexpected secure cookie length
" + keyLength);
+ }
+
+ final byte[] buffer = new byte[BUFFER_SIZE];
--- End diff --
Does this oppose an upper limit on the cookie size?
> Implement Service-Level Authorization
> -
>
> Key: FLINK-3930
> URL: https://issues.apache.org/jira/browse/FLINK-3930
> Project: Flink
> Issue Type: New Feature
> Components: Security
>Reporter: Eron Wright
>Assignee: Vijay Srinivasaraghavan
> Labels: security
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
> design doc._
> Service-level authorization is the initial authorization mechanism to ensure
> clients (or servers) connecting to the Flink cluster are authorized to do so.
> The purpose is to prevent a cluster from being used by an unauthorized
> user, whether to execute jobs, disrupt cluster functionality, or gain access
> to secrets stored within the cluster.
> Implement service-level authorization as described in the design doc.
> - Introduce a shared secret cookie
> - Enable Akka security cookie
> - Implement data transfer authentication
> - Secure the web dashboard
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[
https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588408#comment-15588408
]
ASF GitHub Bot commented on FLINK-3930:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r84035615
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyMessage.java
---
@@ -527,13 +591,17 @@ void readFrom(ByteBuf buffer) throws Exception {
static class CloseRequest extends NettyMessage {
private static final byte ID = 5;
+ String secureCookie = "";
--- End diff --
Initialization can never used.
> Implement Service-Level Authorization
> -
>
> Key: FLINK-3930
> URL: https://issues.apache.org/jira/browse/FLINK-3930
> Project: Flink
> Issue Type: New Feature
> Components: Security
>Reporter: Eron Wright
>Assignee: Vijay Srinivasaraghavan
> Labels: security
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
> design doc._
> Service-level authorization is the initial authorization mechanism to ensure
> clients (or servers) connecting to the Flink cluster are authorized to do so.
> The purpose is to prevent a cluster from being used by an unauthorized
> user, whether to execute jobs, disrupt cluster functionality, or gain access
> to secrets stored within the cluster.
> Implement service-level authorization as described in the design doc.
> - Introduce a shared secret cookie
> - Enable Akka security cookie
> - Implement data transfer authentication
> - Secure the web dashboard
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[
https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588397#comment-15588397
]
ASF GitHub Bot commented on FLINK-3930:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r84035178
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyMessage.java
---
@@ -369,21 +423,24 @@ void readFrom(ByteBuf buffer) throws Exception {
InputChannelID receiverId;
+ String secureCookie = "";
--- End diff --
The `""` initialization is always overridden, please remove.
> Implement Service-Level Authorization
> -
>
> Key: FLINK-3930
> URL: https://issues.apache.org/jira/browse/FLINK-3930
> Project: Flink
> Issue Type: New Feature
> Components: Security
>Reporter: Eron Wright
>Assignee: Vijay Srinivasaraghavan
> Labels: security
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
> design doc._
> Service-level authorization is the initial authorization mechanism to ensure
> clients (or servers) connecting to the Flink cluster are authorized to do so.
> The purpose is to prevent a cluster from being used by an unauthorized
> user, whether to execute jobs, disrupt cluster functionality, or gain access
> to secrets stored within the cluster.
> Implement service-level authorization as described in the design doc.
> - Introduce a shared secret cookie
> - Enable Akka security cookie
> - Implement data transfer authentication
> - Secure the web dashboard
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588421#comment-15588421 ] ASF GitHub Bot commented on FLINK-3930: --- Github user mxm commented on a diff in the pull request: https://github.com/apache/flink/pull/2425#discussion_r84039769 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/blob/BlobClientTest.java --- @@ -48,19 +49,18 @@ private static final int TEST_BUFFER_SIZE = 17 * 1000; /** The instance of the BLOB server used during the tests. */ - private static BlobServer BLOB_SERVER; + protected static BlobServer BLOB_SERVER; - /** The blob service client and server configuration */ - private static Configuration blobServiceConfig; --- End diff -- These renamings are not necessary. Please revert. > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security cookie > - Implement data transfer authentication > - Secure the web dashboard -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[
https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588368#comment-15588368
]
ASF GitHub Bot commented on FLINK-3930:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r84021839
--- Diff:
flink-clients/src/main/java/org/apache/flink/client/cli/CliFrontendParser.java
---
@@ -90,6 +90,9 @@
"directory is optional. If no directory is specified,
the configured default " +
"directory (" + ConfigConstants.SAVEPOINT_DIRECTORY_KEY
+ ") is used.");
+ static final Option SECURE_COOKIE_OPTION = new Option("k", "cookie",
true,
+ "Secure cookie to authenticate");
--- End diff --
Description string should be updated (see above).
> Implement Service-Level Authorization
> -
>
> Key: FLINK-3930
> URL: https://issues.apache.org/jira/browse/FLINK-3930
> Project: Flink
> Issue Type: New Feature
> Components: Security
>Reporter: Eron Wright
>Assignee: Vijay Srinivasaraghavan
> Labels: security
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
> design doc._
> Service-level authorization is the initial authorization mechanism to ensure
> clients (or servers) connecting to the Flink cluster are authorized to do so.
> The purpose is to prevent a cluster from being used by an unauthorized
> user, whether to execute jobs, disrupt cluster functionality, or gain access
> to secrets stored within the cluster.
> Implement service-level authorization as described in the design doc.
> - Introduce a shared secret cookie
> - Enable Akka security cookie
> - Implement data transfer authentication
> - Secure the web dashboard
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[
https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588392#comment-15588392
]
ASF GitHub Bot commented on FLINK-3930:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r84034738
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyMessage.java
---
@@ -292,9 +344,10 @@ ByteBuf write(ByteBufAllocator allocator) throws
IOException {
ByteBuf result = null;
ObjectOutputStream oos = null;
+ final String NO_SECURE_COOKIE= "";
--- End diff --
Please consolidate the variable with same definition above.
> Implement Service-Level Authorization
> -
>
> Key: FLINK-3930
> URL: https://issues.apache.org/jira/browse/FLINK-3930
> Project: Flink
> Issue Type: New Feature
> Components: Security
>Reporter: Eron Wright
>Assignee: Vijay Srinivasaraghavan
> Labels: security
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
> design doc._
> Service-level authorization is the initial authorization mechanism to ensure
> clients (or servers) connecting to the Flink cluster are authorized to do so.
> The purpose is to prevent a cluster from being used by an unauthorized
> user, whether to execute jobs, disrupt cluster functionality, or gain access
> to secrets stored within the cluster.
> Implement service-level authorization as described in the design doc.
> - Introduce a shared secret cookie
> - Enable Akka security cookie
> - Implement data transfer authentication
> - Secure the web dashboard
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588367#comment-15588367 ] ASF GitHub Bot commented on FLINK-3930: --- Github user mxm commented on a diff in the pull request: https://github.com/apache/flink/pull/2425#discussion_r84020927 --- Diff: docs/setup/cli.md --- @@ -239,6 +241,7 @@ Action "run" compiles and runs a program. Zookeeper sub-paths for high availability mode Options for yarn-cluster mode: + -k,--cookie Secure cookie to authenticate --- End diff -- Why is this added here again? This option is already present in the general options. > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security cookie > - Implement data transfer authentication > - Secure the web dashboard -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[
https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588393#comment-15588393
]
ASF GitHub Bot commented on FLINK-3930:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r84033402
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyMessage.java
---
@@ -57,24 +61,37 @@
// constructor in order to work with the generic deserializer.
//
- static final int HEADER_LENGTH = 4 + 4 + 1; // frame length (4), magic
number (4), msg ID (1)
+ static final int HEADER_LENGTH = 4 + 4 + 4 + 1; // frame length (4),
magic number (4), Cookie (4), msg ID (1)
static final int MAGIC_NUMBER = 0xBADC0FFE;
+ static final int BUFFER_SIZE = 65536;
+
+ static final Charset DEFAULT_CHARSET = Charset.forName("utf-8");
+
abstract ByteBuf write(ByteBufAllocator allocator) throws Exception;
abstract void readFrom(ByteBuf buffer) throws Exception;
//
- private static ByteBuf allocateBuffer(ByteBufAllocator allocator, byte
id) {
- return allocateBuffer(allocator, id, 0);
+ private static ByteBuf allocateBuffer(ByteBufAllocator allocator, byte
id, String secureCookie) {
+ return allocateBuffer(allocator, id, secureCookie, 0);
}
- private static ByteBuf allocateBuffer(ByteBufAllocator allocator, byte
id, int length) {
+ private static ByteBuf allocateBuffer(ByteBufAllocator allocator, byte
id, String secureCookie, int length) {
+ secureCookie = (secureCookie == null || secureCookie.length()
== 0) ? "": secureCookie;
+ length+=secureCookie.getBytes().length;
--- End diff --
This seems inefficient, always getting the byte array. The length should be
supplied and just be calculated once.
> Implement Service-Level Authorization
> -
>
> Key: FLINK-3930
> URL: https://issues.apache.org/jira/browse/FLINK-3930
> Project: Flink
> Issue Type: New Feature
> Components: Security
>Reporter: Eron Wright
>Assignee: Vijay Srinivasaraghavan
> Labels: security
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
> design doc._
> Service-level authorization is the initial authorization mechanism to ensure
> clients (or servers) connecting to the Flink cluster are authorized to do so.
> The purpose is to prevent a cluster from being used by an unauthorized
> user, whether to execute jobs, disrupt cluster functionality, or gain access
> to secrets stored within the cluster.
> Implement service-level authorization as described in the design doc.
> - Introduce a shared secret cookie
> - Enable Akka security cookie
> - Implement data transfer authentication
> - Secure the web dashboard
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[
https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588417#comment-15588417
]
ASF GitHub Bot commented on FLINK-3930:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r84041744
--- Diff:
flink-yarn/src/main/java/org/apache/flink/yarn/cli/FlinkYarnSessionCli.java ---
@@ -442,8 +453,10 @@ public static void runInteractiveCli(YarnClusterClient
yarnCluster, boolean read
case "quit":
case "stop":
yarnCluster.shutdownCluster();
+ if
(yarnCluster.hasBeenShutdown()) {
+
removeAppState(applicationId);
--- End diff --
Cleanup is handled in the shutdown method. Please remove this. The cluster
client is always in state running if the user calls stop.
> Implement Service-Level Authorization
> -
>
> Key: FLINK-3930
> URL: https://issues.apache.org/jira/browse/FLINK-3930
> Project: Flink
> Issue Type: New Feature
> Components: Security
>Reporter: Eron Wright
>Assignee: Vijay Srinivasaraghavan
> Labels: security
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
> design doc._
> Service-level authorization is the initial authorization mechanism to ensure
> clients (or servers) connecting to the Flink cluster are authorized to do so.
> The purpose is to prevent a cluster from being used by an unauthorized
> user, whether to execute jobs, disrupt cluster functionality, or gain access
> to secrets stored within the cluster.
> Implement service-level authorization as described in the design doc.
> - Introduce a shared secret cookie
> - Enable Akka security cookie
> - Implement data transfer authentication
> - Secure the web dashboard
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[
https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588414#comment-15588414
]
ASF GitHub Bot commented on FLINK-3930:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r84037462
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/PartitionRequestQueue.java
---
@@ -215,7 +222,8 @@ private void handleException(Channel channel, Throwable
cause) throws IOExceptio
releaseAllResources();
if (channel.isActive()) {
- channel.writeAndFlush(new
NettyMessage.ErrorResponse(cause)).addListener(ChannelFutureListener.CLOSE);
+ channel.writeAndFlush(new
NettyMessage.ErrorResponse(cause))
+
.addListener(ChannelFutureListener.CLOSE);
--- End diff --
just reformatting, no code changes (but i still have to check).
> Implement Service-Level Authorization
> -
>
> Key: FLINK-3930
> URL: https://issues.apache.org/jira/browse/FLINK-3930
> Project: Flink
> Issue Type: New Feature
> Components: Security
>Reporter: Eron Wright
>Assignee: Vijay Srinivasaraghavan
> Labels: security
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
> design doc._
> Service-level authorization is the initial authorization mechanism to ensure
> clients (or servers) connecting to the Flink cluster are authorized to do so.
> The purpose is to prevent a cluster from being used by an unauthorized
> user, whether to execute jobs, disrupt cluster functionality, or gain access
> to secrets stored within the cluster.
> Implement service-level authorization as described in the design doc.
> - Introduce a shared secret cookie
> - Enable Akka security cookie
> - Implement data transfer authentication
> - Secure the web dashboard
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588369#comment-15588369 ] ASF GitHub Bot commented on FLINK-3930: --- Github user mxm commented on a diff in the pull request: https://github.com/apache/flink/pull/2425#discussion_r84020724 --- Diff: docs/setup/cli.md --- @@ -217,6 +217,8 @@ Action "run" compiles and runs a program. java.net.URLClassLoader}. -d,--detached If present, runs the job in detached mode + -k,--cookie Secure cookie to +authenticate --- End diff -- The description could be a bit more elaborate. For example, > String to authorize Akka-based RPC communication > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security cookie > - Implement data transfer authentication > - Secure the web dashboard -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[
https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588398#comment-15588398
]
ASF GitHub Bot commented on FLINK-3930:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r84034660
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyMessage.java
---
@@ -226,8 +277,9 @@ ByteBuf write(ByteBufAllocator allocator) throws
IOException {
int length = 16 + 4 + 1 + 4 + buffer.getSize();
ByteBuf result = null;
+ final String NO_SECURE_COOKIE= "";
--- End diff --
space after variable name
> Implement Service-Level Authorization
> -
>
> Key: FLINK-3930
> URL: https://issues.apache.org/jira/browse/FLINK-3930
> Project: Flink
> Issue Type: New Feature
> Components: Security
>Reporter: Eron Wright
>Assignee: Vijay Srinivasaraghavan
> Labels: security
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
> design doc._
> Service-level authorization is the initial authorization mechanism to ensure
> clients (or servers) connecting to the Flink cluster are authorized to do so.
> The purpose is to prevent a cluster from being used by an unauthorized
> user, whether to execute jobs, disrupt cluster functionality, or gain access
> to secrets stored within the cluster.
> Implement service-level authorization as described in the design doc.
> - Introduce a shared secret cookie
> - Enable Akka security cookie
> - Implement data transfer authentication
> - Secure the web dashboard
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[
https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588395#comment-15588395
]
ASF GitHub Bot commented on FLINK-3930:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r84033982
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyMessage.java
---
@@ -112,9 +129,9 @@ public void write(ChannelHandlerContext ctx, Object
msg, ChannelPromise promise)
// Create the frame length decoder here as it depends on the
encoder
//
- //
+--+--++++
- // | FRAME LENGTH (4) | MAGIC NUMBER (4) | ID (1) || CUSTOM
MESSAGE |
- //
+--+--++++
+ //
+--+--+++++
+ // | FRAME LENGTH (4) | MAGIC NUMBER (4) | COOKIE (4) | ID (1)
|| CUSTOM MESSAGE |
+ //
+--+--+++++
static LengthFieldBasedFrameDecoder createFrameLengthDecoder() {
return new
LengthFieldBasedFrameDecoder(Integer.MAX_VALUE, 0, 4, -4, 4);
--- End diff --
You didn't adjust the frame decoder. How is this supposed to work?
> Implement Service-Level Authorization
> -
>
> Key: FLINK-3930
> URL: https://issues.apache.org/jira/browse/FLINK-3930
> Project: Flink
> Issue Type: New Feature
> Components: Security
>Reporter: Eron Wright
>Assignee: Vijay Srinivasaraghavan
> Labels: security
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
> design doc._
> Service-level authorization is the initial authorization mechanism to ensure
> clients (or servers) connecting to the Flink cluster are authorized to do so.
> The purpose is to prevent a cluster from being used by an unauthorized
> user, whether to execute jobs, disrupt cluster functionality, or gain access
> to secrets stored within the cluster.
> Implement service-level authorization as described in the design doc.
> - Introduce a shared secret cookie
> - Enable Akka security cookie
> - Implement data transfer authentication
> - Secure the web dashboard
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588375#comment-15588375 ] ASF GitHub Bot commented on FLINK-3930: --- Github user mxm commented on a diff in the pull request: https://github.com/apache/flink/pull/2425#discussion_r84025817 --- Diff: flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/WebRuntimeMonitor.java --- @@ -148,7 +148,7 @@ private MetricFetcher metricFetcher; public WebRuntimeMonitor( - Configuration config, + final Configuration config, --- End diff -- `final` modifier seems redundant here. > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security cookie > - Implement data transfer authentication > - Secure the web dashboard -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[
https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588407#comment-15588407
]
ASF GitHub Bot commented on FLINK-3930:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r84041916
--- Diff:
flink-yarn/src/main/java/org/apache/flink/yarn/cli/FlinkYarnSessionCli.java ---
@@ -607,6 +680,18 @@ public int run(String[] args) {
return 1;
}
+ boolean securityEnabled =
yarnDescriptor.getFlinkConfiguration()
+
.getBoolean(ConfigConstants.SECURITY_ENABLED,
+
DEFAULT_SECURITY_ENABLED);
+ LOG.debug("Security Enabled ? {}", securityEnabled);
+
+ //override cookie configuration if supplied through CLI
+ if(securityEnabled && secureCookieArg != null) {
+ LOG.debug("Secure cookie is provided as CLI
argument and will be used");
+
yarnDescriptor.getFlinkConfiguration().setBoolean(ConfigConstants.SECURITY_ENABLED,
true);
+
yarnDescriptor.getFlinkConfiguration().setString(ConfigConstants.SECURITY_COOKIE,
secureCookieArg);
+ }
--- End diff --
So much duplicate code here. Could be extracted to a method to avoid that.
> Implement Service-Level Authorization
> -
>
> Key: FLINK-3930
> URL: https://issues.apache.org/jira/browse/FLINK-3930
> Project: Flink
> Issue Type: New Feature
> Components: Security
>Reporter: Eron Wright
>Assignee: Vijay Srinivasaraghavan
> Labels: security
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
> design doc._
> Service-level authorization is the initial authorization mechanism to ensure
> clients (or servers) connecting to the Flink cluster are authorized to do so.
> The purpose is to prevent a cluster from being used by an unauthorized
> user, whether to execute jobs, disrupt cluster functionality, or gain access
> to secrets stored within the cluster.
> Implement service-level authorization as described in the design doc.
> - Introduce a shared secret cookie
> - Enable Akka security cookie
> - Implement data transfer authentication
> - Secure the web dashboard
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[
https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588401#comment-15588401
]
ASF GitHub Bot commented on FLINK-3930:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r84034881
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyMessage.java
---
@@ -369,21 +423,24 @@ void readFrom(ByteBuf buffer) throws Exception {
InputChannelID receiverId;
+ String secureCookie = "";
+
public PartitionRequest() {
}
- PartitionRequest(ResultPartitionID partitionId, int queueIndex,
InputChannelID receiverId) {
+ PartitionRequest(ResultPartitionID partitionId, int queueIndex,
InputChannelID receiverId, String secureCookie) {
this.partitionId = partitionId;
this.queueIndex = queueIndex;
this.receiverId = receiverId;
+ this.secureCookie = (secureCookie == null ||
secureCookie.length() == 0) ? "": secureCookie;
--- End diff --
These checks should not be necessary. Please remove.
> Implement Service-Level Authorization
> -
>
> Key: FLINK-3930
> URL: https://issues.apache.org/jira/browse/FLINK-3930
> Project: Flink
> Issue Type: New Feature
> Components: Security
>Reporter: Eron Wright
>Assignee: Vijay Srinivasaraghavan
> Labels: security
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
> design doc._
> Service-level authorization is the initial authorization mechanism to ensure
> clients (or servers) connecting to the Flink cluster are authorized to do so.
> The purpose is to prevent a cluster from being used by an unauthorized
> user, whether to execute jobs, disrupt cluster functionality, or gain access
> to secrets stored within the cluster.
> Implement service-level authorization as described in the design doc.
> - Introduce a shared secret cookie
> - Enable Akka security cookie
> - Implement data transfer authentication
> - Secure the web dashboard
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[
https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588377#comment-15588377
]
ASF GitHub Bot commented on FLINK-3930:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r84026759
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobCache.java ---
@@ -272,4 +284,8 @@ private void closeSilently(Closeable closeable) {
}
}
}
+
+ /* Secure cookie to authenticate */
--- End diff --
> authorize
> Implement Service-Level Authorization
> -
>
> Key: FLINK-3930
> URL: https://issues.apache.org/jira/browse/FLINK-3930
> Project: Flink
> Issue Type: New Feature
> Components: Security
>Reporter: Eron Wright
>Assignee: Vijay Srinivasaraghavan
> Labels: security
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
> design doc._
> Service-level authorization is the initial authorization mechanism to ensure
> clients (or servers) connecting to the Flink cluster are authorized to do so.
> The purpose is to prevent a cluster from being used by an unauthorized
> user, whether to execute jobs, disrupt cluster functionality, or gain access
> to secrets stored within the cluster.
> Implement service-level authorization as described in the design doc.
> - Introduce a shared secret cookie
> - Enable Akka security cookie
> - Implement data transfer authentication
> - Secure the web dashboard
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[
https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588413#comment-15588413
]
ASF GitHub Bot commented on FLINK-3930:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r84037454
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/PartitionRequestQueue.java
---
@@ -170,7 +176,8 @@ else if (currentPartitionQueue.isReleased()) {
}
}
else {
- BufferResponse resp = new
BufferResponse(buffer, currentPartitionQueue.getSequenceNumber(),
currentPartitionQueue.getReceiverId());
+ BufferResponse resp = new
BufferResponse(buffer, currentPartitionQueue.getSequenceNumber(),
+
currentPartitionQueue.getReceiverId());
--- End diff --
just reformatting, no code changes (but i still have to check).
> Implement Service-Level Authorization
> -
>
> Key: FLINK-3930
> URL: https://issues.apache.org/jira/browse/FLINK-3930
> Project: Flink
> Issue Type: New Feature
> Components: Security
>Reporter: Eron Wright
>Assignee: Vijay Srinivasaraghavan
> Labels: security
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
> design doc._
> Service-level authorization is the initial authorization mechanism to ensure
> clients (or servers) connecting to the Flink cluster are authorized to do so.
> The purpose is to prevent a cluster from being used by an unauthorized
> user, whether to execute jobs, disrupt cluster functionality, or gain access
> to secrets stored within the cluster.
> Implement service-level authorization as described in the design doc.
> - Introduce a shared secret cookie
> - Enable Akka security cookie
> - Implement data transfer authentication
> - Secure the web dashboard
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[
https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588382#comment-15588382
]
ASF GitHub Bot commented on FLINK-3930:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r84031908
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyMessage.java
---
@@ -57,24 +61,37 @@
// constructor in order to work with the generic deserializer.
//
- static final int HEADER_LENGTH = 4 + 4 + 1; // frame length (4), magic
number (4), msg ID (1)
+ static final int HEADER_LENGTH = 4 + 4 + 4 + 1; // frame length (4),
magic number (4), Cookie (4), msg ID (1)
static final int MAGIC_NUMBER = 0xBADC0FFE;
+ static final int BUFFER_SIZE = 65536;
+
+ static final Charset DEFAULT_CHARSET = Charset.forName("utf-8");
--- End diff --
This should be read from somewhere else. If this is fixed anyways, then
please just use this in the allocateBuffer method.
> Implement Service-Level Authorization
> -
>
> Key: FLINK-3930
> URL: https://issues.apache.org/jira/browse/FLINK-3930
> Project: Flink
> Issue Type: New Feature
> Components: Security
>Reporter: Eron Wright
>Assignee: Vijay Srinivasaraghavan
> Labels: security
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
> design doc._
> Service-level authorization is the initial authorization mechanism to ensure
> clients (or servers) connecting to the Flink cluster are authorized to do so.
> The purpose is to prevent a cluster from being used by an unauthorized
> user, whether to execute jobs, disrupt cluster functionality, or gain access
> to secrets stored within the cluster.
> Implement service-level authorization as described in the design doc.
> - Introduce a shared secret cookie
> - Enable Akka security cookie
> - Implement data transfer authentication
> - Secure the web dashboard
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[ https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588365#comment-15588365 ] ASF GitHub Bot commented on FLINK-3930: --- Github user mxm commented on a diff in the pull request: https://github.com/apache/flink/pull/2425#discussion_r84021449 --- Diff: docs/setup/yarn_setup.md --- @@ -101,13 +101,19 @@ Usage: Optional -D Dynamic properties -d,--detached Start detached + -id,--applicationIdAttach to running YARN session + -j,--jar Path to Flink jar file -jm,--jobManagerMemory Memory for JobManager Container [in MB] - -nm,--name Set a custom name for the application on YARN + -k,--cookieSecure cookie to authenticate --- End diff -- Should not be present in the Yarn options. > Implement Service-Level Authorization > - > > Key: FLINK-3930 > URL: https://issues.apache.org/jira/browse/FLINK-3930 > Project: Flink > Issue Type: New Feature > Components: Security >Reporter: Eron Wright >Assignee: Vijay Srinivasaraghavan > Labels: security > Original Estimate: 672h > Remaining Estimate: 672h > > _This issue is part of a series of improvements detailed in the [Secure Data > Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing] > design doc._ > Service-level authorization is the initial authorization mechanism to ensure > clients (or servers) connecting to the Flink cluster are authorized to do so. > The purpose is to prevent a cluster from being used by an unauthorized > user, whether to execute jobs, disrupt cluster functionality, or gain access > to secrets stored within the cluster. > Implement service-level authorization as described in the design doc. > - Introduce a shared secret cookie > - Enable Akka security cookie > - Implement data transfer authentication > - Secure the web dashboard -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[
https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588390#comment-15588390
]
ASF GitHub Bot commented on FLINK-3930:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r84032768
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyMessage.java
---
@@ -57,24 +61,37 @@
// constructor in order to work with the generic deserializer.
//
- static final int HEADER_LENGTH = 4 + 4 + 1; // frame length (4), magic
number (4), msg ID (1)
+ static final int HEADER_LENGTH = 4 + 4 + 4 + 1; // frame length (4),
magic number (4), Cookie (4), msg ID (1)
static final int MAGIC_NUMBER = 0xBADC0FFE;
+ static final int BUFFER_SIZE = 65536;
+
+ static final Charset DEFAULT_CHARSET = Charset.forName("utf-8");
+
abstract ByteBuf write(ByteBufAllocator allocator) throws Exception;
abstract void readFrom(ByteBuf buffer) throws Exception;
//
- private static ByteBuf allocateBuffer(ByteBufAllocator allocator, byte
id) {
- return allocateBuffer(allocator, id, 0);
+ private static ByteBuf allocateBuffer(ByteBufAllocator allocator, byte
id, String secureCookie) {
+ return allocateBuffer(allocator, id, secureCookie, 0);
}
- private static ByteBuf allocateBuffer(ByteBufAllocator allocator, byte
id, int length) {
+ private static ByteBuf allocateBuffer(ByteBufAllocator allocator, byte
id, String secureCookie, int length) {
+ secureCookie = (secureCookie == null || secureCookie.length()
== 0) ? "": secureCookie;
--- End diff --
This check seems unncecessary. The cookie should always be a valid String
when it is passed. Also, what is the purpose of assigning `""` when
`secureCookie.length() == 0`?
> Implement Service-Level Authorization
> -
>
> Key: FLINK-3930
> URL: https://issues.apache.org/jira/browse/FLINK-3930
> Project: Flink
> Issue Type: New Feature
> Components: Security
>Reporter: Eron Wright
>Assignee: Vijay Srinivasaraghavan
> Labels: security
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
> design doc._
> Service-level authorization is the initial authorization mechanism to ensure
> clients (or servers) connecting to the Flink cluster are authorized to do so.
> The purpose is to prevent a cluster from being used by an unauthorized
> user, whether to execute jobs, disrupt cluster functionality, or gain access
> to secrets stored within the cluster.
> Implement service-level authorization as described in the design doc.
> - Introduce a shared secret cookie
> - Enable Akka security cookie
> - Implement data transfer authentication
> - Secure the web dashboard
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[
https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588372#comment-15588372
]
ASF GitHub Bot commented on FLINK-3930:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r84024531
--- Diff:
flink-core/src/main/java/org/apache/flink/configuration/GlobalConfiguration.java
---
@@ -144,8 +144,15 @@ private static Configuration loadYAMLResource(File
file) {
continue;
}
- LOG.debug("Loading configuration
property: {}, {}", key, value);
config.setString(key, value);
+
+ //to prevent logging the secure cookie
+
if(key.equals(ConfigConstants.SECURITY_COOKIE) && value != null) {
+ value = "**";
--- End diff --
Could you also do that for the other security-related passwords?
> Implement Service-Level Authorization
> -
>
> Key: FLINK-3930
> URL: https://issues.apache.org/jira/browse/FLINK-3930
> Project: Flink
> Issue Type: New Feature
> Components: Security
>Reporter: Eron Wright
>Assignee: Vijay Srinivasaraghavan
> Labels: security
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
> design doc._
> Service-level authorization is the initial authorization mechanism to ensure
> clients (or servers) connecting to the Flink cluster are authorized to do so.
> The purpose is to prevent a cluster from being used by an unauthorized
> user, whether to execute jobs, disrupt cluster functionality, or gain access
> to secrets stored within the cluster.
> Implement service-level authorization as described in the design doc.
> - Introduce a shared secret cookie
> - Enable Akka security cookie
> - Implement data transfer authentication
> - Secure the web dashboard
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization
[
https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588396#comment-15588396
]
ASF GitHub Bot commented on FLINK-3930:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r84035425
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyMessage.java
---
@@ -492,19 +553,22 @@ public void readFrom(ByteBuf buffer) {
InputChannelID receiverId;
+ String secureCookie = "";
--- End diff --
The `""` initialization is always overridden, please remove.
> Implement Service-Level Authorization
> -
>
> Key: FLINK-3930
> URL: https://issues.apache.org/jira/browse/FLINK-3930
> Project: Flink
> Issue Type: New Feature
> Components: Security
>Reporter: Eron Wright
>Assignee: Vijay Srinivasaraghavan
> Labels: security
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
> design doc._
> Service-level authorization is the initial authorization mechanism to ensure
> clients (or servers) connecting to the Flink cluster are authorized to do so.
> The purpose is to prevent a cluster from being used by an unauthorized
> user, whether to execute jobs, disrupt cluster functionality, or gain access
> to secrets stored within the cluster.
> Implement service-level authorization as described in the design doc.
> - Introduce a shared secret cookie
> - Enable Akka security cookie
> - Implement data transfer authentication
> - Secure the web dashboard
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
