Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/3692
@tillrohrmann Please let me know if you are waiting for any clarifications
before this PR could be merged.
---
If your project is set up for it, you can reply to this email and have your
reply
GitHub user vijikarthi opened a pull request:
https://github.com/apache/flink/pull/3692
FLINK-5974 Added configurations to support mesos-dns hostname resolution
This PR addresses FLINK-5974 requirements which takes care of handling
dynamic host name resolution for JM and TM
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/3600
The changes looks good to me.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/3600
The default ZK SASL client behavior is to enable SASL client and to be in
consistent it makes sense for us to leave the default option enabled.
---
If your project is set up for it, you can
Github user vijikarthi commented on a diff in the pull request:
https://github.com/apache/flink/pull/3600#discussion_r108284815
--- Diff:
flink-core/src/main/java/org/apache/flink/configuration/SecurityOptions.java ---
@@ -55,6 +55,10 @@
// ZooKeeper Security Options
Github user vijikarthi commented on a diff in the pull request:
https://github.com/apache/flink/pull/3566#discussion_r106829135
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/util/ZooKeeperUtils.java
---
@@ -89,6 +90,7 @@ public static CuratorFramework
Github user vijikarthi closed the pull request at:
https://github.com/apache/flink/pull/2425
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2425
@StephanEwen It's absolutely fine with me and I will cancel this PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2425
@StephanEwen The shared secret serves can be considered as an additional
security extension on top of TLS integration, thus it designates only an
authorized identity to execute actions
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/3486
I think the patch looks good. Is there a specific protocol version + cipher
suite combination sets that the user should be aware which needs to be
documented?
---
If your project is set up
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2425
@StephanEwen, @mxm I have updated the documentation changes as suggested,
moved common code from BlobUtils to SecurityContext, added new ConfigOptions
class for security configurations lookup
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2425
@StephanEwen @mxm - Could you please review the proposed change and let me
know if you are okay with it.
---
If your project is set up for it, you can reply to this email and have your
reply
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2425
>
The cookie is added to every single message/buffer that is transferred.
That is too much - securing the integrity of the stream is responsibility of
the encryption layer. The cookie sho
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2425
@mxm - Sorry that I have missed to address some of your comments. Attached
patch that includes Netty code null precondition validation and fixes the Blob
service cookie length issue. Please take
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2734
@mxm - Could you please take a look?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
GitHub user vijikarthi opened a pull request:
https://github.com/apache/flink/pull/2734
Keytab & TLS support for Flink on Mesos Setup
This PR addresses below issues
- FLINK-4826 (Keytab support on Mesos environment)
- FLINK-4918 (TLS support for Mesos Artifact Se
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2425
Addressed multiple application support/Yarn configuration file changes as
part of FLINK-4950 patch.
---
If your project is set up for it, you can reply to this email and have your
reply appear
Github user vijikarthi commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r85167083
--- Diff:
flink-yarn/src/main/java/org/apache/flink/yarn/cli/FlinkYarnSessionCli.java ---
@@ -108,6 +111,11 @@
private final Options
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2425
@mxm - Please take a look when you get a chance?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2425
Thanks @mxm for the review. I will incorporate your feedback and attach the
patch.
>
When security is enabled, encryption should also be turned on by default.
Otherwise
Github user vijikarthi commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r84169656
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyMessage.java
---
@@ -57,24 +61,37 @@
// constructor in order
Github user vijikarthi commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r84175964
--- Diff:
flink-yarn/src/main/java/org/apache/flink/yarn/cli/FlinkYarnSessionCli.java ---
@@ -442,8 +453,10 @@ public static void runInteractiveCli
Github user vijikarthi commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r84174244
--- Diff:
flink-yarn/src/main/java/org/apache/flink/yarn/cli/FlinkYarnSessionCli.java ---
@@ -108,6 +111,11 @@
private final Options
Github user vijikarthi commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r84170399
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyMessage.java
---
@@ -112,9 +129,9 @@ public void write
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2425
Resolved merge conflicts and squashed commits to rebase with master
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2589
Are we waiting for any additional review?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2425
@rmetzger can you please take a look at the updated patch
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2589
Thanks @mxm . I have just rebased it against the master. Could you please
merge the code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2589
@mxm Can you take a look at this PR?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
GitHub user vijikarthi opened a pull request:
https://github.com/apache/flink/pull/2589
FLINK-3932 State Backend Security
This PR addresses ZK authorization (ACLs) requirement of FLINK-3932 and its
dependency FLINK-4667 (Yarn session CLI not using correct ZK namespace in
secure
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2425
Addressed [FLINK-4635] Netty data transfer authentication (missing piece of
FLINK-3930)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2425
Thanks @StephanEwen
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2425
@rmetzger @StephanEwen are you guys waiting for any inputs from my side?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2425
@rmetzger I have added internals documentation section and provided details
on how secure cookie is implemented. I will address the missing Netty data
transfer secure cookie part in FLINK-4635
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2275
@mxm I have addressed some of the review feedback and rebased to upstream
master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well
Github user vijikarthi commented on a diff in the pull request:
https://github.com/apache/flink/pull/2275#discussion_r79221254
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/security/SecurityContext.java
---
@@ -155,6 +157,58 @@ public static void install
Github user vijikarthi commented on a diff in the pull request:
https://github.com/apache/flink/pull/2275#discussion_r79221107
--- Diff:
flink-clients/src/main/java/org/apache/flink/client/CliFrontend.java ---
@@ -161,6 +161,8 @@ public CliFrontend(String configDir) throws
Github user vijikarthi commented on a diff in the pull request:
https://github.com/apache/flink/pull/2275#discussion_r79190433
--- Diff:
flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java ---
@@ -1233,6 +1239,9 @@
/** ZooKeeper default leader port
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2275
>
@vijikarthi Thanks for the update. Great to see the tests are passing now.
I'm curious, why did this issue only appear on Travis and not locally?
Kafka/ZK connection is unusua
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2275
@mxm The issue is apparently due to ZK client API implementation which
lookup the system configuration property to determined the type of event
(SASLAuthenticated/SysConnected) that it should
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2275
@nielsbasjes Thanks for the link. The issue however is related to ZK SASL
client API implementation and it took a while to figure out the actual cause.
---
If your project is set up for it, you
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2275
Thanks, will try that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2275
hmmm.. it's getting complicated. will try to debug the issue. Do you know
why on Travis it has to fail but not on Jenkins?
How do I simulate this on Travis for my own testing
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2275
I have noticed the issue occasionally in master branch too but very
inconsistent. I just rebased against the latest master and ran "mvn clean
verify". I don't see any errors. I have tr
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2425
>
How is the secret transferred to the TaskManagers on YARN?
Cookie is transferred to TM container through container environment
variable and further gets populated to in-memory Fl
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2275
rebased again with the latest master
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2425
>
T2-3 is not about the web interface netty, its about the data transfer netty
In Flink, we are using netty for (at least) three things:
- Akka is using Netty. This is addres
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2275
@mxm I believe the ZK timeout issue occurs from
LocalFlinkMiniClusterITCase->testLocalFlinkMiniClusterWithMultipleTaskManagers
test case but it is not consistent. I ran the Kafka test case al
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2275
@mxm, I was able to reproduce this issue on some other machine. The issue
is that KRB5 config file which is mounted as a local resource is not visible
though we set the system property
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2275
@mxm, I am not seeing the error messages when I run the Yarn test. Could
you please run "secure" test case alone and share the logs?
>
mvn test integratio
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2425
>
According to the design document, netty authentication is also part of this
JIRA. Why was it not addressed?
The netty layer is addressed as part of web layer authentication (T
Github user vijikarthi commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r77230955
--- Diff:
flink-yarn/src/main/java/org/apache/flink/yarn/cli/FlinkYarnSessionCli.java ---
@@ -682,6 +774,91 @@ public static File
Github user vijikarthi commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r77227679
--- Diff:
flink-runtime/src/test/java/org/apache/flink/runtime/blob/BlobClientSecureTest.java
---
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache
Github user vijikarthi commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r77226096
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobServer.java ---
@@ -426,4 +440,11 @@ void unregisterConnection
Github user vijikarthi commented on a diff in the pull request:
https://github.com/apache/flink/pull/2425#discussion_r77221958
--- Diff:
flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/HttpRequestHandler.java
---
@@ -99,7 +110,43 @@ public void channelRead0
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2275
@mxm I have rebased the code to the latest master. Please take a look.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2425
@mxm - The patch is available for your review. Please take a look.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2275
>
YARNSessionFIFOSecuredITCase gives me the following:
17:49:58,097 INFO SecurityLogger.org.apache.hadoop.ipc.Server - Auth
successful for appattempt_1471880990715_0001_000
GitHub user vijikarthi opened a pull request:
https://github.com/apache/flink/pull/2425
FLINK-3930 Added shared secret based authorization for Flink service â¦
This PR addresses FLINK-3930 requirements. It enables shared secret based
secure cookie authorization for the following
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2275
>
It seems like the privileged port issues can be circumvented by setting
conf.getBoolean("dfs.datanode.require.secure.ports", false)?
It is not supported yet https
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2275
> If we have to use privileged ports then we won't be able to use our CI
system. Are you sure it can only run in privileged mode? Is it not possible to
change the port binding to port >
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2275
> YARNSessionFIFOSecuredITCase gives me the following:
17:49:58,097 INFO SecurityLogger.org.apache.hadoop.ipc.Server - Auth
successful for appattempt_1471880990715_0001_01 (auth:SIM
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2275
The `RollingFileSink` error might be due to
https://issues.apache.org/jira/browse/HDFS-9213. The secure MiniDFS
cluster requires privileged ports to be used and we need to enable the java
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2275
@mxm - Could you please take a look and let me know if you need any further
changes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2275
Thanks Max. I have reorganized the secure test case and removed custom
JRunner implementation for Kafka. Kept single secure test case for HDFS, Kafka
and Yarn modules. Please take a look
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2275
@mxm - Could you please let me know if you are okay with the modifications
to the integration test case scenarios that I have mentioned. I am open to keep
just 3 classes for each scenarios (HDFS
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2275
@mxm - I have addressed most of the feedback and pushed the changes. Please
take a look when you get a chance.
Regarding the secure test cases,
- HDFS and Yarn are handled through
Github user vijikarthi commented on a diff in the pull request:
https://github.com/apache/flink/pull/2275#discussion_r73245570
--- Diff:
flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java ---
@@ -1016,6 +1016,23 @@
/** The environment variable
Github user vijikarthi commented on a diff in the pull request:
https://github.com/apache/flink/pull/2275#discussion_r73037620
--- Diff: docs/internals/flink_security.md ---
@@ -0,0 +1,87 @@
+---
+title: "Flink Security"
+# Top navigation
+top-nav-group:
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2275
@mxm - Thanks for your feedback and here is my response to some of your
comments.
- Do we need to run all the Yarn tests normally and secured? We already
have problems with our test
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2275
@nielsbasjes - In most deployments, the KRB5 configuration file will be
located in a well known (for e.g., /etc/krb5.conf) but in scenarios where
custom location needs to be provided, we could
Github user vijikarthi commented on a diff in the pull request:
https://github.com/apache/flink/pull/2275#discussion_r73028461
--- Diff:
flink-yarn/src/main/java/org/apache/flink/yarn/YarnTaskManagerRunner.java ---
@@ -75,34 +84,47 @@ public static void runYarnTaskManager(String
Github user vijikarthi commented on a diff in the pull request:
https://github.com/apache/flink/pull/2275#discussion_r73027093
--- Diff:
flink-test-utils-parent/flink-test-utils/src/main/java/org/apache/flink/test/util/RunTypeSelectionRunner.java
---
@@ -0,0 +1,54
Github user vijikarthi commented on a diff in the pull request:
https://github.com/apache/flink/pull/2275#discussion_r73023852
--- Diff:
flink-test-utils-parent/flink-test-utils/src/main/java/org/apache/flink/test/util/RunTypeHolder.java
---
@@ -0,0 +1,36
Github user vijikarthi commented on a diff in the pull request:
https://github.com/apache/flink/pull/2275#discussion_r73023816
--- Diff:
flink-streaming-connectors/flink-connector-kafka-base/src/test/java/org/apache/flink/streaming/connectors/kafka/KafkaTestBase.java
Github user vijikarthi commented on a diff in the pull request:
https://github.com/apache/flink/pull/2275#discussion_r73023705
--- Diff:
flink-test-utils-parent/flink-test-utils/src/main/java/org/apache/flink/test/util/RunTypeHolder.java
---
@@ -0,0 +1,36
Github user vijikarthi commented on a diff in the pull request:
https://github.com/apache/flink/pull/2275#discussion_r73020806
--- Diff:
flink-streaming-connectors/flink-connector-kafka-base/src/test/java/org/apache/flink/streaming/connectors/kafka/KafkaShortRetentionTestBase.java
Github user vijikarthi commented on a diff in the pull request:
https://github.com/apache/flink/pull/2275#discussion_r73017860
--- Diff:
flink-streaming-connectors/flink-connector-filesystem/src/test/java/org/apache/flink/streaming/connectors/fs/RollingSinkSecuredITCase.java
Github user vijikarthi commented on a diff in the pull request:
https://github.com/apache/flink/pull/2275#discussion_r73015565
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/security/SecurityContext.java
---
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache
Github user vijikarthi commented on a diff in the pull request:
https://github.com/apache/flink/pull/2275#discussion_r73014755
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/security/SecurityContext.java
---
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache
Github user vijikarthi commented on a diff in the pull request:
https://github.com/apache/flink/pull/2275#discussion_r73013795
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/security/JaasConfiguration.java
---
@@ -0,0 +1,158 @@
+/*
+ * Licensed
Github user vijikarthi commented on a diff in the pull request:
https://github.com/apache/flink/pull/2275#discussion_r73013657
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/security/JaasConfiguration.java
---
@@ -0,0 +1,158 @@
+/*
+ * Licensed
Github user vijikarthi commented on a diff in the pull request:
https://github.com/apache/flink/pull/2275#discussion_r73012870
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/security/JaasConfiguration.java
---
@@ -0,0 +1,158 @@
+/*
+ * Licensed
Github user vijikarthi commented on a diff in the pull request:
https://github.com/apache/flink/pull/2275#discussion_r73012593
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/security/JaasConfiguration.java
---
@@ -0,0 +1,158 @@
+/*
+ * Licensed
Github user vijikarthi commented on a diff in the pull request:
https://github.com/apache/flink/pull/2275#discussion_r73012055
--- Diff:
flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java ---
@@ -1016,6 +1016,23 @@
/** The environment variable
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2275
Thanks @mxm for the review and feedback. I will respond to the comments and
incorporate any changes required.
---
If your project is set up for it, you can reply to this email and have your
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2275
Team, please let me know if any additional details are required to
kick-start the review process?
---
If your project is set up for it, you can reply to this email and have your
reply appear
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2275
Adding some more cotext to the implementation details. which is based on
the design proposal
(https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing
GitHub user vijikarthi opened a pull request:
https://github.com/apache/flink/pull/2275
FLINK-3929 Support for Kerberos Authentication with Keytab Credential
This PR addresses FLINK-3929 requirements:
1) Added Keytab support to Flink (Standalone and Yarn mode deployment)
2
Github user vijikarthi commented on the pull request:
https://github.com/apache/flink/commit/c8fed99e3e85a4d27c6134cfa3e07fb3a8e1da2a#commitcomment-17935989
In pom.xml:
In pom.xml on line 972:
This version (2.18.1) has some issues running scoped test case. For example
it does
90 matches
Mail list logo
