Re: Reminder: Help required to fix security vulnerabilities in Flink Docker image

2024-06-23 Thread Gabor Somogyi
Hi Elakiya, I've just double checked the story and seems like the latest 1.17 gosu release is not vulnerable. Can you please try it out on your side? Alexis has written down how you can bump the docker version locally: ---CUT-HERE--- ENV GOSU_VERSION 1.17 ---CUT-HERE--- Please report back and

Re: Setting uid hash for non-legacy sinks

2024-06-12 Thread Gabor Somogyi
lva > > On Mon, Jun 10, 2024 at 9:49 AM Gabor Somogyi > wrote: > >> YW, ping me back whether it works because it's a nifty feature. >> >> G >> >> On Mon, Jun 10, 2024 at 9:26 AM Salva Alcántara >> wrote: >> >>> Thanks Gabor, I will gi

Re: Setting uid hash for non-legacy sinks

2024-06-10 Thread Gabor Somogyi
YW, ping me back whether it works because it's a nifty feature. G On Mon, Jun 10, 2024 at 9:26 AM Salva Alcántara wrote: > Thanks Gabor, I will give it a try! > > On Mon, Jun 10, 2024 at 12:01 AM Gabor Somogyi > wrote: > >> Now I see the intention and then you must

Re: Setting uid hash for non-legacy sinks

2024-06-09 Thread Gabor Somogyi
ly the > same strategy for generating uids to compute the corresponding uidHash for > each suboperator. Maybe you can further investigate it and fire a JIRA > issue on it. > > Best, > Zhanghao Chen > -- > *From:* Salva Alcántara > *Sent:* Sunday, June 9, 2

Re: Setting uid hash for non-legacy sinks

2024-06-07 Thread Gabor Somogyi
Hi Salva, Just wondering why not good to set the uid like this? ``` output.sinkTo(outputSink).uid("my-human-readable-sink-uid"); ``` >From the mentioned UID Flink is going to make the hash which is consistent from UID -> HASH transformation perspective. BR, G On Fri, Jun 7, 2024 at 7:54 AM

Re: SecurityManager in Flink

2024-03-06 Thread Gabor Somogyi
Hi Kirti, Not sure what is the exact issue here but I'm not convinced that having FlinkSecurityManager is going to solve it. Here is the condition however: * cluster.intercept-user-system-exit != DISABLED (this must be changed) * cluster.processes.halt-on-fatal-error == false (this is good by

Re: Securing Keytab File in Flink

2023-09-15 Thread Gabor Somogyi
Hi Chirag, Couple things can be done to reduce the attack surface (including but not limited to): * Use delegation tokens where only JM needs the keytab file: https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/security/security-delegation-token/ * Limit the access rights of the

Re: Keytab Setup on Kubernetes

2023-09-07 Thread Gabor Somogyi
cable in Flink 1.16? > > Thanks > > On Tuesday, 5 September, 2023 at 07:15:07 pm IST, Gabor Somogyi < > gabor.g.somo...@gmail.com> wrote: > > > hi Chirag, > > Flink now supports 2 ways to have TGT which is a Kerberos ticket and has > nothing to do with th

Re: Keytab Setup on Kubernetes

2023-09-05 Thread Gabor Somogyi
hi Chirag, Flink now supports 2 ways to have TGT which is a Kerberos ticket and has nothing to do with the "until 7 days renewable" HDFS TGS ticket (with default config). * Keytab: if one mounts a keytab for at least the JobManager pod then it can create TGT infinitely (or until the user's

Re: Encryption of parameters in flink-conf.yaml

2023-05-09 Thread Gabor Somogyi
hi Anuj, As Martijn said IAM is the preferred option but if you've no other way than access keys then environment variables is a better choice. Such case conf doesn't contain plain text keys. Just a side note, putting `s3a.access.key` into Flink conf file is not configuring Hadoop S3. The way

Re: Facing issue when using S3 in Flink 1.17

2023-04-20 Thread Gabor Somogyi
Hi Sriram, This has been fixed in https://issues.apache.org/jira/browse/FLINK-31839 G On Thu, Apr 20, 2023 at 4:57 PM Sriram Ganesh wrote: > Hi Team, > > I am using S3 as FileSystem to write data from Flink. I am getting the > below error in Flink 1.17. The same code works in Flink 1.16.

Re: Delegation Tokens config - Upgrade from 1.16.x to 1.17.0

2023-04-07 Thread Gabor Somogyi
Hi Arthur, Delegation tokens were enabled all the time which is not changed since it would be a breaking change. I would personally turn it off by default but it's important to keep original behavior. The manager is loading providers at the very beginning of the init process. It loads and

Re: Flink application mode/S3A Exception after upgrading from to Flink 1.16.0

2023-01-27 Thread Gabor Somogyi
The min supported version was 2.8.5 but in 1.17 it's gonna be 2.10.1 so one can downgrade. G On Fri, Jan 27, 2023, 20:42 Leon Xu wrote: > Thank you Mate. > Yeah this looks like the root cause. A follow-up question, do you know if > Flink 1.16 will have a hard dependency on Hadoop 3.3.x? or can

Re: Re: [ANNOUNCE] Apache Flink 1.15.1 released

2022-07-12 Thread Gabor Somogyi
In order to provide a hotfix please set "taskmanager.resource-id" to something which doesn't contain special any character. G On Tue, Jul 12, 2022 at 11:59 AM Gabor Somogyi wrote: > Flink tried to create the following dir: tm_localhost:50329-fc0146 > Colon is

Re: Re: [ANNOUNCE] Apache Flink 1.15.1 released

2022-07-12 Thread Gabor Somogyi
Flink tried to create the following dir: tm_localhost:50329-fc0146 Colon is allowed on linux but not on windows and that's the reason of the exception. BR, G On Tue, Jul 12, 2022 at 11:30 AM wrote: > ... > 2022-07-12 11:25:08,448 INFO > akka.remote.Remoting

Re: Reply:DelegationTokenManager

2022-06-21 Thread Gabor Somogyi
Thanks for pinging me! Yes, this is my main target to finish this feature however there are major code parts which are still missing. Please have a look at the umbrella jira to get better understanding: https://issues.apache.org/jira/browse/FLINK-21232 In general it's not advised to use it for

Re: Question about plain password in flink-conf.yaml

2022-01-18 Thread Gabor Somogyi
export SSL_PASSWORD=secret flink run -yDsecurity.ssl.rest.*-password=$SSL_PASSWORD ... app.jar Such way the code which starts the workload can store the passwords in a centrally protected area. This still can be hacked but at least not stored in plain text file. BR, G On Tue, Jan 18, 2022 at

Re: mutual authentication with ssl

2021-11-26 Thread Gabor Somogyi
Hi Raul, On all systems keystore is needed normally on the server side and truststore on client side. As a result it's highly advised to use different config files in these places. It's easy to see why it would be a security leak when keystore would be available in client side (client can fake a

Re: kerberos token expire

2021-07-06 Thread Gabor Somogyi
As Yangze stated ticket cache will be expired after its lifespan. Please be aware that when keytab is used then Flink obtains delegation tokens which will be never ever used. The fact that delegation token handling is not functioning is a known issue and working on it to fix it. w/o delegation