Re: Production deployment of Flink

Hi! We recommend using the community supported Flink Kubernetes Operator: https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-release-1.7/docs/try-flink-kubernetes-operator/quick-start/ Cheers, Gyula On Thu, Dec 7, 2023 at 6:33 PM Tauseef Janvekar wrote: > Hi Al, > > I am using

Feature flag functionality on flink

Hi, We would like to enable sort of a feature flag functionality for flink jobs. The idea would be to use broadcast state reading from a configuration topic and then ALL operators with logic would listen to this state. This documentation:

Re: [DISCUSS] Change the default restart-strategy to exponential-delay

Hey Rui, +1 for changing the default restart strategy to exponential-delay. This is something all users eventually run into. They end up changing the restart strategy to exponential-delay. I think the current defaults are quite balanced. Restarts happen quickly enough unless there are consecutive

Re: [DISCUSS] Change the default restart-strategy to exponential-delay

Hey Rui, +1 for changing the default restart strategy to exponential-delay. This is something all users eventually run into. They end up changing the restart strategy to exponential-delay. I think the current defaults are quite balanced. Restarts happen quickly enough unless there are consecutive

Production deployment of Flink

Hi Al, I am using flink in my local setup and it works just fine - I installed it using confluent example training course. Here I had to manually execute start-cluster.sh and othe steps to start task managers. We installed flink on kubernetes using bitnami helm chart and it works just fine. But

RE: Reading text file from S3

Thank you Jaehyeon Kim. A workaround consists on exporting was credentials as environment variables using https://github.com/linaro-its/aws2-wrap The command below will set the environment variables AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN, which makes the Flink code work.

Unable to locate full stderr logs while using Flink Operator

Hi all! I've just deployed an Apache Beam job using FlinkRunner in k8s and found that the job failed and has the following field: error: >- {"type":"org.apache.flink.util.SerializedThrowable","message":"org.apache.flink.client.program.ProgramInvocationException: The main method caused an

Re: Query on using two sinks for a Flink job (Flink SQL)

Hi Chen/ Feng, Thanks for pointing out the mistake I made, after correcting the query I am able to run the job with two sinks successfully. Thanks, Elakiya On Thu, Dec 7, 2023 at 4:37 AM Chen Yu wrote: > Hi Chen, > You should tell flink which table to insert by “INSERT INTO XXX SELECT >

keyby mapState use question

Hi all: KeyBy process function EventKeyedBroadcastProcessFunction { private transient mapstate = null; public void open(Configuration parameters) throws Exception { // initial map state } public void processElement() { // can't get onTimer() function set state key value }