Obtaining Flink Conf in User Code

Hi all, Is it possible to obtain the Flink configuration in the user code? I've tried the Configuration parameter in the open method of rich functions and StreamExecutionEnvironment.getConfig().getGlobalJobParameters()--both do not give the configs in the flink configuration. Best, Mason

Re: Triggers for windowed aggregations in Table API

Thanks Guowei and Caizhi. As Guowei noted, I am using Table API and it seems that it does not support triggers at the moment. Is there a plan to support custom triggers in Table API/SQL too? Also, if I follow Guowei's suggestion, should I use DataStream for other parts of the aggregate computation

pyflink table to datastream

I have a question about how the conversion from Table API to Datastream API actually works under the covers. If I have a Table API operation that creates a random id, like: SELECT id, CAST(UUID() AS VARCHAR) as random_id FROM table ...then I convert this table to a datastream with

RE: [External] Re: Flink on Kubernetes

OK – I will increase the value to something higher and see how it does in recovering. Thank you for your help! ___ Julian Cardarelli CEO T (800) 961-1549 ejul...@thentia.com LinkedIn DISCLAIMER ​ ​Neither Thentia Corporation, nor its directors, officers, shareholders, representatives,

Re: Cleaning old incremental checkpoint files

Hi Robin, It's not easy to clean incremental checkpoints as different job instances have different checkpoint sub-directory (due to different job id). You could analysis your checkpoint metadata to see what files are still useful in older checkpoint directory. BTW, I also think of a possible

Re: [Question] Basic Python examples.wordcount on local FlinkRunner

Thanks Dian, that seemed to do the trick. I built a Docker Image simply using: FROM apache/beam_python3.8_sdk:2.31.0 COPY flink_data/input.txt . I specified nothing in the Pipeline options other than: "--runner=FlinkRunner", "--flink_master=localhost:8081", "--environment_type=DOCKER",

ApacheCon is just 3 weeks away!

[You are receiving this email because you are subscribed to the user list of one or more Apache project.] Dear Apache enthusiast, ApacheCon is our annual convention, featuring content related to our many software projects. This year, it will be held on September 21-23. Registration is free

Re: Deploying Flink on Kubernetes with fractional CPU and different limits and requests

Hi Alexis, Thanks for sharing more thoughts about resource configuration. Your suggestions make a lot of sense to me. I believe it could also help others especially for those who are more familiar with K8s and tend to use pod template as far as possible. I have created a ticket for this

RE: Deploying Flink on Kubernetes with fractional CPU and different limits and requests

Hi Yang, I understand the issue, and yes, if Flink memory must be specified in the configuration anyway, it’s probably better to leave memory configuration in the templates empty. For the CPU case I still think the template’s requests/limits should have priority if they are specified. The

Re: FLINK-14316 happens on version 1.13.2

Hi Yun, Thanks alot. I am running a test, and facing the "Job Leader lost leadership..." issue, and also the checkpointing timeout at the same time,, not sure whether those 2 things related to each other. regarding your question: 1. GC looks ok. 2. seems like once the "Job Leader lost

Re: [Question] Basic Python examples.wordcount on local FlinkRunner

This seems more like a Beam issue although it uses Flink runner. It would be helpful to also send it to the Beam user mailing list. Regarding to this issue itself, could you check is input.txt accessible in the Docker container? Regards, Dian > 2021年9月3日 上午5:19，Adam Pearce 写道： > > Hello

Re: Deploying Flink on Kubernetes with fractional CPU and different limits and requests

Hi Alexis Thanks for your valuable inputs. First, I want to share why Flink has to overwrite the resources which are defined in the pod template. You could the fields that will be overwritten by Flink here[1]. I think the major reason is that Flink need to ensure the consistency between Flink