Re: Flink runner configuration for closure cleaner
On Wed, Apr 14, 2021 at 3:21 PM Kyle Weaver wrote: > Looking at the code, `--flink-conf-dir` is only read when executing the >> runner directly. When running via the pipeline, there seems to be no way to >> set that value. > > > You're right, --flink-conf-dir is not usable in all situations. Can you > try setting the FLINK_CONF_DIR environment variable to the directory > containing your flink.conf instead? > > That did work, thank you. > >> On Tue, Apr 6, 2021 at 5:39 PM Kyle Weaver wrote: >> >>> I don't think this will require Beam to have its own configuration >>> option. You should be able to set the property >>> "pipeline.closure-cleaner-level" in your flink.conf and then pass it to >>> Beam using Beam's "--flink-conf-dir" pipeline option. >>> >>> On Tue, Apr 6, 2021 at 2:28 PM Raman Gupta >>> wrote: >>> >>>> Hello all: I created https://issues.apache.org/jira/browse/BEAM-12055 >>>> because I'm having an issue with using the Flink runner locally, due to >>>> https://issues.apache.org/jira/browse/FLINK-15773. >>>> >>>> Does anyone see any reason why Beam's Flink runner should not provide a >>>> configuration option that can disable the Flink closure cleaner? I'm not >>>> familiar with this myself, and the only reason I would use it is as a >>>> work-around for FLINK-15773 in my local dev environment. >>>> >>>> Regards, >>>> Raman >>>> >>>>
Re: Flink runner configuration for closure cleaner
Looking at the code, `--flink-conf-dir` is only read when executing the runner directly. When running via the pipeline, there seems to be no way to set that value. On Tue, Apr 6, 2021 at 5:39 PM Kyle Weaver wrote: > I don't think this will require Beam to have its own configuration option. > You should be able to set the property "pipeline.closure-cleaner-level" in > your flink.conf and then pass it to Beam using Beam's "--flink-conf-dir" > pipeline option. > > On Tue, Apr 6, 2021 at 2:28 PM Raman Gupta wrote: > >> Hello all: I created https://issues.apache.org/jira/browse/BEAM-12055 >> because I'm having an issue with using the Flink runner locally, due to >> https://issues.apache.org/jira/browse/FLINK-15773. >> >> Does anyone see any reason why Beam's Flink runner should not provide a >> configuration option that can disable the Flink closure cleaner? I'm not >> familiar with this myself, and the only reason I would use it is as a >> work-around for FLINK-15773 in my local dev environment. >> >> Regards, >> Raman >> >>
Flink runner configuration for closure cleaner
Hello all: I created https://issues.apache.org/jira/browse/BEAM-12055 because I'm having an issue with using the Flink runner locally, due to https://issues.apache.org/jira/browse/FLINK-15773. Does anyone see any reason why Beam's Flink runner should not provide a configuration option that can disable the Flink closure cleaner? I'm not familiar with this myself, and the only reason I would use it is as a work-around for FLINK-15773 in my local dev environment. Regards, Raman
