[
https://issues.apache.org/jira/browse/BEAM-9855?focusedWorklogId=508877&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-508877
]
ASF GitHub Bot logged work on BEAM-9855:
----------------------------------------
Author: ASF GitHub Bot
Created on: 08/Nov/20 10:43
Start Date: 08/Nov/20 10:43
Worklog Time Spent: 10m
Work Description: mxm commented on pull request #13116:
URL: https://github.com/apache/beam/pull/13116#issuecomment-723559227
There is currently no universal way to set the configuration per-job. Unless
a per-job cluster is used. In this case a configuration file can be supplied
for the per-job cluster. There is no way to explicitly configure a state
backend via a flag to the main program. Even if we allowed for a configuration
file parameter, we would rely on that configuration file to be present during
runtime which we can't assume. Further, if we allowed a direct configuration
YAML string, we would also have to explicitly set the state backend on the
`StreamExecutionEnvironment`, otherwise during runtime the cluster's default
state backend would be used instead of the configured one (the default state
backend can only be configured during job creation time). The intention here is
to allow setting and overriding the default state backend. Thus, it makes sense
(for now) to have an explicit way to configure the state backend.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 508877)
Time Spent: 2h 40m (was: 2.5h)
> Make it easier to configure a Flink state backend
> -------------------------------------------------
>
> Key: BEAM-9855
> URL: https://issues.apache.org/jira/browse/BEAM-9855
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
> Reporter: Maximilian Michels
> Assignee: Maximilian Michels
> Priority: P3
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> We should make it easier to configure a Flink state backend. At the moment,
> users have to either (1) configure the default state backend in their Flink
> cluster, or make sure (2a) they include the dependency in their Gradle/Maven
> project (e.g.
> {{"org.apache.flink:flink-statebackend-rocksdb_2.11:$flink_version"}} for
> RocksDB) (2b) set the state backend factory in the {{FlinkPipelineOptions}.
> The drawback of option (2) is that it only works in Java due to the factory
> specification being in Java.
> We can make it easier by simple adding pipeline options for the state backend
> name and the checkpoint directory which will be enough for configuring the
> state backend. We can add the RocksDB state backend as a default dependency.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
