Hi Max, Thanks for getting back to me. Will create a ticket, my username is catlynk.
Cheers, Catlyn On Tue, Aug 27, 2019 at 3:50 AM Maximilian Michels <[email protected]> wrote: > Hi Catlyn, > > This option has never worked outside the Java SDK where it originates > from. For the upcoming Beam 2.16.0 release, we have replaced this option > with a factory class: > > https://github.com/apache/beam/blob/da6c1a8f435f5583811785050808a2311db94047/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java#L152 > > We could bundle a default factory class for the RocksDB state backend > and make this easier to configure from Python. Do you mind opening a > JIRA issue for this? I can give you permissions for > https://issues.apache.org/jira/projects/BEAM/issues if you create an > account. > > In any case, you could also configure the Flink cluster to use RocksDB. > Beam will use whatever state backend is configured. > > Thanks, > Max > > On 23.08.19 02:57, Catlyn Kong wrote: > > Hi all, > > > > I'm experimenting with checkpoints/savepoints in Beam (version 2.14) > > when using a Flink (version 1.6.4) runner. Flink was able to take > > periodic checkpoints when I setup the flink-conf.yaml correctly. But I > > was thinking if it's possible to set the StateBackend on a per job level > > by flagging the --state_backend=RocksDBStateBackend option since it's > > said to be supported here > > <https://beam.apache.org/documentation/runners/flink/>. > > > > But instead I got the following error: > > RuntimeError: Pipeline failed in state FAILED: > > com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Cannot > > construct instance of `org.apache.flink.runtime.state.StateBackend` (no > > Creators, like default construct, exist): abstract types either need to > > be mapped to concrete types, have custom deserializer, or contain > > additional type information > > at [Source: (String)""RocksDBStateBackend""; line: 1, column: 1] > > > > I then saw that there's: > > @JasonIgnore > > StateBackend getStateBackend(); > > > > I'm wondering if this is not supported in python yet? If yes then do we > > have plans to support this in the near future? > > > > Best, > > Catlyn >
