@Sruthi: You'll have to implement your own factory. If you include the
mentioned dependency, e.g. if you use Gradle:
runtimeOnly
"org.apache.flink:flink-statebackend-rocksdb_2.11:$flink_version"
Then create the factory:
options.setStateBackend(o -> new RocksDBStateBackend("file://..."));
Alternatively, you can configure the Flink cluster to use RocksDB as the
default.
@Ismael: Good idea. I think we ought to make it easier to use a
different state backend. Since the setup is quite minimal. Tracked here:
https://jira.apache.org/jira/browse/BEAM-9855
-Max
On 28.04.20 13:38, Ismaël Mejía wrote:
> Max would it make sense to make the rocksdb runtime only at the runner
> level just to hint its use. I assume that most Flink users might want
> to have RocksDB as the default state backend?
>
> runtimeOnly
> "org.apache.flink:flink-statebackend-rocksdb_2.11:$flink_version"
>
>
> On Tue, Apr 28, 2020 at 1:14 PM Sruthi Sree Kumar
> <sruthisreekumar2...@gmail.com> wrote:
>>
>> Hello,
>>
>> I have looked the Beam code. The statebackend that we pass should be an
>> instance of FlinkStateBackendFactory . But there is no implementation for
>> the Interface.
>> None of the FlinkStateBackend Implements this interface.
>>
>> So even when I try to pass the default MemoryStateBackend as an argument, it
>> throws an error.
>>
>> But I could manage to specify the statebackend (filesystem) using the config
>> file in the config directory specified by the env variable
>> ENV_FLINK_CONF_DIR.
>>
>> Regards,
>> Sruthi
>>
>> On Tue, Apr 28, 2020 at 11:35 AM Maximilian Michels <m...@apache.org> wrote:
>>>
>>> Hi Sruthi,
>>>
>>> Not possible out-of-the-box at the moment. You'll have to add the
>>> RocksDB Flink dependency in flink_runner.gradle, e.g.:
>>>
>>> compile "org.apache.flink:flink-statebackend-rocksdb_2.11:$flink_version"
>>>
>>> Also in the Flink config you have to set
>>>
>>> state.backend: rocksdb
>>>
>>> Then you can run Nexmark against the cluster, e.g.
>>>
>>> ./gradlew :sdks:java:testing:nexmark:run \
>>> -Pnexmark.runner=":runners:flink:1.10" \
>>> -Pnexmark.args="
>>> --runner=FlinkRunner
>>> --flinkMaster=<flink_master:8081>
>>> --streaming=true
>>> --shutdownSourcesOnFinalWatermark=true
>>> --query=12
>>> --suite=SMOKE
>>> --manageResources=false
>>> --monitorJobs=true
>>> --enforceEncodability=true
>>> --enforceImmutability=true"
>>>
>>>
>>> Admittedly, this is a bit complicated. We could make that easier without
>>> much work.
>>>
>>> Cheers,
>>> Max
>>>
>>> On 28.04.20 10:26, Sruthi Sree Kumar wrote:
>>>> Hello,
>>>>
>>>> Is it possible to run the nexmark queries by specifying a
>>>> state-backed(Ex: RocksDB) ?
>>>>
>>>>
>>>> Regards,
>>>> Sruthi
