Hi all

We already found the programmatic way to configure RocksDB was not so friendly 
in our Alibaba's production environment, and refactored it by wrapping customer 
configurations for RocksDB. We pre-wrapped configurations such as block cache 
size, whether to cache index&filter into block cache size, number of background 
flush threads, and could get the them from TaskManagerConfiguration if users 
want to change the default values. But I think our community's way using 
options factory could also make sense if sophisticated user want to make 
fine-grained tuning which might use configuration not included in our 
pre-wrapped configuration. We might combine these two kind of operations 
together.

I'd like to create a JIRA about this improvement, what's your opinions?

Best
Yun Tang

________________________________
From: Thomas Weise <t...@apache.org>
Sent: Tuesday, November 13, 2018 1:53
To: dev@flink.apache.org
Cc: u...@apache.org
Subject: Re: How to use RocksDBStateBackend predefined options

Sounds good. Perhaps it would also be good to allow the user to specify an
options factory in flink-conf.yaml for more flexibility?

Thomas

On Mon, Nov 12, 2018 at 9:48 AM Stefan Richter <s.rich...@data-artisans.com>
wrote:

> Hi,
>
> Ufuk is right, for historical reasons there is currently only the
> programatic way but I think nothing speaks fundamentally against offering
> configuration via config in the future (maybe just a lot of config keys
> must be introduced to cover all options).
>
> Best,
> Stefan
>
> > On 9. Nov 2018, at 22:52, Ufuk Celebi <u...@apache.org> wrote:
> >
> > Hey Thomas,
> >
> > On Fri, Nov 9, 2018 at 6:07 PM Thomas Weise <t...@apache.org> wrote:
> >> Is there a way to activate the predefined options via configuration /
> flink-
> >> conf.yaml? Or only programmatically, like in [4]? The difficulty with
> the
> >> programmatic route (assuming this works now), is that in my case the
> client
> >> is Beam and I'm not writing the code that submits the job.
> >
> > AFAIK no. You can only do it programmatically at the moment [1].
> > Having the option to configure all settings through the configuration
> > file seems to be a valid feature request to me.
> >
> > @Stefan Richter (cc'd): Are there any reasons that speak against
> > exposing these options via the configuration file in your opinion?
> >
> > Best,
> >
> > Ufuk
> >
> > [1] Looking at the code in [2] and [3], the only options that are
> > exposed via the configuration file are local directories and the timer
> > service factory.
> > [2]
> https://github.com/apache/flink/blob/master/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBOptions.java
> > [3]
> https://github.com/apache/flink/blob/master/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBStateBackend.java
>
>

Reply via email to