Hi Guys,
I am able to overcome the physical memory consumption issue by setting the
following options of RocksDB:
*DBOptions: *
along with the FLASH_SSD_OPTIONS added the following:
maxBackgroundCompactions(4)
*ColumnFamilyOptions:*
max_buffer_size : 512 MB
block_cache_size : 128 MB
max_write_buffer_number : 5
minimum_buffer_number_to_merge : 2
cacheIndexAndFilterBlocks : true
optimizeFilterForHits: true
I am going to modify these options to get the desired performance.
I have set the DbLogDir("/var/tmp/") and StatsSumpPeriodicSec but I only
got the configurations set in the log file present in var/tmp/
Where will I get the RocksDB statistics if I set createStatistics ?
Let me know if any other configurations will help to get better
performance. Now the physical memory is slowly getting increased and I can
see the drop in the graph ( this means flushing is taking place at regular
intervals )
Regards,
Vinay Patil
On Thu, Jun 29, 2017 at 9:13 PM, Vinay Patil <vinay18.pa...@gmail.com>
wrote:
> The state size is not that huge. On the Flink UI when it showed the data
> sent as 4GB , the physical memory usage was close to 90GB ..
>
> I will re-run by setting the Flushing options of RocksDB because I am
> facing this issue on 1.2.0 as well.
>
> Regards,
> Vinay Patil
>
> On Thu, Jun 29, 2017 at 9:03 PM, Aljoscha Krettek <aljos...@apache.org>
> wrote:
>
>> Yup, I got that. I’m just wondering whether this occurs only with enabled
>> checkpointing or also when checkpointing is disabled.
>>
>> On 29. Jun 2017, at 17:31, Vinay Patil <vinay18.pa...@gmail.com> wrote:
>>
>> Hi Aljoscha,
>>
>> Yes I have tried with 1.2.1 and 1.3.0 , facing the same issue.
>>
>> The issue is not of Heap memory , it is of the Off-Heap memory that is
>> getting used ( please refer to the earlier snapshot I have attached in
>> which the graph keeps on growing ).
>>
>>
>> Regards,
>> Vinay Patil
>>
>> On Thu, Jun 29, 2017 at 8:55 PM, Aljoscha Krettek <aljos...@apache.org>
>> wrote:
>>
>>> Just a quick remark: Flink 1.3.0 and 1.2.1 always use FRocksDB, you
>>> shouldn’t manually specify that.
>>>
>>> On 29. Jun 2017, at 17:20, Vinay Patil <vinay18.pa...@gmail.com> wrote:
>>>
>>> Hi Gerry,
>>>
>>> Even I have faced this issue on 1.3.0 even by using FRocksDB and
>>> enabling incremental checkpointing.
>>>
>>> You can add FRocksDB dependency as shown here :
>>> https://github.com/apache/flink/pull/3704
>>>
>>> We will have to set some RocksDB parameters to get this working.
>>>
>>> @Stefan or @Stephan : can you please help in resolving this issue
>>>
>>> Regards,
>>> Vinay Patil
>>>
>>> On Thu, Jun 29, 2017 at 6:01 PM, gerryzhou [via Apache Flink User
>>> Mailing List archive.] <ml+s2336050n1406...@n4.nabble.com> wrote:
>>>
>>>> Hi, Vinay,
>>>> I observed a similar problem in flink 1.3.0 with rocksdb. I wonder
>>>> how to use FRocksDB as you mentioned above. Thanks.
>>>>
>>>> ------------------------------
>>>> If you reply to this email, your message will be added to the
>>>> discussion below:
>>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nab
>>>> ble.com/Re-Checkpointing-with-RocksDB-as-statebackend-tp1175
>>>> 2p14063.html
>>>> To start a new topic under Apache Flink User Mailing List archive.,
>>>> email ml+s2336050n1...@n4.nabble.com
>>>> To unsubscribe from Apache Flink User Mailing List archive., click here
>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=dmluYXkxOC5wYXRpbEBnbWFpbC5jb218MXwxODExMDE2NjAx>
>>>> .
>>>> NAML
>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>>
>>>
>>>
>>>
>>
>>
>
