Re: Checkpointing with RocksDB as statebackend

et this working. >>> >>> @Stefan or @Stephan : can you please help in resolving this issue >>> >>> Regards, >>> Vinay Patil >>> >>> On Thu, Jun 29, 2017 at 6:01 PM, gerryzhou [via Apache Flink User Mailing >>> List archive.]

Re: Checkpointing with RocksDB as statebackend

t; >>> @Stefan or @Stephan : can you please help in resolving this issue >>> >>> Regards, >>> Vinay Patil >>> >>> On Thu, Jun 29, 2017 at 6:01 PM, gerryzhou [via Apache Flink User >>> Mailing List archive.] wrote: >>> >>

Re: Checkpointing with RocksDB as statebackend

a similar problem in flink 1.3.0 with rocksdb. I wonder >>> how to use FRocksDB as you mentioned above. Thanks. >>> >>> -- >>> If you reply to this email, your message will be added to the discussion >>> below: >>> http

Re: Checkpointing with RocksDB as statebackend

ailing >> List archive.] > <mailto:ml+s2336050n1406...@n4.nabble.com>> wrote: >> Hi, Vinay, >> I observed a similar problem in flink 1.3.0 with rocksdb. I wonder how >> to use FRocksDB as you mentioned above. Thanks. >> >> If you reply to th

Re: Checkpointing with RocksDB as statebackend

ioned above. Thanks. >> >> -- >> If you reply to this email, your message will be added to the discussion >> below: >> http://apache-flink-user-mailing-list-archive.2336050.n4. >> nabble.com/Re-Checkpointing-with-RocksDB-as-statebackend

Re: Checkpointing with RocksDB as statebackend

sdb. I wonder how > to use FRocksDB as you mentioned above. Thanks. > > If you reply to this email, your message will be added to the discussion > below: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Re-Checkpointing-with-RocksDB-as-statebackend-tp11752p14063.

Re: Checkpointing with RocksDB as statebackend

d above. Thanks. > > > > -- > View this message in context: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Re-Checkpointing-with-RocksDB-as-statebackend-tp11752p14063.html > Sent from the Apache Flink User Mailing List archive. mailing list archive at > Nabble.com.

Re: Checkpointing with RocksDB as statebackend

e. Thanks. > > -- > If you reply to this email, your message will be added to the discussion > below: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Re- > Checkpointing-with-RocksDB-as-statebackend-tp11752p14063.html > To start

Re: Checkpointing with RocksDB as statebackend

Hi, Vinay, I observed a similar problem in flink 1.3.0 with rocksdb. I wonder how to use FRocksDB as you mentioned above. Thanks. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Re-Checkpointing-with-RocksDB-as-statebackend

Re: Checkpointing with RocksDB as statebackend

disk. >>> > >>> > I have attached the snapshot for reference. >>> > >>> > Also the data processed till now is only 17GB and above 120GB memory is >>> > getting used. >>> > >>> > Is there any change wrt RocksDB configurations >>> > >>> > <http://apache-flink-user-mailing-list-archive.2336050.n4.na >>> bble.com/file/n14013/TM_Memory_Usage.png> >>> > >>> > Regards, >>> > Vinay Patil >>> > >>> > >>> > >>> > -- >>> > View this message in context: http://apache-flink-user-maili >>> ng-list-archive.2336050.n4.nabble.com/Re-Checkpointing-with- >>> RocksDB-as-statebackend-tp11752p14013.html >>> > Sent from the Apache Flink User Mailing List archive. mailing list >>> archive at Nabble.com. >>> >>> >> >

Re: Checkpointing with RocksDB as statebackend

> Also the data processed till now is only 17GB and above 120GB memory is >> > getting used. >> > >> > Is there any change wrt RocksDB configurations >> > >> > <http://apache-flink-user-mailing-list-archive.2336050.n4.na >> bble.com/file/n14013/TM_Mem

Re: Checkpointing with RocksDB as statebackend

4013/TM_Memory_Usage.png> > > > > Regards, > > Vinay Patil > > > > > > > > -- > > View this message in context: http://apache-flink-user-maili > ng-list-archive.2336050.n4.nabble.com/Re-Checkpointing- > with-RocksDB-as-statebackend-tp11752p14013.html > > Sent from the Apache Flink User Mailing List archive. mailing list > archive at Nabble.com. > >

Re: Checkpointing with RocksDB as statebackend

3/TM_Memory_Usage.png> > > > Regards, > Vinay Patil > > > > -- > View this message in context: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Re-Checkpointing-with-RocksDB-as-statebackend-tp11752p14013.html > Sent from the Apache Flink User Mailing List archive. mailing list archive at > Nabble.com.

Re: Checkpointing with RocksDB as statebackend

RocksDB configurations <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/file/n14013/TM_Memory_Usage.png> Regards, Vinay Patil -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Re-Checkpointing-with-RocksDB-as-stateb

Re: Checkpointing with RocksDB as statebackend

;> >>>>>>>>>>> Best, >>>>>>>>>>> Stefan >>>>>>>>>>> >>>>>>>>>>> Am 14.03.2017 um 15:31 schrieb Vishnu Viswanath <[hidden email

Re: Checkpointing with RocksDB as statebackend

t would cause a >> FileNotFound exception which would fail the checkpoint. >> >> >> >> Stephan, >> >> >> >> Currently my aws fork contains some very specific assumptions about the >> pipeline that will in general only hold for my pipeline. Th

Re: Checkpointing with RocksDB as statebackend

>>>> >>>>>>>>> A certain access pattern in RocksDB starts being so slow after a >>>>>>>>> certain size-per-key that it basically brings down the streaming >>>>>>>>> program >>>&g

Re: Checkpointing with RocksDB as statebackend

ings, >>>>>>>> Stephan >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Mar 1, 2017 at 12:10 PM, Stephan Ewen <[hidden email] >>>>>>>> <http:///user/SendEmail.jtp?type=node&node=12209&

Re: Checkpointing with RocksDB as statebackend

;>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> >>>>>>>>> Vinay Patil >>>>>>>>> >>&

Re: Checkpointing with RocksDB as statebackend

gt;>>> Stephan >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Mar 1, 2017 at 12:10 PM, Stephan Ewen <[hidden email] >>>>>>>> <http:///user/SendEmail.jtp?type=node&node=12209&i=3>> wrote: >>>>&

Re: Checkpointing with RocksDB as statebackend

; >>>>>>>> @vinay Can you try to not set the buffer timeout at all? I am >>>>>>>> actually not sure what would be the effect of setting it to a negative >>>>>>>> value, that can be a cause of problems... >>>>>&g

Re: Checkpointing with RocksDB as statebackend

gt; On Mon, Feb 27, 2017 at 7:44 PM, Seth Wiesman <[hidden email] >>>>>>> <http:///user/SendEmail.jtp?type=node&node=12209&i=4>> wrote: >>>>>>> >>>>>>>> Vinay, >>>>>>>> >>>>>>&

Re: Checkpointing with RocksDB as statebackend

;>> >>>>>>> Vinay, >>>>>>> >>>>>>> >>>>>>> >>>>>>> The bucketing sink performs rename operations during the checkpoint >>>>>>> and if it tries to rename

Re: Checkpointing with RocksDB as statebackend

HI Stephan, >>>>>> >>>>>> Just to avoid the confusion here, I am using S3 sink for writing the >>>>>> data, and using HDFS for storing checkpoints. >>>>>> >>>>>> There are 2 cor

Re: Checkpointing with RocksDB as statebackend

>>>> >>>>> >>>>> >>>>> Currently my aws fork contains some very specific assumptions about >>>>> the pipeline that will in general only hold for my pipeline. This is >>>>> because there were still some

Re: Checkpointing with RocksDB as statebackend

;>> pipeline that will in general only hold for my pipeline. This is because >>>> there were still some open questions that I had about how to solve >>>> consistency issues in the general case. I will comment on the Jira issue >>>> with more specific. >>

Re: Checkpointing with RocksDB as statebackend

to:vinay18.pa...@gmail.com>> > Reply-To: "user@flink.apache.org <mailto:user@flink.apache.org>" > mailto:user@flink.apache.org>> > Date: Monday, February 27, 2017 at 1:05 PM > To: "user@flink.apache.org <mailto:user@flink.apache.org>" >

Re: Checkpointing with RocksDB as statebackend

he >>> pipeline that will in general only hold for my pipeline. This is because >>> there were still some open questions that I had about how to solve >>> consistency issues in the general case. I will comment on the Jira issue >>> with more specific. >>

Re: Checkpointing with RocksDB as statebackend

many minutes to >> rename which would stall the entire pipeline. The only viable solution I >> could find was to write a custom sink which understands S3. Each writer >> will write file locally and then copy it to S3 on checkpoint. By only >> interacting with S3 once per file it c

Re: Checkpointing with RocksDB as statebackend

gt; > > *From: *vinay patil > *Reply-To: *"user@flink.apache.org" > *Date: *Monday, February 27, 2017 at 1:05 PM > *To: *"user@flink.apache.org" > > *Subject: *Re: Checkpointing with RocksDB as statebackend > > > > Hi Seth, > > Thank you fo

Re: Checkpointing with RocksDB as statebackend

;user@flink.apache.org" Date: Monday, February 27, 2017 at 1:05 PM To: "user@flink.apache.org" Subject: Re: Checkpointing with RocksDB as statebackend Hi Seth, Thank you for your suggestion. But if the issue is only related to S3, then why does this happen when I replace the S3 sink

Re: Checkpointing with RocksDB as statebackend

>> *From: *vinay patil <[hidden email] >> <http:///user/SendEmail.jtp?type=node&node=11943&i=1>> >> *Reply-To: *"[hidden email] >> <http:///user/SendEmail.jtp?type=node&node=11943&i=2>" <[hidden email] >> <http:///user/SendEmail.jt

Re: Checkpointing with RocksDB as statebackend

By only > interacting with S3 once per file it can circumvent consistency issues all > together. > > > > Hope this helps, > > > > Seth Wiesman > > > > *From: *vinay patil > *Reply-To: *"user@flink.apache.org" > *Date: *Saturday, February 25, 2017 at

Re: Checkpointing with RocksDB as statebackend

, Seth Wiesman From: vinay patil Reply-To: "user@flink.apache.org" Date: Saturday, February 25, 2017 at 10:50 AM To: "user@flink.apache.org" Subject: Re: Checkpointing with RocksDB as statebackend HI Stephan, Just to avoid the confusion here, I am using S3 sink for writing

Re: Checkpointing with RocksDB as statebackend

next >>>>> 16minutes the >>>>> pipeline is stuck , I don't see any progress beyond 15M because of >>>>> checkpoints getting failed consistently. >>>>> >>>>> <http://apache-flink-user-mailing-list-archive.2336050.n

Re: Checkpointing with RocksDB as statebackend

t;>> 3 it >>>> is stuck at the Kafka source after 50% >>>> (The data sent till now by Kafka source 1 is 65GB and sent by source 2 >>>> is >>>> 15GB ) >>>> >>>> Within 10minutes 15M records were processed, and for the nex

Re: Checkpointing with RocksDB as statebackend

15GB ) >>> >>> Within 10minutes 15M records were processed, and for the next 16minutes >>> the >>> pipeline is stuck , I don't see any progress beyond 15M because of >>> checkpoints getting failed consistently. >>> >>> <http://apach

Re: Checkpointing with RocksDB as statebackend

336050.n4. >> nabble.com/file/n11882/Checkpointing_Failed.png> >> >> >> >> -- >> View this message in context: http://apache-flink-user-maili >> ng-list-archive.2336050.n4.nabble.com/Re-Checkpointing- >> with-RocksDB-as-statebackend-tp11752p11882.

Re: Checkpointing with RocksDB as statebackend

ly. > > <http://apache-flink-user-mailing-list-archive.2336050. > n4.nabble.com/file/n11882/Checkpointing_Failed.png> > > > > -- > View this message in context: http://apache-flink-user- > mailing-list-archive.2336050.n4.nabble.com/Re- > Checkpointing-with-RocksDB

Re: Checkpointing with RocksDB as statebackend

pache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Re-Checkpointing-with-RocksDB-as-statebackend-tp11752p11882.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: Checkpointing with RocksDB as statebackend

y are done asynchronously. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Re-Checkpointing-with-RocksDB-as-statebackend-tp11752p11879.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: Checkpointing with RocksDB as statebackend

-- >> If you reply to this email, your message will be added to the discussion >> below: >> http://apache-flink-user-mailing-list-archive.2336050.n4. >> nabble.com/Re-Checkpointing-with-RocksDB-as-statebackend- >> tp11752p11831.html >> To start a new topic under Apac

Re: Checkpointing with RocksDB as statebackend

ally distributed among all TM's. >> Why does this happen ? >> >> -- >> If you reply to this email, your message will be added to the discussion >> below: >> http://apache-flink-user-mailing-list-archive.2336050.n4. >> nabble.com/

Re: Checkpointing with RocksDB as statebackend

x27;s. > Why does this happen ? > > -- > If you reply to this email, your message will be added to the discussion > below: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Re- > Checkpointing-with-RocksDB-as-statebackend-tp11752p11831.html > To start a new topic

Re: Checkpointing with RocksDB as statebackend

TM's. Why does this happen ? -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Re-Checkpointing-with-RocksDB-as-statebackend-tp11752p11831.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: Checkpointing with RocksDB as statebackend

> Regards, > Vinay Patil > > > > -- > View this message in context: http://apache-flink-user- > mailing-list-archive.2336050.n4.nabble.com/Re- > Checkpointing-with-RocksDB-as-statebackend-tp11752p11799.html > Sent from the Apache Flink User Mailing List archive. mailing list archive > at Nabble.com. >

Re: Checkpointing with RocksDB as statebackend

) Regards, Vinay Patil -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Re-Checkpointing-with-RocksDB-as-statebackend-tp11752p11799.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: Checkpointing with RocksDB as statebackend

DB to store >>>>>> necessary indices. To avoid the unlimited growth in the memory >>>>>> consumption, you can put these indices into block cache (set >>>>>> CacheIndexAndFilterBlock to true) and properly set the block cache size. >>

Re: Checkpointing with RocksDB as statebackend

FilterBlock >>>>> to >>>>> true) and properly set the block cache size. >>>>> >>>>> You can also increase the number of backgroud threads to improve the >>>>> performance of flushes and compactions (via MaxBackgroundFlu

Re: Checkpointing with RocksDB as statebackend

et the block cache size. >>>> >>>> You can also increase the number of backgroud threads to improve the >>>> performance of flushes and compactions (via MaxBackgroundFlushes and >>>> MaxBackgroudCompactions). >>>> >>>> In YARN clusters, task managers will be killed if their memory >>>> util

Re: Checkpointing with RocksDB as statebackend

age in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Re-Checkpointing-with-RocksDB-as-statebackend-tp11752p11759.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: Checkpointing with RocksDB as statebackend

does not count the >>> memory used by RocksDB in the allocation. We are working on fine-grained >>> resource allocation (see FLINK-5131). It may help to avoid such problems. >>> >>> May the information helps you. >>> >>> Regards, >>> Xia

Re: Checkpointing with RocksDB as statebackend

t; >> May the information helps you. >> >> Regards, >> Xiaogang >> >> >> ------------------ >> 发件人：Vinay Patil <[hidden email] >> <http:///user/SendEmail.jtp?type=node&node=11731&i=0>

Re: Checkpointing with RocksDB as statebackend

jtp?type=node&node=11731&i=0>> > 发送时间：2017年2月17日(星期五) 21:19 > 收件人：user <[hidden email] > <http:///user/SendEmail.jtp?type=node&node=11731&i=1>> > 主 题：Re: Checkpointing with RocksDB as statebackend > > Hi Guys, > > There seems to be some issue w

Re: Checkpointing with RocksDB as statebackend

Hi Guys, There seems to be some issue with RocksDB memory utilization. Within few minutes of job run the physical memory usage increases by 4-5 GB and it keeps on increasing. I have tried different options for Max Buffer Size(30MB, 64MB, 128MB , 512MB) and Min Buffer to Merge as 2, but the physic

Re: Checkpointing with RocksDB as statebackend

I think its more of related to RocksDB, I am also not aware about RocksDB but reading the tuning guide to understand the important values that can be set Regards, Vinay Patil On Thu, Feb 16, 2017 at 5:48 PM, Stefan Richter [via Apache Flink User Mailing List archive.] wrote: > What kind of prob

Re: Checkpointing with RocksDB as statebackend

What kind of problem are we talking about? S3 related or RocksDB related. I am not aware of problems with RocksDB per se. I think seeing logs for this would be very helpful. > Am 16.02.2017 um 11:56 schrieb Aljoscha Krettek : > > +Stefan Richter and +Stephan

Re: Checkpointing with RocksDB as statebackend

Hi Aljoscha, Which problem you are referring to ? I am seeing unexpected stalls in between for a long time. Also one thing I have observed with FLASH_SSD_OPTIMIZED option is that it is using more amount of physical memory and not flushing the data to storage. I am trying to figure out the best

Re: Checkpointing with RocksDB as statebackend

+Stefan Richter and +Stephan Ewen could this be the same problem that you recently saw when working with other people? On Wed, 15 Feb 2017 at 17:23 Vinay Patil wrote: > Hi Guys, > > Can anyone please help me with this issue > > Regards, > Vinay Patil > > On Wed, Feb 15, 2017 at 6:17 PM, Vinay

Re: Checkpointing with RocksDB as statebackend

Hi Guys, Can anyone please help me with this issue Regards, Vinay Patil On Wed, Feb 15, 2017 at 6:17 PM, Vinay Patil wrote: > Hi Ted, > > I have 3 boxes in my pipeline , 1st and 2nd box containing source and s3 > sink and the 3rd box is window operator followed by chained operators and a > s3

Re: Checkpointing with RocksDB as statebackend

Hi Ted, I have 3 boxes in my pipeline , 1st and 2nd box containing source and s3 sink and the 3rd box is window operator followed by chained operators and a s3 sink So in the details link section I can see that that S3 sink is taking time for the acknowledgement and it is not even going to the wi

Re: Checkpointing with RocksDB as statebackend

What did the More Details link say ? Thanks > On Feb 15, 2017, at 3:11 AM, vinay patil wrote: > > Hi, > > I have kept the checkpointing interval to 6secs and minimum pause between > checkpoints to 5secs, while testing the pipeline I have observed that that > for some checkpoints it is taking