[jira] [Comment Edited] (FLINK-8794) When using BucketingSink, it happens that one of the files is always in the [.in-progress] state
[ https://issues.apache.org/jira/browse/FLINK-8794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16415491#comment-16415491 ] Steve Loughran edited comment on FLINK-8794 at 3/27/18 12:06 PM: - That's amazon EMR's problem. Switch to their "consistent s3" offering for the bucket you are using as the sink Link: https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan-consistent-view.html if you open any issue with the EMR team, and it is just a read-after-write list inconsistency, they'll just point at that was (Author: ste...@apache.org): That's amazon EMR's problem. Switch to their "consistent s3" offering for the bucket you are using as the sink > When using BucketingSink, it happens that one of the files is always in the > [.in-progress] state > > > Key: FLINK-8794 > URL: https://issues.apache.org/jira/browse/FLINK-8794 > Project: Flink > Issue Type: Improvement > Components: filesystem-connector >Affects Versions: 1.4.0, 1.4.1 >Reporter: yanxiaobin >Priority: Major > > When using BucketingSink, it happens that one of the files is always in the > [.in-progress] state. And this state has never changed after that. The > underlying use of S3 as storage. > > {code:java} > // code placeholder > {code} > 2018-02-28 11:58:42 147341619 {color:#d04437}_part-28-0.in-progress{color} > 2018-02-28 12:06:27 147315059 part-0-0 > 2018-02-28 12:06:27 147462359 part-1-0 > 2018-02-28 12:06:27 147316006 part-10-0 > 2018-02-28 12:06:28 147349854 part-100-0 > 2018-02-28 12:06:27 147421625 part-101-0 > 2018-02-28 12:06:27 147443830 part-102-0 > 2018-02-28 12:06:27 147372801 part-103-0 > 2018-02-28 12:06:27 147343670 part-104-0 > .. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (FLINK-8794) When using BucketingSink, it happens that one of the files is always in the [.in-progress] state
[ https://issues.apache.org/jira/browse/FLINK-8794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393956#comment-16393956 ] yanxiaobin edited comment on FLINK-8794 at 3/10/18 1:53 AM: About : 1.What I described above is that there will be such a situation when there is no failure in this job. I think I've found the problem. I found through log that filesystem's rename method has been executed without any exception, but the filename hasn't changed, so I think it should be S3's problem. This should not be a problem with Flink. was (Author: backlight): About : 1.What I described above is that there will be such a situation when there is no failure in this job. I found through log that filesystem's rename method has been executed without any exception, but the filename hasn't changed, so I think it should be S3's problem. This should not be a problem with Flink. > When using BucketingSink, it happens that one of the files is always in the > [.in-progress] state > > > Key: FLINK-8794 > URL: https://issues.apache.org/jira/browse/FLINK-8794 > Project: Flink > Issue Type: Improvement > Components: filesystem-connector >Affects Versions: 1.4.0, 1.4.1 >Reporter: yanxiaobin >Priority: Major > > When using BucketingSink, it happens that one of the files is always in the > [.in-progress] state. And this state has never changed after that. The > underlying use of S3 as storage. > > {code:java} > // code placeholder > {code} > 2018-02-28 11:58:42 147341619 {color:#d04437}_part-28-0.in-progress{color} > 2018-02-28 12:06:27 147315059 part-0-0 > 2018-02-28 12:06:27 147462359 part-1-0 > 2018-02-28 12:06:27 147316006 part-10-0 > 2018-02-28 12:06:28 147349854 part-100-0 > 2018-02-28 12:06:27 147421625 part-101-0 > 2018-02-28 12:06:27 147443830 part-102-0 > 2018-02-28 12:06:27 147372801 part-103-0 > 2018-02-28 12:06:27 147343670 part-104-0 > .. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (FLINK-8794) When using BucketingSink, it happens that one of the files is always in the [.in-progress] state
[ https://issues.apache.org/jira/browse/FLINK-8794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16384954#comment-16384954 ] yanxiaobin edited comment on FLINK-8794 at 3/4/18 4:13 AM: --- yes. If we allow for different directory that should be already enough.So how do we solve this problem? And when using BucketingSink, it happens that one of the files is always in the [.in-progress] state. And this state has never changed after that. was (Author: backlight): yes. If we allow for different directory that should be already enough.So how do we solve this problem? Adn when using BucketingSink, it happens that one of the files is always in the [.in-progress] state. And this state has never changed after that. > When using BucketingSink, it happens that one of the files is always in the > [.in-progress] state > > > Key: FLINK-8794 > URL: https://issues.apache.org/jira/browse/FLINK-8794 > Project: Flink > Issue Type: Bug > Components: filesystem-connector >Affects Versions: 1.4.0, 1.4.1 >Reporter: yanxiaobin >Priority: Major > > When using BucketingSink, it happens that one of the files is always in the > [.in-progress] state. And this state has never changed after that. The > underlying use of S3 as storage. > > {code:java} > // code placeholder > {code} > 2018-02-28 11:58:42 147341619 {color:#d04437}_part-28-0.in-progress{color} > 2018-02-28 12:06:27 147315059 part-0-0 > 2018-02-28 12:06:27 147462359 part-1-0 > 2018-02-28 12:06:27 147316006 part-10-0 > 2018-02-28 12:06:28 147349854 part-100-0 > 2018-02-28 12:06:27 147421625 part-101-0 > 2018-02-28 12:06:27 147443830 part-102-0 > 2018-02-28 12:06:27 147372801 part-103-0 > 2018-02-28 12:06:27 147343670 part-104-0 > .. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (FLINK-8794) When using BucketingSink, it happens that one of the files is always in the [.in-progress] state
[ https://issues.apache.org/jira/browse/FLINK-8794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383356#comment-16383356 ] yanxiaobin edited comment on FLINK-8794 at 3/2/18 8:40 AM: --- hi, [~pnowojski] Thank you for your suggestion! The downstream processor can ignore the files with "*pending" or "*in-progress" sufixes and "_" prefix, but I don't think it's a good way to deal with it. We can change this behaviour/add an option for BucketingSink to use temporary "in-progress" and "pending" directories instead of prefixes, but the temporary "in-progress" and "pending" directories is still also a subdirectory of the base directory, and the downstream processor may still read the base directory recursively, It also results in reading redundant dirty data. I think the temporary data produced during the program should be isolated from the final output data. Thanks! Also [~kkl0u] could you elaborate why rescaling forced us to keep lingering files? was (Author: backlight): hi, [~pnowojski] ! Thank you for your suggestion! The downstream processor can ignore the files with "*pending" or "*in-progress" sufixes and "_" prefix, but I don't think it's a good way to deal with it. We can change this behaviour/add an option for BucketingSink to use temporary "in-progress" and "pending" directories instead of prefixes, but the temporary "in-progress" and "pending" directories is still also a subdirectory of the base directory, and the downstream processor may still read the base directory recursively, It also results in reading redundant dirty data. I think the temporary data produced during the program should be isolated from the final output data. Thanks! Also [~kkl0u] could you elaborate why rescaling forced us to keep lingering files? > When using BucketingSink, it happens that one of the files is always in the > [.in-progress] state > > > Key: FLINK-8794 > URL: https://issues.apache.org/jira/browse/FLINK-8794 > Project: Flink > Issue Type: Bug > Components: filesystem-connector >Affects Versions: 1.4.0, 1.4.1 >Reporter: yanxiaobin >Priority: Major > > When using BucketingSink, it happens that one of the files is always in the > [.in-progress] state. And this state has never changed after that. The > underlying use of S3 as storage. > > {code:java} > // code placeholder > {code} > 2018-02-28 11:58:42 147341619 {color:#d04437}_part-28-0.in-progress{color} > 2018-02-28 12:06:27 147315059 part-0-0 > 2018-02-28 12:06:27 147462359 part-1-0 > 2018-02-28 12:06:27 147316006 part-10-0 > 2018-02-28 12:06:28 147349854 part-100-0 > 2018-02-28 12:06:27 147421625 part-101-0 > 2018-02-28 12:06:27 147443830 part-102-0 > 2018-02-28 12:06:27 147372801 part-103-0 > 2018-02-28 12:06:27 147343670 part-104-0 > .. -- This message was sent by Atlassian JIRA (v7.6.3#76005)