[ https://issues.apache.org/jira/browse/FLINK-8794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16383356#comment-16383356 ]
yanxiaobin edited comment on FLINK-8794 at 3/2/18 8:40 AM: ----------------------------------------------------------- hi, [~pnowojski] Thank you for your suggestion! The downstream processor can ignore the files with "*pending" or "*in-progress" sufixes and "_" prefix, but I don't think it's a good way to deal with it. We can change this behaviour/add an option for BucketingSink to use temporary "in-progress" and "pending" directories instead of prefixes, but the temporary "in-progress" and "pending" directories is still also a subdirectory of the base directory, and the downstream processor may still read the base directory recursively, It also results in reading redundant dirty data. I think the temporary data produced during the program should be isolated from the final output data. Thanks! Also [~kkl0u] could you elaborate why rescaling forced us to keep lingering files? was (Author: backlight): hi, [~pnowojski] ! Thank you for your suggestion! The downstream processor can ignore the files with "*pending" or "*in-progress" sufixes and "_" prefix, but I don't think it's a good way to deal with it. We can change this behaviour/add an option for BucketingSink to use temporary "in-progress" and "pending" directories instead of prefixes, but the temporary "in-progress" and "pending" directories is still also a subdirectory of the base directory, and the downstream processor may still read the base directory recursively, It also results in reading redundant dirty data. I think the temporary data produced during the program should be isolated from the final output data. Thanks! Also [~kkl0u] could you elaborate why rescaling forced us to keep lingering files? > When using BucketingSink, it happens that one of the files is always in the > [.in-progress] state > ------------------------------------------------------------------------------------------------ > > Key: FLINK-8794 > URL: https://issues.apache.org/jira/browse/FLINK-8794 > Project: Flink > Issue Type: Bug > Components: filesystem-connector > Affects Versions: 1.4.0, 1.4.1 > Reporter: yanxiaobin > Priority: Major > > When using BucketingSink, it happens that one of the files is always in the > [.in-progress] state. And this state has never changed after that. The > underlying use of S3 as storage. > > {code:java} > // code placeholder > {code} > 2018-02-28 11:58:42 147341619 {color:#d04437}_part-28-0.in-progress{color} > 2018-02-28 12:06:27 147315059 part-0-0 > 2018-02-28 12:06:27 147462359 part-1-0 > 2018-02-28 12:06:27 147316006 part-10-0 > 2018-02-28 12:06:28 147349854 part-100-0 > 2018-02-28 12:06:27 147421625 part-101-0 > 2018-02-28 12:06:27 147443830 part-102-0 > 2018-02-28 12:06:27 147372801 part-103-0 > 2018-02-28 12:06:27 147343670 part-104-0 > ...... -- This message was sent by Atlassian JIRA (v7.6.3#76005)