Re: Pending parquet file with Bucking Sink

Hi, Your analysis is correct. If the program ends before we can do a checkpoint files will never be moved to "final" state. We could move all files to "final" stage when the Sink is closing but the problem here is that Flink currently doesn't provide a way for user functions (which Sinks are) t

Re: Pending parquet file with Bucking Sink

Hi Vipul, Thanks for the information. Yes, I do have checkpointing enabled with 10 millisecs. I think the issue here is that the stream ended before the checkpoint reached. This is a testing code that the DataStream only have 5 events then it ended. Once the stream ended, the checkpoint is no

Re: Pending parquet file with Bucking Sink

Hi Tao, Is checkpointing enabled in your app? The pending files should be moved to non-pending files after checkpoint interval. Please take a look at this link

Pending parquet file with Bucking Sink

Hi All, Do you guys write parquet file using Bucking Sink? I run into an issue with all the parquet files are in the pending status. Any ideas? processedStream is a DataStream of NDEvent. Output files are all like this one "_part-0-0.pending" val parquetSink = new BucketingSink[NDEvent]("/tmp