Re: fllink 1.7.1 and RollingFileSink

Awesome, thanks! Will open a new thread. But yes the inprogress file was helpful. On Thu, Feb 14, 2019, 7:50 AM Kostas Kloudas Hi Vishal, > > For the StreamingFileSink vs Rolling/BucketingSink: > - you can use the StreamingFileSink instead of the Rolling/BucketingSink. > You can see the

Re: fllink 1.7.1 and RollingFileSink

Hi Vishal, For the StreamingFileSink vs Rolling/BucketingSink: - you can use the StreamingFileSink instead of the Rolling/BucketingSink. You can see the StreamingFileSink as an evolution of the previous two. In the StreamingFileSink the files in Pending state are not renamed, but they keep

Re: fllink 1.7.1 and RollingFileSink

Thanks Fabian, more questions 1. I had on k8s standlone job env.getCheckpointConfig().setFailOnCheckpointingErrors(true)// the default. The job failed on chkpoint and I would have imagined that under HA the job would restore from the last checkpoint but it did not ( The UI showed the job had

Re: fllink 1.7.1 and RollingFileSink

Hi Vishal, Kostas (in CC) should be able to help here. Best, Fabian Am Mo., 11. Feb. 2019 um 00:05 Uhr schrieb Vishal Santoshi < vishal.santo...@gmail.com>: > Any one ? > > On Sun, Feb 10, 2019 at 2:07 PM Vishal Santoshi > wrote: > >> You don't have to. Thank you for the input. >> >> On Sun,

Re: fllink 1.7.1 and RollingFileSink

Any one ? On Sun, Feb 10, 2019 at 2:07 PM Vishal Santoshi wrote: > You don't have to. Thank you for the input. > > On Sun, Feb 10, 2019 at 1:56 PM Timothy Victor wrote: > >> My apologies for not seeing your use case properly. The constraint on >> rolling policy is only applicable for bulk

Re: fllink 1.7.1 and RollingFileSink

You don't have to. Thank you for the input. On Sun, Feb 10, 2019 at 1:56 PM Timothy Victor wrote: > My apologies for not seeing your use case properly. The constraint on > rolling policy is only applicable for bulk formats such as Parquet as > highlighted in the docs. > > As for your

Re: fllink 1.7.1 and RollingFileSink

My apologies for not seeing your use case properly. The constraint on rolling policy is only applicable for bulk formats such as Parquet as highlighted in the docs. As for your questions, I'll have to defer to others more familiar with it. I mostly just use bulk formats such as avro and

Re: fllink 1.7.1 and RollingFileSink

That said the in the DefaultRollingPolicy it seems the check is on the file size ( mimics the check shouldRollOnEVent()). I guess the question is Is the call to shouldRollOnCheckPoint. done by the checkpointing thread ? Are the calls to the other 2 methods shouldRollOnEVent and

Re: fllink 1.7.1 and RollingFileSink

Thanks for the quick reply. I am confused. If this was a more full featured BucketingSink ,I would imagine that based on shouldRollOnEvent and shouldRollOnEvent, an in progress file could go into pending phase and on checkpoint the pending part file would be finalized. For exactly once any

Re: fllink 1.7.1 and RollingFileSink

I think the only rolling policy that can be used is CheckpointRollingPolicy to ensure exactly once. Tim On Sun, Feb 10, 2019, 9:13 AM Vishal Santoshi Can StreamingFileSink be used instead of > https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/connectors/filesystem_sink.html, >

fllink 1.7.1 and RollingFileSink

Can StreamingFileSink be used instead of https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/connectors/filesystem_sink.html, even though it looks it could. This code for example StreamingFileSink .forRowFormat(new Path(PATH), new