Hi Israel, thanks for reaching out to the Flink community. As Guowei said, the StreamingFileSink can currently only recover from faults if it writes to HDFS or S3. Other file systems are currently not supported if you need fault tolerance.
Maybe Klou can tell you more about the background and what is needed to make it work with other file systems. He is one of the original authors of the StreamingFileSink. Cheers, Till On Thu, May 28, 2020 at 4:39 PM Israel Ekpo <israele...@gmail.com> wrote: > Guowei, > > What do we need to do to add support for it? > > How do I get started on that? > > > > On Wed, May 27, 2020 at 8:53 PM Guowei Ma <guowei....@gmail.com> wrote: > >> Hi, >> I think the StreamingFileSink could not support Azure currently. >> You could find more detailed info from here[1]. >> >> [1] https://issues.apache.org/jira/browse/FLINK-17444 >> Best, >> Guowei >> >> >> Israel Ekpo <israele...@gmail.com> 于2020年5月28日周四 上午6:04写道： >> >>> You can assign the task to me and I will like to collaborate with >>> someone to fix it. >>> >>> On Wed, May 27, 2020 at 5:52 PM Israel Ekpo <israele...@gmail.com> >>> wrote: >>> >>>> Some users are running into issues when using Azure Blob Storage for >>>> the StreamFileSink >>>> >>>> https://issues.apache.org/jira/browse/FLINK-17989 >>>> >>>> The issue is because certain packages are relocated in the POM file and >>>> some classes are dropped in the final shaded jar >>>> >>>> I have attempted to comment out the relocated and recompile the source >>>> but I keep hitting roadblocks of other relocation and filtration each time >>>> I update a specific pom file >>>> >>>> How can this be addressed so that these users can be unblocked? Why are >>>> the classes filtered out? What is the workaround? I can work on the patch >>>> if I have some guidance. >>>> >>>> This is an issue in Flink 1.9 and 1.10 and I believe 1.11 has the same >>>> issue but I am yet to confirm >>>> >>>> Thanks. >>>> >>>> >>>> >>>
