Hi Puneet, this is a known limitation and unfortunately `web.upload.dir` currently works only with the local system :( There are multiple issues covering this already, I guess FLINK-16544 [1] summarizes the current state well.
This is something we want to address with the future releases. We've briefly discussed it with Chesnay and the most straightforward / solid approach would be making the "rest submission" stateless (having a single endpoint that takes the jar and runs it right away). We'll also try to make the documentation more explicit in terms of which "file-based" config options only work with the local filesystem for the 1.15 release. [1] https://issues.apache.org/jira/browse/FLINK-16544 Best, D. On Mon, Jan 10, 2022 at 5:11 PM Puneet Duggal <puneetduggal1...@gmail.com> wrote: > Hi Piotr, > > Thank you for your immediate reply. I went through this thread and it was > also mentioned that flink required s3-filesystem related jars which are > present in my HA flink cluster. Also as mentioned in Apache Flink > Documentation for Amazon S3 integration , > > > https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/filesystems/s3/ > > It is mentioned that we can use s3 in all locations where Flink expects > Filesystem URI. (including high availability and RocksDB State Backend). > > Regards, > Puneet Duggal > > On 10-Jan-2022, at 7:13 PM, Piotr Nowojski <pnowoj...@apache.org> wrote: > > Hi Puneet, > > Have you seen this thread before? [1]. It looks like the same issue and > especially this part might be the key: > > > Be aware that the filesystem used by the FileUploadHandler > > is java.nio.file.FileSystem and not > > Flink's org.apache.flink.core.fs.FileSystem for which we provide > different > > FileSystem implementations. > > Best, > Piotrek > > [1] https://www.mail-archive.com/user@flink.apache.org/msg38043.html > > > > pon., 10 sty 2022 o 08:19 Puneet Duggal <puneetduggal1...@gmail.com> > napisał(a): > >> Hi, >> >> Currently i am working with flink HA cluster with 3 job managers and 3 >> zookeeper nodes. Also i am persisting my checkpoints to s3 and hence >> already configured required flink-s3 jars during flink job manager and task >> manager process startup. Now i have configured a variable >> >> web.upload.dir: s3p://d11-flink-job-manager-load/jars >> >> Expectation is that jar upload via rest apis will be uploaded to this >> location and hence is accessible to all 3 job managers (which eventually >> will help in job submission as all 3 job managers will have record of >> uploaded jar to this location). But while uploading the jar, I am facing >> following Illegal Argument Exception which i am not sure why. Also above >> provided s3 location was created before job manager process was even >> started. >> >> *2022-01-09 18:12:46,790 WARN >> org.apache.flink.runtime.rest.FileUploadHandler [] - File >> upload failed.* >> *java.lang.IllegalArgumentException: UploadDirectory is not absolute.* >> at >> org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:138) >> ~[flink-dist_2.12-1.13.1.jar:1.13.1] >> at >> org.apache.flink.runtime.rest.handler.FileUploads.<init>(FileUploads.java:59) >> ~[flink-dist_2.12-1.13.1.jar:1.13.1] >> at >> org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:186) >> ~[flink-dist_2.12-1.13.1.jar:1.13.1] >> at >> org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:69) >> ~[flink-dist_2.12-1.13.1.jar:1.13.1] >> at >> org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) >> ~[flink-dist_2.12-1.13.1.jar:1.13.1] >> at >> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) >> ~[flink-dist_2.12-1.13.1.jar:1.13.1] >> at >> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) >> ~[flink-dist_2.12-1.13.1.jar:1.13.1] >> at >> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) >> ~[flink-dist_2.12-1.13.1.jar:1.13.1] >> at >> org.apache.flink.shaded.netty4.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) >> ~[flink-dist_2.12-1.13.1.jar:1.13.1] >> >> >> >> >
