[
https://issues.apache.org/jira/browse/FLINK-4115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15350678#comment-15350678
]
ASF GitHub Bot commented on FLINK-4115:
---------------------------------------
Github user aljoscha commented on the issue:
https://github.com/apache/flink/pull/2157
Thanks for the PR! I'll have a look and try and figure out if it's the
right way to go.
> FsStateBackend filesystem verification can cause classpath exceptions
> ---------------------------------------------------------------------
>
> Key: FLINK-4115
> URL: https://issues.apache.org/jira/browse/FLINK-4115
> Project: Flink
> Issue Type: Bug
> Components: Local Runtime
> Affects Versions: 1.1.0
> Reporter: Josh Forman-Gornall
> Priority: Minor
>
> In the constructor of FsStateBackend, the FileSystem for the checkpoint
> directory is initialised and it is verified that the checkpoint path exists.
> This verification happens in the Flink client program when submitting a job
> and can cause classpath issues if classes required to access the file system
> are not available in the client's classpath.
> For example, if we run Flink on YARN over AWS EMR using RocksDBStateBackend
> and an s3:// checkpoint directory, we get the below ClassNotFoundException.
> This is because the jars needed to use the EMR file system are available only
> in the YARN context and not when submitting the job via the Flink client.
> {noformat}
> java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> com.amazon.ws.emr.hadoop.fs.EmrFileSystem not found
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2227)
> at
> org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.getHadoopWrapperClassNameForFileSystem(HadoopFileSystem.java:460)
> at
> org.apache.flink.core.fs.FileSystem.getHadoopWrapperClassNameForFileSystem(FileSystem.java:352)
> at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:280)
> at
> org.apache.flink.runtime.state.filesystem.FsStateBackend.validateAndNormalizeUri(FsStateBackend.java:383)
> at
> org.apache.flink.runtime.state.filesystem.FsStateBackend.<init>(FsStateBackend.java:175)
> at
> org.apache.flink.runtime.state.filesystem.FsStateBackend.<init>(FsStateBackend.java:144)
> at
> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.<init>(RocksDBStateBackend.java:205)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
