[
https://issues.apache.org/jira/browse/FLINK-23888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17412004#comment-17412004
]
Konstantinos Papalias commented on FLINK-23888:
-----------------------------------------------
[~chesnay] the issue should be re-producible if you try to enable
high-availability using `azure-fs-hadoop`.
Is there any reason to include the shaded libraries in the first in comparison
to AWS S3 plug-in? Following the same approach as AWS S3 the plug-in is cleaner
and the issue goes away for high-availability.
Without this we cannot even start the pipeline with HA on Azure.
Any help or insights are appreciated :)
> Flink azure fs doesn't work
> ---------------------------
>
> Key: FLINK-23888
> URL: https://issues.apache.org/jira/browse/FLINK-23888
> Project: Flink
> Issue Type: Bug
> Components: FileSystems
> Affects Versions: 1.13.1, 1.13.2
> Reporter: Liviu Firu
> Priority: Major
> Attachments: pom.xml
>
>
> A working pipeline on AWS S3 doesn't work with BlobStorage.
> Flink deployed in kubernetes with HA . The following is the configuration :
> high-availability:
> org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
> high-availability.storageDir: wasbs://***
> state.backend: rocksdb
> state.checkpoints.dir: wasb://***
> I always got LinkageError.
> I believe the issue is that azure / hadoop classes are shaded with
> "org.apache.flink" pattern and always get loaded via parent not via plugin
> class loader.
> Snapshot of the log :
> 2021-08-18 11:56:03,557 INFO org.apache.flink.fs.azurefs.AzureFSFactory
> [] - Trying to load and instantiate Azure File System
> 2021-08-18 11:56:03,747 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Shutting
> StandaloneApplicationClusterEntryPoint down with application status FAILED.
> Diagnostics java.lang.LinkageError: loader
> org.apache.flink.core.plugin.PluginLoader$PluginClassLoader @61bcbcce
> attempted duplicate class definition for
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.LocalFileSystem.
> (org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.LocalFileSystem is
> in unnamed module of loader
> org.apache.flink.core.plugin.PluginLoader$PluginClassLoader @61bcbcce, parent
> loader 'platform')
> at java.base/java.lang.ClassLoader.defineClass1(Native Method)
> at java.base/java.lang.ClassLoader.defineClass(Unknown Source)
> at java.base/java.security.SecureClassLoader.defineClass(Unknown
> Source)
> at java.base/java.net.URLClassLoader.defineClass(Unknown Source)
> at java.base/java.net.URLClassLoader$1.run(Unknown Source)
> at java.base/java.net.URLClassLoader$1.run(Unknown Source)
> at java.base/java.security.AccessController.doPrivileged(Native
> Method)
> at java.base/java.net.URLClassLoader.findClass(Unknown Source)
> at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
> at
> org.apache.flink.core.plugin.PluginLoader$PluginClassLoader.loadClass(PluginLoader.java:171)
> at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
> at
> org.apache.flink.fs.azure.common.HadoopConfigLoader.loadHadoopConfigFromFlink(HadoopConfigLoader.java:96)
> at
> org.apache.flink.fs.azure.common.HadoopConfigLoader.getOrLoadHadoopConfig(HadoopConfigLoader.java:82)
> at
> org.apache.flink.fs.azurefs.AbstractAzureFSFactory.createInitializedAzureFS(AbstractAzureFSFactory.java:85)
> at
> org.apache.flink.fs.azurefs.AbstractAzureFSFactory.create(AbstractAzureFSFactory.java:79)
> at
> org.apache.flink.core.fs.PluginFileSystemFactory.create(PluginFileSystemFactory.java:62)
> at
> org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:506)
> at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:407)
> at org.apache.flink.core.fs.Path.getFileSystem(Path.java:274)
> at
> org.apache.flink.runtime.blob.BlobUtils.createFileSystemBlobStore(BlobUtils.java:89)
> at
> org.apache.flink.runtime.blob.BlobUtils.createBlobStoreFromConfig(BlobUtils.java:76)
> at
> org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory.createHAServices(KubernetesHaServicesFactory.java:40)
> at
> org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createCustomHAServices(HighAvailabilityServicesUtils.java:265)
> at
> org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createHighAvailabilityServices(HighAvailabilityServicesUtils.java:124)
> at
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.createHaServices(ClusterEntrypoint.java:353)
> at
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.initializeServices(ClusterEntrypoint.java:311)
> at
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:239)
> at
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$1(ClusterEntrypoint.java:189)
> at
> org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
> at
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:186)
> at
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:600)
> at
> org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.main(StandaloneApplicationClusterEntryPoint.java:85)
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
