[
https://issues.apache.org/jira/browse/FLINK-26224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl resolved FLINK-26224.
-----------------------------------
Resolution: Won't Fix
This is caused by the fork relying on presto-only for the s3 scheme. It needs
to be fixed in the fork and not Flink
> HAJobRunOnHadoopS3FileSystemITCase fails with
> UnsupportedFileSystemSchemeException
> ----------------------------------------------------------------------------------
>
> Key: FLINK-26224
> URL: https://issues.apache.org/jira/browse/FLINK-26224
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / FileSystem
> Affects Versions: 1.15.0
> Reporter: Matthias Pohl
> Priority: Major
>
> We're observing occassional errors in a Flink fork based on {{master}} in
> {{HAJobRunOnHadoopS3FileSystemITCase}} where the Hadoop S3 FileSystem wasn't
> loaded properly causing the {{UnsupportedFileSystemSchemeException}}:
> {code}
> Feb 16 19:02:52 [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0,
> Time elapsed: 7.391 s <<< FAILURE! - in
> org.apache.flink.fs.s3hadoop.HAJobRunOnHadoopS3FileSystemITCase
> Feb 16 19:02:52 [ERROR]
> org.apache.flink.fs.s3hadoop.HAJobRunOnHadoopS3FileSystemITCase.testJobExecutionInHaMode
> Time elapsed: 1.89 s <<< ERROR!
> Feb 16 19:02:52 java.io.IOException: Could not create FileSystem for highly
> available storage path (s3://bucket-cqycdk/test-cluster/default)
> Feb 16 19:02:52 at
> org.apache.flink.runtime.blob.BlobUtils.createFileSystemBlobStore(BlobUtils.java:102)
> Feb 16 19:02:52 at
> org.apache.flink.runtime.blob.BlobUtils.createBlobStoreFromConfig(BlobUtils.java:86)
> Feb 16 19:02:52 at
> org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createZooKeeperHaServices(HighAvailabilityServicesUtils.java:87)
> Feb 16 19:02:52 at
> org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createAvailableOrEmbeddedServices(HighAvailabilityServicesUtils.java:70)
> Feb 16 19:02:52 at
> org.apache.flink.runtime.minicluster.MiniCluster.createHighAvailabilityServices(MiniCluster.java:549)
> Feb 16 19:02:52 at
> org.apache.flink.runtime.minicluster.MiniCluster.start(MiniCluster.java:397)
> Feb 16 19:02:52 at
> org.apache.flink.runtime.testutils.MiniClusterResource.startMiniCluster(MiniClusterResource.java:245)
> Feb 16 19:02:52 at
> org.apache.flink.runtime.testutils.MiniClusterResource.before(MiniClusterResource.java:105)
> Feb 16 19:02:52 at
> org.apache.flink.runtime.testutils.MiniClusterExtension.before(MiniClusterExtension.java:56)
> Feb 16 19:02:52 at
> org.apache.flink.core.testutils.EachCallbackWrapper.beforeEach(EachCallbackWrapper.java:45)
> Feb 16 19:02:52 at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeBeforeEachCallbacks$2(TestMethodTestDescriptor.java:163)
> Feb 16 19:02:52 at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeBeforeMethodsOrCallbacksUntilExceptionOccurs$6(TestMethodTestDescriptor.java:199)
> Feb 16 19:02:52 at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> Feb 16 19:02:52 at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeBeforeMethodsOrCallbacksUntilExceptionOccurs(TestMethodTestDescriptor.java:199)
> Feb 16 19:02:52 at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeBeforeEachCallbacks(TestMethodTestDescriptor.java:162)
> Feb 16 19:02:52 at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:129)
> Feb 16 19:02:52 at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:66)
> Feb 16 19:02:52 at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
> Feb 16 19:02:52 at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> Feb 16 19:02:52 at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
> Feb 16 19:02:52 at
> org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
> Feb 16 19:02:52 at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
> Feb 16 19:02:52 at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> Feb 16 19:02:52 at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
> Feb 16 19:02:52 at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
> Feb 16 19:02:52 at java.util.ArrayList.forEach(ArrayList.java:1259)
> Feb 16 19:02:52 at
> org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
> Feb 16 19:02:52 at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
> Feb 16 19:02:52 at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> Feb 16 19:02:52 at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
> Feb 16 19:02:52 at
> org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
> Feb 16 19:02:52 at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
> Feb 16 19:02:52 at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> Feb 16 19:02:52 at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
> Feb 16 19:02:52 at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
> Feb 16 19:02:52 at java.util.ArrayList.forEach(ArrayList.java:1259)
> Feb 16 19:02:52 at
> org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
> Feb 16 19:02:52 at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
> Feb 16 19:02:52 at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> Feb 16 19:02:52 at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
> Feb 16 19:02:52 at
> org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
> Feb 16 19:02:52 at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
> Feb 16 19:02:52 at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> Feb 16 19:02:52 at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
> Feb 16 19:02:52 at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
> Feb 16 19:02:52 at
> org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
> Feb 16 19:02:52 at
> org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
> Feb 16 19:02:52 at
> org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
> Feb 16 19:02:52 at
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:107)
> Feb 16 19:02:52 at
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
> Feb 16 19:02:52 at
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
> Feb 16 19:02:52 at
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
> Feb 16 19:02:52 at
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
> Feb 16 19:02:52 at
> org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
> Feb 16 19:02:52 at
> org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
> Feb 16 19:02:52 at
> org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
> Feb 16 19:02:52 at
> org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:53)
> Feb 16 19:02:52 at
> org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.execute(JUnitPlatformProvider.java:188)
> Feb 16 19:02:52 at
> org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:154)
> Feb 16 19:02:52 at
> org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:124)
> Feb 16 19:02:52 at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:428)
> Feb 16 19:02:52 at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
> Feb 16 19:02:52 at
> org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:562)
> Feb 16 19:02:52 at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:548)
> Feb 16 19:02:52 Caused by:
> org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find
> a file system implementation for scheme 's3'. The scheme is directly
> supported by Flink through the following plugin: flink-s3-fs-presto. Please
> ensure that each plugin resides within its own subfolder within the plugins
> directory. See
> https://ci.apache.org/projects/flink/flink-docs-stable/ops/plugins.html for
> more information. If you want to use a Hadoop file system for that scheme,
> please add the scheme to the configuration fs.allowed-fallback-filesystems.
> For a full list of supported file systems, please see
> https://nightlies.apache.org/flink/flink-docs-stable/ops/filesystems/.
> Feb 16 19:02:52 at
> org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:514)
> Feb 16 19:02:52 at
> org.apache.flink.core.fs.FileSystem.get(FileSystem.java:408)
> Feb 16 19:02:52 at
> org.apache.flink.core.fs.Path.getFileSystem(Path.java:274)
> Feb 16 19:02:52 at
> org.apache.flink.runtime.blob.BlobUtils.createFileSystemBlobStore(BlobUtils.java:99)
> Feb 16 19:02:52 ... 63 more
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
