yes I copied to plugin folder but not sure same jar I see in /opt as well by
default
root@d852f125da1f:/opt/flink/plugins# lsREADME.txt
flink-s3-fs-hadoop-1.13.1.jar metrics-datadog metrics-influx
metrics-prometheus metrics-statsdexternal-resource-gpu
flink-s3-fs-presto-1.13.1.jar metrics-graphite metrics-jmx metrics-slf4j
I need help sooner on this
On Wednesday, September 8, 2021, 09:26:46 AM EDT, Dhiru
<[email protected]> wrote:
yes I copied to plugin folder but not sure same jar I see in /opt as well by
default
root@d852f125da1f:/opt/flink/plugins# lsREADME.txt
flink-s3-fs-hadoop-1.13.1.jar metrics-datadog metrics-influx
metrics-prometheus metrics-statsdexternal-resource-gpu
flink-s3-fs-presto-1.13.1.jar metrics-graphite metrics-jmx metrics-slf4j
On Wednesday, September 8, 2021, 02:58:38 AM EDT, Martijn Visser
<[email protected]> wrote:
Hi,
Have you copied the correct JAR [1] to the plugins directory?
Best regards,
Martijn
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/filesystems/s3.html
On Wed, 8 Sept 2021 at 04:27, Dhiru <[email protected]> wrote:
Need to configure aws S3 getting this error
org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find
a file system implementation for scheme 's3'. The scheme is directly supported
by Flink through the following plugins: flink-s3-fs-hadoop, flink-s3-fs-presto.
Please ensure that each plugin resides within its own subfolder within the
plugins directory. See
https://ci.apache.org/projects/flink/flink-docs-stable/ops/plugins.html for
more information. If you want to use a Hadoop file system for that scheme,
please add the scheme to the configuration fs.allowed-fallback-filesystems. For
a full list of supported file systems, please see
https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems/. at
org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:473)
at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:389) at
org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink$RowFormatBuilder.createBuckets(StreamingFileSink.java:260)
at
org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink.initializeState(StreamingFileSink.java:396)
at
org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:185)
at
org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:167)
at
org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)
at
org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.initializeOperatorState(StreamOperatorStateHandler.java:106)
at
org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:260)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:290)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$1(StreamTask.java:506)
at
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:92)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:475)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:526)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721) at
org.apache.flink.runtime.taskmanager.Task.run(Task.java:546) at
java.base/java.lang.Thread.run(Unknown Source)
image used flink:1.11.3-scala_2.12-java11
some of the part which I added for flink-conf.yaml # REQUIRED: set storage
location for job metadata in remote storage high-availability.storageDir:
s3://msc-sandbox-flink/test state.backend: filesystem
state.checkpoints.dir: s3://msc-sandbox-flink/test
state.backend.fs.checkpointdir: s3://msc-sandbox-flink/test
s3.path.style.access: true
I am trying to deploy jobmanager and taskmanager using eks , please let me know
if I need to do any more support for adding s3
