Thanks David. It worked after adding the jar inside a folder.

On Sat, Feb 1, 2020 at 2:37 AM David Magalhães <[email protected]>
wrote:

> Did you put each inside a different folder with their name? Like
> /opt/flink/plugins/s3-fs-presto/flink-s3-fs-presto-1.9.1.jar ?
>
> check
> https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems/s3.html
>
> On Sat, Feb 1, 2020, 07:42 Navneeth Krishnan <[email protected]>
> wrote:
>
>> Hi Arvid,
>>
>> Thanks for the response.
>>
>> I have both the jars under /opt/flink/plugins but I'm still getting the
>> same error message. Also can someone please provide some pointers on how
>> entropy works. How should I setup the directory structure?
>>
>> In the link that you have provided there is a aws-credentials.jar file,
>> not sure where to get the jar since I don't see it in maven repo.
>>
>> *Plugins:*
>>
>> flink-s3-fs-hadoop-1.9.1.jar
>>
>> flink-s3-fs-presto-1.9.1.jar
>>
>>
>> *flink-conf:*
>>
>> high-availability.storageDir: s3p://.../recovery
>>
>>
>>
>> *Error message:*
>>
>> Shutting StandaloneSessionClusterEntrypoint down with application status
>> FAILED. Diagnostics java.io.IOException: Could not create FileSystem for
>> highly available storage (high-availability.storageDir)
>>
>> Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException:
>> Could not find a file system implementation for scheme 's3p'. The scheme is
>> not directly supported by Flink and no Hadoop file system to support this
>> scheme could be loaded.
>>
>> Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException:
>> Hadoop is not in the classpath/dependencies.
>>
>>
>> Thanks
>>
>>
>>
>> On Thu, Jan 30, 2020 at 2:37 AM Arvid Heise <[email protected]> wrote:
>>
>>> Hi Navneeth,
>>>
>>> did you follow the plugin folder structure? [1]
>>>
>>> There is another plugin called flink-s3-fs-presto that you can use.
>>> If you want to use both plugins, use s3a:// for s3-fs-hadoop (output)
>>> and s3p:// for s3-fs-presto (checkpointing).
>>>
>>> [1]
>>> https://ci.apache.org/projects/flink/flink-docs-master/ops/plugins.html#isolation-and-plugin-structure
>>>
>>> On Thu, Jan 30, 2020 at 10:26 AM Navneeth Krishnan <
>>> [email protected]> wrote:
>>>
>>>> Hi All,
>>>>
>>>> I'm trying to migrate from NFS to S3 for checkpointing and I'm facing
>>>> few issues. I have flink running in docker with flink-s3-fs-hadoop jar
>>>> copied to plugins folder. Even after having the jar I'm getting the
>>>> following error: Caused by:
>>>> org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Hadoop is
>>>> not in the classpath/dependencies. Am I missing something?
>>>>
>>>> In the documentation it says "Presto is the recommended file system
>>>> for checkpointing to S3". How can I enable this? Is there a specific
>>>> configuration that I need to do for this?
>>>>
>>>> Also, I couldn't figure out how the entropy injection works. Should I
>>>> create the bucket with checkpoints folder and flink will automatically
>>>> inject an entropy and create a per job checkpoint folder or should I create
>>>> it?
>>>>
>>>> bucket/checkpoints/_entropy_/dashboard-job/
>>>>
>>>> s3.entropy.key: _entropy_
>>>> s3.entropy.length: 4 (default)
>>>>
>>>> Thanks
>>>>
>>>

Reply via email to