Re: EXT :Re: StreamingFileSink cannot get AWS S3 credentials

[Vinay Patil]

Hi Stephan.,

Yes, we tried setting fs.s3a.aws.credentials.provider but we are getting
class not found exception for InstanceProfileCredentialsProvider because of
shading issue.


Regards,
Vinay Patil


On Thu, Jan 17, 2019 at 3:02 PM Stephan Ewen  wrote:

> Regarding configurations: According to the code [1]
> <https://github.com/apache/flink/blob/master/flink-filesystems/flink-s3-fs-hadoop/src/main/java/org/apache/flink/fs/s3hadoop/S3FileSystemFactory.java#L50>,
> all config keys starting with "s3", "s3a" and "fs.s3a" are forwarded from
> the flink-conf.yaml to the Hadoop file systems.
>
> Regarding profile-based authentication: Have you tried to set the
> credentials provider explicitly, by setting "
> fs.s3a.aws.credentials.provider" ?
>
> [1]
> https://github.com/apache/flink/blob/master/flink-filesystems/flink-s3-fs-hadoop/src/main/java/org/apache/flink/fs/s3hadoop/S3FileSystemFactory.java#L50
>
> On Thu, Jan 17, 2019 at 12:57 AM Martin, Nick  wrote:
>
>> Does that mean that the full set of fs.s3a.<…> configs in core-site.xml
>> will be picked up from flink-conf.yaml by flink? Or only certain configs
>> involved with authentication?
>>
>>
>>
>> *From:* Till Rohrmann [mailto:trohrm...@apache.org]
>> *Sent:* Wednesday, January 16, 2019 3:43 AM
>> *To:* Vinay Patil 
>> *Cc:* Kostas Kloudas ; Dawid Wysakowicz <
>> dwysakow...@apache.org>; Taher Koitawala [via Apache Flink User Mailing
>> List archive.] ; user <
>> user@flink.apache.org>
>> *Subject:* EXT :Re: StreamingFileSink cannot get AWS S3 credentials
>>
>>
>>
>> I haven't configured this myself but I would guess that you need to set
>> the parameters defined here under S3A Authentication methods [1]. If the
>> environment variables don't work, then I would try to set the
>> authentication properties.
>>
>>
>>
>> [1]
>> https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html#S3A
>>
>>
>>
>> Cheers,
>> Till
>>
>>
>>
>> On Wed, Jan 16, 2019 at 11:09 AM Vinay Patil 
>> wrote:
>>
>> Hi Till,
>>
>>
>>
>> Can you please let us know the configurations that we need to set for
>> Profile based credential provider in flink-conf.yaml
>>
>>
>>
>> Exporting AWS_PROFILE property on EMR did not work.
>>
>>
>>
>> Regards,
>>
>> Vinay Patil
>>
>>
>>
>>
>>
>> On Wed, Jan 16, 2019 at 3:05 PM Till Rohrmann 
>> wrote:
>>
>> The old BucketingSink was using Hadoop's S3 filesystem directly whereas
>> the new StreamingFileSink uses Flink's own FileSystem which need to be
>> configured via the flink-conf.yaml.
>>
>>
>>
>> Cheers,
>>
>> Till
>>
>>
>>
>> On Wed, Jan 16, 2019 at 10:31 AM Vinay Patil 
>> wrote:
>>
>> Hi Till,
>>
>>
>>
>> We are not providing `fs.s3a.access.key: access_key`, `fs.s3a.secret.key:
>> secret_key` in flink-conf.yaml as we are using Profile based credentials
>> provider. The older BucketingSink code is able to get the credentials and
>> write to S3. We are facing this issue only with StreamingFileSink. We tried
>> adding fs.s3a.impl to core-site.xml when the default configurations were
>> not working.
>>
>>
>>
>> Regards,
>>
>> Vinay Patil
>>
>>
>>
>>
>>
>> On Wed, Jan 16, 2019 at 2:55 PM Till Rohrmann 
>> wrote:
>>
>> Hi Vinay,
>>
>>
>>
>> Flink's file systems are self contained and won't respect the
>> core-site.xml if I'm not mistaken. Instead you have to set the credentials
>> in the flink configuration flink-conf.yaml via `fs.s3a.access.key:
>> access_key`, `fs.s3a.secret.key: secret_key` and so on [1]. Have you tried
>> this out?
>>
>>
>>
>> This has been fixed with Flink 1.6.2 and 1.7.0 [2].
>>
>>
>>
>> [1]
>> https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems.html#built-in-file-systems
>>
>> [2] https://issues.apache.org/jira/browse/FLINK-10383
>>
>>
>>
>> Cheers,
>>
>> Till
>>
>>
>>
>> On Wed, Jan 16, 2019 at 10:10 AM Kostas Kloudas <
>> k.klou...@da-platform.com> wrote:
>>
>> Hi Taher,
>>
>>
>>
>> So you are using the same configuration files and everything and the only
>> thing you change is the "s3://" to "s3a://" and the sink cannot find the
>> cred

Re: EXT :Re: StreamingFileSink cannot get AWS S3 credentials

[Stephan Ewen]

Regarding configurations: According to the code [1]
<https://github.com/apache/flink/blob/master/flink-filesystems/flink-s3-fs-hadoop/src/main/java/org/apache/flink/fs/s3hadoop/S3FileSystemFactory.java#L50>,
all config keys starting with "s3", "s3a" and "fs.s3a" are forwarded from
the flink-conf.yaml to the Hadoop file systems.

Regarding profile-based authentication: Have you tried to set the
credentials provider explicitly, by setting "fs.s3a.aws.credentials.provider"
?

[1]
https://github.com/apache/flink/blob/master/flink-filesystems/flink-s3-fs-hadoop/src/main/java/org/apache/flink/fs/s3hadoop/S3FileSystemFactory.java#L50

On Thu, Jan 17, 2019 at 12:57 AM Martin, Nick  wrote:

> Does that mean that the full set of fs.s3a.<…> configs in core-site.xml
> will be picked up from flink-conf.yaml by flink? Or only certain configs
> involved with authentication?
>
>
>
> *From:* Till Rohrmann [mailto:trohrm...@apache.org]
> *Sent:* Wednesday, January 16, 2019 3:43 AM
> *To:* Vinay Patil 
> *Cc:* Kostas Kloudas ; Dawid Wysakowicz <
> dwysakow...@apache.org>; Taher Koitawala [via Apache Flink User Mailing
> List archive.] ; user <
> user@flink.apache.org>
> *Subject:* EXT :Re: StreamingFileSink cannot get AWS S3 credentials
>
>
>
> I haven't configured this myself but I would guess that you need to set
> the parameters defined here under S3A Authentication methods [1]. If the
> environment variables don't work, then I would try to set the
> authentication properties.
>
>
>
> [1]
> https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html#S3A
>
>
>
> Cheers,
> Till
>
>
>
> On Wed, Jan 16, 2019 at 11:09 AM Vinay Patil 
> wrote:
>
> Hi Till,
>
>
>
> Can you please let us know the configurations that we need to set for
> Profile based credential provider in flink-conf.yaml
>
>
>
> Exporting AWS_PROFILE property on EMR did not work.
>
>
>
> Regards,
>
> Vinay Patil
>
>
>
>
>
> On Wed, Jan 16, 2019 at 3:05 PM Till Rohrmann 
> wrote:
>
> The old BucketingSink was using Hadoop's S3 filesystem directly whereas
> the new StreamingFileSink uses Flink's own FileSystem which need to be
> configured via the flink-conf.yaml.
>
>
>
> Cheers,
>
> Till
>
>
>
> On Wed, Jan 16, 2019 at 10:31 AM Vinay Patil 
> wrote:
>
> Hi Till,
>
>
>
> We are not providing `fs.s3a.access.key: access_key`, `fs.s3a.secret.key:
> secret_key` in flink-conf.yaml as we are using Profile based credentials
> provider. The older BucketingSink code is able to get the credentials and
> write to S3. We are facing this issue only with StreamingFileSink. We tried
> adding fs.s3a.impl to core-site.xml when the default configurations were
> not working.
>
>
>
> Regards,
>
> Vinay Patil
>
>
>
>
>
> On Wed, Jan 16, 2019 at 2:55 PM Till Rohrmann 
> wrote:
>
> Hi Vinay,
>
>
>
> Flink's file systems are self contained and won't respect the
> core-site.xml if I'm not mistaken. Instead you have to set the credentials
> in the flink configuration flink-conf.yaml via `fs.s3a.access.key:
> access_key`, `fs.s3a.secret.key: secret_key` and so on [1]. Have you tried
> this out?
>
>
>
> This has been fixed with Flink 1.6.2 and 1.7.0 [2].
>
>
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems.html#built-in-file-systems
>
> [2] https://issues.apache.org/jira/browse/FLINK-10383
>
>
>
> Cheers,
>
> Till
>
>
>
> On Wed, Jan 16, 2019 at 10:10 AM Kostas Kloudas 
> wrote:
>
> Hi Taher,
>
>
>
> So you are using the same configuration files and everything and the only
> thing you change is the "s3://" to "s3a://" and the sink cannot find the
> credentials?
>
> Could you please provide the logs of the Task Managers?
>
>
>
> Cheers,
>
> Kostas
>
>
>
> On Wed, Jan 16, 2019 at 9:13 AM Dawid Wysakowicz 
> wrote:
>
> Forgot to cc ;)
>
> On 16/01/2019 08:51, Vinay Patil wrote:
>
> Hi,
>
>
>
> Can someone please help on this issue. We have even tried to set
> fs.s3a.impl in core-site.xml, still its not working.
>
>
>
> Regards,
>
> Vinay Patil
>
>
>
>
>
> On Fri, Jan 11, 2019 at 5:03 PM Taher Koitawala [via Apache Flink User
> Mailing List archive.]  wrote:
>
> Hi All,
>
>  We have implemented S3 sink in the following way:
>
>
>
> StreamingFileSink sink= StreamingFileSink.forBulkFormat(new
> Path("s3a://mybucket/myfolder/output/"),
> ParquetAvroWriters.forGenericRecord(schema))
>
> .with

RE: EXT :Re: StreamingFileSink cannot get AWS S3 credentials

[Martin, Nick]

Does that mean that the full set of fs.s3a.<…> configs in core-site.xml will be 
picked up from flink-conf.yaml by flink? Or only certain configs involved with 
authentication?

From: Till Rohrmann [mailto:trohrm...@apache.org]
Sent: Wednesday, January 16, 2019 3:43 AM
To: Vinay Patil 
Cc: Kostas Kloudas ; Dawid Wysakowicz 
; Taher Koitawala [via Apache Flink User Mailing List 
archive.] ; user 
Subject: EXT :Re: StreamingFileSink cannot get AWS S3 credentials

I haven't configured this myself but I would guess that you need to set the 
parameters defined here under S3A Authentication methods [1]. If the 
environment variables don't work, then I would try to set the authentication 
properties.

[1] 
https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html#S3A

Cheers,
Till

On Wed, Jan 16, 2019 at 11:09 AM Vinay Patil 
mailto:vinay18.pa...@gmail.com>> wrote:
Hi Till,

Can you please let us know the configurations that we need to set for Profile 
based credential provider in flink-conf.yaml

Exporting AWS_PROFILE property on EMR did not work.

Regards,
Vinay Patil


On Wed, Jan 16, 2019 at 3:05 PM Till Rohrmann 
mailto:trohrm...@apache.org>> wrote:
The old BucketingSink was using Hadoop's S3 filesystem directly whereas the new 
StreamingFileSink uses Flink's own FileSystem which need to be configured via 
the flink-conf.yaml.

Cheers,
Till

On Wed, Jan 16, 2019 at 10:31 AM Vinay Patil 
mailto:vinay18.pa...@gmail.com>> wrote:
Hi Till,

We are not providing `fs.s3a.access.key: access_key`, `fs.s3a.secret.key: 
secret_key` in flink-conf.yaml as we are using Profile based credentials 
provider. The older BucketingSink code is able to get the credentials and write 
to S3. We are facing this issue only with StreamingFileSink. We tried adding 
fs.s3a.impl to core-site.xml when the default configurations were not working.

Regards,
Vinay Patil


On Wed, Jan 16, 2019 at 2:55 PM Till Rohrmann 
mailto:trohrm...@apache.org>> wrote:
Hi Vinay,

Flink's file systems are self contained and won't respect the core-site.xml if 
I'm not mistaken. Instead you have to set the credentials in the flink 
configuration flink-conf.yaml via `fs.s3a.access.key: access_key`, 
`fs.s3a.secret.key: secret_key` and so on [1]. Have you tried this out?

This has been fixed with Flink 1.6.2 and 1.7.0 [2].

[1] 
https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems.html#built-in-file-systems
[2] https://issues.apache.org/jira/browse/FLINK-10383

Cheers,
Till

On Wed, Jan 16, 2019 at 10:10 AM Kostas Kloudas 
mailto:k.klou...@da-platform.com>> wrote:
Hi Taher,

So you are using the same configuration files and everything and the only thing 
you change is the "s3://" to "s3a://" and the sink cannot find the credentials?
Could you please provide the logs of the Task Managers?

Cheers,
Kostas

On Wed, Jan 16, 2019 at 9:13 AM Dawid Wysakowicz 
mailto:dwysakow...@apache.org>> wrote:

Forgot to cc ;)
On 16/01/2019 08:51, Vinay Patil wrote:
Hi,

Can someone please help on this issue. We have even tried to set fs.s3a.impl in 
core-site.xml, still its not working.

Regards,
Vinay Patil


On Fri, Jan 11, 2019 at 5:03 PM Taher Koitawala [via Apache Flink User Mailing 
List archive.] 
mailto:ml%2bs2336050n25464...@n4.nabble.com>>
 wrote:
Hi All,
 We have implemented S3 sink in the following way:

StreamingFileSink sink= StreamingFileSink.forBulkFormat(new 
Path("s3a://mybucket/myfolder/output/"), 
ParquetAvroWriters.forGenericRecord(schema))
.withBucketCheckInterval(50l).withBucketAssigner(new 
CustomBucketAssigner()).build();

The problem we are facing is that StreamingFileSink is initializing 
S3AFileSystem class to write to s3 and is not able to find the s3 credentials 
to write data, However other flink application on the same cluster use "s3://" 
paths are able to write data to the same s3 bucket and folders, we are only 
facing this issue with StreamingFileSink.

Regards,
Taher Koitawala
GS Lab Pune
+91 8407979163


If you reply to this email, your message will be added to the discussion below:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/StreamingFileSink-cannot-get-AWS-S3-credentials-tp25464.html
To start a new topic under Apache Flink User Mailing List archive., email 
ml+s2336050n1...@n4.nabble.com<mailto:ml%2bs2336050n1...@n4.nabble.com>
To unsubscribe from Apache Flink User Mailing List archive., click 
here<http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code=1=dmluYXkxOC5wYXRpbEBnbWFpbC5jb218MXwxODExMDE2NjAx>.
NAML<http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer=instant_html%21nabble%3Aemail.naml=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace=no

Re: StreamingFileSink cannot get AWS S3 credentials

[Till Rohrmann]

I haven't configured this myself but I would guess that you need to set the
parameters defined here under S3A Authentication methods [1]. If the
environment variables don't work, then I would try to set the
authentication properties.

[1]
https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html#S3A

Cheers,
Till

On Wed, Jan 16, 2019 at 11:09 AM Vinay Patil 
wrote:

> Hi Till,
>
> Can you please let us know the configurations that we need to set for
> Profile based credential provider in flink-conf.yaml
>
> Exporting AWS_PROFILE property on EMR did not work.
>
> Regards,
> Vinay Patil
>
>
> On Wed, Jan 16, 2019 at 3:05 PM Till Rohrmann 
> wrote:
>
>> The old BucketingSink was using Hadoop's S3 filesystem directly whereas
>> the new StreamingFileSink uses Flink's own FileSystem which need to be
>> configured via the flink-conf.yaml.
>>
>> Cheers,
>> Till
>>
>> On Wed, Jan 16, 2019 at 10:31 AM Vinay Patil 
>> wrote:
>>
>>> Hi Till,
>>>
>>> We are not providing `fs.s3a.access.key: access_key`,
>>> `fs.s3a.secret.key: secret_key` in flink-conf.yaml as we are using Profile
>>> based credentials provider. The older BucketingSink code is able to get the
>>> credentials and write to S3. We are facing this issue only with
>>> StreamingFileSink. We tried adding fs.s3a.impl to core-site.xml when the
>>> default configurations were not working.
>>>
>>> Regards,
>>> Vinay Patil
>>>
>>>
>>> On Wed, Jan 16, 2019 at 2:55 PM Till Rohrmann 
>>> wrote:
>>>
 Hi Vinay,

 Flink's file systems are self contained and won't respect the
 core-site.xml if I'm not mistaken. Instead you have to set the credentials
 in the flink configuration flink-conf.yaml via `fs.s3a.access.key:
 access_key`, `fs.s3a.secret.key: secret_key` and so on [1]. Have you tried
 this out?

 This has been fixed with Flink 1.6.2 and 1.7.0 [2].

 [1]
 https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems.html#built-in-file-systems
 [2] https://issues.apache.org/jira/browse/FLINK-10383

 Cheers,
 Till

 On Wed, Jan 16, 2019 at 10:10 AM Kostas Kloudas <
 k.klou...@da-platform.com> wrote:

> Hi Taher,
>
> So you are using the same configuration files and everything and the
> only thing you change is the "s3://" to "s3a://" and the sink cannot find
> the credentials?
> Could you please provide the logs of the Task Managers?
>
> Cheers,
> Kostas
>
> On Wed, Jan 16, 2019 at 9:13 AM Dawid Wysakowicz <
> dwysakow...@apache.org> wrote:
>
>> Forgot to cc ;)
>> On 16/01/2019 08:51, Vinay Patil wrote:
>>
>> Hi,
>>
>> Can someone please help on this issue. We have even tried to set
>> fs.s3a.impl in core-site.xml, still its not working.
>>
>> Regards,
>> Vinay Patil
>>
>>
>> On Fri, Jan 11, 2019 at 5:03 PM Taher Koitawala [via Apache Flink
>> User Mailing List archive.] 
>> wrote:
>>
>>> Hi All,
>>>  We have implemented S3 sink in the following way:
>>>
>>> StreamingFileSink sink= StreamingFileSink.forBulkFormat(new
>>> Path("s3a://mybucket/myfolder/output/"),
>>> ParquetAvroWriters.forGenericRecord(schema))
>>> .withBucketCheckInterval(50l).withBucketAssigner(new
>>> CustomBucketAssigner()).build();
>>>
>>> The problem we are facing is that StreamingFileSink is initializing
>>> S3AFileSystem class to write to s3 and is not able to find the s3
>>> credentials to write data, However other flink application on the
>>> same cluster use "s3://" paths are able to write data to the same s3 
>>> bucket
>>> and folders, we are only facing this issue with StreamingFileSink.
>>>
>>> Regards,
>>> Taher Koitawala
>>> GS Lab Pune
>>> +91 8407979163
>>>
>>>
>>> --
>>> If you reply to this email, your message will be added to the
>>> discussion below:
>>>
>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/StreamingFileSink-cannot-get-AWS-S3-credentials-tp25464.html
>>> To start a new topic under Apache Flink User Mailing List archive.,
>>> email ml+s2336050n1...@n4.nabble.com
>>> To unsubscribe from Apache Flink User Mailing List archive., click
>>> here
>>> 
>>> .
>>> NAML
>>> 
>>>
>>

Re: StreamingFileSink cannot get AWS S3 credentials

[Vinay Patil]

Hi Till,

Can you please let us know the configurations that we need to set for
Profile based credential provider in flink-conf.yaml

Exporting AWS_PROFILE property on EMR did not work.

Regards,
Vinay Patil


On Wed, Jan 16, 2019 at 3:05 PM Till Rohrmann  wrote:

> The old BucketingSink was using Hadoop's S3 filesystem directly whereas
> the new StreamingFileSink uses Flink's own FileSystem which need to be
> configured via the flink-conf.yaml.
>
> Cheers,
> Till
>
> On Wed, Jan 16, 2019 at 10:31 AM Vinay Patil 
> wrote:
>
>> Hi Till,
>>
>> We are not providing `fs.s3a.access.key: access_key`, `fs.s3a.secret.key:
>> secret_key` in flink-conf.yaml as we are using Profile based credentials
>> provider. The older BucketingSink code is able to get the credentials and
>> write to S3. We are facing this issue only with StreamingFileSink. We tried
>> adding fs.s3a.impl to core-site.xml when the default configurations were
>> not working.
>>
>> Regards,
>> Vinay Patil
>>
>>
>> On Wed, Jan 16, 2019 at 2:55 PM Till Rohrmann 
>> wrote:
>>
>>> Hi Vinay,
>>>
>>> Flink's file systems are self contained and won't respect the
>>> core-site.xml if I'm not mistaken. Instead you have to set the credentials
>>> in the flink configuration flink-conf.yaml via `fs.s3a.access.key:
>>> access_key`, `fs.s3a.secret.key: secret_key` and so on [1]. Have you tried
>>> this out?
>>>
>>> This has been fixed with Flink 1.6.2 and 1.7.0 [2].
>>>
>>> [1]
>>> https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems.html#built-in-file-systems
>>> [2] https://issues.apache.org/jira/browse/FLINK-10383
>>>
>>> Cheers,
>>> Till
>>>
>>> On Wed, Jan 16, 2019 at 10:10 AM Kostas Kloudas <
>>> k.klou...@da-platform.com> wrote:
>>>
 Hi Taher,

 So you are using the same configuration files and everything and the
 only thing you change is the "s3://" to "s3a://" and the sink cannot find
 the credentials?
 Could you please provide the logs of the Task Managers?

 Cheers,
 Kostas

 On Wed, Jan 16, 2019 at 9:13 AM Dawid Wysakowicz <
 dwysakow...@apache.org> wrote:

> Forgot to cc ;)
> On 16/01/2019 08:51, Vinay Patil wrote:
>
> Hi,
>
> Can someone please help on this issue. We have even tried to set
> fs.s3a.impl in core-site.xml, still its not working.
>
> Regards,
> Vinay Patil
>
>
> On Fri, Jan 11, 2019 at 5:03 PM Taher Koitawala [via Apache Flink User
> Mailing List archive.]  wrote:
>
>> Hi All,
>>  We have implemented S3 sink in the following way:
>>
>> StreamingFileSink sink= StreamingFileSink.forBulkFormat(new
>> Path("s3a://mybucket/myfolder/output/"),
>> ParquetAvroWriters.forGenericRecord(schema))
>> .withBucketCheckInterval(50l).withBucketAssigner(new
>> CustomBucketAssigner()).build();
>>
>> The problem we are facing is that StreamingFileSink is initializing
>> S3AFileSystem class to write to s3 and is not able to find the s3
>> credentials to write data, However other flink application on the
>> same cluster use "s3://" paths are able to write data to the same s3 
>> bucket
>> and folders, we are only facing this issue with StreamingFileSink.
>>
>> Regards,
>> Taher Koitawala
>> GS Lab Pune
>> +91 8407979163
>>
>>
>> --
>> If you reply to this email, your message will be added to the
>> discussion below:
>>
>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/StreamingFileSink-cannot-get-AWS-S3-credentials-tp25464.html
>> To start a new topic under Apache Flink User Mailing List archive.,
>> email ml+s2336050n1...@n4.nabble.com
>> To unsubscribe from Apache Flink User Mailing List archive., click
>> here
>> 
>> .
>> NAML
>> 
>>
>

Re: StreamingFileSink cannot get AWS S3 credentials

[Till Rohrmann]

The old BucketingSink was using Hadoop's S3 filesystem directly whereas the
new StreamingFileSink uses Flink's own FileSystem which need to be
configured via the flink-conf.yaml.

Cheers,
Till

On Wed, Jan 16, 2019 at 10:31 AM Vinay Patil 
wrote:

> Hi Till,
>
> We are not providing `fs.s3a.access.key: access_key`, `fs.s3a.secret.key:
> secret_key` in flink-conf.yaml as we are using Profile based credentials
> provider. The older BucketingSink code is able to get the credentials and
> write to S3. We are facing this issue only with StreamingFileSink. We tried
> adding fs.s3a.impl to core-site.xml when the default configurations were
> not working.
>
> Regards,
> Vinay Patil
>
>
> On Wed, Jan 16, 2019 at 2:55 PM Till Rohrmann 
> wrote:
>
>> Hi Vinay,
>>
>> Flink's file systems are self contained and won't respect the
>> core-site.xml if I'm not mistaken. Instead you have to set the credentials
>> in the flink configuration flink-conf.yaml via `fs.s3a.access.key:
>> access_key`, `fs.s3a.secret.key: secret_key` and so on [1]. Have you tried
>> this out?
>>
>> This has been fixed with Flink 1.6.2 and 1.7.0 [2].
>>
>> [1]
>> https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems.html#built-in-file-systems
>> [2] https://issues.apache.org/jira/browse/FLINK-10383
>>
>> Cheers,
>> Till
>>
>> On Wed, Jan 16, 2019 at 10:10 AM Kostas Kloudas <
>> k.klou...@da-platform.com> wrote:
>>
>>> Hi Taher,
>>>
>>> So you are using the same configuration files and everything and the
>>> only thing you change is the "s3://" to "s3a://" and the sink cannot find
>>> the credentials?
>>> Could you please provide the logs of the Task Managers?
>>>
>>> Cheers,
>>> Kostas
>>>
>>> On Wed, Jan 16, 2019 at 9:13 AM Dawid Wysakowicz 
>>> wrote:
>>>
 Forgot to cc ;)
 On 16/01/2019 08:51, Vinay Patil wrote:

 Hi,

 Can someone please help on this issue. We have even tried to set
 fs.s3a.impl in core-site.xml, still its not working.

 Regards,
 Vinay Patil


 On Fri, Jan 11, 2019 at 5:03 PM Taher Koitawala [via Apache Flink User
 Mailing List archive.]  wrote:

> Hi All,
>  We have implemented S3 sink in the following way:
>
> StreamingFileSink sink= StreamingFileSink.forBulkFormat(new
> Path("s3a://mybucket/myfolder/output/"),
> ParquetAvroWriters.forGenericRecord(schema))
> .withBucketCheckInterval(50l).withBucketAssigner(new
> CustomBucketAssigner()).build();
>
> The problem we are facing is that StreamingFileSink is initializing
> S3AFileSystem class to write to s3 and is not able to find the s3
> credentials to write data, However other flink application on the
> same cluster use "s3://" paths are able to write data to the same s3 
> bucket
> and folders, we are only facing this issue with StreamingFileSink.
>
> Regards,
> Taher Koitawala
> GS Lab Pune
> +91 8407979163
>
>
> --
> If you reply to this email, your message will be added to the
> discussion below:
>
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/StreamingFileSink-cannot-get-AWS-S3-credentials-tp25464.html
> To start a new topic under Apache Flink User Mailing List archive.,
> email ml+s2336050n1...@n4.nabble.com
> To unsubscribe from Apache Flink User Mailing List archive., click
> here
> 
> .
> NAML
> 
>

Re: StreamingFileSink cannot get AWS S3 credentials

[Vinay Patil]

Hi Till,

We are not providing `fs.s3a.access.key: access_key`, `fs.s3a.secret.key:
secret_key` in flink-conf.yaml as we are using Profile based credentials
provider. The older BucketingSink code is able to get the credentials and
write to S3. We are facing this issue only with StreamingFileSink. We tried
adding fs.s3a.impl to core-site.xml when the default configurations were
not working.

Regards,
Vinay Patil


On Wed, Jan 16, 2019 at 2:55 PM Till Rohrmann  wrote:

> Hi Vinay,
>
> Flink's file systems are self contained and won't respect the
> core-site.xml if I'm not mistaken. Instead you have to set the credentials
> in the flink configuration flink-conf.yaml via `fs.s3a.access.key:
> access_key`, `fs.s3a.secret.key: secret_key` and so on [1]. Have you tried
> this out?
>
> This has been fixed with Flink 1.6.2 and 1.7.0 [2].
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems.html#built-in-file-systems
> [2] https://issues.apache.org/jira/browse/FLINK-10383
>
> Cheers,
> Till
>
> On Wed, Jan 16, 2019 at 10:10 AM Kostas Kloudas 
> wrote:
>
>> Hi Taher,
>>
>> So you are using the same configuration files and everything and the only
>> thing you change is the "s3://" to "s3a://" and the sink cannot find the
>> credentials?
>> Could you please provide the logs of the Task Managers?
>>
>> Cheers,
>> Kostas
>>
>> On Wed, Jan 16, 2019 at 9:13 AM Dawid Wysakowicz 
>> wrote:
>>
>>> Forgot to cc ;)
>>> On 16/01/2019 08:51, Vinay Patil wrote:
>>>
>>> Hi,
>>>
>>> Can someone please help on this issue. We have even tried to set
>>> fs.s3a.impl in core-site.xml, still its not working.
>>>
>>> Regards,
>>> Vinay Patil
>>>
>>>
>>> On Fri, Jan 11, 2019 at 5:03 PM Taher Koitawala [via Apache Flink User
>>> Mailing List archive.]  wrote:
>>>
 Hi All,
  We have implemented S3 sink in the following way:

 StreamingFileSink sink= StreamingFileSink.forBulkFormat(new
 Path("s3a://mybucket/myfolder/output/"),
 ParquetAvroWriters.forGenericRecord(schema))
 .withBucketCheckInterval(50l).withBucketAssigner(new
 CustomBucketAssigner()).build();

 The problem we are facing is that StreamingFileSink is initializing
 S3AFileSystem class to write to s3 and is not able to find the s3
 credentials to write data, However other flink application on the same
 cluster use "s3://" paths are able to write data to the same s3 bucket and
 folders, we are only facing this issue with StreamingFileSink.

 Regards,
 Taher Koitawala
 GS Lab Pune
 +91 8407979163


 --
 If you reply to this email, your message will be added to the
 discussion below:

 http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/StreamingFileSink-cannot-get-AWS-S3-credentials-tp25464.html
 To start a new topic under Apache Flink User Mailing List archive.,
 email ml+s2336050n1...@n4.nabble.com
 To unsubscribe from Apache Flink User Mailing List archive., click here
 
 .
 NAML
 

>>>

Re: StreamingFileSink cannot get AWS S3 credentials

[Kostas Kloudas]

Actually Till is right.

Sorry, my fault, I did not read your second email where Vinay mentions the
core-site.xml.

Cheers,
Kostas

On Wed, Jan 16, 2019 at 10:25 AM Till Rohrmann  wrote:

> Hi Vinay,
>
> Flink's file systems are self contained and won't respect the
> core-site.xml if I'm not mistaken. Instead you have to set the credentials
> in the flink configuration flink-conf.yaml via `fs.s3a.access.key:
> access_key`, `fs.s3a.secret.key: secret_key` and so on [1]. Have you tried
> this out?
>
> This has been fixed with Flink 1.6.2 and 1.7.0 [2].
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems.html#built-in-file-systems
> [2] https://issues.apache.org/jira/browse/FLINK-10383
>
> Cheers,
> Till
>
> On Wed, Jan 16, 2019 at 10:10 AM Kostas Kloudas 
> wrote:
>
>> Hi Taher,
>>
>> So you are using the same configuration files and everything and the only
>> thing you change is the "s3://" to "s3a://" and the sink cannot find the
>> credentials?
>> Could you please provide the logs of the Task Managers?
>>
>> Cheers,
>> Kostas
>>
>> On Wed, Jan 16, 2019 at 9:13 AM Dawid Wysakowicz 
>> wrote:
>>
>>> Forgot to cc ;)
>>> On 16/01/2019 08:51, Vinay Patil wrote:
>>>
>>> Hi,
>>>
>>> Can someone please help on this issue. We have even tried to set
>>> fs.s3a.impl in core-site.xml, still its not working.
>>>
>>> Regards,
>>> Vinay Patil
>>>
>>>
>>> On Fri, Jan 11, 2019 at 5:03 PM Taher Koitawala [via Apache Flink User
>>> Mailing List archive.]  wrote:
>>>
 Hi All,
  We have implemented S3 sink in the following way:

 StreamingFileSink sink= StreamingFileSink.forBulkFormat(new
 Path("s3a://mybucket/myfolder/output/"),
 ParquetAvroWriters.forGenericRecord(schema))
 .withBucketCheckInterval(50l).withBucketAssigner(new
 CustomBucketAssigner()).build();

 The problem we are facing is that StreamingFileSink is initializing
 S3AFileSystem class to write to s3 and is not able to find the s3
 credentials to write data, However other flink application on the same
 cluster use "s3://" paths are able to write data to the same s3 bucket and
 folders, we are only facing this issue with StreamingFileSink.

 Regards,
 Taher Koitawala
 GS Lab Pune
 +91 8407979163


 --
 If you reply to this email, your message will be added to the
 discussion below:

 http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/StreamingFileSink-cannot-get-AWS-S3-credentials-tp25464.html
 To start a new topic under Apache Flink User Mailing List archive.,
 email ml+s2336050n1...@n4.nabble.com
 To unsubscribe from Apache Flink User Mailing List archive., click here
 
 .
 NAML
 

>>>

Re: StreamingFileSink cannot get AWS S3 credentials

[Till Rohrmann]

Hi Vinay,

Flink's file systems are self contained and won't respect the core-site.xml
if I'm not mistaken. Instead you have to set the credentials in the flink
configuration flink-conf.yaml via `fs.s3a.access.key: access_key`,
`fs.s3a.secret.key: secret_key` and so on [1]. Have you tried this out?

This has been fixed with Flink 1.6.2 and 1.7.0 [2].

[1]
https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems.html#built-in-file-systems
[2] https://issues.apache.org/jira/browse/FLINK-10383

Cheers,
Till

On Wed, Jan 16, 2019 at 10:10 AM Kostas Kloudas 
wrote:

> Hi Taher,
>
> So you are using the same configuration files and everything and the only
> thing you change is the "s3://" to "s3a://" and the sink cannot find the
> credentials?
> Could you please provide the logs of the Task Managers?
>
> Cheers,
> Kostas
>
> On Wed, Jan 16, 2019 at 9:13 AM Dawid Wysakowicz 
> wrote:
>
>> Forgot to cc ;)
>> On 16/01/2019 08:51, Vinay Patil wrote:
>>
>> Hi,
>>
>> Can someone please help on this issue. We have even tried to set
>> fs.s3a.impl in core-site.xml, still its not working.
>>
>> Regards,
>> Vinay Patil
>>
>>
>> On Fri, Jan 11, 2019 at 5:03 PM Taher Koitawala [via Apache Flink User
>> Mailing List archive.]  wrote:
>>
>>> Hi All,
>>>  We have implemented S3 sink in the following way:
>>>
>>> StreamingFileSink sink= StreamingFileSink.forBulkFormat(new
>>> Path("s3a://mybucket/myfolder/output/"),
>>> ParquetAvroWriters.forGenericRecord(schema))
>>> .withBucketCheckInterval(50l).withBucketAssigner(new
>>> CustomBucketAssigner()).build();
>>>
>>> The problem we are facing is that StreamingFileSink is initializing
>>> S3AFileSystem class to write to s3 and is not able to find the s3
>>> credentials to write data, However other flink application on the same
>>> cluster use "s3://" paths are able to write data to the same s3 bucket and
>>> folders, we are only facing this issue with StreamingFileSink.
>>>
>>> Regards,
>>> Taher Koitawala
>>> GS Lab Pune
>>> +91 8407979163
>>>
>>>
>>> --
>>> If you reply to this email, your message will be added to the discussion
>>> below:
>>>
>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/StreamingFileSink-cannot-get-AWS-S3-credentials-tp25464.html
>>> To start a new topic under Apache Flink User Mailing List archive.,
>>> email ml+s2336050n1...@n4.nabble.com
>>> To unsubscribe from Apache Flink User Mailing List archive., click here
>>> 
>>> .
>>> NAML
>>> 
>>>
>>

Re: StreamingFileSink cannot get AWS S3 credentials

[Kostas Kloudas]

Hi Taher,

So you are using the same configuration files and everything and the only
thing you change is the "s3://" to "s3a://" and the sink cannot find the
credentials?
Could you please provide the logs of the Task Managers?

Cheers,
Kostas

On Wed, Jan 16, 2019 at 9:13 AM Dawid Wysakowicz 
wrote:

> Forgot to cc ;)
> On 16/01/2019 08:51, Vinay Patil wrote:
>
> Hi,
>
> Can someone please help on this issue. We have even tried to set
> fs.s3a.impl in core-site.xml, still its not working.
>
> Regards,
> Vinay Patil
>
>
> On Fri, Jan 11, 2019 at 5:03 PM Taher Koitawala [via Apache Flink User
> Mailing List archive.]  wrote:
>
>> Hi All,
>>  We have implemented S3 sink in the following way:
>>
>> StreamingFileSink sink= StreamingFileSink.forBulkFormat(new
>> Path("s3a://mybucket/myfolder/output/"),
>> ParquetAvroWriters.forGenericRecord(schema))
>> .withBucketCheckInterval(50l).withBucketAssigner(new
>> CustomBucketAssigner()).build();
>>
>> The problem we are facing is that StreamingFileSink is initializing
>> S3AFileSystem class to write to s3 and is not able to find the s3
>> credentials to write data, However other flink application on the same
>> cluster use "s3://" paths are able to write data to the same s3 bucket and
>> folders, we are only facing this issue with StreamingFileSink.
>>
>> Regards,
>> Taher Koitawala
>> GS Lab Pune
>> +91 8407979163
>>
>>
>> --
>> If you reply to this email, your message will be added to the discussion
>> below:
>>
>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/StreamingFileSink-cannot-get-AWS-S3-credentials-tp25464.html
>> To start a new topic under Apache Flink User Mailing List archive., email
>> ml+s2336050n1...@n4.nabble.com
>> To unsubscribe from Apache Flink User Mailing List archive., click here
>> 
>> .
>> NAML
>> 
>>
>

Re: StreamingFileSink cannot get AWS S3 credentials

[Dawid Wysakowicz]

Forgot to cc ;)

On 16/01/2019 08:51, Vinay Patil wrote:
> Hi,
>
> Can someone please help on this issue. We have even tried to set
> fs.s3a.impl in core-site.xml, still its not working.
>
> Regards,
> Vinay Patil
>
>
> On Fri, Jan 11, 2019 at 5:03 PM Taher Koitawala [via Apache Flink User
> Mailing List archive.]  > wrote:
>
> Hi All,
>          We have implemented S3 sink in the following way:
>
> StreamingFileSink sink= StreamingFileSink.forBulkFormat(new
> Path("s3a://mybucket/myfolder/output/"),
> ParquetAvroWriters.forGenericRecord(schema))
> .withBucketCheckInterval(50l).withBucketAssigner(new
> CustomBucketAssigner()).build();
>
> The problem we are facing is that StreamingFileSink is
> initializing S3AFileSystem class to write to s3 and is not able to
> find the s3 credentials to write data, However
> other flink application on the same cluster use "s3://" paths are
> able to write data to the same s3 bucket and folders, we are only
> facing this issue with StreamingFileSink.
>
> Regards,
> Taher Koitawala
> GS Lab Pune
> +91 8407979163
>
>
> 
> If you reply to this email, your message will be added to the
> discussion below:
> 
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/StreamingFileSink-cannot-get-AWS-S3-credentials-tp25464.html
>
> To start a new topic under Apache Flink User Mailing List
> archive., email ml+s2336050n1...@n4.nabble.com
> 
> To unsubscribe from Apache Flink User Mailing List archive., click
> here
> 
> .
> NAML
> 
> 
>
>


signature.asc
Description: OpenPGP digital signature

Re: StreamingFileSink cannot get AWS S3 credentials

[Dawid Wysakowicz]

Hi,

I cc Kostas who should be able to help you.

Best,

Dawid

On 16/01/2019 08:51, Vinay Patil wrote:
> Hi,
>
> Can someone please help on this issue. We have even tried to set
> fs.s3a.impl in core-site.xml, still its not working.
>
> Regards,
> Vinay Patil
>
>
> On Fri, Jan 11, 2019 at 5:03 PM Taher Koitawala [via Apache Flink User
> Mailing List archive.]  > wrote:
>
> Hi All,
>          We have implemented S3 sink in the following way:
>
> StreamingFileSink sink= StreamingFileSink.forBulkFormat(new
> Path("s3a://mybucket/myfolder/output/"),
> ParquetAvroWriters.forGenericRecord(schema))
> .withBucketCheckInterval(50l).withBucketAssigner(new
> CustomBucketAssigner()).build();
>
> The problem we are facing is that StreamingFileSink is
> initializing S3AFileSystem class to write to s3 and is not able to
> find the s3 credentials to write data, However
> other flink application on the same cluster use "s3://" paths are
> able to write data to the same s3 bucket and folders, we are only
> facing this issue with StreamingFileSink.
>
> Regards,
> Taher Koitawala
> GS Lab Pune
> +91 8407979163
>
>
> 
> If you reply to this email, your message will be added to the
> discussion below:
> 
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/StreamingFileSink-cannot-get-AWS-S3-credentials-tp25464.html
>
> To start a new topic under Apache Flink User Mailing List
> archive., email ml+s2336050n1...@n4.nabble.com
> 
> To unsubscribe from Apache Flink User Mailing List archive., click
> here
> 
> .
> NAML
> 
> 
>
>


signature.asc
Description: OpenPGP digital signature

Re: StreamingFileSink cannot get AWS S3 credentials

[Vinay Patil]

Hi,

Can someone please help on this issue. We have even tried to set
fs.s3a.impl in core-site.xml, still its not working.

Regards,
Vinay Patil


On Fri, Jan 11, 2019 at 5:03 PM Taher Koitawala [via Apache Flink User
Mailing List archive.]  wrote:

> Hi All,
>  We have implemented S3 sink in the following way:
>
> StreamingFileSink sink= StreamingFileSink.forBulkFormat(new
> Path("s3a://mybucket/myfolder/output/"),
> ParquetAvroWriters.forGenericRecord(schema))
> .withBucketCheckInterval(50l).withBucketAssigner(new
> CustomBucketAssigner()).build();
>
> The problem we are facing is that StreamingFileSink is initializing
> S3AFileSystem class to write to s3 and is not able to find the s3
> credentials to write data, However other flink application on the same
> cluster use "s3://" paths are able to write data to the same s3 bucket and
> folders, we are only facing this issue with StreamingFileSink.
>
> Regards,
> Taher Koitawala
> GS Lab Pune
> +91 8407979163
>
>
> --
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/StreamingFileSink-cannot-get-AWS-S3-credentials-tp25464.html
> To start a new topic under Apache Flink User Mailing List archive., email
> ml+s2336050n1...@n4.nabble.com
> To unsubscribe from Apache Flink User Mailing List archive., click here
> 
> .
> NAML
> 
>