RE: Use different S3 access key for different S3 bucket

[Qing Lim]

Thank you for the tips, I will try these out

From: Josh Mahonin 
Sent: 18 January 2024 21:07
To: Qing Lim 
Cc: Jun Qin ; User 
Subject: Re: Use different S3 access key for different S3 bucket

Oops my syntax was a bit off there, as shown in the Hadoop docs, it looks like:
fs.s3a.bucket..

Josh

This e-mail and any attachments are confidential to the addressee(s) and may 
contain information that is legally privileged and/or confidential. If you are 
not the intended recipient of this e-mail you are hereby notified that any 
dissemination, distribution, or copying of its content is strictly prohibited. 
If you have received this message in error, please notify the sender by return 
e-mail and destroy the message and all copies in your possession.

To find out more details about how we may collect, use and share your personal 
information, please see https://www.mwam.com/privacy-policy. This includes 
details of how calls you make to us may be recorded in order for us to comply 
with our legal and regulatory obligations.

To the extent that the contents of this email constitutes a financial 
promotion, please note that it is issued only to and/or directed only at 
persons who are professional clients or eligible counterparties as defined in 
the FCA Rules. Any investment products or services described in this email are 
available only to professional clients and eligible counterparties. Persons who 
are not professional clients or eligible counterparties should not rely or act 
on the contents of this email.

Marshall Wace LLP is authorised and regulated by the Financial Conduct 
Authority. Marshall Wace LLP is a limited liability partnership registered in 
England and Wales with registered number OC302228 and registered office at 
George House, 131 Sloane Street, London, SW1X 9AT. If you are receiving this 
e-mail as a client, or an investor in an investment vehicle, managed or advised 
by Marshall Wace North America L.P., the sender of this e-mail is communicating 
with you in the sender's capacity as an associated or related person of 
Marshall Wace North America L.P. (�MWNA�), which is registered with the US 
Securities and Exchange Commission (�SEC�) as an investment adviser.  
Registration with the SEC does not imply that MWNA or its employees possess a 
certain level of skill or training.

Re: Use different S3 access key for different S3 bucket

[Josh Mahonin via user]

Oops my syntax was a bit off there, as shown in the Hadoop docs, it looks
like:
fs.s3a.bucket..

Josh

>

Re: Use different S3 access key for different S3 bucket

[Josh Mahonin via user]

Hi Qing,

You may have some luck with using per-bucket S3 configuration. Assuming
you're using the flink-s3-fs-hadoop plugin, you should be able to apply
different access keys to different buckets, eg:
https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html#Configuring_different_S3_buckets_with_Per-Bucket_Configuration

I'm not certain if the 's3..' syntax will work, you may
need to set the 'fs.s3a..' values directly.

Josh

On Thu, Jan 18, 2024 at 6:02 AM Qing Lim  wrote:

> Hi Jun
>
>
>
> I am indeed talking about processing two different tables, but I don’t see
> any option that allow configuring credentials at Flink table level, do you
> know where is it documented?
>
>
>
> Today we are setting the credentials via Flink conf yaml, which is
> documented here:
> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/filesystems/s3/#configure-access-credentials
>
>
>
> Thanks
>
>
>
> *From:* Jun Qin 
> *Sent:* 18 January 2024 10:51
> *To:* User ; Qing Lim 
> *Subject:* Re: Use different S3 access key for different S3 bucket
>
>
>
> Hi Qing
>
> The S3 credentials are associated with Flink SQL tables.
>
> I assume you are talking about processing/joining from two different
> tables, backed up by two different S3 buckets. If so, you can provide
> different credentials for different tables, then use the two tables in your
> pipeline.
>
>
>
> Jun
>
> On 18. Jan 2024 at 11:32 +0100, Qing Lim , wrote:
>
> Hi, I am using Flink SQL to create table backed by S3 buckets.
>
>
>
> We are not using AWS S3, so we have to use access key and secret for Auth.
>
>
>
> My pipeline depends on 2 different buckets, each requires different
> credentials, can flink support this?
>
>
>
>
>
>
>
> *Qing Lim* | Marshall Wace LLP, George House, 131 Sloane Street, London,
> SW1X 9AT | E-mail: q@mwam.com | Tel: +44 207 925 4865
>
>
>
> This e-mail and any attachments are confidential to the addressee(s) and
> may contain information that is legally privileged and/or confidential. If
> you are not the intended recipient of this e-mail you are hereby notified
> that any dissemination, distribution, or copying of its content is strictly
> prohibited. If you have received this message in error, please notify the
> sender by return e-mail and destroy the message and all copies in your
> possession.
>
>
> To find out more details about how we may collect, use and share your
> personal information, please see https://www.mwam.com/privacy-policy.
> This includes details of how calls you make to us may be recorded in order
> for us to comply with our legal and regulatory obligations.
>
>
> To the extent that the contents of this email constitutes a financial
> promotion, please note that it is issued only to and/or directed only at
> persons who are professional clients or eligible counterparties as defined
> in the FCA Rules. Any investment products or services described in this
> email are available only to professional clients and eligible
> counterparties. Persons who are not professional clients or eligible
> counterparties should not rely or act on the contents of this email.
>
>
> Marshall Wace LLP is authorised and regulated by the Financial Conduct
> Authority. Marshall Wace LLP is a limited liability partnership registered
> in England and Wales with registered number OC302228 and registered office
> at George House, 131 Sloane Street, London, SW1X 9AT. If you are receiving
> this e-mail as a client, or an investor in an investment vehicle, managed
> or advised by Marshall Wace North America L.P., the sender of this e-mail
> is communicating with you in the sender's capacity as an associated or
> related person of Marshall Wace North America L.P. ("MWNA"), which is
> registered with the US Securities and Exchange Commission ("SEC") as an
> investment adviser. Registration with the SEC does not imply that MWNA or
> its employees possess a certain level of skill or training.
>
>

RE: Use different S3 access key for different S3 bucket

[Qing Lim]

Hi Jun

I am indeed talking about processing two different tables, but I don’t see any 
option that allow configuring credentials at Flink table level, do you know 
where is it documented?

Today we are setting the credentials via Flink conf yaml, which is documented 
here: 
https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/filesystems/s3/#configure-access-credentials

Thanks

From: Jun Qin 
Sent: 18 January 2024 10:51
To: User ; Qing Lim 
Subject: Re: Use different S3 access key for different S3 bucket

Hi Qing

The S3 credentials are associated with Flink SQL tables.

I assume you are talking about processing/joining from two different tables, 
backed up by two different S3 buckets. If so, you can provide different 
credentials for different tables, then use the two tables in your pipeline.

Jun
On 18. Jan 2024 at 11:32 +0100, Qing Lim 
mailto:q@mwam.com>>, wrote:

Hi, I am using Flink SQL to create table backed by S3 buckets.

We are not using AWS S3, so we have to use access key and secret for Auth.

My pipeline depends on 2 different buckets, each requires different 
credentials, can flink support this?



Qing Lim | Marshall Wace LLP, George House, 131 Sloane Street, London, SW1X 9AT 
| E-mail: q@mwam.com<mailto:q@mwam.com> | Tel: +44 207 925 4865


This e-mail and any attachments are confidential to the addressee(s) and may 
contain information that is legally privileged and/or confidential. If you are 
not the intended recipient of this e-mail you are hereby notified that any 
dissemination, distribution, or copying of its content is strictly prohibited. 
If you have received this message in error, please notify the sender by return 
e-mail and destroy the message and all copies in your possession.

To find out more details about how we may collect, use and share your personal 
information, please see https://www.mwam.com/privacy-policy. This includes 
details of how calls you make to us may be recorded in order for us to comply 
with our legal and regulatory obligations.

To the extent that the contents of this email constitutes a financial 
promotion, please note that it is issued only to and/or directed only at 
persons who are professional clients or eligible counterparties as defined in 
the FCA Rules. Any investment products or services described in this email are 
available only to professional clients and eligible counterparties. Persons who 
are not professional clients or eligible counterparties should not rely or act 
on the contents of this email.

Marshall Wace LLP is authorised and regulated by the Financial Conduct 
Authority. Marshall Wace LLP is a limited liability partnership registered in 
England and Wales with registered number OC302228 and registered office at 
George House, 131 Sloane Street, London, SW1X 9AT. If you are receiving this 
e-mail as a client, or an investor in an investment vehicle, managed or advised 
by Marshall Wace North America L.P., the sender of this e-mail is communicating 
with you in the sender's capacity as an associated or related person of 
Marshall Wace North America L.P. ("MWNA"), which is registered with the US 
Securities and Exchange Commission ("SEC") as an investment adviser. 
Registration with the SEC does not imply that MWNA or its employees possess a 
certain level of skill or training.

Re: Use different S3 access key for different S3 bucket

[Jun Qin]

Hi Qing

The S3 credentials are associated with Flink SQL tables.

I assume you are talking about processing/joining from two different tables, 
backed up by two different S3 buckets. If so, you can provide different 
credentials for different tables, then use the two tables in your pipeline.

Jun
On 18. Jan 2024 at 11:32 +0100, Qing Lim , wrote:
> Hi, I am using Flink SQL to create table backed by S3 buckets.
>
> We are not using AWS S3, so we have to use access key and secret for Auth.
>
> My pipeline depends on 2 different buckets, each requires different 
> credentials, can flink support this?
>
>
>
> Qing Lim | Marshall Wace LLP, George House, 131 Sloane Street, London, SW1X 
> 9AT | E-mail: q@mwam.com | Tel: +44 207 925 4865
>
> This e-mail and any attachments are confidential to the addressee(s) and may 
> contain information that is legally privileged and/or confidential. If you 
> are not the intended recipient of this e-mail you are hereby notified that 
> any dissemination, distribution, or copying of its content is strictly 
> prohibited. If you have received this message in error, please notify the 
> sender by return e-mail and destroy the message and all copies in your 
> possession.
>
> To find out more details about how we may collect, use and share your 
> personal information, please see https://www.mwam.com/privacy-policy. This 
> includes details of how calls you make to us may be recorded in order for us 
> to comply with our legal and regulatory obligations.
>
> To the extent that the contents of this email constitutes a financial 
> promotion, please note that it is issued only to and/or directed only at 
> persons who are professional clients or eligible counterparties as defined in 
> the FCA Rules. Any investment products or services described in this email 
> are available only to professional clients and eligible counterparties. 
> Persons who are not professional clients or eligible counterparties should 
> not rely or act on the contents of this email.
>
> Marshall Wace LLP is authorised and regulated by the Financial Conduct 
> Authority. Marshall Wace LLP is a limited liability partnership registered in 
> England and Wales with registered number OC302228 and registered office at 
> George House, 131 Sloane Street, London, SW1X 9AT. If you are receiving this 
> e-mail as a client, or an investor in an investment vehicle, managed or 
> advised by Marshall Wace North America L.P., the sender of this e-mail is 
> communicating with you in the sender's capacity as an associated or related 
> person of Marshall Wace North America L.P. ("MWNA"), which is registered with 
> the US Securities and Exchange Commission ("SEC") as an investment adviser. 
> Registration with the SEC does not imply that MWNA or its employees possess a 
> certain level of skill or training.

Use different S3 access key for different S3 bucket

[Qing Lim]

Hi, I am using Flink SQL to create table backed by S3 buckets.

We are not using AWS S3, so we have to use access key and secret for Auth.

My pipeline depends on 2 different buckets, each requires different 
credentials, can flink support this?



Qing Lim | Marshall Wace LLP, George House, 131 Sloane Street, London, SW1X 9AT 
| E-mail: q@mwam.com | Tel: +44 207 925 4865


This e-mail and any attachments are confidential to the addressee(s) and may 
contain information that is legally privileged and/or confidential. If you are 
not the intended recipient of this e-mail you are hereby notified that any 
dissemination, distribution, or copying of its content is strictly prohibited. 
If you have received this message in error, please notify the sender by return 
e-mail and destroy the message and all copies in your possession.

To find out more details about how we may collect, use and share your personal 
information, please see https://www.mwam.com/privacy-policy. This includes 
details of how calls you make to us may be recorded in order for us to comply 
with our legal and regulatory obligations.

To the extent that the contents of this email constitutes a financial 
promotion, please note that it is issued only to and/or directed only at 
persons who are professional clients or eligible counterparties as defined in 
the FCA Rules. Any investment products or services described in this email are 
available only to professional clients and eligible counterparties. Persons who 
are not professional clients or eligible counterparties should not rely or act 
on the contents of this email.

Marshall Wace LLP is authorised and regulated by the Financial Conduct 
Authority. Marshall Wace LLP is a limited liability partnership registered in 
England and Wales with registered number OC302228 and registered office at 
George House, 131 Sloane Street, London, SW1X 9AT. If you are receiving this 
e-mail as a client, or an investor in an investment vehicle, managed or advised 
by Marshall Wace North America L.P., the sender of this e-mail is communicating 
with you in the sender's capacity as an associated or related person of 
Marshall Wace North America L.P. (“MWNA”), which is registered with the US 
Securities and Exchange Commission (“SEC”) as an investment adviser.  
Registration with the SEC does not imply that MWNA or its employees possess a 
certain level of skill or training.