Hi Timothy,
unfortunetaly, this is not supported yet. However, the effort will be
tracked under the following ticket:
https://issues.apache.org/jira/browse/FLINK-19589
I will loop-in Arvid (in CC) which might help you in contributing the
missing functioniality.
Regards,
Timo
On 10.12.21 23:48, Timothy James wrote:
Hi,
The Hadoop s3a library itself supports some properties we need, but the
"FileSystem SQL Connector" (via FileSystemTableFactory) does not pass
connector options for these to the "Hadoop/Presto S3 File Systems
plugins" (via S3FileSystemFactory).
Instead, only Job-global Flink config values are passed to Hadoop s3a.
That won't work for us: we need to vary these values per Flink SQL
table, and not override our config for other use of S3 (such as Flink
checkpointing).
Contrast this with the Kafka connector, which supports an analogous
"properties.*" prefixed pass-through mechanism, and the Kinesis
connector, which supports all the specific properties we would need out
of the box.
Our current intent is to alter FileSystemTableFactory to follow the
"properties.*" approach used by the Kafka connector.
*** ➡️ Our questions for you: ⬅️
- Know of anything like this? Anybody solved this?
- Know of anything that's going to break this approach?
- What are we missing?
For context, our particular use case requires options like:
- fs.s3a.assumed.role.arn
- fs.s3a.aws.credentials.provider, (or some other mechanism to pass
externalId)
We imagine there would be other use cases for this, and if we build it
ourselves there's the possibility of contributing it to the Flink repo
for everybody.
Relevant documentation:
-
https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/table/filesystem/
<https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/table/filesystem/>
-
https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/filesystems/s3/#hadooppresto-s3-file-systems-plugins
<https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/filesystems/s3/#hadooppresto-s3-file-systems-plugins>
-
https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html
<https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html>
-
https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/table/kafka/#properties
<https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/table/kafka/#properties>
-
https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/table/kinesis/#aws-credentials-role-externalid
<https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/table/kinesis/#aws-credentials-role-externalid>
Thank you!
Tim James
Decodable.co
