I have successfully connected Azure blob storage to Flink-1.1.
Below are the steps necessary:
- Add hadoop-azure-2.7.2.jar (assuming you are using a Hadoop 2.7 Flink
binary) and azure-storage-4.3.0.jar to <flinkdir>/lib, and set file
permissions / ownership accordingly.
- Add the following to a file 'core-site.xml'
<property>
<name>fs.wasb.impl</name>
<value>org.apache.hadoop.fs.azure.NativeAzureFileSystem</value>
</property>
<property>
<name>fs.wasbs.impl</name>
<value>org.apache.hadoop.fs.azure.NativeAzureFileSystem</value>
</property>
<property>
<name>fs.azure.account.key.STORAGEACCOUNTNAME.blob.core.windows.net
</name>
<value>ACCOUNTKEY</value>
</property>
- Update the parameter fs.hdfs.hadoopconf: to the path for the directory
where core-site.xml is located.
- Restart Flink
It is now possible to read from blobs (block and page) by referencing
'wasb://containern...@storageaccountname.blob.core.windows.net/PATH' or
'wasbs://containern...@storageaccountname.blob.core.windows.net/PATH'
Regards,
Lau
On 16 August 2016 at 14:37, MIkkel Islay <my.inputstr...@gmail.com> wrote:
>
> Hello,
>
> I would like to access data in Azure blob storage from Flink, via the
Azure storage HDFS-compatibility interface.
> That is feasible from Apache Drill, and I am thinking something similar
should be doable from Flink. A documentation page on eternal storage
connectors for Flink exist, but it was written pre 1.0.
> Does anyone have experience with setting up a Azure blob connector?
>
> Mikkel
--
Lau Sennels
Founder, scaling biologist
https://dk.linkedin.com/pub/lau-sennels/a9/3b5/196
