[jira] [Commented] (FLINK-10865) Implement Flink's own Aliyun OSS filesystem
[ https://issues.apache.org/jira/browse/FLINK-10865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16714576#comment-16714576 ] ASF GitHub Bot commented on FLINK-10865: wujinhu commented on issue #7123: [FLINK-10865] Add Aliyun OSS file systems without Hadoop dependencies URL: https://github.com/apache/flink/pull/7123#issuecomment-445776844 @StefanRRichter Cool, thanks，I will focus on FLINK-11012 now. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement Flink's own Aliyun OSS filesystem > --- > > Key: FLINK-10865 > URL: https://issues.apache.org/jira/browse/FLINK-10865 > Project: Flink > Issue Type: New Feature > Components: filesystem-connector >Affects Versions: 1.6.2 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Labels: pull-request-available > Fix For: 1.8.0 > > > Aliyun OSS is widely used among China’s cloud users, and Hadoop supports > Aliyun OSS since 2.9.1. > Open this jira to wrap AliyunOSSFileSystem in flink(similar to s3 support), > so that user can read from & write to OSS more easily in flink. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10865) Implement Flink's own Aliyun OSS filesystem
[
https://issues.apache.org/jira/browse/FLINK-10865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16714549#comment-16714549
]
ASF GitHub Bot commented on FLINK-10865:
asfgit closed pull request #7123: [FLINK-10865] Add Aliyun OSS file systems
without Hadoop dependencies
URL: https://github.com/apache/flink/pull/7123
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/docs/ops/deployment/oss.md b/docs/ops/deployment/oss.md
new file mode 100644
index 000..543d5a2f176
--- /dev/null
+++ b/docs/ops/deployment/oss.md
@@ -0,0 +1,233 @@
+---
+title: "Aliyun Open Storage Service (OSS)"
+nav-title: Aliyun OSS
+nav-parent_id: deployment
+nav-pos: 9
+---
+
+
+* ToC
+{:toc}
+
+
+## OSS: Open Storage Service
+
+[Aliyun Open Storage Service](https://www.aliyun.com/product/oss) (Aliyun OSS)
is widely used especially among China’s cloud users, and it provides cloud
object storage for a variety of use cases.
+
+[Hadoop file
system](http://hadoop.apache.org/docs/current/hadoop-aliyun/tools/hadoop-aliyun/index.html)
supports OSS since version 2.9.1. Now, you can also use OSS with Fink for
**reading** and **writing data**.
+
+You can access OSS objects like this:
+
+{% highlight plain %}
+oss:///
+{% endhighlight %}
+
+Below shows how to use OSS with Flink:
+
+{% highlight java %}
+// Read from OSS bucket
+env.readTextFile("oss:///");
+
+// Write to OSS bucket
+dataSet.writeAsText("oss:///")
+
+{% endhighlight %}
+
+There are two ways to use OSS with Flink, our shaded `flink-oss-fs-hadoop`
will cover most scenarios. However, you may need to set up a specific Hadoop
OSS FileSystem implementation if you want use OSS as YARN's resource storage
dir ([This patch](https://issues.apache.org/jira/browse/HADOOP-15919) enables
YARN to use OSS). Both ways are described below.
+
+### Shaded Hadoop OSS file system (recommended)
+
+In order to use `flink-oss-fs-hadoop`, copy the respective JAR file from the
opt directory to the lib directory of your Flink distribution before starting
Flink, e.g.
+
+{% highlight bash %}
+cp ./opt/flink-oss-fs-hadoop-{{ site.version }}.jar ./lib/
+{% endhighlight %}
+
+`flink-oss-fs-hadoop` registers default FileSystem wrappers for URIs with the
oss:// scheme.
+
+ Configurations setup
+After setting up the OSS FileSystem wrapper, you need to add some
configurations to make sure that Flink is allowed to access your OSS buckets.
+
+In order to use OSS with Flink more easily, you can use the same configuration
keys in `flink-conf.yaml` as in Hadoop's `core-site.xml`
+
+You can see the configuration keys in the [Hadoop OSS
documentation](http://hadoop.apache.org/docs/current/hadoop-aliyun/tools/hadoop-aliyun/index.html).
+
+There are some required configurations that must be added to `flink-conf.yaml`
(**Other configurations defined in Hadoop OSS documentation are advanced
configurations which used by performance tuning**):
+
+{% highlight yaml %}
+fs.oss.endpoint: Aliyun OSS endpoint to connect to
+fs.oss.accessKeyId: Aliyun access key ID
+fs.oss.accessKeySecret: Aliyun access key secret
+{% endhighlight %}
+
+### Hadoop-provided OSS file system - manual setup
+This setup is a bit more complex and we recommend using our shaded Hadoop file
systems instead (see above) unless required otherwise, e.g. for using OSS as
YARN’s resource storage dir via the fs.defaultFS configuration property in
Hadoop’s core-site.xml.
+
+ Set OSS FileSystem
+You need to point Flink to a valid Hadoop configuration, which contains the
following properties in core-site.xml:
+
+{% highlight xml %}
+
+
+
+fs.oss.impl
+org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem
+
+
+
+fs.oss.endpoint
+Aliyun OSS endpoint to connect to
+Aliyun OSS endpoint to connect to. An up-to-date list is
provided in the Aliyun OSS Documentation.
+
+
+
+fs.oss.accessKeyId
+Aliyun access key ID
+
+
+
+fs.oss.accessKeySecret
+Aliyun access key secret
+
+
+
+fs.oss.buffer.dir
+/tmp/oss
+
+
+
+
+
+{% endhighlight %}
+
+ Hadoop Configuration
+
+You can specify the [Hadoop configuration](../config.html#hdfs) in various
ways pointing Flink to
+the path of the Hadoop configuration directory, for example
+- by setting the environment variable `HADOOP_CONF_DIR`, or
+- by setting the `fs.hdfs.hadoopconf` configuration option in
`flink-conf.yaml`:
+{% highlight yaml %}
+fs.hdfs.hadoopconf: /path/to/etc/hadoop
+{% endhighlight %}
+
+This registers `/path/to/etc/hadoop` as Hadoop's configuration directory with
Flink. Flink will look for the `core-site.xml` and `hdfs-site.xml` files in the
specified directory.
+
+ Provide OSS FileSystem Dependency
+
+You can
[jira] [Commented] (FLINK-10865) Implement Flink's own Aliyun OSS filesystem
[ https://issues.apache.org/jira/browse/FLINK-10865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713561#comment-16713561 ] ASF GitHub Bot commented on FLINK-10865: wujinhu commented on issue #7123: [FLINK-10865] Add Aliyun OSS file systems without Hadoop dependencies URL: https://github.com/apache/flink/pull/7123#issuecomment-445435454 @StefanRRichter Thanks, I have rebased master branch and revert merge history, I think this is what you mean. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement Flink's own Aliyun OSS filesystem > --- > > Key: FLINK-10865 > URL: https://issues.apache.org/jira/browse/FLINK-10865 > Project: Flink > Issue Type: New Feature > Components: filesystem-connector >Affects Versions: 1.6.2 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Labels: pull-request-available > > Aliyun OSS is widely used among China’s cloud users, and Hadoop supports > Aliyun OSS since 2.9.1. > Open this jira to wrap AliyunOSSFileSystem in flink(similar to s3 support), > so that user can read from & write to OSS more easily in flink. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10865) Implement Flink's own Aliyun OSS filesystem
[ https://issues.apache.org/jira/browse/FLINK-10865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16712985#comment-16712985 ] ASF GitHub Bot commented on FLINK-10865: StefanRRichter commented on issue #7123: [FLINK-10865] Add Aliyun OSS file systems without Hadoop dependencies URL: https://github.com/apache/flink/pull/7123#issuecomment-445270894 Hi, sorry for the late reply I but I was not available for the last week. Your commits are still using merges, see all the `Merge branch 'master' of` cases in the history of this PR. What we require is that your changes are all rebased on top of master and not merged with master. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement Flink's own Aliyun OSS filesystem > --- > > Key: FLINK-10865 > URL: https://issues.apache.org/jira/browse/FLINK-10865 > Project: Flink > Issue Type: New Feature > Components: filesystem-connector >Affects Versions: 1.6.2 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Labels: pull-request-available > > Aliyun OSS is widely used among China’s cloud users, and Hadoop supports > Aliyun OSS since 2.9.1. > Open this jira to wrap AliyunOSSFileSystem in flink(similar to s3 support), > so that user can read from & write to OSS more easily in flink. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10865) Implement Flink's own Aliyun OSS filesystem
[ https://issues.apache.org/jira/browse/FLINK-10865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16704407#comment-16704407 ] ASF GitHub Bot commented on FLINK-10865: wujinhu commented on issue #7123: [FLINK-10865] Add Aliyun OSS file systems without Hadoop dependencies URL: https://github.com/apache/flink/pull/7123#issuecomment-443130876 @StefanRRichter Please help to merge this PR, so that I can move on :) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement Flink's own Aliyun OSS filesystem > --- > > Key: FLINK-10865 > URL: https://issues.apache.org/jira/browse/FLINK-10865 > Project: Flink > Issue Type: New Feature > Components: filesystem-connector >Affects Versions: 1.6.2 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Labels: pull-request-available > > Aliyun OSS is widely used among China’s cloud users, and Hadoop supports > Aliyun OSS since 2.9.1. > Open this jira to wrap AliyunOSSFileSystem in flink(similar to s3 support), > so that user can read from & write to OSS more easily in flink. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10865) Implement Flink's own Aliyun OSS filesystem
[ https://issues.apache.org/jira/browse/FLINK-10865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701418#comment-16701418 ] ASF GitHub Bot commented on FLINK-10865: wujinhu commented on issue #7123: [FLINK-10865] Add Aliyun OSS file systems without Hadoop dependencies URL: https://github.com/apache/flink/pull/7123#issuecomment-442328461 @StefanRRichter The commits are OK now, pls merge this PR, thanks.:) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement Flink's own Aliyun OSS filesystem > --- > > Key: FLINK-10865 > URL: https://issues.apache.org/jira/browse/FLINK-10865 > Project: Flink > Issue Type: New Feature > Components: filesystem-connector >Affects Versions: 1.6.2 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Labels: pull-request-available > > Aliyun OSS is widely used among China’s cloud users, and Hadoop supports > Aliyun OSS since 2.9.1. > Open this jira to wrap AliyunOSSFileSystem in flink(similar to s3 support), > so that user can read from & write to OSS more easily in flink. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10865) Implement Flink's own Aliyun OSS filesystem
[ https://issues.apache.org/jira/browse/FLINK-10865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700123#comment-16700123 ] ASF GitHub Bot commented on FLINK-10865: wujinhu edited a comment on issue #7123: [FLINK-10865] Add Aliyun OSS file systems without Hadoop dependencies URL: https://github.com/apache/flink/pull/7123#issuecomment-441991022 @StefanRRichter The commits are ok now. I do not have rights to change assignee and I have sent an email to apply for the permission. I think it will be ok soon.:) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement Flink's own Aliyun OSS filesystem > --- > > Key: FLINK-10865 > URL: https://issues.apache.org/jira/browse/FLINK-10865 > Project: Flink > Issue Type: New Feature > Components: filesystem-connector >Affects Versions: 1.6.2 >Reporter: wujinhu >Priority: Major > Labels: pull-request-available > > Aliyun OSS is widely used among China’s cloud users, and Hadoop supports > Aliyun OSS since 2.9.1. > Open this jira to wrap AliyunOSSFileSystem in flink(similar to s3 support), > so that user can read from & write to OSS more easily in flink. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10865) Implement Flink's own Aliyun OSS filesystem
[ https://issues.apache.org/jira/browse/FLINK-10865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700119#comment-16700119 ] ASF GitHub Bot commented on FLINK-10865: wujinhu commented on issue #7123: [FLINK-10865] Add Aliyun OSS file systems without Hadoop dependencies URL: https://github.com/apache/flink/pull/7123#issuecomment-441991022 @StefanRRichter The commits are ok now. I do not have rights to change assignee and I have sent an email to apply the permission. I think it will be ok soon.:) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement Flink's own Aliyun OSS filesystem > --- > > Key: FLINK-10865 > URL: https://issues.apache.org/jira/browse/FLINK-10865 > Project: Flink > Issue Type: New Feature > Components: filesystem-connector >Affects Versions: 1.6.2 >Reporter: wujinhu >Priority: Major > Labels: pull-request-available > > Aliyun OSS is widely used among China’s cloud users, and Hadoop supports > Aliyun OSS since 2.9.1. > Open this jira to wrap AliyunOSSFileSystem in flink(similar to s3 support), > so that user can read from & write to OSS more easily in flink. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10865) Implement Flink's own Aliyun OSS filesystem
[ https://issues.apache.org/jira/browse/FLINK-10865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700096#comment-16700096 ] ASF GitHub Bot commented on FLINK-10865: StefanRRichter commented on issue #7123: [FLINK-10865] Add Aliyun OSS file systems without Hadoop dependencies URL: https://github.com/apache/flink/pull/7123#issuecomment-441981345 Alright, that sounds good, I opened FLINK-11012 for the test issue but for some reason was unable to assign your username. Can you try to assign it to you please? I will then merge once the commits are ready. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement Flink's own Aliyun OSS filesystem > --- > > Key: FLINK-10865 > URL: https://issues.apache.org/jira/browse/FLINK-10865 > Project: Flink > Issue Type: New Feature > Components: filesystem-connector >Affects Versions: 1.6.2 >Reporter: wujinhu >Priority: Major > Labels: pull-request-available > > Aliyun OSS is widely used among China’s cloud users, and Hadoop supports > Aliyun OSS since 2.9.1. > Open this jira to wrap AliyunOSSFileSystem in flink(similar to s3 support), > so that user can read from & write to OSS more easily in flink. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10865) Implement Flink's own Aliyun OSS filesystem
[ https://issues.apache.org/jira/browse/FLINK-10865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700092#comment-16700092 ] ASF GitHub Bot commented on FLINK-10865: wujinhu edited a comment on issue #7123: [FLINK-10865] Add Aliyun OSS file systems without Hadoop dependencies URL: https://github.com/apache/flink/pull/7123#issuecomment-441976550 @StefanRRichter Thanks for your comments, they are very helpful. IMO, we should open another jira to optimize the tests, because it will change tests in **flink-s3-fs-hadoop/flink-swift-fs-hadoop/flink-mapr-fs** and beyond this pr's scope. Could you pls open another jira and assign it to me? I think i can work on it after this change merged. Besides, I will fix the commits. :) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement Flink's own Aliyun OSS filesystem > --- > > Key: FLINK-10865 > URL: https://issues.apache.org/jira/browse/FLINK-10865 > Project: Flink > Issue Type: New Feature > Components: filesystem-connector >Affects Versions: 1.6.2 >Reporter: wujinhu >Priority: Major > Labels: pull-request-available > > Aliyun OSS is widely used among China’s cloud users, and Hadoop supports > Aliyun OSS since 2.9.1. > Open this jira to wrap AliyunOSSFileSystem in flink(similar to s3 support), > so that user can read from & write to OSS more easily in flink. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10865) Implement Flink's own Aliyun OSS filesystem
[ https://issues.apache.org/jira/browse/FLINK-10865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700084#comment-16700084 ] ASF GitHub Bot commented on FLINK-10865: wujinhu commented on issue #7123: [FLINK-10865] Add Aliyun OSS file systems without Hadoop dependencies URL: https://github.com/apache/flink/pull/7123#issuecomment-441978520 @StefanRRichter Here is the test result with my Aliyun access key and secret. It's sad we do not have public keys that can share now. --- T E S T S --- Running org.apache.flink.fs.osshadoop.HadoopOSSFileSystemITCase log4j:WARN No appenders could be found for logger (org.apache.flink.fs.osshadoop.HadoopOSSFileSystemITCase). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.91 sec - in org.apache.flink.fs.osshadoop.HadoopOSSFileSystemITCase Results : Tests run: 2, Failures: 0, Errors: 0, Skipped: 0 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement Flink's own Aliyun OSS filesystem > --- > > Key: FLINK-10865 > URL: https://issues.apache.org/jira/browse/FLINK-10865 > Project: Flink > Issue Type: New Feature > Components: filesystem-connector >Affects Versions: 1.6.2 >Reporter: wujinhu >Priority: Major > Labels: pull-request-available > > Aliyun OSS is widely used among China’s cloud users, and Hadoop supports > Aliyun OSS since 2.9.1. > Open this jira to wrap AliyunOSSFileSystem in flink(similar to s3 support), > so that user can read from & write to OSS more easily in flink. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10865) Implement Flink's own Aliyun OSS filesystem
[ https://issues.apache.org/jira/browse/FLINK-10865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700075#comment-16700075 ] ASF GitHub Bot commented on FLINK-10865: wujinhu commented on issue #7123: [FLINK-10865] Add Aliyun OSS file systems without Hadoop dependencies URL: https://github.com/apache/flink/pull/7123#issuecomment-441976550 @StefanRRichter Thanks for your comments, they are very helpful. IMO, we should open another jira to optimize the tests, because this change will change tests in **flink-s3-fs-hadoop/flink-swift-fs-hadoop/flink-mapr-fs** and beyond this pr's scope. Could you pls open another jira and assign it to me? I think i can work on it after this change merged. Besides, I will fix the commits. :) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement Flink's own Aliyun OSS filesystem > --- > > Key: FLINK-10865 > URL: https://issues.apache.org/jira/browse/FLINK-10865 > Project: Flink > Issue Type: New Feature > Components: filesystem-connector >Affects Versions: 1.6.2 >Reporter: wujinhu >Priority: Major > Labels: pull-request-available > > Aliyun OSS is widely used among China’s cloud users, and Hadoop supports > Aliyun OSS since 2.9.1. > Open this jira to wrap AliyunOSSFileSystem in flink(similar to s3 support), > so that user can read from & write to OSS more easily in flink. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10865) Implement Flink's own Aliyun OSS filesystem
[ https://issues.apache.org/jira/browse/FLINK-10865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700061#comment-16700061 ] ASF GitHub Bot commented on FLINK-10865: StefanRRichter commented on issue #7123: [FLINK-10865] Add Aliyun OSS file systems without Hadoop dependencies URL: https://github.com/apache/flink/pull/7123#issuecomment-441971705 Thanks for updating the docs! Any opinion about my comment on a generalized test? Please also fix all commits so that they are rebased on master without merges, otherwise we cannot merge it. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement Flink's own Aliyun OSS filesystem > --- > > Key: FLINK-10865 > URL: https://issues.apache.org/jira/browse/FLINK-10865 > Project: Flink > Issue Type: New Feature > Components: filesystem-connector >Affects Versions: 1.6.2 >Reporter: wujinhu >Priority: Major > Labels: pull-request-available > > Aliyun OSS is widely used among China’s cloud users, and Hadoop supports > Aliyun OSS since 2.9.1. > Open this jira to wrap AliyunOSSFileSystem in flink(similar to s3 support), > so that user can read from & write to OSS more easily in flink. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10865) Implement Flink's own Aliyun OSS filesystem
[ https://issues.apache.org/jira/browse/FLINK-10865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699330#comment-16699330 ] ASF GitHub Bot commented on FLINK-10865: StefanRRichter edited a comment on issue #7123: [FLINK-10865] Add Aliyun OSS file systems without Hadoop dependencies URL: https://github.com/apache/flink/pull/7123#issuecomment-441723504 Thank you for this contribution. I think overall it looks good to me, in particular I like how detailed your documentation is. My main comments are: - Some fixes for the documentation - Thinking about a generalized test base for testing the file systems Please also bring your commits in order so that there is no merge and everything is rebased on the latest master instead. Thanks! Disclaimer: I also could not try out the test case because I have no access to a Aliyun OSS fs. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement Flink's own Aliyun OSS filesystem > --- > > Key: FLINK-10865 > URL: https://issues.apache.org/jira/browse/FLINK-10865 > Project: Flink > Issue Type: New Feature > Components: filesystem-connector >Affects Versions: 1.6.2 >Reporter: wujinhu >Priority: Major > Labels: pull-request-available > > Aliyun OSS is widely used among China’s cloud users, and Hadoop supports > Aliyun OSS since 2.9.1. > Open this jira to wrap AliyunOSSFileSystem in flink(similar to s3 support), > so that user can read from & write to OSS more easily in flink. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10865) Implement Flink's own Aliyun OSS filesystem
[ https://issues.apache.org/jira/browse/FLINK-10865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699327#comment-16699327 ] ASF GitHub Bot commented on FLINK-10865: StefanRRichter commented on issue #7123: [FLINK-10865] Add Aliyun OSS file systems without Hadoop dependencies URL: https://github.com/apache/flink/pull/7123#issuecomment-441723504 Thank you for this contribution. I think overall it looks good to me, in particular I like how detailed your documentation is. My main comments are: - Some fixes for the documentation - Thinking about a generalized test base for testing the file systems Please also bring your commits in order so that there is no merge and everything is rebased on the latest master instead. Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement Flink's own Aliyun OSS filesystem > --- > > Key: FLINK-10865 > URL: https://issues.apache.org/jira/browse/FLINK-10865 > Project: Flink > Issue Type: New Feature > Components: filesystem-connector >Affects Versions: 1.6.2 >Reporter: wujinhu >Priority: Major > Labels: pull-request-available > > Aliyun OSS is widely used among China’s cloud users, and Hadoop supports > Aliyun OSS since 2.9.1. > Open this jira to wrap AliyunOSSFileSystem in flink(similar to s3 support), > so that user can read from & write to OSS more easily in flink. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10865) Implement Flink's own Aliyun OSS filesystem
[
https://issues.apache.org/jira/browse/FLINK-10865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699322#comment-16699322
]
ASF GitHub Bot commented on FLINK-10865:
StefanRRichter commented on a change in pull request #7123: [FLINK-10865] Add
Aliyun OSS file systems without Hadoop dependencies
URL: https://github.com/apache/flink/pull/7123#discussion_r236345803
##
File path: docs/ops/deployment/oss.md
##
@@ -0,0 +1,233 @@
+---
+title: "Aliyun Open Storage Service (OSS)"
+nav-title: Aliyun OSS
+nav-parent_id: deployment
+nav-pos: 9
+---
+
+
+* ToC
+{:toc}
+
+
+## OSS: Open Storage Service
+
+[Aliyun Open Storage Service](https://www.aliyun.com/product/oss) (Aliyun OSS)
is widely used especially among China’s cloud users, and it provides cloud
object storage for a variety of use cases.
+
+[Hadoop file
system](http://hadoop.apache.org/docs/current/hadoop-aliyun/tools/hadoop-aliyun/index.html)
supports OSS since version 2.9.1. Now, you can also use OSS with Fink for
**reading** and **writing data**.
+
+You can access OSS objects like this:
+
+{% highlight plain %}
+oss:///
+{% endhighlight %}
+
+Below shows how to use OSS with Flink:
+
+{% highlight java %}
+// Read from OSS bucket
+env.readTextFile("oss:///");
+
+// Write to OSS bucket
+dataSet.writeAsText("oss:///")
+
+{% endhighlight %}
+
+There are two ways to use OSS with Flink, our shaded `flink-oss-fs-hadoop`
will cover most scenarios. However, you may need to set up a specific Hadoop
OSS FileSystem implementation if you want use OSS as YARN's resource storage
dir([This patch](https://issues.apache.org/jira/browse/HADOOP-15919) enables
YARN to use OSS). Both ways are described below.
+
+### Shaded Hadoop OSS file system (recommended)
+
+In order to use `flink-oss-fs-hadoop`, copy the respective JAR file from the
opt directory to the lib directory of your Flink distribution before starting
Flink, e.g.
+
+{% highlight bash %}
+cp ./opt/flink-oss-fs-hadoop-{{ site.version }}.jar ./lib/
+{% endhighlight %}
+
+`flink-oss-fs-hadoop` registers default FileSystem wrappers for URIs with the
oss:// scheme.
+
+ Configurations setup
+After setting up the OSS FileSystem wrapper, you need to add some
configurations to make sure that Flink is allowed to access your OSS buckets.
+
+In order to use OSS with Flink more easily, you can use the same configuration
keys in `flink-conf.yaml` with which in Hadoop's `core-site.xml`
+
+You can see configuration keys in [Hadoop OSS
documentation](http://hadoop.apache.org/docs/current/hadoop-aliyun/tools/hadoop-aliyun/index.html).
+
+There are some required configurations must be added to
`flink-conf.yaml`(**Other configurations defined in Hadoop OSS documentation
are advanced configurations which used by performance tuning**):
Review comment:
... configurations **that** must ...
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Implement Flink's own Aliyun OSS filesystem
> ---
>
> Key: FLINK-10865
> URL: https://issues.apache.org/jira/browse/FLINK-10865
> Project: Flink
> Issue Type: New Feature
> Components: filesystem-connector
>Affects Versions: 1.6.2
>Reporter: wujinhu
>Priority: Major
> Labels: pull-request-available
>
> Aliyun OSS is widely used among China’s cloud users, and Hadoop supports
> Aliyun OSS since 2.9.1.
> Open this jira to wrap AliyunOSSFileSystem in flink(similar to s3 support),
> so that user can read from & write to OSS more easily in flink.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10865) Implement Flink's own Aliyun OSS filesystem
[
https://issues.apache.org/jira/browse/FLINK-10865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699320#comment-16699320
]
ASF GitHub Bot commented on FLINK-10865:
StefanRRichter commented on a change in pull request #7123: [FLINK-10865] Add
Aliyun OSS file systems without Hadoop dependencies
URL: https://github.com/apache/flink/pull/7123#discussion_r236348730
##
File path: docs/ops/deployment/oss.md
##
@@ -0,0 +1,233 @@
+---
+title: "Aliyun Open Storage Service (OSS)"
+nav-title: Aliyun OSS
+nav-parent_id: deployment
+nav-pos: 9
+---
+
+
+* ToC
+{:toc}
+
+
+## OSS: Open Storage Service
+
+[Aliyun Open Storage Service](https://www.aliyun.com/product/oss) (Aliyun OSS)
is widely used especially among China’s cloud users, and it provides cloud
object storage for a variety of use cases.
+
+[Hadoop file
system](http://hadoop.apache.org/docs/current/hadoop-aliyun/tools/hadoop-aliyun/index.html)
supports OSS since version 2.9.1. Now, you can also use OSS with Fink for
**reading** and **writing data**.
+
+You can access OSS objects like this:
+
+{% highlight plain %}
+oss:///
+{% endhighlight %}
+
+Below shows how to use OSS with Flink:
+
+{% highlight java %}
+// Read from OSS bucket
+env.readTextFile("oss:///");
+
+// Write to OSS bucket
+dataSet.writeAsText("oss:///")
+
+{% endhighlight %}
+
+There are two ways to use OSS with Flink, our shaded `flink-oss-fs-hadoop`
will cover most scenarios. However, you may need to set up a specific Hadoop
OSS FileSystem implementation if you want use OSS as YARN's resource storage
dir([This patch](https://issues.apache.org/jira/browse/HADOOP-15919) enables
YARN to use OSS). Both ways are described below.
+
+### Shaded Hadoop OSS file system (recommended)
+
+In order to use `flink-oss-fs-hadoop`, copy the respective JAR file from the
opt directory to the lib directory of your Flink distribution before starting
Flink, e.g.
+
+{% highlight bash %}
+cp ./opt/flink-oss-fs-hadoop-{{ site.version }}.jar ./lib/
+{% endhighlight %}
+
+`flink-oss-fs-hadoop` registers default FileSystem wrappers for URIs with the
oss:// scheme.
+
+ Configurations setup
+After setting up the OSS FileSystem wrapper, you need to add some
configurations to make sure that Flink is allowed to access your OSS buckets.
+
+In order to use OSS with Flink more easily, you can use the same configuration
keys in `flink-conf.yaml` with which in Hadoop's `core-site.xml`
+
+You can see configuration keys in [Hadoop OSS
documentation](http://hadoop.apache.org/docs/current/hadoop-aliyun/tools/hadoop-aliyun/index.html).
+
+There are some required configurations must be added to
`flink-conf.yaml`(**Other configurations defined in Hadoop OSS documentation
are advanced configurations which used by performance tuning**):
+
+{% highlight yaml %}
+fs.oss.endpoint: Aliyun OSS endpoint to connect to
+fs.oss.accessKeyId: Aliyun access key ID
+fs.oss.accessKeySecret: Aliyun access key secret
+{% endhighlight %}
+
+### Hadoop-provided OSS file system - manual setup
+This setup is a bit more complex and we recommend using our shaded Hadoop file
systems instead (see above) unless required otherwise, e.g. for using OSS as
YARN’s resource storage dir via the fs.defaultFS configuration property in
Hadoop’s core-site.xml.
+
+ Set OSS FileSystem
+You need to point Flink to a valid Hadoop configuration, which contains the
following properties in core-site.xml:
+
+{% highlight xml %}
+
+
+
+fs.oss.impl
+org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem
+
+
+
+fs.oss.endpoint
+Aliyun OSS endpoint to connect to
+Aliyun OSS endpoint to connect to. An up-to-date list is
provided in the Aliyun OSS Documentation.
+
+
+
+fs.oss.accessKeyId
+Aliyun access key ID
+
+
+
+fs.oss.accessKeySecret
+Aliyun access key secret
+
+
+
+fs.oss.buffer.dir
+/tmp/oss
+
+
+
+
+
+{% endhighlight %}
+
+ Hadoop Configuration
+
+You can specify the [Hadoop configuration](../config.html#hdfs) in various
ways pointing Flink to
+the path of the Hadoop configuration directory, for example
+- by setting the environment variable `HADOOP_CONF_DIR`, or
+- by setting the `fs.hdfs.hadoopconf` configuration option in
`flink-conf.yaml`:
+{% highlight yaml %}
+fs.hdfs.hadoopconf: /path/to/etc/hadoop
+{% endhighlight %}
+
+This registers `/path/to/etc/hadoop` as Hadoop's configuration directory with
Flink. Flink will look for the `core-site.xml` and `hdfs-site.xml` files in the
specified directory.
+
+ Provide OSS FileSystem Dependency
+
+You can find Hadoop OSS FileSystem are packaged in the hadoop-aliyun artifact.
This JAR and all its dependencies need to be added to Flink’s classpath, i.e.
the class path of both Job and TaskManagers.
+
+There are multiple ways of adding JARs to Flink’s class path, the easiest
being simply to drop the JARs in Flink’s lib folder. You
[jira] [Commented] (FLINK-10865) Implement Flink's own Aliyun OSS filesystem
[
https://issues.apache.org/jira/browse/FLINK-10865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699321#comment-16699321
]
ASF GitHub Bot commented on FLINK-10865:
StefanRRichter commented on a change in pull request #7123: [FLINK-10865] Add
Aliyun OSS file systems without Hadoop dependencies
URL: https://github.com/apache/flink/pull/7123#discussion_r236348352
##
File path: docs/ops/deployment/oss.md
##
@@ -0,0 +1,233 @@
+---
+title: "Aliyun Open Storage Service (OSS)"
+nav-title: Aliyun OSS
+nav-parent_id: deployment
+nav-pos: 9
+---
+
+
+* ToC
+{:toc}
+
+
+## OSS: Open Storage Service
+
+[Aliyun Open Storage Service](https://www.aliyun.com/product/oss) (Aliyun OSS)
is widely used especially among China’s cloud users, and it provides cloud
object storage for a variety of use cases.
+
+[Hadoop file
system](http://hadoop.apache.org/docs/current/hadoop-aliyun/tools/hadoop-aliyun/index.html)
supports OSS since version 2.9.1. Now, you can also use OSS with Fink for
**reading** and **writing data**.
+
+You can access OSS objects like this:
+
+{% highlight plain %}
+oss:///
+{% endhighlight %}
+
+Below shows how to use OSS with Flink:
+
+{% highlight java %}
+// Read from OSS bucket
+env.readTextFile("oss:///");
+
+// Write to OSS bucket
+dataSet.writeAsText("oss:///")
+
+{% endhighlight %}
+
+There are two ways to use OSS with Flink, our shaded `flink-oss-fs-hadoop`
will cover most scenarios. However, you may need to set up a specific Hadoop
OSS FileSystem implementation if you want use OSS as YARN's resource storage
dir([This patch](https://issues.apache.org/jira/browse/HADOOP-15919) enables
YARN to use OSS). Both ways are described below.
+
+### Shaded Hadoop OSS file system (recommended)
+
+In order to use `flink-oss-fs-hadoop`, copy the respective JAR file from the
opt directory to the lib directory of your Flink distribution before starting
Flink, e.g.
+
+{% highlight bash %}
+cp ./opt/flink-oss-fs-hadoop-{{ site.version }}.jar ./lib/
+{% endhighlight %}
+
+`flink-oss-fs-hadoop` registers default FileSystem wrappers for URIs with the
oss:// scheme.
+
+ Configurations setup
+After setting up the OSS FileSystem wrapper, you need to add some
configurations to make sure that Flink is allowed to access your OSS buckets.
+
+In order to use OSS with Flink more easily, you can use the same configuration
keys in `flink-conf.yaml` with which in Hadoop's `core-site.xml`
+
+You can see configuration keys in [Hadoop OSS
documentation](http://hadoop.apache.org/docs/current/hadoop-aliyun/tools/hadoop-aliyun/index.html).
+
+There are some required configurations must be added to
`flink-conf.yaml`(**Other configurations defined in Hadoop OSS documentation
are advanced configurations which used by performance tuning**):
+
+{% highlight yaml %}
+fs.oss.endpoint: Aliyun OSS endpoint to connect to
+fs.oss.accessKeyId: Aliyun access key ID
+fs.oss.accessKeySecret: Aliyun access key secret
+{% endhighlight %}
+
+### Hadoop-provided OSS file system - manual setup
+This setup is a bit more complex and we recommend using our shaded Hadoop file
systems instead (see above) unless required otherwise, e.g. for using OSS as
YARN’s resource storage dir via the fs.defaultFS configuration property in
Hadoop’s core-site.xml.
+
+ Set OSS FileSystem
+You need to point Flink to a valid Hadoop configuration, which contains the
following properties in core-site.xml:
+
+{% highlight xml %}
+
+
+
+fs.oss.impl
+org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem
+
+
+
+fs.oss.endpoint
+Aliyun OSS endpoint to connect to
+Aliyun OSS endpoint to connect to. An up-to-date list is
provided in the Aliyun OSS Documentation.
+
+
+
+fs.oss.accessKeyId
+Aliyun access key ID
+
+
+
+fs.oss.accessKeySecret
+Aliyun access key secret
+
+
+
+fs.oss.buffer.dir
+/tmp/oss
+
+
+
+
+
+{% endhighlight %}
+
+ Hadoop Configuration
+
+You can specify the [Hadoop configuration](../config.html#hdfs) in various
ways pointing Flink to
+the path of the Hadoop configuration directory, for example
+- by setting the environment variable `HADOOP_CONF_DIR`, or
+- by setting the `fs.hdfs.hadoopconf` configuration option in
`flink-conf.yaml`:
+{% highlight yaml %}
+fs.hdfs.hadoopconf: /path/to/etc/hadoop
+{% endhighlight %}
+
+This registers `/path/to/etc/hadoop` as Hadoop's configuration directory with
Flink. Flink will look for the `core-site.xml` and `hdfs-site.xml` files in the
specified directory.
+
+ Provide OSS FileSystem Dependency
+
+You can find Hadoop OSS FileSystem are packaged in the hadoop-aliyun artifact.
This JAR and all its dependencies need to be added to Flink’s classpath, i.e.
the class path of both Job and TaskManagers.
+
+There are multiple ways of adding JARs to Flink’s class path, the easiest
being simply to drop the JARs in Flink’s lib folder. You
[jira] [Commented] (FLINK-10865) Implement Flink's own Aliyun OSS filesystem
[
https://issues.apache.org/jira/browse/FLINK-10865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699317#comment-16699317
]
ASF GitHub Bot commented on FLINK-10865:
StefanRRichter commented on a change in pull request #7123: [FLINK-10865] Add
Aliyun OSS file systems without Hadoop dependencies
URL: https://github.com/apache/flink/pull/7123#discussion_r236345508
##
File path: docs/ops/deployment/oss.md
##
@@ -0,0 +1,233 @@
+---
+title: "Aliyun Open Storage Service (OSS)"
+nav-title: Aliyun OSS
+nav-parent_id: deployment
+nav-pos: 9
+---
+
+
+* ToC
+{:toc}
+
+
+## OSS: Open Storage Service
+
+[Aliyun Open Storage Service](https://www.aliyun.com/product/oss) (Aliyun OSS)
is widely used especially among China’s cloud users, and it provides cloud
object storage for a variety of use cases.
+
+[Hadoop file
system](http://hadoop.apache.org/docs/current/hadoop-aliyun/tools/hadoop-aliyun/index.html)
supports OSS since version 2.9.1. Now, you can also use OSS with Fink for
**reading** and **writing data**.
+
+You can access OSS objects like this:
+
+{% highlight plain %}
+oss:///
+{% endhighlight %}
+
+Below shows how to use OSS with Flink:
+
+{% highlight java %}
+// Read from OSS bucket
+env.readTextFile("oss:///");
+
+// Write to OSS bucket
+dataSet.writeAsText("oss:///")
+
+{% endhighlight %}
+
+There are two ways to use OSS with Flink, our shaded `flink-oss-fs-hadoop`
will cover most scenarios. However, you may need to set up a specific Hadoop
OSS FileSystem implementation if you want use OSS as YARN's resource storage
dir([This patch](https://issues.apache.org/jira/browse/HADOOP-15919) enables
YARN to use OSS). Both ways are described below.
+
+### Shaded Hadoop OSS file system (recommended)
+
+In order to use `flink-oss-fs-hadoop`, copy the respective JAR file from the
opt directory to the lib directory of your Flink distribution before starting
Flink, e.g.
+
+{% highlight bash %}
+cp ./opt/flink-oss-fs-hadoop-{{ site.version }}.jar ./lib/
+{% endhighlight %}
+
+`flink-oss-fs-hadoop` registers default FileSystem wrappers for URIs with the
oss:// scheme.
+
+ Configurations setup
+After setting up the OSS FileSystem wrapper, you need to add some
configurations to make sure that Flink is allowed to access your OSS buckets.
+
+In order to use OSS with Flink more easily, you can use the same configuration
keys in `flink-conf.yaml` with which in Hadoop's `core-site.xml`
+
+You can see configuration keys in [Hadoop OSS
documentation](http://hadoop.apache.org/docs/current/hadoop-aliyun/tools/hadoop-aliyun/index.html).
Review comment:
keys -> **the** keys
and also: in **the** Hadoop...
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Implement Flink's own Aliyun OSS filesystem
> ---
>
> Key: FLINK-10865
> URL: https://issues.apache.org/jira/browse/FLINK-10865
> Project: Flink
> Issue Type: New Feature
> Components: filesystem-connector
>Affects Versions: 1.6.2
>Reporter: wujinhu
>Priority: Major
> Labels: pull-request-available
>
> Aliyun OSS is widely used among China’s cloud users, and Hadoop supports
> Aliyun OSS since 2.9.1.
> Open this jira to wrap AliyunOSSFileSystem in flink(similar to s3 support),
> so that user can read from & write to OSS more easily in flink.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10865) Implement Flink's own Aliyun OSS filesystem
[
https://issues.apache.org/jira/browse/FLINK-10865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699323#comment-16699323
]
ASF GitHub Bot commented on FLINK-10865:
StefanRRichter commented on a change in pull request #7123: [FLINK-10865] Add
Aliyun OSS file systems without Hadoop dependencies
URL: https://github.com/apache/flink/pull/7123#discussion_r236347738
##
File path: docs/ops/deployment/oss.md
##
@@ -0,0 +1,233 @@
+---
+title: "Aliyun Open Storage Service (OSS)"
+nav-title: Aliyun OSS
+nav-parent_id: deployment
+nav-pos: 9
+---
+
+
+* ToC
+{:toc}
+
+
+## OSS: Open Storage Service
+
+[Aliyun Open Storage Service](https://www.aliyun.com/product/oss) (Aliyun OSS)
is widely used especially among China’s cloud users, and it provides cloud
object storage for a variety of use cases.
+
+[Hadoop file
system](http://hadoop.apache.org/docs/current/hadoop-aliyun/tools/hadoop-aliyun/index.html)
supports OSS since version 2.9.1. Now, you can also use OSS with Fink for
**reading** and **writing data**.
+
+You can access OSS objects like this:
+
+{% highlight plain %}
+oss:///
+{% endhighlight %}
+
+Below shows how to use OSS with Flink:
+
+{% highlight java %}
+// Read from OSS bucket
+env.readTextFile("oss:///");
+
+// Write to OSS bucket
+dataSet.writeAsText("oss:///")
+
+{% endhighlight %}
+
+There are two ways to use OSS with Flink, our shaded `flink-oss-fs-hadoop`
will cover most scenarios. However, you may need to set up a specific Hadoop
OSS FileSystem implementation if you want use OSS as YARN's resource storage
dir([This patch](https://issues.apache.org/jira/browse/HADOOP-15919) enables
YARN to use OSS). Both ways are described below.
+
+### Shaded Hadoop OSS file system (recommended)
+
+In order to use `flink-oss-fs-hadoop`, copy the respective JAR file from the
opt directory to the lib directory of your Flink distribution before starting
Flink, e.g.
+
+{% highlight bash %}
+cp ./opt/flink-oss-fs-hadoop-{{ site.version }}.jar ./lib/
+{% endhighlight %}
+
+`flink-oss-fs-hadoop` registers default FileSystem wrappers for URIs with the
oss:// scheme.
+
+ Configurations setup
+After setting up the OSS FileSystem wrapper, you need to add some
configurations to make sure that Flink is allowed to access your OSS buckets.
+
+In order to use OSS with Flink more easily, you can use the same configuration
keys in `flink-conf.yaml` with which in Hadoop's `core-site.xml`
+
+You can see configuration keys in [Hadoop OSS
documentation](http://hadoop.apache.org/docs/current/hadoop-aliyun/tools/hadoop-aliyun/index.html).
+
+There are some required configurations must be added to
`flink-conf.yaml`(**Other configurations defined in Hadoop OSS documentation
are advanced configurations which used by performance tuning**):
+
+{% highlight yaml %}
+fs.oss.endpoint: Aliyun OSS endpoint to connect to
+fs.oss.accessKeyId: Aliyun access key ID
+fs.oss.accessKeySecret: Aliyun access key secret
+{% endhighlight %}
+
+### Hadoop-provided OSS file system - manual setup
+This setup is a bit more complex and we recommend using our shaded Hadoop file
systems instead (see above) unless required otherwise, e.g. for using OSS as
YARN’s resource storage dir via the fs.defaultFS configuration property in
Hadoop’s core-site.xml.
+
+ Set OSS FileSystem
+You need to point Flink to a valid Hadoop configuration, which contains the
following properties in core-site.xml:
+
+{% highlight xml %}
+
+
+
+fs.oss.impl
+org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem
+
+
+
+fs.oss.endpoint
+Aliyun OSS endpoint to connect to
+Aliyun OSS endpoint to connect to. An up-to-date list is
provided in the Aliyun OSS Documentation.
+
+
+
+fs.oss.accessKeyId
+Aliyun access key ID
+
+
+
+fs.oss.accessKeySecret
+Aliyun access key secret
+
+
+
+fs.oss.buffer.dir
+/tmp/oss
+
+
+
+
+
+{% endhighlight %}
+
+ Hadoop Configuration
+
+You can specify the [Hadoop configuration](../config.html#hdfs) in various
ways pointing Flink to
+the path of the Hadoop configuration directory, for example
+- by setting the environment variable `HADOOP_CONF_DIR`, or
+- by setting the `fs.hdfs.hadoopconf` configuration option in
`flink-conf.yaml`:
+{% highlight yaml %}
+fs.hdfs.hadoopconf: /path/to/etc/hadoop
+{% endhighlight %}
+
+This registers `/path/to/etc/hadoop` as Hadoop's configuration directory with
Flink. Flink will look for the `core-site.xml` and `hdfs-site.xml` files in the
specified directory.
+
+ Provide OSS FileSystem Dependency
+
+You can find Hadoop OSS FileSystem are packaged in the hadoop-aliyun artifact.
This JAR and all its dependencies need to be added to Flink’s classpath, i.e.
the class path of both Job and TaskManagers.
+
+There are multiple ways of adding JARs to Flink’s class path, the easiest
being simply to drop the JARs in Flink’s lib folder. You
[jira] [Commented] (FLINK-10865) Implement Flink's own Aliyun OSS filesystem
[
https://issues.apache.org/jira/browse/FLINK-10865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699319#comment-16699319
]
ASF GitHub Bot commented on FLINK-10865:
StefanRRichter commented on a change in pull request #7123: [FLINK-10865] Add
Aliyun OSS file systems without Hadoop dependencies
URL: https://github.com/apache/flink/pull/7123#discussion_r236345238
##
File path: docs/ops/deployment/oss.md
##
@@ -0,0 +1,233 @@
+---
+title: "Aliyun Open Storage Service (OSS)"
+nav-title: Aliyun OSS
+nav-parent_id: deployment
+nav-pos: 9
+---
+
+
+* ToC
+{:toc}
+
+
+## OSS: Open Storage Service
+
+[Aliyun Open Storage Service](https://www.aliyun.com/product/oss) (Aliyun OSS)
is widely used especially among China’s cloud users, and it provides cloud
object storage for a variety of use cases.
+
+[Hadoop file
system](http://hadoop.apache.org/docs/current/hadoop-aliyun/tools/hadoop-aliyun/index.html)
supports OSS since version 2.9.1. Now, you can also use OSS with Fink for
**reading** and **writing data**.
+
+You can access OSS objects like this:
+
+{% highlight plain %}
+oss:///
+{% endhighlight %}
+
+Below shows how to use OSS with Flink:
+
+{% highlight java %}
+// Read from OSS bucket
+env.readTextFile("oss:///");
+
+// Write to OSS bucket
+dataSet.writeAsText("oss:///")
+
+{% endhighlight %}
+
+There are two ways to use OSS with Flink, our shaded `flink-oss-fs-hadoop`
will cover most scenarios. However, you may need to set up a specific Hadoop
OSS FileSystem implementation if you want use OSS as YARN's resource storage
dir([This patch](https://issues.apache.org/jira/browse/HADOOP-15919) enables
YARN to use OSS). Both ways are described below.
+
+### Shaded Hadoop OSS file system (recommended)
+
+In order to use `flink-oss-fs-hadoop`, copy the respective JAR file from the
opt directory to the lib directory of your Flink distribution before starting
Flink, e.g.
+
+{% highlight bash %}
+cp ./opt/flink-oss-fs-hadoop-{{ site.version }}.jar ./lib/
+{% endhighlight %}
+
+`flink-oss-fs-hadoop` registers default FileSystem wrappers for URIs with the
oss:// scheme.
+
+ Configurations setup
+After setting up the OSS FileSystem wrapper, you need to add some
configurations to make sure that Flink is allowed to access your OSS buckets.
+
+In order to use OSS with Flink more easily, you can use the same configuration
keys in `flink-conf.yaml` with which in Hadoop's `core-site.xml`
Review comment:
the same configuration keys ... **as in** Hadoop's ...
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Implement Flink's own Aliyun OSS filesystem
> ---
>
> Key: FLINK-10865
> URL: https://issues.apache.org/jira/browse/FLINK-10865
> Project: Flink
> Issue Type: New Feature
> Components: filesystem-connector
>Affects Versions: 1.6.2
>Reporter: wujinhu
>Priority: Major
> Labels: pull-request-available
>
> Aliyun OSS is widely used among China’s cloud users, and Hadoop supports
> Aliyun OSS since 2.9.1.
> Open this jira to wrap AliyunOSSFileSystem in flink(similar to s3 support),
> so that user can read from & write to OSS more easily in flink.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10865) Implement Flink's own Aliyun OSS filesystem
[
https://issues.apache.org/jira/browse/FLINK-10865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699324#comment-16699324
]
ASF GitHub Bot commented on FLINK-10865:
StefanRRichter commented on a change in pull request #7123: [FLINK-10865] Add
Aliyun OSS file systems without Hadoop dependencies
URL: https://github.com/apache/flink/pull/7123#discussion_r236348045
##
File path: docs/ops/deployment/oss.md
##
@@ -0,0 +1,233 @@
+---
+title: "Aliyun Open Storage Service (OSS)"
+nav-title: Aliyun OSS
+nav-parent_id: deployment
+nav-pos: 9
+---
+
+
+* ToC
+{:toc}
+
+
+## OSS: Open Storage Service
+
+[Aliyun Open Storage Service](https://www.aliyun.com/product/oss) (Aliyun OSS)
is widely used especially among China’s cloud users, and it provides cloud
object storage for a variety of use cases.
+
+[Hadoop file
system](http://hadoop.apache.org/docs/current/hadoop-aliyun/tools/hadoop-aliyun/index.html)
supports OSS since version 2.9.1. Now, you can also use OSS with Fink for
**reading** and **writing data**.
+
+You can access OSS objects like this:
+
+{% highlight plain %}
+oss:///
+{% endhighlight %}
+
+Below shows how to use OSS with Flink:
+
+{% highlight java %}
+// Read from OSS bucket
+env.readTextFile("oss:///");
+
+// Write to OSS bucket
+dataSet.writeAsText("oss:///")
+
+{% endhighlight %}
+
+There are two ways to use OSS with Flink, our shaded `flink-oss-fs-hadoop`
will cover most scenarios. However, you may need to set up a specific Hadoop
OSS FileSystem implementation if you want use OSS as YARN's resource storage
dir([This patch](https://issues.apache.org/jira/browse/HADOOP-15919) enables
YARN to use OSS). Both ways are described below.
+
+### Shaded Hadoop OSS file system (recommended)
+
+In order to use `flink-oss-fs-hadoop`, copy the respective JAR file from the
opt directory to the lib directory of your Flink distribution before starting
Flink, e.g.
+
+{% highlight bash %}
+cp ./opt/flink-oss-fs-hadoop-{{ site.version }}.jar ./lib/
+{% endhighlight %}
+
+`flink-oss-fs-hadoop` registers default FileSystem wrappers for URIs with the
oss:// scheme.
+
+ Configurations setup
+After setting up the OSS FileSystem wrapper, you need to add some
configurations to make sure that Flink is allowed to access your OSS buckets.
+
+In order to use OSS with Flink more easily, you can use the same configuration
keys in `flink-conf.yaml` with which in Hadoop's `core-site.xml`
+
+You can see configuration keys in [Hadoop OSS
documentation](http://hadoop.apache.org/docs/current/hadoop-aliyun/tools/hadoop-aliyun/index.html).
+
+There are some required configurations must be added to
`flink-conf.yaml`(**Other configurations defined in Hadoop OSS documentation
are advanced configurations which used by performance tuning**):
+
+{% highlight yaml %}
+fs.oss.endpoint: Aliyun OSS endpoint to connect to
+fs.oss.accessKeyId: Aliyun access key ID
+fs.oss.accessKeySecret: Aliyun access key secret
+{% endhighlight %}
+
+### Hadoop-provided OSS file system - manual setup
+This setup is a bit more complex and we recommend using our shaded Hadoop file
systems instead (see above) unless required otherwise, e.g. for using OSS as
YARN’s resource storage dir via the fs.defaultFS configuration property in
Hadoop’s core-site.xml.
+
+ Set OSS FileSystem
+You need to point Flink to a valid Hadoop configuration, which contains the
following properties in core-site.xml:
+
+{% highlight xml %}
+
+
+
+fs.oss.impl
+org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem
+
+
+
+fs.oss.endpoint
+Aliyun OSS endpoint to connect to
+Aliyun OSS endpoint to connect to. An up-to-date list is
provided in the Aliyun OSS Documentation.
+
+
+
+fs.oss.accessKeyId
+Aliyun access key ID
+
+
+
+fs.oss.accessKeySecret
+Aliyun access key secret
+
+
+
+fs.oss.buffer.dir
+/tmp/oss
+
+
+
+
+
+{% endhighlight %}
+
+ Hadoop Configuration
+
+You can specify the [Hadoop configuration](../config.html#hdfs) in various
ways pointing Flink to
+the path of the Hadoop configuration directory, for example
+- by setting the environment variable `HADOOP_CONF_DIR`, or
+- by setting the `fs.hdfs.hadoopconf` configuration option in
`flink-conf.yaml`:
+{% highlight yaml %}
+fs.hdfs.hadoopconf: /path/to/etc/hadoop
+{% endhighlight %}
+
+This registers `/path/to/etc/hadoop` as Hadoop's configuration directory with
Flink. Flink will look for the `core-site.xml` and `hdfs-site.xml` files in the
specified directory.
+
+ Provide OSS FileSystem Dependency
+
+You can find Hadoop OSS FileSystem are packaged in the hadoop-aliyun artifact.
This JAR and all its dependencies need to be added to Flink’s classpath, i.e.
the class path of both Job and TaskManagers.
+
+There are multiple ways of adding JARs to Flink’s class path, the easiest
being simply to drop the JARs in Flink’s lib folder. You
[jira] [Commented] (FLINK-10865) Implement Flink's own Aliyun OSS filesystem
[
https://issues.apache.org/jira/browse/FLINK-10865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699318#comment-16699318
]
ASF GitHub Bot commented on FLINK-10865:
StefanRRichter commented on a change in pull request #7123: [FLINK-10865] Add
Aliyun OSS file systems without Hadoop dependencies
URL: https://github.com/apache/flink/pull/7123#discussion_r236344254
##
File path:
flink-filesystems/flink-oss-fs-hadoop/src/test/java/org/apache/flink/fs/osshadoop/HadoopOSSFileSystemITCase.java
##
@@ -0,0 +1,113 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.fs.osshadoop;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.fs.FSDataInputStream;
+import org.apache.flink.core.fs.FSDataOutputStream;
+import org.apache.flink.core.fs.FileSystem;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.util.TestLogger;
+
+import org.junit.Assume;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import java.io.BufferedReader;
+import java.io.InputStreamReader;
+import java.io.OutputStreamWriter;
+import java.nio.charset.StandardCharsets;
+import java.util.UUID;
+
+import static junit.framework.TestCase.assertEquals;
+import static junit.framework.TestCase.assertTrue;
+
+/**
+ * Unit tests for the OSS file system support via AliyunOSSFileSystem.
+ * These tests do actually read from or write to OSS.
+ */
+public class HadoopOSSFileSystemITCase extends TestLogger {
+
+ private static final String ENDPOINT =
System.getenv("ARTIFACTS_OSS_ENDPOINT");
+ private static final String BUCKET =
System.getenv("ARTIFACTS_OSS_BUCKET");
+ private static final String TEST_DATA_DIR = "tests-" +
UUID.randomUUID();
+ private static final String ACCESS_KEY =
System.getenv("ARTIFACTS_OSS_ACCESS_KEY");
+ private static final String SECRET_KEY =
System.getenv("ARTIFACTS_OSS_SECRET_KEY");
+
+ @BeforeClass
+ public static void checkIfCredentialsArePresent() {
+ Assume.assumeTrue("Aliyun OSS endpoint not configured, skipping
test...", ENDPOINT != null);
+ Assume.assumeTrue("Aliyun OSS bucket not configured, skipping
test...", BUCKET != null);
+ Assume.assumeTrue("Aliyun OSS access key not configured,
skipping test...", ACCESS_KEY != null);
+ Assume.assumeTrue("Aliyun OSS secret key not configured,
skipping test...", SECRET_KEY != null);
+ }
+
+ @Test
+ public void testReadAndWrite() throws Exception {
Review comment:
I found that there are more than one file system IT case that does similar
things like testing a read-write cycle, listing directories, etc.
I would suggest at this point to introduce an `AbstractFileSystemITCase`
that serves as a base for all thoise tests and the tests only implement the
setup and path creation. This would help us to deduplicate code. We could also
separate tests for read/write and directory listing, because maybe not all file
systems support the concept of a directory.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Implement Flink's own Aliyun OSS filesystem
> ---
>
> Key: FLINK-10865
> URL: https://issues.apache.org/jira/browse/FLINK-10865
> Project: Flink
> Issue Type: New Feature
> Components: filesystem-connector
>Affects Versions: 1.6.2
>Reporter: wujinhu
>Priority: Major
> Labels: pull-request-available
>
> Aliyun OSS is widely used among China’s cloud users, and Hadoop supports
> Aliyun OSS since 2.9.1.
> Open this jira to wrap AliyunOSSFileSystem in flink(similar to s3 support),
> so that user can read from & write to OSS more easily in flink.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10865) Implement Flink's own Aliyun OSS filesystem
[ https://issues.apache.org/jira/browse/FLINK-10865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16689419#comment-16689419 ] ASF GitHub Bot commented on FLINK-10865: wujinhu opened a new pull request #7123: [FLINK-10865] Add Aliyun OSS file systems without Hadoop dependencies URL: https://github.com/apache/flink/pull/7123 ## What is the purpose of the change This PR will add an implementation of a file system that read from & write to Aliyun OSS so that users can use OSS with Flink without depending on Hadoop. In this way, users will find it is more easily to use OSS with Flink now. This implementation wraps **AliyunOSSFileSystem** and shade its dependencies. However, the wrapped jar is not in Flink's lib directory, users need to copy the jar which built in **opt** directory to **lib** directory. ## Brief change log - Adds **flink-filesystems/flink-oss-fs-hadoop** ## Verifying this change This implementation adds some tests to test instantiation and some reads & writes & lists operations(test communications with Aliyun OSS). However, in order to run these tests, someone need to have his own Aliyun access key id and access key secret. Then, set environment variables below： `export ARTIFACTS_OSS_ENDPOINT=` `export ARTIFACTS_OSS_BUCKET=` `export ARTIFACTS_OSS_ACCESS_KEY=` `export ARTIFACTS_OSS_SECRET_KEY=` These tests are skipped by default. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (**yes** / no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (**yes** / no / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (**yes** / no) - If yes, how is the feature documented? (not applicable / **docs** / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement Flink's own Aliyun OSS filesystem > --- > > Key: FLINK-10865 > URL: https://issues.apache.org/jira/browse/FLINK-10865 > Project: Flink > Issue Type: New Feature > Components: filesystem-connector >Affects Versions: 1.6.2 >Reporter: wujinhu >Priority: Major > Labels: pull-request-available > > Aliyun OSS is widely used among China’s cloud users, and Hadoop supports > Aliyun OSS since 2.9.1. > Open this jira to wrap AliyunOSSFileSystem in flink(similar to s3 support), > so that user can read from & write to OSS more easily in flink. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
