This is an automated email from the ASF dual-hosted git repository.
vinoth pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/asf-site by this push:
new cc07b93 Travis CI build asf-site
cc07b93 is described below
commit cc07b93e36e7f40174ebade447fd27bd4ed230b0
Author: CI <[email protected]>
AuthorDate: Sun Oct 18 09:28:55 2020 +0000
Travis CI build asf-site
---
content/assets/js/lunr/lunr-store.js | 12 ++-
content/cn/docs/cloud.html | 2 +
.../cn/docs/{cloud.html => ibm_cos_hoodie.html} | 111 ++++++++++++++------
content/docs/cloud.html | 2 +
content/docs/{cloud.html => ibm_cos_hoodie.html} | 112 +++++++++++++++------
content/sitemap.xml | 8 ++
6 files changed, 189 insertions(+), 58 deletions(-)
diff --git a/content/assets/js/lunr/lunr-store.js
b/content/assets/js/lunr/lunr-store.js
index 4b71d3c..bc1859c 100644
--- a/content/assets/js/lunr/lunr-store.js
+++ b/content/assets/js/lunr/lunr-store.js
@@ -969,6 +969,16 @@ var store = [{
"tags": [],
"url": "https://hudi.apache.org/docs/cos_hoodie.html";,
"teaser":"https://hudi.apache.org/assets/images/500x300.png"},{
+ "title": "IBM Cloud Object Storage Filesystem",
+ "excerpt":"In this page, we explain how to get your Hudi spark job to
store into IBM Cloud Object Storage. IBM COS configs There are two
configurations required for Hudi-IBM Cloud Object Storage compatibility: Adding
IBM COS Credentials for Hudi Adding required Jars to classpath IBM Cloud Object
Storage Credentials Simplest...","categories": [],
+ "tags": [],
+ "url": "https://hudi.apache.org/cn/docs/ibm_cos_hoodie.html";,
+ "teaser":"https://hudi.apache.org/assets/images/500x300.png"},{
+ "title": "IBM Cloud Object Storage Filesystem",
+ "excerpt":"In this page, we explain how to get your Hudi spark job to
store into IBM Cloud Object Storage. IBM COS configs There are two
configurations required for Hudi-IBM Cloud Object Storage compatibility: Adding
IBM COS Credentials for Hudi Adding required Jars to classpath IBM Cloud Object
Storage Credentials Simplest...","categories": [],
+ "tags": [],
+ "url": "https://hudi.apache.org/docs/ibm_cos_hoodie.html";,
+ "teaser":"https://hudi.apache.org/assets/images/500x300.png"},{
"title": "Quick-Start Guide",
"excerpt":"本指南通过使用spark-shell简要介绍了Hudi功能。使用Spark数据源，我们将通过代码段展示如何插入和更新Hudi的默认存储类型数据集：
写时复制。每次写操作之后，我们还将展示如何读取快照和增量数据。 设置spark-shell
Hudi适用于Spark-2.x版本。您可以按照此处的说明设置spark。 在提取的目录中，使用spark-shell运行Hudi：
bin/spark-shell --packages org.apache.hudi:hudi-spark-bundle:0.5.0-incubating
--conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer'
设置表名、基本路径和数据生成器来为本指南生成记录。 import org.apache.hudi.QuickstartUtils._ import
scala.collection.JavaConversions._ import org.apache.spark.sql.Sa [...]
"tags": [],
@@ -1075,7 +1085,7 @@ var store = [{
"url": "https://hudi.apache.org/docs/deployment.html";,
"teaser":"https://hudi.apache.org/assets/images/500x300.png"},{
"title": "云储存",
- "excerpt":"与云存储连接 无论使用RDD/WriteClient API还是数据源，以下信息都有助于配置对云存储的访问。
AWS S3 S3和Hudi协同工作所需的配置。 Google Cloud Storage GCS和Hudi协同工作所需的配置。
Alibaba Cloud OSS 阿里云和Hudi协同工作所需的配置。 Microsoft Azure Azure和Hudi协同工作所需的配置。
Tencent Cloud Object Storage COS和Hudi协同工作所需的配置。 ","categories": [],
+ "excerpt":"与云存储连接 无论使用RDD/WriteClient API还是数据源，以下信息都有助于配置对云存储的访问。
AWS S3 S3和Hudi协同工作所需的配置。 Google Cloud Storage GCS和Hudi协同工作所需的配置。
Alibaba Cloud OSS 阿里云和Hudi协同工作所需的配置。 Microsoft Azure Azure和Hudi协同工作所需的配置。
Tencent Cloud Object Storage COS和Hudi协同工作所需的配置。 IBM Cloud Object Storage
IBM Cloud Object Storage和Hudi协同工作所需的配置。 ","categories": [],
"tags": [],
"url": "https://hudi.apache.org/cn/docs/cloud.html";,
"teaser":"https://hudi.apache.org/assets/images/500x300.png"},{
diff --git a/content/cn/docs/cloud.html b/content/cn/docs/cloud.html
index d536b73..8b3306f 100644
--- a/content/cn/docs/cloud.html
+++ b/content/cn/docs/cloud.html
@@ -375,6 +375,8 @@ GCS和Hudi协同工作所需的配置。</li>
Azure和Hudi协同工作所需的配置。</li>
<li><a href="/cn/docs/cos_hoodie.html">Tencent Cloud Object Storage</a> <br
/>
COS和Hudi协同工作所需的配置。</li>
+ <li><a href="/cn/docs/ibm_cos_hoodie.html">IBM Cloud Object Storage</a> <br
/>
+IBM Cloud Object Storage和Hudi协同工作所需的配置。</li>
</ul>
</section>
diff --git a/content/cn/docs/cloud.html b/content/cn/docs/ibm_cos_hoodie.html
similarity index 62%
copy from content/cn/docs/cloud.html
copy to content/cn/docs/ibm_cos_hoodie.html
index d536b73..3a43832 100644
--- a/content/cn/docs/cloud.html
+++ b/content/cn/docs/ibm_cos_hoodie.html
@@ -3,23 +3,23 @@
<head>
<meta charset="utf-8">
-<!-- begin _includes/seo.html --><title>云储存 - Apache Hudi</title>
-<meta name="description" content="与云存储连接">
+<!-- begin _includes/seo.html --><title>IBM Cloud Object Storage Filesystem -
Apache Hudi</title>
+<meta name="description" content="In this page, we explain how to get your
Hudi spark job to store into IBM Cloud Object Storage.">
<meta property="og:type" content="article">
<meta property="og:locale" content="en_US">
<meta property="og:site_name" content="">
-<meta property="og:title" content="云储存">
-<meta property="og:url" content="https://hudi.apache.org/cn/docs/cloud.html";>
+<meta property="og:title" content="IBM Cloud Object Storage Filesystem">
+<meta property="og:url"
content="https://hudi.apache.org/cn/docs/ibm_cos_hoodie.html";>
- <meta property="og:description" content="与云存储连接">
+ <meta property="og:description" content="In this page, we explain how to get
your Hudi spark job to store into IBM Cloud Object Storage.">
- <meta property="article:modified_time" content="2019-06-16T21:59:57-04:00">
+ <meta property="article:modified_time" content="2020-10-01T17:38:24-04:00">
@@ -284,7 +284,7 @@
- <li><a href="/cn/docs/cloud.html" class="active">云储存</a></li>
+ <li><a href="/cn/docs/cloud.html" class="">云储存</a></li>
@@ -342,7 +342,7 @@
<div class="page__inner-wrap">
<header>
- <h1 id="page-title" class="page__title" itemprop="headline">云储存
+ <h1 id="page-title" class="page__title" itemprop="headline">IBM
Cloud Object Storage Filesystem
</h1>
<!-- Output author details if some exist. -->
@@ -351,30 +351,85 @@
<section class="page__content" itemprop="text">
- <aside class="sidebar__right sticky">
- <nav class="toc">
- <header><h4 class="nav__title"><i class="fas fa-file-alt"></i> IN
THIS PAGE</h4></header>
- <ul class="toc__menu">
- <li><a href="#与云存储连接">与云存储连接</a></li>
-</ul>
- </nav>
- </aside>
+ <style>
+ .page {
+ padding-right: 0 !important;
+ }
+ </style>
- <h2 id="与云存储连接">与云存储连接</h2>
+ <p>In this page, we explain how to get your Hudi spark job to store
into IBM Cloud Object Storage.</p>
+
+<h2 id="ibm-cos-configs">IBM COS configs</h2>
+
+<p>There are two configurations required for Hudi-IBM Cloud Object Storage
compatibility:</p>
+
+<ul>
+ <li>Adding IBM COS Credentials for Hudi</li>
+ <li>Adding required Jars to classpath</li>
+</ul>
+
+<h3 id="ibm-cloud-object-storage-credentials">IBM Cloud Object Storage
Credentials</h3>
+
+<p>Simplest way to use Hudi with IBM Cloud Object Storage, is to configure
your <code class="highlighter-rouge">SparkSession</code> or <code
class="highlighter-rouge">SparkContext</code> with IBM Cloud Object Storage
credentials using <a href="https://github.com/CODAIT/stocator";>Stocator</a>
storage connector for Spark. Hudi will automatically pick this up and talk to
IBM Cloud Object Storage.</p>
+
+<p>Alternatively, add the required configs in your core-site.xml from where
Hudi can fetch them. Replace the <code
class="highlighter-rouge">fs.defaultFS</code> with your IBM Cloud Object
Storage bucket name and Hudi should be able to read/write from the bucket.</p>
+
+<p>For example, using HMAC keys and service name <code
class="highlighter-rouge">myCOS</code>:</p>
+<div class="language-xml highlighter-rouge"><div class="highlight"><pre
class="highlight"><code> <span class="nt"><property></span>
+ <span class="nt"><name></span>fs.defaultFS<span
class="nt"></name></span>
+ <span class="nt"><value></span>cos://myBucket.myCOS<span
class="nt"></value></span>
+ <span class="nt"></property></span>
+
+ <span class="nt"><property></span>
+ <span class="nt"><name></span>fs.cos.flat.list<span
class="nt"></name></span>
+ <span class="nt"><value></span>true<span
class="nt"></value></span>
+ <span class="nt"></property></span>
+
+ <span class="nt"><property></span>
+ <span class="nt"><name></span>fs.stocator.scheme.list<span
class="nt"></name></span>
+ <span class="nt"><value></span>cos<span
class="nt"></value></span>
+ <span class="nt"></property></span>
+
+ <span class="nt"><property></span>
+ <span class="nt"><name></span>fs.cos.impl<span
class="nt"></name></span>
+ <span
class="nt"><value></span>com.ibm.stocator.fs.ObjectStoreFileSystem<span
class="nt"></value></span>
+ <span class="nt"></property></span>
+
+ <span class="nt"><property></span>
+ <span class="nt"><name></span>fs.stocator.cos.impl<span
class="nt"></name></span>
+ <span
class="nt"><value></span>com.ibm.stocator.fs.cos.COSAPIClient<span
class="nt"></value></span>
+ <span class="nt"></property></span>
+
+ <span class="nt"><property></span>
+ <span class="nt"><name></span>fs.stocator.cos.scheme<span
class="nt"></name></span>
+ <span class="nt"><value></span>cos<span
class="nt"></value></span>
+ <span class="nt"></property></span>
+
+ <span class="nt"><property></span>
+ <span class="nt"><name></span>fs.cos.myCos.access.key<span
class="nt"></name></span>
+ <span class="nt"><value></span>ACCESS KEY<span
class="nt"></value></span>
+ <span class="nt"></property></span>
+
+ <span class="nt"><property></span>
+ <span class="nt"><name></span>fs.cos.myCos.endpoint<span
class="nt"></name></span>
+ <span
class="nt"><value></span>http://s3-api.us-geo.objectstorage.softlayer.net<span
class="nt"></value></span>
+ <span class="nt"></property></span>
+
+ <span class="nt"><property></span>
+ <span class="nt"><name></span>fs.cos.myCos.secret.key<span
class="nt"></name></span>
+ <span class="nt"><value></span>SECRET KEY<span
class="nt"></value></span>
+ <span class="nt"></property></span>
+
+</code></pre></div></div>
+
+<p>For more options see Stocator <a
href="https://github.com/CODAIT/stocator/blob/master/README.md";>documentation</a>.</p>
+
+<h3 id="ibm-cloud-object-storage-libs">IBM Cloud Object Storage Libs</h3>
-<p>无论使用RDD/WriteClient API还是数据源，以下信息都有助于配置对云存储的访问。</p>
+<p>IBM Cloud Object Storage hadoop libraries to add to our classpath</p>
<ul>
- <li><a href="/cn/docs/s3_hoodie.html">AWS S3</a> <br />
-S3和Hudi协同工作所需的配置。</li>
- <li><a href="/cn/docs/gcs_hoodie.html">Google Cloud Storage</a> <br />
-GCS和Hudi协同工作所需的配置。</li>
- <li><a href="/cn/docs/oss_hoodie.html">Alibaba Cloud OSS</a> <br />
-阿里云和Hudi协同工作所需的配置。</li>
- <li><a href="/cn/docs/azure_hoodie.html">Microsoft Azure</a> <br />
-Azure和Hudi协同工作所需的配置。</li>
- <li><a href="/cn/docs/cos_hoodie.html">Tencent Cloud Object Storage</a> <br
/>
-COS和Hudi协同工作所需的配置。</li>
+ <li>com.ibm.stocator:stocator:1.1.3</li>
</ul>
</section>
diff --git a/content/docs/cloud.html b/content/docs/cloud.html
index 7e7e06f..b0a89e7 100644
--- a/content/docs/cloud.html
+++ b/content/docs/cloud.html
@@ -387,6 +387,8 @@ Configurations required for OSS and Hudi
co-operability.</li>
Configurations required for Azure and Hudi co-operability.</li>
<li><a href="/docs/cos_hoodie.html">Tencent Cloud Object Storage</a> <br />
Configurations required for COS and Hudi co-operability.</li>
+ <li><a href="/docs/ibm_cos_hoodie.html">IBM Cloud Object Storage</a> <br />
+ Configurations required for IBM Cloud Object Storage and Hudi
co-operability.</li>
</ul>
</section>
diff --git a/content/docs/cloud.html b/content/docs/ibm_cos_hoodie.html
similarity index 62%
copy from content/docs/cloud.html
copy to content/docs/ibm_cos_hoodie.html
index 7e7e06f..4d23156 100644
--- a/content/docs/cloud.html
+++ b/content/docs/ibm_cos_hoodie.html
@@ -3,23 +3,23 @@
<head>
<meta charset="utf-8">
-<!-- begin _includes/seo.html --><title>Cloud Storage - Apache Hudi</title>
-<meta name="description" content="Talking to Cloud Storage">
+<!-- begin _includes/seo.html --><title>IBM Cloud Object Storage Filesystem -
Apache Hudi</title>
+<meta name="description" content="In this page, we explain how to get your
Hudi spark job to store into IBM Cloud Object Storage.">
<meta property="og:type" content="article">
<meta property="og:locale" content="en_US">
<meta property="og:site_name" content="">
-<meta property="og:title" content="Cloud Storage">
-<meta property="og:url" content="https://hudi.apache.org/docs/cloud.html";>
+<meta property="og:title" content="IBM Cloud Object Storage Filesystem">
+<meta property="og:url"
content="https://hudi.apache.org/docs/ibm_cos_hoodie.html";>
- <meta property="og:description" content="Talking to Cloud Storage">
+ <meta property="og:description" content="In this page, we explain how to get
your Hudi spark job to store into IBM Cloud Object Storage.">
- <meta property="article:modified_time" content="2019-06-16T21:59:57-04:00">
+ <meta property="article:modified_time" content="2020-10-01T17:38:24-04:00">
@@ -284,7 +284,7 @@
- <li><a href="/docs/cloud.html" class="active">Cloud
Storage</a></li>
+ <li><a href="/docs/cloud.html" class="">Cloud Storage</a></li>
@@ -353,7 +353,7 @@
<div class="page__inner-wrap">
<header>
- <h1 id="page-title" class="page__title" itemprop="headline">Cloud
Storage
+ <h1 id="page-title" class="page__title" itemprop="headline">IBM
Cloud Object Storage Filesystem
</h1>
<!-- Output author details if some exist. -->
@@ -362,31 +362,85 @@
<section class="page__content" itemprop="text">
- <aside class="sidebar__right sticky">
- <nav class="toc">
- <header><h4 class="nav__title"><i class="fas fa-file-alt"></i> IN
THIS PAGE</h4></header>
- <ul class="toc__menu">
- <li><a href="#talking-to-cloud-storage">Talking to Cloud Storage</a></li>
-</ul>
- </nav>
- </aside>
+ <style>
+ .page {
+ padding-right: 0 !important;
+ }
+ </style>
- <h2 id="talking-to-cloud-storage">Talking to Cloud Storage</h2>
+ <p>In this page, we explain how to get your Hudi spark job to store
into IBM Cloud Object Storage.</p>
+
+<h2 id="ibm-cos-configs">IBM COS configs</h2>
+
+<p>There are two configurations required for Hudi-IBM Cloud Object Storage
compatibility:</p>
+
+<ul>
+ <li>Adding IBM COS Credentials for Hudi</li>
+ <li>Adding required Jars to classpath</li>
+</ul>
+
+<h3 id="ibm-cloud-object-storage-credentials">IBM Cloud Object Storage
Credentials</h3>
+
+<p>Simplest way to use Hudi with IBM Cloud Object Storage, is to configure
your <code class="highlighter-rouge">SparkSession</code> or <code
class="highlighter-rouge">SparkContext</code> with IBM Cloud Object Storage
credentials using <a href="https://github.com/CODAIT/stocator";>Stocator</a>
storage connector for Spark. Hudi will automatically pick this up and talk to
IBM Cloud Object Storage.</p>
+
+<p>Alternatively, add the required configs in your <code
class="highlighter-rouge">core-site.xml</code> from where Hudi can fetch them.
Replace the <code class="highlighter-rouge">fs.defaultFS</code> with your IBM
Cloud Object Storage bucket name and Hudi should be able to read/write from the
bucket.</p>
+
+<p>For example, using HMAC keys and service name <code
class="highlighter-rouge">myCOS</code>:</p>
+<div class="language-xml highlighter-rouge"><div class="highlight"><pre
class="highlight"><code> <span class="nt"><property></span>
+ <span class="nt"><name></span>fs.defaultFS<span
class="nt"></name></span>
+ <span class="nt"><value></span>cos://myBucket.myCOS<span
class="nt"></value></span>
+ <span class="nt"></property></span>
+
+ <span class="nt"><property></span>
+ <span class="nt"><name></span>fs.cos.flat.list<span
class="nt"></name></span>
+ <span class="nt"><value></span>true<span
class="nt"></value></span>
+ <span class="nt"></property></span>
+
+ <span class="nt"><property></span>
+ <span class="nt"><name></span>fs.stocator.scheme.list<span
class="nt"></name></span>
+ <span class="nt"><value></span>cos<span
class="nt"></value></span>
+ <span class="nt"></property></span>
+
+ <span class="nt"><property></span>
+ <span class="nt"><name></span>fs.cos.impl<span
class="nt"></name></span>
+ <span
class="nt"><value></span>com.ibm.stocator.fs.ObjectStoreFileSystem<span
class="nt"></value></span>
+ <span class="nt"></property></span>
+
+ <span class="nt"><property></span>
+ <span class="nt"><name></span>fs.stocator.cos.impl<span
class="nt"></name></span>
+ <span
class="nt"><value></span>com.ibm.stocator.fs.cos.COSAPIClient<span
class="nt"></value></span>
+ <span class="nt"></property></span>
+
+ <span class="nt"><property></span>
+ <span class="nt"><name></span>fs.stocator.cos.scheme<span
class="nt"></name></span>
+ <span class="nt"><value></span>cos<span
class="nt"></value></span>
+ <span class="nt"></property></span>
+
+ <span class="nt"><property></span>
+ <span class="nt"><name></span>fs.cos.myCos.access.key<span
class="nt"></name></span>
+ <span class="nt"><value></span>ACCESS KEY<span
class="nt"></value></span>
+ <span class="nt"></property></span>
+
+ <span class="nt"><property></span>
+ <span class="nt"><name></span>fs.cos.myCos.endpoint<span
class="nt"></name></span>
+ <span
class="nt"><value></span>http://s3-api.us-geo.objectstorage.softlayer.net<span
class="nt"></value></span>
+ <span class="nt"></property></span>
+
+ <span class="nt"><property></span>
+ <span class="nt"><name></span>fs.cos.myCos.secret.key<span
class="nt"></name></span>
+ <span class="nt"><value></span>SECRET KEY<span
class="nt"></value></span>
+ <span class="nt"></property></span>
+
+</code></pre></div></div>
+
+<p>For more options see Stocator <a
href="https://github.com/CODAIT/stocator/blob/master/README.md";>documentation</a>.</p>
+
+<h3 id="ibm-cloud-object-storage-libs">IBM Cloud Object Storage Libs</h3>
-<p>Immaterial of whether RDD/WriteClient APIs or Datasource is used, the
following information helps configure access
-to cloud stores.</p>
+<p>IBM Cloud Object Storage hadoop libraries to add to our classpath</p>
<ul>
- <li><a href="/docs/s3_hoodie">AWS S3</a> <br />
-Configurations required for S3 and Hudi co-operability.</li>
- <li><a href="/docs/gcs_hoodie">Google Cloud Storage</a> <br />
-Configurations required for GCS and Hudi co-operability.</li>
- <li><a href="/docs/oss_hoodie.html">Alibaba Cloud OSS</a> <br />
-Configurations required for OSS and Hudi co-operability.</li>
- <li><a href="/docs/azure_hoodie.html">Microsoft Azure</a> <br />
-Configurations required for Azure and Hudi co-operability.</li>
- <li><a href="/docs/cos_hoodie.html">Tencent Cloud Object Storage</a> <br />
- Configurations required for COS and Hudi co-operability.</li>
+ <li>com.ibm.stocator:stocator:1.1.3</li>
</ul>
</section>
diff --git a/content/sitemap.xml b/content/sitemap.xml
index 0d9935d..53b9400 100644
--- a/content/sitemap.xml
+++ b/content/sitemap.xml
@@ -777,6 +777,14 @@
<lastmod>2020-04-21T17:38:24-04:00</lastmod>
</url>
<url>
+<loc>https://hudi.apache.org/cn/docs/ibm_cos_hoodie.html</loc>
+<lastmod>2020-10-01T17:38:24-04:00</lastmod>
+</url>
+<url>
+<loc>https://hudi.apache.org/docs/ibm_cos_hoodie.html</loc>
+<lastmod>2020-10-01T17:38:24-04:00</lastmod>
+</url>
+<url>
<loc>https://hudi.apache.org/cn/docs/quick-start-guide.html</loc>
<lastmod>2019-12-30T14:59:57-05:00</lastmod>
</url>
![https://hudi.apache.org/assets/images/500x300.png"},{](https://hudi.apache.org/assets/images/500x300.png"},{){kind=link}