This is an automated email from the ASF dual-hosted git repository.
vinoth pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/incubator-hudi.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 97b3106 Refreshing site content
97b3106 is described below
commit 97b3106520c489612dd2187eb9ce4796d5f5c49f
Author: Vinoth Chandar <[email protected]>
AuthorDate: Sat Mar 9 13:18:07 2019 -0800
Refreshing site content
---
content/.gitignore | 1 -
content/404.html | 6 +-
content/admin_guide.html | 50 ++-
content/community.html | 11 +-
content/comparison.html | 15 +-
content/concepts.html | 8 +-
content/configurations.html | 489 ++++++++++++++-------
content/contributing.html | 8 +-
content/css/customstyles.css | 4 +-
content/css/theme-blue.css | 2 +-
content/feed.xml | 6 +-
content/gcs_hoodie.html | 16 +-
...ommit_duration.png => hudi_commit_duration.png} | Bin
.../{hoodie_intro_1.png => hudi_intro_1.png} | Bin
...ie_log_format_v2.png => hudi_log_format_v2.png} | Bin
...uery_perf_hive.png => hudi_query_perf_hive.png} | Bin
..._perf_presto.png => hudi_query_perf_presto.png} | Bin
...ry_perf_spark.png => hudi_query_perf_spark.png} | Bin
.../{hoodie_upsert_dag.png => hudi_upsert_dag.png} | Bin
...odie_upsert_perf1.png => hudi_upsert_perf1.png} | Bin
...odie_upsert_perf2.png => hudi_upsert_perf2.png} | Bin
content/implementation.html | 22 +-
content/incremental_processing.html | 36 +-
content/index.html | 10 +-
content/js/mydoc_scroll.html | 6 +-
content/migration_guide.html | 15 +-
content/news.html | 8 +-
content/news_archive.html | 6 +-
content/powered_by.html | 7 +-
content/privacy.html | 6 +-
content/quickstart.html | 26 +-
content/s3_hoodie.html | 19 +-
content/search.json | 40 +-
content/sql_queries.html | 8 +-
content/strata-talk.html | 6 +-
content/use_cases.html | 19 +-
36 files changed, 558 insertions(+), 292 deletions(-)
diff --git a/content/.gitignore b/content/.gitignore
deleted file mode 100644
index e43b0f9..0000000
--- a/content/.gitignore
+++ /dev/null
@@ -1 +0,0 @@
-.DS_Store
diff --git a/content/404.html b/content/404.html
index 9491810..fedef9b 100644
--- a/content/404.html
+++ b/content/404.html
@@ -4,7 +4,7 @@
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="">
-<meta name="keywords" content=" ">
+<meta name="keywords" content="">
<title>Page Not Found | Hudi</title>
<link rel="stylesheet" href="css/syntax.css">
@@ -149,6 +149,10 @@
<li><a
href="https://cwiki.apache.org/confluence/pages/viewrecentblogposts.action?key=HUDI";
target="_blank">Blog</a></li>
+
+ <li><a
href="https://projects.apache.org/project.html?incubator-hudi";
target="_blank">Team</a></li>
+
+
</ul>
</li>
diff --git a/content/admin_guide.html b/content/admin_guide.html
index 470a219..3625cee 100644
--- a/content/admin_guide.html
+++ b/content/admin_guide.html
@@ -4,7 +4,7 @@
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="This section offers an overview of tools
available to operate an ecosystem of Hudi datasets">
-<meta name="keywords" content=" admin">
+<meta name="keywords" content="hudi, administration, operation, devops">
<title>Admin Guide | Hudi</title>
<link rel="stylesheet" href="css/syntax.css">
@@ -149,6 +149,10 @@
<li><a
href="https://cwiki.apache.org/confluence/pages/viewrecentblogposts.action?key=HUDI";
target="_blank">Blog</a></li>
+
+ <li><a
href="https://projects.apache.org/project.html?incubator-hudi";
target="_blank">Team</a></li>
+
+
</ul>
</li>
@@ -355,11 +359,11 @@
<h2 id="admin-cli">Admin CLI</h2>
-<p>Once hoodie has been built via <code class="highlighter-rouge">mvn clean
install -DskipTests</code>, the shell can be fired by via <code
class="highlighter-rouge">cd hoodie-cli && ./hoodie-cli.sh</code>.
-A hoodie dataset resides on HDFS, in a location referred to as the
<strong>basePath</strong> and we would need this location in order to connect
to a Hoodie dataset.
-Hoodie library effectively manages this HDFS dataset internally, using .hoodie
subfolder to track all metadata</p>
+<p>Once hudi has been built, the shell can be fired by via <code
class="highlighter-rouge">cd hoodie-cli && ./hoodie-cli.sh</code>.
+A hudi dataset resides on DFS, in a location referred to as the
<strong>basePath</strong> and we would need this location in order to connect
to a Hudi dataset.
+Hudi library effectively manages this dataset internally, using .hoodie
subfolder to track all metadata</p>
-<p>To initialize a hoodie table, use the following command.</p>
+<p>To initialize a hudi table, use the following command.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>18/09/06 15:56:52
INFO annotation.AutowiredAnnotationBeanPostProcessor: JSR-330
'javax.inject.Inject' annotation found and supported for autowiring
============================================
@@ -380,7 +384,7 @@ hoodie->create --path /user/hive/warehouse/table1
--tableName hoodie_table_1
</code></pre>
</div>
-<p>To see the description of hoodie table, use the command:</p>
+<p>To see the description of hudi table, use the command:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>
hoodie:hoodie_table_1->desc
@@ -398,7 +402,7 @@ hoodie:hoodie_table_1->desc
</code></pre>
</div>
-<p>Following is a sample command to connect to a Hoodie dataset contains uber
trips.</p>
+<p>Following is a sample command to connect to a Hudi dataset contains uber
trips.</p>
<div class="highlighter-rouge"><pre
class="highlight"><code>hoodie:trips->connect --path /app/uber/trips
@@ -447,7 +451,7 @@ hoodie:trips->
<h4 id="inspecting-commits">Inspecting Commits</h4>
-<p>The task of upserting or inserting a batch of incoming records is known as
a <strong>commit</strong> in Hoodie. A commit provides basic atomicity
guarantees such that only commited data is available for querying.
+<p>The task of upserting or inserting a batch of incoming records is known as
a <strong>commit</strong> in Hudi. A commit provides basic atomicity guarantees
such that only commited data is available for querying.
Each commit has a monotonically increasing string/number called the
<strong>commit number</strong>. Typically, this is the time at which we started
the commit.</p>
<p>To view some basic information about the last 10 commits,</p>
@@ -464,7 +468,7 @@ hoodie:trips->
</code></pre>
</div>
-<p>At the start of each write, Hoodie also writes a .inflight commit to the
.hoodie folder. You can use the timestamp there to estimate how long the commit
has been inflight</p>
+<p>At the start of each write, Hudi also writes a .inflight commit to the
.hoodie folder. You can use the timestamp there to estimate how long the commit
has been inflight</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ hdfs dfs -ls
/app/uber/trips/.hoodie/*.inflight
-rw-r--r-- 3 vinoth supergroup 321984 2016-10-05 23:18
/app/uber/trips/.hoodie/20161005225920.inflight
@@ -522,7 +526,7 @@ order (See Concepts). The below commands allow users to
view the file-slices for
<h4 id="statistics">Statistics</h4>
-<p>Since Hoodie directly manages file sizes for HDFS dataset, it might be good
to get an overall picture</p>
+<p>Since Hudi directly manages file sizes for DFS dataset, it might be good to
get an overall picture</p>
<div class="highlighter-rouge"><pre
class="highlight"><code>hoodie:trips->stats filesizes --partitionPath
2016/09/01 --sortBy "95th" --desc true --limit 10
________________________________________________________________________________________________
@@ -534,7 +538,7 @@ order (See Concepts). The below commands allow users to
view the file-slices for
</code></pre>
</div>
-<p>In case of Hoodie write taking much longer, it might be good to see the
write amplification for any sudden increases</p>
+<p>In case of Hudi write taking much longer, it might be good to see the write
amplification for any sudden increases</p>
<div class="highlighter-rouge"><pre
class="highlight"><code>hoodie:trips->stats wa
__________________________________________________________________________
@@ -547,7 +551,7 @@ order (See Concepts). The below commands allow users to
view the file-slices for
<h4 id="archived-commits">Archived Commits</h4>
-<p>In order to limit the amount of growth of .commit files on HDFS, Hoodie
archives older .commit files (with due respect to the cleaner policy) into a
commits.archived file.
+<p>In order to limit the amount of growth of .commit files on DFS, Hudi
archives older .commit files (with due respect to the cleaner policy) into a
commits.archived file.
This is a sequence file that contains a mapping from commitNumber => json
with raw information about the commit (same that is nicely rolled up above).</p>
<h4 id="compactions">Compactions</h4>
@@ -692,7 +696,7 @@ No File renames needed to unschedule pending compaction.
Operation successful.</
<div class="highlighter-rouge"><pre class="highlight"><code>
##### Repair Compaction
-The above compaction unscheduling operations could sometimes fail partially
(e:g -> HDFS temporarily unavailable). With
+The above compaction unscheduling operations could sometimes fail partially
(e:g -> DFS temporarily unavailable). With
partial failures, the compaction operation could become inconsistent with the
state of file-slices. When you run
`compaction validate`, you can notice invalid compaction operations if there
is one. In these cases, the repair
command comes to the rescue, it will rearrange the file-slices so that there
is no loss and the file-slices are
@@ -710,7 +714,7 @@ Compaction successfully repaired
<h2 id="metrics">Metrics</h2>
-<p>Once the Hoodie Client is configured with the right datasetname and
environment for metrics, it produces the following graphite metrics, that aid
in debugging hoodie datasets</p>
+<p>Once the Hudi Client is configured with the right datasetname and
environment for metrics, it produces the following graphite metrics, that aid
in debugging hudi datasets</p>
<ul>
<li><strong>Commit Duration</strong> - This is amount of time it took to
successfully commit a batch of records</li>
@@ -722,29 +726,29 @@ Compaction successfully repaired
<p>These metrics can then be plotted on a standard tool like grafana. Below is
a sample commit duration chart.</p>
-<figure><img class="docimage" src="images/hoodie_commit_duration.png"
alt="hoodie_commit_duration.png" style="max-width: 1000px" /></figure>
+<figure><img class="docimage" src="images/hudi_commit_duration.png"
alt="hudi_commit_duration.png" style="max-width: 1000px" /></figure>
<h2 id="troubleshooting-failures">Troubleshooting Failures</h2>
-<p>Section below generally aids in debugging Hoodie failures. Off the bat, the
following metadata is added to every record to help triage issues easily using
standard Hadoop SQL engines (Hive/Presto/Spark)</p>
+<p>Section below generally aids in debugging Hudi failures. Off the bat, the
following metadata is added to every record to help triage issues easily using
standard Hadoop SQL engines (Hive/Presto/Spark)</p>
<ul>
- <li><strong>_hoodie_record_key</strong> - Treated as a primary key within
each HDFS partition, basis of all updates/inserts</li>
+ <li><strong>_hoodie_record_key</strong> - Treated as a primary key within
each DFS partition, basis of all updates/inserts</li>
<li><strong>_hoodie_commit_time</strong> - Last commit that touched this
record</li>
<li><strong>_hoodie_file_name</strong> - Actual file name containing the
record (super useful to triage duplicates)</li>
<li><strong>_hoodie_partition_path</strong> - Path from basePath that
identifies the partition containing this record</li>
</ul>
-<div class="bs-callout bs-callout-warning">Note that as of now, Hoodie assumes
the application passes in the same deterministic partitionpath for a given
recordKey. i.e the uniqueness of record key is only enforced within each
partition</div>
+<div class="bs-callout bs-callout-warning">Note that as of now, Hudi assumes
the application passes in the same deterministic partitionpath for a given
recordKey. i.e the uniqueness of record key is only enforced within each
partition</div>
<h4 id="missing-records">Missing records</h4>
<p>Please check if there were any write errors using the admin commands above,
during the window at which the record could have been written.
-If you do find errors, then the record was not actually written by Hoodie, but
handed back to the application to decide what to do with it.</p>
+If you do find errors, then the record was not actually written by Hudi, but
handed back to the application to decide what to do with it.</p>
<h4 id="duplicates">Duplicates</h4>
-<p>First of all, please confirm if you do indeed have duplicates
<strong>AFTER</strong> ensuring the query is accessing the Hoodie datasets <a
href="sql_queries.html">properly</a> .</p>
+<p>First of all, please confirm if you do indeed have duplicates
<strong>AFTER</strong> ensuring the query is accessing the Hudi datasets <a
href="sql_queries.html">properly</a> .</p>
<ul>
<li>If confirmed, please use the metadata fields above, to identify the
physical files & partition files containing the records .</li>
@@ -754,10 +758,10 @@ If you do find errors, then the record was not actually
written by Hoodie, but h
<h4 id="spark-failures">Spark failures</h4>
-<p>Typical upsert() DAG looks like below. Note that Hoodie client also caches
intermediate RDDs to intelligently profile workload and size files and spark
parallelism.
+<p>Typical upsert() DAG looks like below. Note that Hudi client also caches
intermediate RDDs to intelligently profile workload and size files and spark
parallelism.
Also Spark UI shows sortByKey twice due to the probe job also being shown,
nonetheless its just a single sort.</p>
-<figure><img class="docimage" src="images/hoodie_upsert_dag.png"
alt="hoodie_upsert_dag.png" style="max-width: 1000px" /></figure>
+<figure><img class="docimage" src="images/hudi_upsert_dag.png"
alt="hudi_upsert_dag.png" style="max-width: 1000px" /></figure>
<p>At a high level, there are two steps</p>
@@ -777,7 +781,7 @@ Also Spark UI shows sortByKey twice due to the probe job
also being shown, nonet
<li>Job 7 : Actual writing of data (update + insert + insert turned to
updates to maintain file size)</li>
</ul>
-<p>Depending on the exception source (Hoodie/Spark), the above knowledge of
the DAG can be used to pinpoint the actual issue. The most often encountered
failures result from YARN/HDFS temporary failures.
+<p>Depending on the exception source (Hudi/Spark), the above knowledge of the
DAG can be used to pinpoint the actual issue. The most often encountered
failures result from YARN/DFS temporary failures.
In the future, a more sophisticated debug/management UI would be added to the
project, that can help automate some of this debugging.</p>
diff --git a/content/community.html b/content/community.html
index 39488eb..34196f3 100644
--- a/content/community.html
+++ b/content/community.html
@@ -4,7 +4,7 @@
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="">
-<meta name="keywords" content=" usecases">
+<meta name="keywords" content="hudi, use cases, big data, apache">
<title>Community | Hudi</title>
<link rel="stylesheet" href="css/syntax.css">
@@ -149,6 +149,10 @@
<li><a
href="https://cwiki.apache.org/confluence/pages/viewrecentblogposts.action?key=HUDI";
target="_blank">Blog</a></li>
+
+ <li><a
href="https://projects.apache.org/project.html?incubator-hudi";
target="_blank">Team</a></li>
+
+
</ul>
</li>
@@ -355,7 +359,7 @@
<tbody>
<tr>
<td>For any general questions, user support, development discussions</td>
- <td>Dev Mailing list (<a
href="mailto:dev-subscribe@hudi.apache.org">Subscribe</a>,
<a
href="mailto:dev-unsubscribe@hudi.apache.or&
[...]
+ <td>Dev Mailing list (<a
href="mailto:dev-subscribe@hudi.apache.org">Subscribe</a>,
<a
href="mailto:dev-unsubscribe@hudi.apache.or&
[...]
</tr>
<tr>
<td>For reporting bugs or issues or discover known issues</td>
@@ -389,9 +393,10 @@ Apache Hudi follows the typical Apache vulnerability
handling <a href="https://a
<li>Ask (and/or) answer questions on our support channels listed above.</li>
<li>Review code or HIPs</li>
<li>Help improve documentation</li>
+ <li>Author blogs on our wiki</li>
<li>Testing; Improving out-of-box experience by reporting bugs</li>
<li>Share new ideas/directions to pursue or propose a new HIP</li>
- <li>Contributing code to the project</li>
+ <li>Contributing code to the project (<a
href="https://issues.apache.org/jira/issues/?jql=project+%3D+HUDI+AND+component+%3D+newbie";>newbie
JIRAs</a>)</li>
</ul>
<h4 id="code-contributions">Code Contributions</h4>
diff --git a/content/comparison.html b/content/comparison.html
index 34082e0..59bcf75 100644
--- a/content/comparison.html
+++ b/content/comparison.html
@@ -4,7 +4,7 @@
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="">
-<meta name="keywords" content=" usecases">
+<meta name="keywords" content="apache, hudi, kafka, kudu, hive, hbase, stream
processing">
<title>Comparison | Hudi</title>
<link rel="stylesheet" href="css/syntax.css">
@@ -149,6 +149,10 @@
<li><a
href="https://cwiki.apache.org/confluence/pages/viewrecentblogposts.action?key=HUDI";
target="_blank">Blog</a></li>
+
+ <li><a
href="https://projects.apache.org/project.html?incubator-hudi";
target="_blank">Team</a></li>
+
+
</ul>
</li>
@@ -341,7 +345,7 @@
- <p>Apache Hudi fills a big void for processing data on top of HDFS, and thus
mostly co-exists nicely with these technologies. However,
+ <p>Apache Hudi fills a big void for processing data on top of DFS, and thus
mostly co-exists nicely with these technologies. However,
it would be useful to understand how Hudi fits into the current big data
ecosystem, contrasting it with a few related systems
and bring out the different tradeoffs these systems have accepted in their
design.</p>
@@ -380,16 +384,15 @@ just for analytics. Finally, HBase does not support
incremental processing primi
<p>A popular question, we get is : “How does Hudi relate to stream processing
systems?”, which we will try to answer here. Simply put, Hudi can integrate with
batch (<code class="highlighter-rouge">copy-on-write storage</code>) and
streaming (<code class="highlighter-rouge">merge-on-read storage</code>) jobs
of today, to store the computed results in Hadoop. For Spark apps, this can
happen via direct
integration of Hudi library with Spark/Spark streaming DAGs. In case of
Non-Spark processing systems (eg: Flink, Hive), the processing can be done in
the respective systems
-and later sent into a Hudi table via a Kafka topic/HDFS intermediate file. In
more conceptual level, data processing
+and later sent into a Hudi table via a Kafka topic/DFS intermediate file. In
more conceptual level, data processing
pipelines just consist of three components : <code
class="highlighter-rouge">source</code>, <code
class="highlighter-rouge">processing</code>, <code
class="highlighter-rouge">sink</code>, with users ultimately running queries
against the sink to use the results of the pipeline.
-Hudi can act as either a source or sink, that stores data on HDFS.
Applicability of Hudi to a given stream processing pipeline ultimately boils
down to suitability
+Hudi can act as either a source or sink, that stores data on DFS.
Applicability of Hudi to a given stream processing pipeline ultimately boils
down to suitability
of Presto/SparkSQL/Hive for your queries.</p>
<p>More advanced use cases revolve around the concepts of <a
href="https://www.oreilly.com/ideas/ubers-case-for-incremental-processing-on-hadoop";>incremental
processing</a>, which effectively
uses Hudi even inside the <code class="highlighter-rouge">processing</code>
engine to speed up typical batch pipelines. For e.g: Hudi can be used as a
state store inside a processing DAG (similar
to how <a
href="https://ci.apache.org/projects/flink/flink-docs-release-1.2/ops/state_backends.html#the-rocksdbstatebackend";>rocksDB</a>
is used by Flink). This is an item on the roadmap
-and will eventually happen as a <a
href="https://github.com/uber/hoodie/issues/8";>Beam Runner</a></p>
-
+and will eventually happen as a <a
href="https://issues.apache.org/jira/browse/HUDI-60";>Beam Runner</a></p>
<div class="tags">
diff --git a/content/concepts.html b/content/concepts.html
index 7e85d32..22754c4 100644
--- a/content/concepts.html
+++ b/content/concepts.html
@@ -4,7 +4,7 @@
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="Here we introduce some basic concepts & give
a broad technical overview of Hudi">
-<meta name="keywords" content=" concepts">
+<meta name="keywords" content="hudi, design, storage, views, timeline">
<title>Concepts | Hudi</title>
<link rel="stylesheet" href="css/syntax.css">
@@ -149,6 +149,10 @@
<li><a
href="https://cwiki.apache.org/confluence/pages/viewrecentblogposts.action?key=HUDI";
target="_blank">Blog</a></li>
+
+ <li><a
href="https://projects.apache.org/project.html?incubator-hudi";
target="_blank">Team</a></li>
+
+
</ul>
</li>
@@ -343,7 +347,7 @@
- <p>Apache Hudi (pronounced “Hudi”) provides the following primitives over
datasets on HDFS</p>
+ <p>Apache Hudi (pronounced “Hudi”) provides the following primitives over
datasets on DFS</p>
<ul>
<li>Upsert (how do I change the dataset?)</li>
diff --git a/content/configurations.html b/content/configurations.html
index 5f1adb8..73f66c9 100644
--- a/content/configurations.html
+++ b/content/configurations.html
@@ -4,7 +4,7 @@
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="Here we list all possible configurations and
what they mean">
-<meta name="keywords" content=" configurations">
+<meta name="keywords" content="garbage collection, hudi, jvm, configs, tuning">
<title>Configurations | Hudi</title>
<link rel="stylesheet" href="css/syntax.css">
@@ -149,6 +149,10 @@
<li><a
href="https://cwiki.apache.org/confluence/pages/viewrecentblogposts.action?key=HUDI";
target="_blank">Blog</a></li>
+
+ <li><a
href="https://projects.apache.org/project.html?incubator-hudi";
target="_blank">Team</a></li>
+
+
</ul>
</li>
@@ -343,174 +347,360 @@
- <h3 id="configuration">Configuration</h3>
+ <p>This page covers the different ways of configuring your job to write/read
Hudi datasets.
+At a high level, you can control behaviour at few levels.</p>
+
+<ul>
+ <li><strong><a href="#spark-datasource">Spark Datasource
Configs</a></strong> : These configs control the Hudi Spark Datasource,
providing ability to define keys/partitioning, pick out the write operation,
specify how to merge records or choosing view type to read.</li>
+ <li><strong><a href="#writeclient-configs">WriteClient Configs</a></strong>
: Internally, the Hudi datasource uses a RDD based <code
class="highlighter-rouge">HoodieWriteClient</code> api to actually perform
writes to storage. These configs provide deep control over lower level aspects
like
+ file sizing, compression, parallelism, compaction, write schema, cleaning
etc. Although Hudi provides sane defaults, from time-time these configs may
need to be tweaked to optimize for specific workloads.</li>
+ <li><strong><a href="#PAYLOAD_CLASS_OPT_KEY">RecordPayload
Config</a></strong> : This is the lowest level of customization offered by
Hudi. Record payloads define how to produce new values to upsert based on
incoming new record and
+ stored old record. Hudi provides default implementations such as <code
class="highlighter-rouge">OverwriteWithLatestAvroPayload</code> which simply
update storage with the latest/last-written record.
+ This can be overridden to a custom class extending <code
class="highlighter-rouge">HoodieRecordPayload</code> class, on both datasource
and WriteClient levels.</li>
+</ul>
+
+<h3 id="talking-to-cloud-storage">Talking to Cloud Storage</h3>
+
+<p>Immaterial of whether RDD/WriteClient APIs or Datasource is used, the
following information helps configure access
+to cloud stores.</p>
+
+<ul>
+ <li><a href="s3_hoodie.html">AWS S3</a> <br />
+Configurations required for S3 and Hudi co-operability.</li>
+ <li><a href="gcs_hoodie.html">Google Cloud Storage</a> <br />
+Configurations required for GCS and Hudi co-operability.</li>
+</ul>
+
+<h3 id="spark-datasource">Spark Datasource Configs</h3>
+
+<p>Spark jobs using the datasource can be configured by passing the below
options into the <code class="highlighter-rouge">option(k,v)</code> method as
usual.
+The actual datasource level configs are listed below.</p>
+
+<h4 id="write-options">Write Options</h4>
+
+<p>Additionally, you can pass down any of the WriteClient level configs
directly using <code class="highlighter-rouge">options()</code> or <code
class="highlighter-rouge">option(k,v)</code> methods.</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>inputDF.write()
+.format("com.uber.hoodie")
+.options(clientOpts) // any of the Hudi client opts can be passed in as well
+.option(DataSourceWriteOptions.RECORDKEY_FIELD_OPT_KEY(), "_row_key")
+.option(DataSourceWriteOptions.PARTITIONPATH_FIELD_OPT_KEY(), "partition")
+.option(DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY(), "timestamp")
+.option(HoodieWriteConfig.TABLE_NAME, tableName)
+.mode(SaveMode.Append)
+.save(basePath);
+</code></pre>
+</div>
+
+<p>Options useful for writing datasets via <code
class="highlighter-rouge">write.format.option(...)</code></p>
+
+<ul>
+ <li><a href="#TABLE_NAME_OPT_KEY">TABLE_NAME_OPT_KEY</a><br />
+Property: <code
class="highlighter-rouge">hoodie.datasource.write.table.name</code>
[Required]<br />
+<span style="color:grey">Hive table name, to register the dataset
into.</span></li>
+ <li><a href="#OPERATION_OPT_KEY">OPERATION_OPT_KEY</a><br />
+Property: <code
class="highlighter-rouge">hoodie.datasource.write.operation</code>, Default:
<code class="highlighter-rouge">upsert</code><br />
+<span style="color:grey">whether to do upsert, insert or bulkinsert for the
write operation. Use <code class="highlighter-rouge">bulkinsert</code> to load
new data into a table, and there on use <code
class="highlighter-rouge">upsert</code>/<code
class="highlighter-rouge">insert</code>.
+bulk insert uses a disk based write path to scale to load large inputs without
need to cache it.</span></li>
+ <li><a href="#STORAGE_TYPE_OPT_KEY">STORAGE_TYPE_OPT_KEY</a><br />
+Property: <code
class="highlighter-rouge">hoodie.datasource.write.storage.type</code>, Default:
<code class="highlighter-rouge">COPY_ON_WRITE</code> <br />
+<span style="color:grey">The storage type for the underlying data, for this
write. This can’t change between writes.</span></li>
+ <li><a href="#PRECOMBINE_FIELD_OPT_KEY">PRECOMBINE_FIELD_OPT_KEY</a><br />
+Property: <code
class="highlighter-rouge">hoodie.datasource.write.precombine.field</code>,
Default: <code class="highlighter-rouge">ts</code> <br />
+<span style="color:grey">Field used in preCombining before actual write. When
two records have the same key value,
+we will pick the one with the largest value for the precombine field,
determined by Object.compareTo(..)</span></li>
+ <li><a href="#PAYLOAD_CLASS_OPT_KEY">PAYLOAD_CLASS_OPT_KEY</a><br />
+Property: <code
class="highlighter-rouge">hoodie.datasource.write.payload.class</code>,
Default: <code
class="highlighter-rouge">com.uber.hoodie.OverwriteWithLatestAvroPayload</code>
<br />
+<span style="color:grey">Payload class used. Override this, if you like to
roll your own merge logic, when upserting/inserting.
+This will render any value set for <code
class="highlighter-rouge">PRECOMBINE_FIELD_OPT_VAL</code>
in-effective</span></li>
+ <li><a href="#RECORDKEY_FIELD_OPT_KEY">RECORDKEY_FIELD_OPT_KEY</a><br />
+Property: <code
class="highlighter-rouge">hoodie.datasource.write.recordkey.field</code>,
Default: <code class="highlighter-rouge">uuid</code> <br />
+<span style="color:grey">Record key field. Value to be used as the <code
class="highlighter-rouge">recordKey</code> component of <code
class="highlighter-rouge">HoodieKey</code>. Actual value
+will be obtained by invoking .toString() on the field value. Nested fields can
be specified using
+the dot notation eg: <code class="highlighter-rouge">a.b.c</code></span></li>
+ <li><a
href="#PARTITIONPATH_FIELD_OPT_KEY">PARTITIONPATH_FIELD_OPT_KEY</a><br />
+Property: <code
class="highlighter-rouge">hoodie.datasource.write.partitionpath.field</code>,
Default: <code class="highlighter-rouge">partitionpath</code> <br />
+<span style="color:grey">Partition path field. Value to be used at the <code
class="highlighter-rouge">partitionPath</code> component of <code
class="highlighter-rouge">HoodieKey</code>.
+Actual value ontained by invoking .toString()</span></li>
+ <li><a href="#KEYGENERATOR_CLASS_OPT_KEY">KEYGENERATOR_CLASS_OPT_KEY</a><br
/>
+Property: <code
class="highlighter-rouge">hoodie.datasource.write.keygenerator.class</code>,
Default: <code
class="highlighter-rouge">com.uber.hoodie.SimpleKeyGenerator</code> <br />
+<span style="color:grey">Key generator class, that implements will extract the
key out of incoming <code class="highlighter-rouge">Row</code>
object</span></li>
+ <li><a
href="#COMMIT_METADATA_KEYPREFIX_OPT_KEY">COMMIT_METADATA_KEYPREFIX_OPT_KEY</a><br
/>
+Property: <code
class="highlighter-rouge">hoodie.datasource.write.commitmeta.key.prefix</code>,
Default: <code class="highlighter-rouge">_</code> <br />
+<span style="color:grey">Option keys beginning with this prefix, are
automatically added to the commit/deltacommit metadata.
+This is useful to store checkpointing information, in a consistent way with
the hudi timeline</span></li>
+ <li><a href="#INSERT_DROP_DUPS_OPT_KEY">INSERT_DROP_DUPS_OPT_KEY</a><br />
+Property: <code
class="highlighter-rouge">hoodie.datasource.write.insert.drop.duplicates</code>,
Default: <code class="highlighter-rouge">false</code> <br />
+<span style="color:grey">If set to true, filters out all duplicate records
from incoming dataframe, during insert operations. </span></li>
+ <li><a href="#HIVE_SYNC_ENABLED_OPT_KEY">HIVE_SYNC_ENABLED_OPT_KEY</a><br />
+Property: <code
class="highlighter-rouge">hoodie.datasource.hive_sync.enable</code>, Default:
<code class="highlighter-rouge">false</code> <br />
+<span style="color:grey">When set to true, register/sync the dataset to Apache
Hive metastore</span></li>
+ <li><a href="#HIVE_DATABASE_OPT_KEY">HIVE_DATABASE_OPT_KEY</a><br />
+Property: <code
class="highlighter-rouge">hoodie.datasource.hive_sync.database</code>, Default:
<code class="highlighter-rouge">default</code> <br />
+<span style="color:grey">database to sync to</span></li>
+ <li><a href="#HIVE_TABLE_OPT_KEY">HIVE_TABLE_OPT_KEY</a><br />
+Property: <code
class="highlighter-rouge">hoodie.datasource.hive_sync.table</code>, [Required]
<br />
+<span style="color:grey">table to sync to</span></li>
+ <li><a href="#HIVE_USER_OPT_KEY">HIVE_USER_OPT_KEY</a><br />
+Property: <code
class="highlighter-rouge">hoodie.datasource.hive_sync.username</code>, Default:
<code class="highlighter-rouge">hive</code> <br />
+<span style="color:grey">hive user name to use</span></li>
+ <li><a href="#HIVE_PASS_OPT_KEY">HIVE_PASS_OPT_KEY</a><br />
+Property: <code
class="highlighter-rouge">hoodie.datasource.hive_sync.password</code>, Default:
<code class="highlighter-rouge">hive</code> <br />
+<span style="color:grey">hive password to use</span></li>
+ <li><a href="#HIVE_URL_OPT_KEY">HIVE_URL_OPT_KEY</a><br />
+Property: <code
class="highlighter-rouge">hoodie.datasource.hive_sync.jdbcurl</code>, Default:
<code class="highlighter-rouge">jdbc:hive2://localhost:10000</code> <br />
+<span style="color:grey">Hive metastore url</span></li>
+ <li><a
href="#HIVE_PARTITION_FIELDS_OPT_KEY">HIVE_PARTITION_FIELDS_OPT_KEY</a><br />
+Property: <code
class="highlighter-rouge">hoodie.datasource.hive_sync.partition_fields</code>,
Default: ` ` <br />
+<span style="color:grey">field in the dataset to use for determining hive
partition columns.</span></li>
+ <li><a
href="#HIVE_PARTITION_EXTRACTOR_CLASS_OPT_KEY">HIVE_PARTITION_EXTRACTOR_CLASS_OPT_KEY</a><br
/>
+Property: <code
class="highlighter-rouge">hoodie.datasource.hive_sync.partition_extractor_class</code>,
Default: <code
class="highlighter-rouge">com.uber.hoodie.hive.SlashEncodedDayPartitionValueExtractor</code>
<br />
+<span style="color:grey">Class used to extract partition field values into
hive partition columns.</span></li>
+ <li><a
href="#HIVE_ASSUME_DATE_PARTITION_OPT_KEY">HIVE_ASSUME_DATE_PARTITION_OPT_KEY</a><br
/>
+Property: <code
class="highlighter-rouge">hoodie.datasource.hive_sync.assume_date_partitioning</code>,
Default: <code class="highlighter-rouge">false</code> <br />
+<span style="color:grey">Assume partitioning is yyyy/mm/dd</span></li>
+</ul>
+
+<h4 id="read-options">Read Options</h4>
+
+<p>Options useful for reading datasets via <code
class="highlighter-rouge">read.format.option(...)</code></p>
<ul>
- <li><a href="#HoodieWriteConfig">HoodieWriteConfig</a> <br />
-<span style="color:grey">Top Level Config which is passed in when
HoodieWriteClent is created.</span>
+ <li><a href="#VIEW_TYPE_OPT_KEY">VIEW_TYPE_OPT_KEY</a> <br />
+Property: <code class="highlighter-rouge">hoodie.datasource.view.type</code>,
Default: <code class="highlighter-rouge">read_optimized</code> <br />
+<span style="color:grey">Whether data needs to be read, in incremental mode
(new data since an instantTime)
+(or) Read Optimized mode (obtain latest view, based on columnar data)
+(or) Real time mode (obtain latest view, based on row & columnar
data)</span></li>
+ <li><a href="#BEGIN_INSTANTTIME_OPT_KEY">BEGIN_INSTANTTIME_OPT_KEY</a> <br
/>
+Property: <code
class="highlighter-rouge">hoodie.datasource.read.begin.instanttime</code>,
[Required in incremental mode] <br />
+<span style="color:grey">Instant time to start incrementally pulling data
from. The instanttime here need not
+necessarily correspond to an instant on the timeline. New data written with an
+ <code class="highlighter-rouge">instant_time > BEGIN_INSTANTTIME</code>
are fetched out. For e.g: ‘20170901080000’ will get
+ all new data written after Sep 1, 2017 08:00AM.</span></li>
+ <li><a href="#END_INSTANTTIME_OPT_KEY">END_INSTANTTIME_OPT_KEY</a> <br />
+Property: <code
class="highlighter-rouge">hoodie.datasource.read.end.instanttime</code>,
Default: latest instant (i.e fetches all new data since begin instant time) <br
/>
+<span style="color:grey"> Instant time to limit incrementally fetched data to.
New data written with an
+<code class="highlighter-rouge">instant_time <= END_INSTANTTIME</code> are
fetched out.</span></li>
+</ul>
+
+<h3 id="writeclient-configs">WriteClient Configs</h3>
+
+<p>Jobs programming directly against the RDD level apis can build a <code
class="highlighter-rouge">HoodieWriteConfig</code> object and pass it in to the
<code class="highlighter-rouge">HoodieWriteClient</code> constructor.
+HoodieWriteConfig can be built using a builder pattern as below.</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>HoodieWriteConfig
cfg = HoodieWriteConfig.newBuilder()
+ .withPath(basePath)
+ .forTable(tableName)
+ .withSchema(schemaStr)
+ .withProps(props) // pass raw k,v pairs from a property file.
+
.withCompactionConfig(HoodieCompactionConfig.newBuilder().withXXX(...).build())
+ .withIndexConfig(HoodieIndexConfig.newBuilder().withXXX(...).build())
+ ...
+ .build();
+</code></pre>
+</div>
+
+<p>Following subsections go over different aspects of write configs,
explaining most important configs with their property names, default values.</p>
+
+<ul>
+ <li><a href="#withPath">withPath</a> (hoodie_base_path)
+Property: <code class="highlighter-rouge">hoodie.base.path</code> [Required]
<br />
+<span style="color:grey">Base DFS path under which all the data partitions are
created. Always prefix it explicitly with the storage scheme (e.g hdfs://,
s3:// etc). Hudi stores all the main meta-data about commits, savepoints,
cleaning audit logs etc in .hoodie directory under the base directory.
</span></li>
+ <li><a href="#withSchema">withSchema</a> (schema_str) <br />
+Property: <code class="highlighter-rouge">hoodie.avro.schema</code>
[Required]<br />
+<span style="color:grey">This is the current reader avro schema for the
dataset. This is a string of the entire schema. HoodieWriteClient uses this
schema to pass on to implementations of HoodieRecordPayload to convert from the
source format to avro record. This is also used when re-writing records during
an update. </span></li>
+ <li><a href="#forTable">forTable</a> (table_name)<br />
+Property: <code class="highlighter-rouge">hoodie.table.name</code> [Required]
<br />
+ <span style="color:grey">Table name for the dataset, will be used for
registering with Hive. Needs to be same across runs.</span></li>
+ <li><a href="#withBulkInsertParallelism">withBulkInsertParallelism</a>
(bulk_insert_parallelism = 1500) <br />
+Property: <code
class="highlighter-rouge">hoodie.bulkinsert.shuffle.parallelism</code><br />
+<span style="color:grey">Bulk insert is meant to be used for large initial
imports and this parallelism determines the initial number of files in your
dataset. Tune this to achieve a desired optimal size during initial
import.</span></li>
+ <li><a href="#withParallelism">withParallelism</a>
(insert_shuffle_parallelism = 1500, upsert_shuffle_parallelism = 1500)<br />
+Property: <code
class="highlighter-rouge">hoodie.insert.shuffle.parallelism</code>, <code
class="highlighter-rouge">hoodie.upsert.shuffle.parallelism</code><br />
+<span style="color:grey">Once data has been initially imported, this
parallelism controls initial parallelism for reading input records. Ensure this
value is high enough say: 1 partition for 1 GB of input data</span></li>
+ <li><a href="#combineInput">combineInput</a> (on_insert = false,
on_update=true)<br />
+Property: <code class="highlighter-rouge">hoodie.combine.before.insert</code>,
<code class="highlighter-rouge">hoodie.combine.before.upsert</code><br />
+<span style="color:grey">Flag which first combines the input RDD and merges
multiple partial records into a single record before inserting or updating in
DFS</span></li>
+ <li><a href="#withWriteStatusStorageLevel">withWriteStatusStorageLevel</a>
(level = MEMORY_AND_DISK_SER)<br />
+Property: <code
class="highlighter-rouge">hoodie.write.status.storage.level</code><br />
+<span style="color:grey">HoodieWriteClient.insert and HoodieWriteClient.upsert
returns a persisted RDD[WriteStatus], this is because the Client can choose to
inspect the WriteStatus and choose and commit or not based on the failures.
This is a configuration for the storage level for this RDD </span></li>
+ <li><a href="#withAutoCommit">withAutoCommit</a> (autoCommit = true)<br />
+Property: <code class="highlighter-rouge">hoodie.auto.commit</code><br />
+<span style="color:grey">Should HoodieWriteClient autoCommit after insert and
upsert. The client can choose to turn off auto-commit and commit on a “defined
success condition”</span></li>
+ <li><a href="#withAssumeDatePartitioning">withAssumeDatePartitioning</a>
(assumeDatePartitioning = false)<br />
+Property: ` hoodie.assume.date.partitioning`<br />
+<span style="color:grey">Should HoodieWriteClient assume the data is
partitioned by dates, i.e three levels from base path. This is a stop-gap to
support tables created by versions < 0.3.1. Will be removed eventually
</span></li>
+ <li><a href="#withConsistencyCheckEnabled">withConsistencyCheckEnabled</a>
(enabled = false)<br />
+Property: <code
class="highlighter-rouge">hoodie.consistency.check.enabled</code><br />
+<span style="color:grey">Should HoodieWriteClient perform additional checks to
ensure written files’ are listable on the underlying filesystem/storage. Set
this to true, to workaround S3’s eventual consistency model and ensure all data
written as a part of a commit is faithfully available for queries. </span></li>
+</ul>
+
+<h4 id="index-configs">Index configs</h4>
+<p>Following configs control indexing behavior, which tags incoming records as
either inserts or updates to older records.</p>
+
+<ul>
+ <li><a href="#withIndexConfig">withIndexConfig</a> (HoodieIndexConfig) <br />
+ <span style="color:grey">This is pluggable to have a external index (HBase)
or use the default bloom filter stored in the Parquet files</span>
<ul>
- <li><a href="#withPath">withPath</a> (hoodie_base_path) <br />
- <span style="color:grey">Base HDFS path under which all the data partitions
are created. Hoodie stores all the main meta-data about commits, savepoints,
cleaning audit logs etc in .hoodie directory under the base directory.
</span></li>
- <li><a href="#withSchema">withSchema</a> (schema_str) <br />
- <span style="color:grey">This is the current reader avro schema for the
Hoodie Dataset. This is a string of the entire schema. HoodieWriteClient uses
this schema to pass on to implementations of HoodieRecordPayload to convert
from the source format to avro record. This is also used when re-writing
records during an update. </span></li>
- <li><a href="#withParallelism">withParallelism</a>
(insert_shuffle_parallelism = 200, upsert_shuffle_parallelism = 200) <br />
- <span style="color:grey">Insert DAG uses the insert_parallelism in every
shuffle. Upsert DAG uses the upsert_parallelism in every shuffle. Typical
workload is profiled and once a min parallelism is established, trade off
between latency and cluster usage optimizations this is tuned and have a
conservatively high number to optimize for latency and </span></li>
- <li><a href="#combineInput">combineInput</a> (on_insert = false,
on_update=true) <br />
- <span style="color:grey">Flag which first combines the input RDD and merges
multiple partial records into a single record before inserting or updating in
HDFS</span></li>
- <li><a
href="#withWriteStatusStorageLevel">withWriteStatusStorageLevel</a> (level =
MEMORY_AND_DISK_SER) <br />
- <span style="color:grey">HoodieWriteClient.insert and
HoodieWriteClient.upsert returns a persisted RDD[WriteStatus], this is because
the Client can choose to inspect the WriteStatus and choose and commit or not
based on the failures. This is a configuration for the storage level for this
RDD </span></li>
- <li><a href="#withAutoCommit">withAutoCommit</a> (autoCommit = true) <br
/>
- <span style="color:grey">Should HoodieWriteClient autoCommit after insert
and upsert. The client can choose to turn off auto-commit and commit on a
“defined success condition”</span></li>
- <li><a href="#withAssumeDatePartitioning">withAssumeDatePartitioning</a>
(assumeDatePartitioning = false) <br />
- <span style="color:grey">Should HoodieWriteClient assume the data is
partitioned by dates, i.e three levels from base path. This is a stop-gap to
support tables created by versions < 0.3.1. Will be removed eventually
</span></li>
- <li>
- <p><a
href="#withConsistencyCheckEnabled">withConsistencyCheckEnabled</a> (enabled =
false) <br />
- <span style="color:grey">Should HoodieWriteClient perform additional checks
to ensure written files’ are listable on the underlying filesystem/storage. Set
this to true, to workaround S3’s eventual consistency model and ensure all data
written as a part of a commit is faithfully available for queries. </span></p>
- </li>
- <li><a href="#withIndexConfig">withIndexConfig</a> (HoodieIndexConfig)
<br />
- <span style="color:grey">Hoodie uses a index to help find the FileID which
contains an incoming record key. This is pluggable to have a external index
(HBase) or use the default bloom filter stored in the Parquet files</span>
- <ul>
- <li><a href="#withIndexType">withIndexType</a> (indexType = BLOOM)
<br />
+ <li><a href="#withIndexType">withIndexType</a> (indexType = BLOOM) <br />
+ Property: <code class="highlighter-rouge">hoodie.index.type</code> <br />
<span style="color:grey">Type of index to use. Default is Bloom filter.
Possible options are [BLOOM | HBASE | INMEMORY]. Bloom filters removes the
dependency on a external system and is stored in the footer of the Parquet Data
Files</span></li>
- <li><a href="#bloomFilterNumEntries">bloomFilterNumEntries</a>
(60000) <br />
- <span style="color:grey">Only applies if index type is BLOOM. <br />This is
the number of entries to be stored in the bloom filter. We assume the
maxParquetFileSize is 128MB and averageRecordSize is 1024B and hence we approx
a total of 130K records in a file. The default (60000) is roughly half of this
approximation. <a href="https://github.com/uber/hoodie/issues/70";>#70</a>
tracks computing this dynamically. Warning: Setting this very low, will
generate a lot of false positives and in [...]
- <li><a href="#bloomFilterFPP">bloomFilterFPP</a> (0.000000001) <br />
+ <li><a href="#bloomFilterNumEntries">bloomFilterNumEntries</a>
(numEntries = 60000) <br />
+ Property: <code
class="highlighter-rouge">hoodie.index.bloom.num_entries</code> <br />
+ <span style="color:grey">Only applies if index type is BLOOM. <br />This is
the number of entries to be stored in the bloom filter. We assume the
maxParquetFileSize is 128MB and averageRecordSize is 1024B and hence we approx
a total of 130K records in a file. The default (60000) is roughly half of this
approximation. <a
href="https://issues.apache.org/jira/browse/HUDI-56";>HUDI-56</a> tracks
computing this dynamically. Warning: Setting this very low, will generate a lot
of false positiv [...]
+ <li><a href="#bloomFilterFPP">bloomFilterFPP</a> (fpp = 0.000000001) <br
/>
+ Property: <code class="highlighter-rouge">hoodie.index.bloom.fpp</code> <br
/>
<span style="color:grey">Only applies if index type is BLOOM. <br /> Error
rate allowed given the number of entries. This is used to calculate how many
bits should be assigned for the bloom filter and the number of hash functions.
This is usually set very low (default: 0.000000001), we like to tradeoff disk
space for lower false positives</span></li>
- <li><a href="#bloomIndexPruneByRanges">bloomIndexPruneByRanges</a>
(true) <br />
+ <li><a href="#bloomIndexPruneByRanges">bloomIndexPruneByRanges</a>
(pruneRanges = true) <br />
+ Property: <code
class="highlighter-rouge">hoodie.bloom.index.prune.by.ranges</code> <br />
<span style="color:grey">Only applies if index type is BLOOM. <br /> When
true, range information from files to leveraged speed up index lookups.
Particularly helpful, if the key has a monotonously increasing prefix, such as
timestamp.</span></li>
- <li><a href="#bloomIndexUseCaching">bloomIndexUseCaching</a> (true)
<br />
+ <li><a href="#bloomIndexUseCaching">bloomIndexUseCaching</a> (useCaching
= true) <br />
+ Property: <code
class="highlighter-rouge">hoodie.bloom.index.use.caching</code> <br />
<span style="color:grey">Only applies if index type is BLOOM. <br /> When
true, the input RDD will cached to speed up index lookup by reducing IO for
computing parallelism or affected partitions</span></li>
- <li><a href="#bloomIndexParallelism">bloomIndexParallelism</a> (0)
<br />
+ <li><a href="#bloomIndexParallelism">bloomIndexParallelism</a> (0) <br />
+ Property: <code
class="highlighter-rouge">hoodie.bloom.index.parallelism</code> <br />
<span style="color:grey">Only applies if index type is BLOOM. <br /> This is
the amount of parallelism for index lookup, which involves a Spark Shuffle. By
default, this is auto computed based on input workload
characteristics</span></li>
- <li><a href="#hbaseZkQuorum">hbaseZkQuorum</a> (zkString) <br />
+ <li><a href="#hbaseZkQuorum">hbaseZkQuorum</a> (zkString) [Required]<br
/>
+ Property: <code class="highlighter-rouge">hoodie.index.hbase.zkquorum</code>
<br />
<span style="color:grey">Only application if index type is HBASE. HBase ZK
Quorum url to connect to.</span></li>
- <li><a href="#hbaseZkPort">hbaseZkPort</a> (port) <br />
+ <li><a href="#hbaseZkPort">hbaseZkPort</a> (port) [Required]<br />
+ Property: <code class="highlighter-rouge">hoodie.index.hbase.zkport</code>
<br />
<span style="color:grey">Only application if index type is HBASE. HBase ZK
Quorum port to connect to.</span></li>
- <li><a href="#hbaseTableName">hbaseTableName</a> (tableName) <br />
- <span style="color:grey">Only application if index type is HBASE. HBase
Table name to use as the index. Hoodie stores the row_key and [partition_path,
fileID, commitTime] mapping in the table.</span></li>
- </ul>
- </li>
- <li><a href="#withStorageConfig">withStorageConfig</a>
(HoodieStorageConfig) <br />
- <span style="color:grey">Storage related configs</span>
- <ul>
- <li><a href="#limitFileSize">limitFileSize</a> (size = 120MB) <br />
- <span style="color:grey">Hoodie re-writes a single file during update
(copy_on_write) or a compaction (merge_on_read). This is fundamental unit of
parallelism. It is important that this is aligned with the underlying
filesystem block size. </span></li>
- <li><a href="#parquetBlockSize">parquetBlockSize</a> (rowgroupsize =
120MB) <br />
- <span style="color:grey">Parquet RowGroup size. Its better than this is
aligned with the file size, so that a single column within a file is stored
continuously on disk</span></li>
- <li><a href="#parquetPageSize">parquetPageSize</a> (pagesize = 1MB)
<br />
+ <li><a href="#hbaseTableName">hbaseTableName</a> (tableName)
[Required]<br />
+ Property: <code class="highlighter-rouge">hoodie.index.hbase.table</code>
<br />
+ <span style="color:grey">Only application if index type is HBASE. HBase
Table name to use as the index. Hudi stores the row_key and [partition_path,
fileID, commitTime] mapping in the table.</span></li>
+ </ul>
+ </li>
+</ul>
+
+<h4 id="storage-configs">Storage configs</h4>
+<p>Controls aspects around sizing parquet and log files.</p>
+
+<ul>
+ <li><a href="#withStorageConfig">withStorageConfig</a> (HoodieStorageConfig)
<br />
+ <ul>
+ <li><a href="#limitFileSize">limitFileSize</a> (size = 120MB) <br />
+ Property: <code
class="highlighter-rouge">hoodie.parquet.max.file.size</code> <br />
+ <span style="color:grey">Target size for parquet files produced by Hudi
write phases. For DFS, this needs to be aligned with the underlying filesystem
block size for optimal performance. </span></li>
+ <li><a href="#parquetBlockSize">parquetBlockSize</a> (rowgroupsize =
120MB) <br />
+ Property: <code class="highlighter-rouge">hoodie.parquet.block.size</code>
<br />
+ <span style="color:grey">Parquet RowGroup size. Its better this is same as
the file size, so that a single column within a file is stored continuously on
disk</span></li>
+ <li><a href="#parquetPageSize">parquetPageSize</a> (pagesize = 1MB) <br
/>
+ Property: <code class="highlighter-rouge">hoodie.parquet.page.size</code>
<br />
<span style="color:grey">Parquet page size. Page is the unit of read within
a parquet file. Within a block, pages are compressed seperately. </span></li>
- <li><a href="#logFileMaxSize">logFileMaxSize</a> (logFileSize = 1GB)
<br />
+ <li><a href="#parquetCompressionRatio">parquetCompressionRatio</a>
(parquetCompressionRatio = 0.1) <br />
+ Property: <code
class="highlighter-rouge">hoodie.parquet.compression.ratio</code> <br />
+ <span style="color:grey">Expected compression of parquet data used by Hudi,
when it tries to size new parquet files. Increase this value, if bulk_insert is
producing smaller than expected sized files</span></li>
+ <li><a href="#logFileMaxSize">logFileMaxSize</a> (logFileSize = 1GB) <br
/>
+ Property: <code class="highlighter-rouge">hoodie.logfile.max.size</code> <br
/>
<span style="color:grey">LogFile max size. This is the maximum size allowed
for a log file before it is rolled over to the next version. </span></li>
- <li><a href="#logFileDataBlockMaxSize">logFileDataBlockMaxSize</a>
(dataBlockSize = 256MB) <br />
+ <li><a href="#logFileDataBlockMaxSize">logFileDataBlockMaxSize</a>
(dataBlockSize = 256MB) <br />
+ Property: <code
class="highlighter-rouge">hoodie.logfile.data.block.max.size</code> <br />
<span style="color:grey">LogFile Data block max size. This is the maximum
size allowed for a single data block to be appended to a log file. This helps
to make sure the data appended to the log file is broken up into sizable blocks
to prevent from OOM errors. This size should be greater than the JVM memory.
</span></li>
- </ul>
- </li>
- <li><a href="#withCompactionConfig">withCompactionConfig</a>
(HoodieCompactionConfig) <br />
- <span style="color:grey">Cleaning and configurations related to compaction
techniques</span>
- <ul>
- <li><a href="#withCleanerPolicy">withCleanerPolicy</a> (policy =
KEEP_LATEST_COMMITS) <br />
- <span style="color:grey">Hoodie Cleaning policy. Hoodie will delete older
versions of parquet files to re-claim space. Any Query/Computation referring to
this version of the file will fail. It is good to make sure that the data is
retained for more than the maximum query execution time.</span></li>
- <li><a href="#retainCommits">retainCommits</a>
(no_of_commits_to_retain = 24) <br />
+ <li><a
href="#logFileToParquetCompressionRatio">logFileToParquetCompressionRatio</a>
(logFileToParquetCompressionRatio = 0.35) <br />
+ Property: <code
class="highlighter-rouge">hoodie.logfile.to.parquet.compression.ratio</code>
<br />
+ <span style="color:grey">Expected additional compression as records move
from log files to parquet. Used for merge_on_read storage to send inserts into
log files & control the size of compacted parquet file.</span></li>
+ </ul>
+ </li>
+</ul>
+
+<h4 id="compaction-configs">Compaction configs</h4>
+<p>Configs that control compaction (merging of log files onto a new parquet
base file), cleaning (reclamation of older/unused file groups).</p>
+
+<ul>
+ <li><a href="#withCompactionConfig">withCompactionConfig</a>
(HoodieCompactionConfig) <br />
+ <ul>
+ <li><a href="#withCleanerPolicy">withCleanerPolicy</a> (policy =
KEEP_LATEST_COMMITS) <br />
+ Property: <code class="highlighter-rouge">hoodie.cleaner.policy</code> <br />
+ <span style="color:grey"> Cleaning policy to be used. Hudi will delete older
versions of parquet files to re-claim space. Any Query/Computation referring to
this version of the file will fail. It is good to make sure that the data is
retained for more than the maximum query execution time.</span></li>
+ <li><a href="#retainCommits">retainCommits</a> (no_of_commits_to_retain
= 24) <br />
+ Property: <code
class="highlighter-rouge">hoodie.cleaner.commits.retained</code> <br />
<span style="color:grey">Number of commits to retain. So data will be
retained for num_of_commits * time_between_commits (scheduled). This also
directly translates into how much you can incrementally pull on this
dataset</span></li>
- <li><a href="#archiveCommitsWith">archiveCommitsWith</a> (minCommits
= 96, maxCommits = 128) <br />
- <span style="color:grey">Each commit is a small file in the .hoodie
directory. Since HDFS is not designed to handle multiple small files, hoodie
archives older commits into a sequential log. A commit is published atomically
by a rename of the commit file.</span></li>
- <li><a href="#compactionSmallFileSize">compactionSmallFileSize</a>
(size = 0) <br />
- <span style="color:grey">Small files can always happen because of the number
of insert records in a paritition in a batch. Hoodie has an option to
auto-resolve small files by masking inserts into this partition as updates to
existing small files. The size here is the minimum file size considered as a
“small file size”. This should be less < maxFileSize and setting it to 0,
turns off this feature. </span></li>
- <li><a href="#insertSplitSize">insertSplitSize</a> (size = 500000)
<br />
+ <li><a href="#archiveCommitsWith">archiveCommitsWith</a> (minCommits =
96, maxCommits = 128) <br />
+ Property: <code class="highlighter-rouge">hoodie.keep.min.commits</code>,
<code class="highlighter-rouge">hoodie.keep.max.commits</code> <br />
+ <span style="color:grey">Each commit is a small file in the <code
class="highlighter-rouge">.hoodie</code> directory. Since DFS typically does
not favor lots of small files, Hudi archives older commits into a sequential
log. A commit is published atomically by a rename of the commit
file.</span></li>
+ <li><a href="#compactionSmallFileSize">compactionSmallFileSize</a> (size
= 0) <br />
+ Property: <code
class="highlighter-rouge">hoodie.parquet.small.file.limit</code> <br />
+ <span style="color:grey">This should be less < maxFileSize and setting it
to 0, turns off this feature. Small files can always happen because of the
number of insert records in a partition in a batch. Hudi has an option to
auto-resolve small files by masking inserts into this partition as updates to
existing small files. The size here is the minimum file size considered as a
“small file size”.</span></li>
+ <li><a href="#insertSplitSize">insertSplitSize</a> (size = 500000) <br />
+ Property: <code
class="highlighter-rouge">hoodie.copyonwrite.insert.split.size</code> <br />
<span style="color:grey">Insert Write Parallelism. Number of inserts grouped
for a single partition. Writing out 100MB files, with atleast 1kb records,
means 100K records per file. Default is to overprovision to 500K. To improve
insert latency, tune this to match the number of records in a single file.
Setting this to a low number, will result in small files (particularly when
compactionSmallFileSize is 0)</span></li>
- <li><a href="#autoTuneInsertSplits">autoTuneInsertSplits</a> (true)
<br />
- <span style="color:grey">Should hoodie dynamically compute the
insertSplitSize based on the last 24 commit’s metadata. Turned off by default.
</span></li>
- <li><a href="#approxRecordSize">approxRecordSize</a> () <br />
- <span style="color:grey">The average record size. If specified, hoodie will
use this and not compute dynamically based on the last 24 commit’s metadata. No
value set as default. This is critical in computing the insert parallelism and
bin-packing inserts into small files. See above.</span></li>
- <li><a
href="#withCompactionLazyBlockReadEnabled">withCompactionLazyBlockReadEnabled</a>
(true) <br />
- <span style="color:grey">When a CompactedLogScanner merges all log files,
this config helps to choose whether the logblocks should be read lazily or not.
Choose true to use I/O intensive lazy block reading (low memory usage) or false
for Memory intensive immediate block read (high memory usage)</span></li>
- <li><a
href="#withMaxNumDeltaCommitsBeforeCompaction">withMaxNumDeltaCommitsBeforeCompaction</a>
(maxNumDeltaCommitsBeforeCompaction = 10) <br />
+ <li><a href="#autoTuneInsertSplits">autoTuneInsertSplits</a> (true) <br
/>
+ Property: <code
class="highlighter-rouge">hoodie.copyonwrite.insert.auto.split</code> <br />
+ <span style="color:grey">Should hudi dynamically compute the insertSplitSize
based on the last 24 commit’s metadata. Turned off by default. </span></li>
+ <li><a href="#approxRecordSize">approxRecordSize</a> () <br />
+ Property: <code
class="highlighter-rouge">hoodie.copyonwrite.record.size.estimate</code> <br />
+ <span style="color:grey">The average record size. If specified, hudi will
use this and not compute dynamically based on the last 24 commit’s metadata. No
value set as default. This is critical in computing the insert parallelism and
bin-packing inserts into small files. See above.</span></li>
+ <li><a href="#withInlineCompaction">withInlineCompaction</a>
(inlineCompaction = false) <br />
+ Property: <code class="highlighter-rouge">hoodie.compact.inline</code> <br />
+ <span style="color:grey">When set to true, compaction is triggered by the
ingestion itself, right after a commit/deltacommit action as part of
insert/upsert/bulk_insert</span></li>
+ <li><a
href="#withMaxNumDeltaCommitsBeforeCompaction">withMaxNumDeltaCommitsBeforeCompaction</a>
(maxNumDeltaCommitsBeforeCompaction = 10) <br />
+ Property: <code
class="highlighter-rouge">hoodie.compact.inline.max.delta.commits</code> <br />
<span style="color:grey">Number of max delta commits to keep before
triggering an inline compaction</span></li>
- <li><a
href="#withCompactionReverseLogReadEnabled">withCompactionReverseLogReadEnabled</a>
(false) <br />
+ <li><a
href="#withCompactionLazyBlockReadEnabled">withCompactionLazyBlockReadEnabled</a>
(true) <br />
+ Property: <code
class="highlighter-rouge">hoodie.compaction.lazy.block.read</code> <br />
+ <span style="color:grey">When a CompactedLogScanner merges all log files,
this config helps to choose whether the logblocks should be read lazily or not.
Choose true to use I/O intensive lazy block reading (low memory usage) or false
for Memory intensive immediate block read (high memory usage)</span></li>
+ <li><a
href="#withCompactionReverseLogReadEnabled">withCompactionReverseLogReadEnabled</a>
(false) <br />
+ Property: <code
class="highlighter-rouge">hoodie.compaction.reverse.log.read</code> <br />
<span style="color:grey">HoodieLogFormatReader reads a logfile in the
forward direction starting from pos=0 to pos=file_length. If this config is set
to true, the Reader reads the logfile in reverse direction, from
pos=file_length to pos=0</span></li>
- </ul>
- </li>
- <li><a href="#withMetricsConfig">withMetricsConfig</a>
(HoodieMetricsConfig) <br />
- <span style="color:grey">Hoodie publishes metrics on every commit, clean,
rollback etc.</span>
- <ul>
- <li><a href="#on">on</a> (true) <br />
+ <li><a href="#withCleanerParallelism">withCleanerParallelism</a>
(cleanerParallelism = 200) <br />
+ Property: <code class="highlighter-rouge">hoodie.cleaner.parallelism</code>
<br />
+ <span style="color:grey">Increase this if cleaning becomes slow.</span></li>
+ <li><a href="#withCompactionStrategy">withCompactionStrategy</a>
(compactionStrategy =
com.uber.hoodie.io.compact.strategy.LogFileSizeBasedCompactionStrategy) <br />
+ Property: <code class="highlighter-rouge">hoodie.compaction.strategy</code>
<br />
+ <span style="color:grey">Compaction strategy decides which file groups are
picked up for compaction during each compaction run. By default. Hudi picks the
log file with most accumulated unmerged data</span></li>
+ <li><a
href="#withTargetIOPerCompactionInMB">withTargetIOPerCompactionInMB</a>
(targetIOPerCompactionInMB = 500000) <br />
+ Property: <code class="highlighter-rouge">hoodie.compaction.target.io</code>
<br />
+ <span style="color:grey">Amount of MBs to spend during compaction run for
the LogFileSizeBasedCompactionStrategy. This value helps bound ingestion
latency while compaction is run inline mode.</span></li>
+ <li><a
href="#withTargetPartitionsPerDayBasedCompaction">withTargetPartitionsPerDayBasedCompaction</a>
(targetPartitionsPerCompaction = 10) <br />
+ Property: <code
class="highlighter-rouge">hoodie.compaction.daybased.target</code> <br />
+ <span style="color:grey">Used by
com.uber.hoodie.io.compact.strategy.DayBasedCompactionStrategy to denote the
number of latest partitions to compact during a compaction run.</span></li>
+ <li><a href="#payloadClassName">withPayloadClass</a> (payloadClassName =
com.uber.hoodie.common.model.HoodieAvroPayload) <br />
+ Property: <code
class="highlighter-rouge">hoodie.compaction.payload.class</code> <br />
+ <span style="color:grey">This needs to be same as class used during
insert/upserts. Just like writing, compaction also uses the record payload
class to merge records in the log against each other, merge again with the base
file and produce the final record to be written after compaction.</span></li>
+ </ul>
+ </li>
+</ul>
+
+<h4 id="metrics-configs">Metrics configs</h4>
+<p>Enables reporting of Hudi metrics to graphite.</p>
+
+<ul>
+ <li><a href="#withMetricsConfig">withMetricsConfig</a> (HoodieMetricsConfig)
<br />
+<span style="color:grey">Hudi publishes metrics on every commit, clean,
rollback etc.</span>
+ <ul>
+ <li><a href="#on">on</a> (metricsOn = true) <br />
+ Property: <code class="highlighter-rouge">hoodie.metrics.on</code> <br />
<span style="color:grey">Turn sending metrics on/off. on by
default.</span></li>
- <li><a href="#withReporterType">withReporterType</a> (GRAPHITE) <br
/>
+ <li><a href="#withReporterType">withReporterType</a> (reporterType =
GRAPHITE) <br />
+ Property: <code
class="highlighter-rouge">hoodie.metrics.reporter.type</code> <br />
<span style="color:grey">Type of metrics reporter. Graphite is the default
and the only value suppported.</span></li>
- <li><a href="#toGraphiteHost">toGraphiteHost</a> () <br />
+ <li><a href="#toGraphiteHost">toGraphiteHost</a> (host = localhost) <br
/>
+ Property: <code
class="highlighter-rouge">hoodie.metrics.graphite.host</code> <br />
<span style="color:grey">Graphite host to connect to</span></li>
- <li><a href="#onGraphitePort">onGraphitePort</a> () <br />
+ <li><a href="#onGraphitePort">onGraphitePort</a> (port = 4756) <br />
+ Property: <code
class="highlighter-rouge">hoodie.metrics.graphite.port</code> <br />
<span style="color:grey">Graphite port to connect to</span></li>
- <li><a href="#usePrefix">usePrefix</a> () <br />
- <span style="color:grey">Standard prefix for all metrics</span></li>
- </ul>
- </li>
- <li><a href="#withMemoryConfig">withMemoryConfig</a>
(HoodieMemoryConfig) <br />
- <span style="color:grey">Memory related configs</span>
- <ul>
- <li><a
href="#withMaxMemoryFractionPerPartitionMerge">withMaxMemoryFractionPerPartitionMerge</a>
(maxMemoryFractionPerPartitionMerge = 0.6) <br />
- <span style="color:grey">This fraction is multiplied with the user memory
fraction (1 - spark.memory.fraction) to get a final fraction of heap space to
use during merge </span></li>
- <li><a
href="#withMaxMemorySizePerCompactionInBytes">withMaxMemorySizePerCompactionInBytes</a>
(maxMemorySizePerCompactionInBytes = 1GB) <br />
- <span style="color:grey">HoodieCompactedLogScanner reads logblocks, converts
records to HoodieRecords and then merges these log blocks and records. At any
point, the number of entries in a log block can be less than or equal to the
number of entries in the corresponding parquet file. This can lead to OOM in
the Scanner. Hence, a spillable map helps alleviate the memory pressure. Use
this config to set the max allowable inMemory footprint of the spillable
map.</span></li>
- </ul>
- </li>
- <li>
- <p><a href="s3_hoodie.html">S3Configs</a> (Hoodie S3 Configs) <br />
- <span style="color:grey">Configurations required for S3 and Hoodie
co-operability.</span></p>
- </li>
- <li><a href="gcs_hoodie.html">GCSConfigs</a> (Hoodie GCS Configs) <br />
- <span style="color:grey">Configurations required for GCS and Hoodie
co-operability.</span></li>
+ <li><a href="#usePrefix">usePrefix</a> (prefix = “”) <br />
+ Property: <code
class="highlighter-rouge">hoodie.metrics.graphite.metric.prefix</code> <br />
+ <span style="color:grey">Standard prefix applied to all metrics. This helps
to add datacenter, environment information for e.g</span></li>
</ul>
</li>
- <li><a href="#datasource">Hoodie Datasource</a> <br />
-<span style="color:grey">Configs for datasource</span>
+</ul>
+
+<h4 id="memory-configs">Memory configs</h4>
+<p>Controls memory usage for compaction and merges, performed internally by
Hudi</p>
+
+<ul>
+ <li><a href="#withMemoryConfig">withMemoryConfig</a> (HoodieMemoryConfig)
<br />
+<span style="color:grey">Memory related configs</span>
<ul>
- <li><a href="#writeoptions">write options</a> (write.format.option(…))
<br />
- <span style="color:grey"> Options useful for writing datasets </span>
- <ul>
- <li><a href="#OPERATION_OPT_KEY">OPERATION_OPT_KEY</a> (Default:
upsert) <br />
- <span style="color:grey">whether to do upsert, insert or bulkinsert for the
write operation</span></li>
- <li><a href="#STORAGE_TYPE_OPT_KEY">STORAGE_TYPE_OPT_KEY</a>
(Default: COPY_ON_WRITE) <br />
- <span style="color:grey">The storage type for the underlying data, for this
write. This can’t change between writes.</span></li>
- <li><a href="#TABLE_NAME_OPT_KEY">TABLE_NAME_OPT_KEY</a> (Default:
None (mandatory)) <br />
- <span style="color:grey">Hive table name, to register the dataset
into.</span></li>
- <li><a href="#PRECOMBINE_FIELD_OPT_KEY">PRECOMBINE_FIELD_OPT_KEY</a>
(Default: ts) <br />
- <span style="color:grey">Field used in preCombining before actual write.
When two records have the same key value,
- we will pick the one with the largest value for the precombine field,
determined by Object.compareTo(..)</span></li>
- <li><a href="#PAYLOAD_CLASS_OPT_KEY">PAYLOAD_CLASS_OPT_KEY</a>
(Default: com.uber.hoodie.OverwriteWithLatestAvroPayload) <br />
- <span style="color:grey">Payload class used. Override this, if you like to
roll your own merge logic, when upserting/inserting.
- This will render any value set for <code
class="highlighter-rouge">PRECOMBINE_FIELD_OPT_VAL</code>
in-effective</span></li>
- <li><a href="#RECORDKEY_FIELD_OPT_KEY">RECORDKEY_FIELD_OPT_KEY</a>
(Default: uuid) <br />
- <span style="color:grey">Record key field. Value to be used as the <code
class="highlighter-rouge">recordKey</code> component of <code
class="highlighter-rouge">HoodieKey</code>. Actual value
- will be obtained by invoking .toString() on the field value. Nested fields
can be specified using
- the dot notation eg: <code class="highlighter-rouge">a.b.c</code></span></li>
- <li><a
href="#PARTITIONPATH_FIELD_OPT_KEY">PARTITIONPATH_FIELD_OPT_KEY</a> (Default:
partitionpath) <br />
- <span style="color:grey">Partition path field. Value to be used at the <code
class="highlighter-rouge">partitionPath</code> component of <code
class="highlighter-rouge">HoodieKey</code>.
- Actual value ontained by invoking .toString()</span></li>
- <li><a
href="#KEYGENERATOR_CLASS_OPT_KEY">KEYGENERATOR_CLASS_OPT_KEY</a> (Default:
com.uber.hoodie.SimpleKeyGenerator) <br />
- <span style="color:grey">Key generator class, that implements will extract
the key out of incoming <code class="highlighter-rouge">Row</code>
object</span></li>
- <li><a
href="#COMMIT_METADATA_KEYPREFIX_OPT_KEY">COMMIT_METADATA_KEYPREFIX_OPT_KEY</a>
(Default: <code class="highlighter-rouge">_</code>) <br />
- <span style="color:grey">Option keys beginning with this prefix, are
automatically added to the commit/deltacommit metadata.
- This is useful to store checkpointing information, in a consistent way with
the hoodie timeline</span></li>
- </ul>
- </li>
- <li><a href="#readoptions">read options</a> (read.format.option(…)) <br
/>
- <span style="color:grey">Options useful for reading datasets</span>
- <ul>
- <li><a href="#VIEW_TYPE_OPT_KEY">VIEW_TYPE_OPT_KEY</a> (Default: =
read_optimized) <br />
- <span style="color:grey">Whether data needs to be read, in incremental mode
(new data since an instantTime)
- (or) Read Optimized mode (obtain latest view, based on columnar data)
- (or) Real time mode (obtain latest view, based on row & columnar
data)</span></li>
- <li><a
href="#BEGIN_INSTANTTIME_OPT_KEY">BEGIN_INSTANTTIME_OPT_KEY</a> (Default: None
(Mandatory in incremental mode)) <br />
- <span style="color:grey">Instant time to start incrementally pulling data
from. The instanttime here need not
- necessarily correspond to an instant on the timeline. New data written with
an
- <code class="highlighter-rouge">instant_time > BEGIN_INSTANTTIME</code>
are fetched out. For e.g: ‘20170901080000’ will get
- all new data written after Sep 1, 2017 08:00AM.</span></li>
- <li><a href="#END_INSTANTTIME_OPT_KEY">END_INSTANTTIME_OPT_KEY</a>
(Default: latest instant (i.e fetches all new data since begin instant time))
<br />
- <span style="color:grey"> Instant time to limit incrementally fetched data
to. New data written with an
- <code class="highlighter-rouge">instant_time <= END_INSTANTTIME</code>
are fetched out.</span></li>
- </ul>
- </li>
+ <li><a
href="#withMaxMemoryFractionPerPartitionMerge">withMaxMemoryFractionPerPartitionMerge</a>
(maxMemoryFractionPerPartitionMerge = 0.6) <br />
+ Property: <code
class="highlighter-rouge">hoodie.memory.merge.fraction</code> <br />
+ <span style="color:grey">This fraction is multiplied with the user memory
fraction (1 - spark.memory.fraction) to get a final fraction of heap space to
use during merge </span></li>
+ <li><a
href="#withMaxMemorySizePerCompactionInBytes">withMaxMemorySizePerCompactionInBytes</a>
(maxMemorySizePerCompactionInBytes = 1GB) <br />
+ Property: <code
class="highlighter-rouge">hoodie.memory.compaction.fraction</code> <br />
+ <span style="color:grey">HoodieCompactedLogScanner reads logblocks, converts
records to HoodieRecords and then merges these log blocks and records. At any
point, the number of entries in a log block can be less than or equal to the
number of entries in the corresponding parquet file. This can lead to OOM in
the Scanner. Hence, a spillable map helps alleviate the memory pressure. Use
this config to set the max allowable inMemory footprint of the spillable
map.</span></li>
</ul>
</li>
</ul>
@@ -519,14 +709,11 @@
<p>Writing data via Hudi happens as a Spark job and thus general rules of
spark debugging applies here too. Below is a list of things to keep in mind, if
you are looking to improving performance or reliability.</p>
-<p><strong>Write operations</strong> : Use <code
class="highlighter-rouge">bulkinsert</code> to load new data into a table, and
there on use <code class="highlighter-rouge">upsert</code>/<code
class="highlighter-rouge">insert</code>.
- Difference between them is that bulk insert uses a disk based write path to
scale to load large inputs without need to cache it.</p>
-
-<p><strong>Input Parallelism</strong> : By default, Hoodie tends to
over-partition input (i.e <code
class="highlighter-rouge">withParallelism(1500)</code>), to ensure each Spark
partition stays within the 2GB limit for inputs upto 500GB. Bump this up
accordingly if you have larger inputs. We recommend having shuffle parallelism
<code
class="highlighter-rouge">hoodie.[insert|upsert|bulkinsert].shuffle.parallelism</code>
such that its atleast input_data_size/500MB</p>
+<p><strong>Input Parallelism</strong> : By default, Hudi tends to
over-partition input (i.e <code
class="highlighter-rouge">withParallelism(1500)</code>), to ensure each Spark
partition stays within the 2GB limit for inputs upto 500GB. Bump this up
accordingly if you have larger inputs. We recommend having shuffle parallelism
<code
class="highlighter-rouge">hoodie.[insert|upsert|bulkinsert].shuffle.parallelism</code>
such that its atleast input_data_size/500MB</p>
-<p><strong>Off-heap memory</strong> : Hoodie writes parquet files and that
needs good amount of off-heap memory proportional to schema width. Consider
setting something like <code
class="highlighter-rouge">spark.yarn.executor.memoryOverhead</code> or <code
class="highlighter-rouge">spark.yarn.driver.memoryOverhead</code>, if you are
running into such failures.</p>
+<p><strong>Off-heap memory</strong> : Hudi writes parquet files and that needs
good amount of off-heap memory proportional to schema width. Consider setting
something like <code
class="highlighter-rouge">spark.yarn.executor.memoryOverhead</code> or <code
class="highlighter-rouge">spark.yarn.driver.memoryOverhead</code>, if you are
running into such failures.</p>
-<p><strong>Spark Memory</strong> : Typically, hoodie needs to be able to read
a single file into memory to perform merges or compactions and thus the
executor memory should be sufficient to accomodate this. In addition, Hoodie
caches the input to be able to intelligently place data and thus leaving some
<code class="highlighter-rouge">spark.storage.memoryFraction</code> will
generally help boost performance.</p>
+<p><strong>Spark Memory</strong> : Typically, hudi needs to be able to read a
single file into memory to perform merges or compactions and thus the executor
memory should be sufficient to accomodate this. In addition, Hoodie caches the
input to be able to intelligently place data and thus leaving some <code
class="highlighter-rouge">spark.storage.memoryFraction</code> will generally
help boost performance.</p>
<p><strong>Sizing files</strong> : Set <code
class="highlighter-rouge">limitFileSize</code> above judiciously, to balance
ingest/write latency vs number of files & consequently metadata overhead
associated with it.</p>
diff --git a/content/contributing.html b/content/contributing.html
index 1901952..9c9e61d 100644
--- a/content/contributing.html
+++ b/content/contributing.html
@@ -4,7 +4,7 @@
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="">
-<meta name="keywords" content=" developer setup">
+<meta name="keywords" content="hudi, ide, developer, setup">
<title>Developer Setup | Hudi</title>
<link rel="stylesheet" href="css/syntax.css">
@@ -149,6 +149,10 @@
<li><a
href="https://cwiki.apache.org/confluence/pages/viewrecentblogposts.action?key=HUDI";
target="_blank">Blog</a></li>
+
+ <li><a
href="https://projects.apache.org/project.html?incubator-hudi";
target="_blank">Team</a></li>
+
+
</ul>
</li>
@@ -380,6 +384,8 @@ have an open source license <a
href="https://www.apache.org/legal/resolved.html#
<li>Add adequate tests for your new functionality</li>
<li>[Optional] For involved changes, its best to also run the entire
integration test suite using <code class="highlighter-rouge">mvn clean
install</code></li>
<li>For website changes, please build the site locally & test
navigation, formatting & links thoroughly</li>
+ <li>If your code change changes some aspect of documentation (e.g new
config, default value change),
+please ensure there is a another PR to <a
href="https://github.com/apache/incubator-hudi/blob/asf-site/docs/README.md";>update
the docs</a> as well.</li>
</ul>
</li>
<li>Format commit messages and the pull request title like <code
class="highlighter-rouge">[HUDI-XXX] Fixes bug in Spark Datasource</code>,
diff --git a/content/css/customstyles.css b/content/css/customstyles.css
index d6667a5..56dcdba 100644
--- a/content/css/customstyles.css
+++ b/content/css/customstyles.css
@@ -1,5 +1,5 @@
body {
- font-size:15px;
+ font-size:14px;
}
.bs-callout {
@@ -607,7 +607,7 @@ a.fa.fa-envelope-o.mailto {
font-weight: 600;
}
-h3 {color: #ED1951; font-weight:normal; font-size:130%;}
+h3 {color: #545253; font-weight:normal; font-size:130%;}
h4 {color: #808080; font-weight:normal; font-size:120%; font-style:italic;}
.alert, .callout {
diff --git a/content/css/theme-blue.css b/content/css/theme-blue.css
index 9a923ef..46fbd0d 100644
--- a/content/css/theme-blue.css
+++ b/content/css/theme-blue.css
@@ -5,7 +5,7 @@
}
-h3 {color: #ED1951; }
+h3 {color: #545253; }
h4 {color: #808080; }
.nav-tabs > li.active > a, .nav-tabs > li.active > a:hover, .nav-tabs >
li.active > a:focus {
diff --git a/content/feed.xml b/content/feed.xml
index b21704e..cd76d50 100644
--- a/content/feed.xml
+++ b/content/feed.xml
@@ -5,8 +5,8 @@
<description>Apache Hudi (pronounced “Hoodie”) provides upserts and
incremental processing capaibilities on Big Data</description>
<link>http://0.0.0.0:4000/</link>
<atom:link href="http://0.0.0.0:4000/feed.xml"; rel="self"
type="application/rss+xml"/>
- <pubDate>Mon, 25 Feb 2019 20:49:33 +0000</pubDate>
- <lastBuildDate>Mon, 25 Feb 2019 20:49:33 +0000</lastBuildDate>
+ <pubDate>Sat, 09 Mar 2019 21:08:53 +0000</pubDate>
+ <lastBuildDate>Sat, 09 Mar 2019 21:08:53 +0000</lastBuildDate>
<generator>Jekyll v3.3.1</generator>
<item>
@@ -25,7 +25,7 @@
<item>
<title>Connect with us at Strata San Jose March 2017</title>
- <description><p>We will be presenting Hoodie &amp;
general concepts around how incremental processing works at Uber.
+ <description><p>We will be presenting Hudi &amp; general
concepts around how incremental processing works at Uber.
Catch our talk <strong>“Incremental Processing on Hadoop At
Uber”</strong></p>
</description>
diff --git a/content/gcs_hoodie.html b/content/gcs_hoodie.html
index f90992d..cb96011 100644
--- a/content/gcs_hoodie.html
+++ b/content/gcs_hoodie.html
@@ -4,8 +4,8 @@
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="In this page, we go over how to configure
hudi with Google Cloud Storage.">
-<meta name="keywords" content=" sql hive gcs spark presto">
-<title>GCS Filesystem (experimental) | Hudi</title>
+<meta name="keywords" content="hudi, hive, google cloud, storage, spark,
presto">
+<title>GCS Filesystem | Hudi</title>
<link rel="stylesheet" href="css/syntax.css">
@@ -149,6 +149,10 @@
<li><a
href="https://cwiki.apache.org/confluence/pages/viewrecentblogposts.action?key=HUDI";
target="_blank">Blog</a></li>
+
+ <li><a
href="https://projects.apache.org/project.html?incubator-hudi";
target="_blank">Team</a></li>
+
+
</ul>
</li>
@@ -158,7 +162,7 @@
- <a class="email" title="Submit feedback" href="#"
onclick="javascript:window.location='mailto:[email protected]?subject=Hudi
Documentation feedback&body=I have some feedback about the GCS Filesystem
(experimental) page: ' + window.location.href;"><i class="fa
fa-envelope-o"></i> Feedback</a>
+ <a class="email" title="Submit feedback" href="#"
onclick="javascript:window.location='mailto:[email protected]?subject=Hudi
Documentation feedback&body=I have some feedback about the GCS Filesystem page:
' + window.location.href;"><i class="fa fa-envelope-o"></i> Feedback</a>
<li>
@@ -176,7 +180,7 @@
searchInput:
document.getElementById('search-input'),
resultsContainer:
document.getElementById('results-container'),
dataSource: 'search.json',
- searchResultTemplate: '<li><a href="{url}"
title="GCS Filesystem (experimental)">{title}</a></li>',
+ searchResultTemplate: '<li><a href="{url}"
title="GCS Filesystem">{title}</a></li>',
noResultsText: 'No results found.',
limit: 10,
fuzzy: true,
@@ -327,7 +331,7 @@
<!-- Content Column -->
<div class="col-md-9">
<div class="post-header">
- <h1 class="post-title-main">GCS Filesystem (experimental)</h1>
+ <h1 class="post-title-main">GCS Filesystem</h1>
</div>
@@ -343,7 +347,7 @@
- <p>Hudi works with HDFS by default and GCS <strong>regional</strong> buckets
provide an HDFS API with strong consistency.</p>
+ <p>For Hudi storage on GCS, <strong>regional</strong> buckets provide an DFS
API with strong consistency.</p>
<h2 id="gcs-configs">GCS Configs</h2>
diff --git a/content/images/hoodie_commit_duration.png
b/content/images/hudi_commit_duration.png
similarity index 100%
rename from content/images/hoodie_commit_duration.png
rename to content/images/hudi_commit_duration.png
diff --git a/content/images/hoodie_intro_1.png b/content/images/hudi_intro_1.png
similarity index 100%
rename from content/images/hoodie_intro_1.png
rename to content/images/hudi_intro_1.png
diff --git a/content/images/hoodie_log_format_v2.png
b/content/images/hudi_log_format_v2.png
similarity index 100%
rename from content/images/hoodie_log_format_v2.png
rename to content/images/hudi_log_format_v2.png
diff --git a/content/images/hoodie_query_perf_hive.png
b/content/images/hudi_query_perf_hive.png
similarity index 100%
rename from content/images/hoodie_query_perf_hive.png
rename to content/images/hudi_query_perf_hive.png
diff --git a/content/images/hoodie_query_perf_presto.png
b/content/images/hudi_query_perf_presto.png
similarity index 100%
rename from content/images/hoodie_query_perf_presto.png
rename to content/images/hudi_query_perf_presto.png
diff --git a/content/images/hoodie_query_perf_spark.png
b/content/images/hudi_query_perf_spark.png
similarity index 100%
rename from content/images/hoodie_query_perf_spark.png
rename to content/images/hudi_query_perf_spark.png
diff --git a/content/images/hoodie_upsert_dag.png
b/content/images/hudi_upsert_dag.png
similarity index 100%
rename from content/images/hoodie_upsert_dag.png
rename to content/images/hudi_upsert_dag.png
diff --git a/content/images/hoodie_upsert_perf1.png
b/content/images/hudi_upsert_perf1.png
similarity index 100%
rename from content/images/hoodie_upsert_perf1.png
rename to content/images/hudi_upsert_perf1.png
diff --git a/content/images/hoodie_upsert_perf2.png
b/content/images/hudi_upsert_perf2.png
similarity index 100%
rename from content/images/hoodie_upsert_perf2.png
rename to content/images/hudi_upsert_perf2.png
diff --git a/content/implementation.html b/content/implementation.html
index d649a70..e524ec6 100644
--- a/content/implementation.html
+++ b/content/implementation.html
@@ -4,7 +4,7 @@
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="">
-<meta name="keywords" content=" implementation">
+<meta name="keywords" content="hudi, index, storage, compaction, cleaning,
implementation">
<title>Implementation | Hudi</title>
<link rel="stylesheet" href="css/syntax.css">
@@ -149,6 +149,10 @@
<li><a
href="https://cwiki.apache.org/confluence/pages/viewrecentblogposts.action?key=HUDI";
target="_blank">Blog</a></li>
+
+ <li><a
href="https://projects.apache.org/project.html?incubator-hudi";
target="_blank">Team</a></li>
+
+
</ul>
</li>
@@ -347,7 +351,7 @@ Hudi upsert/insert is merely a Spark DAG, that can be
broken into two big pieces
<ul>
<li>
- <p><strong>Indexing</strong> : A big part of Hoodie’s efficiency comes
from indexing the mapping from record keys to the file ids, to which they
belong to.
+ <p><strong>Indexing</strong> : A big part of Hudi’s efficiency comes from
indexing the mapping from record keys to the file ids, to which they belong to.
This index also helps the <code
class="highlighter-rouge">HoodieWriteClient</code> separate upserted records
into inserts and updates, so they can be treated differently.
<code class="highlighter-rouge">HoodieReadClient</code> supports operations
such as <code class="highlighter-rouge">filterExists</code> (used for
de-duplication of table) and an efficient batch <code
class="highlighter-rouge">read(keys)</code> api, that
can read out the records corresponding to the keys using the index much
quickly, than a typical scan via a query. The index is also atomically
@@ -406,7 +410,7 @@ Any remaining records after that, are again packed into new
file id groups, agai
<p>In the case of Copy-On-Write, a single parquet file constitutes one <code
class="highlighter-rouge">file slice</code> which contains one complete version
of
the file</p>
-<figure><img class="docimage" src="images/hoodie_log_format_v2.png"
alt="hoodie_log_format_v2.png" style="max-width: 1000px" /></figure>
+<figure><img class="docimage" src="images/hudi_log_format_v2.png"
alt="hudi_log_format_v2.png" style="max-width: 1000px" /></figure>
<h4 id="merge-on-read">Merge On Read</h4>
@@ -575,7 +579,7 @@ incremental ingestion (writer at DC6) happened before the
compaction (some time
The below description is with regards to compaction from file-group
perspective.
<ul>
<li><code class="highlighter-rouge">Reader querying at time between
ingestion completion time for DC6 and compaction finish “Tc”</code>:
-Hoodie’s implementation will be changed to become aware of file-groups
currently waiting for compaction and
+Hudi’s implementation will be changed to become aware of file-groups currently
waiting for compaction and
merge log-files corresponding to DC2-DC6 with the base-file corresponding to
SC1. In essence, Hudi will create
a pseudo file-slice by combining the 2 file-slices starting at base-commits
SC1 and SC5 to one.
For file-groups not waiting for compaction, the reader behavior is essentially
the same - read latest file-slice
@@ -602,12 +606,12 @@ the conventional alternatives for achieving these
tasks.</p>
<p>Following shows the speed up obtained for NoSQL ingestion, by switching
from bulk loads off HBase to Parquet to incrementally upserting
on a Hudi dataset, on 5 tables ranging from small to huge.</p>
-<figure><img class="docimage" src="images/hoodie_upsert_perf1.png"
alt="hoodie_upsert_perf1.png" style="max-width: 1000px" /></figure>
+<figure><img class="docimage" src="images/hudi_upsert_perf1.png"
alt="hudi_upsert_perf1.png" style="max-width: 1000px" /></figure>
<p>Given Hudi can build the dataset incrementally, it opens doors for also
scheduling ingesting more frequently thus reducing latency, with
significant savings on the overall compute cost.</p>
-<figure><img class="docimage" src="images/hoodie_upsert_perf2.png"
alt="hoodie_upsert_perf2.png" style="max-width: 1000px" /></figure>
+<figure><img class="docimage" src="images/hudi_upsert_perf2.png"
alt="hudi_upsert_perf2.png" style="max-width: 1000px" /></figure>
<p>Hudi upserts have been stress tested upto 4TB in a single commit across the
t1 table.</p>
@@ -618,15 +622,15 @@ with no impact on queries. Following charts compare the
Hudi vs non-Hudi dataset
<p><strong>Hive</strong></p>
-<figure><img class="docimage" src="images/hoodie_query_perf_hive.png"
alt="hoodie_query_perf_hive.png" style="max-width: 800px" /></figure>
+<figure><img class="docimage" src="images/hudi_query_perf_hive.png"
alt="hudi_query_perf_hive.png" style="max-width: 800px" /></figure>
<p><strong>Spark</strong></p>
-<figure><img class="docimage" src="images/hoodie_query_perf_spark.png"
alt="hoodie_query_perf_spark.png" style="max-width: 1000px" /></figure>
+<figure><img class="docimage" src="images/hudi_query_perf_spark.png"
alt="hudi_query_perf_spark.png" style="max-width: 1000px" /></figure>
<p><strong>Presto</strong></p>
-<figure><img class="docimage" src="images/hoodie_query_perf_presto.png"
alt="hoodie_query_perf_presto.png" style="max-width: 1000px" /></figure>
+<figure><img class="docimage" src="images/hudi_query_perf_presto.png"
alt="hudi_query_perf_presto.png" style="max-width: 1000px" /></figure>
diff --git a/content/incremental_processing.html
b/content/incremental_processing.html
index a694881..c487368 100644
--- a/content/incremental_processing.html
+++ b/content/incremental_processing.html
@@ -4,7 +4,7 @@
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="In this page, we will discuss some available
tools for ingesting data incrementally & consuming the changes.">
-<meta name="keywords" content=" incremental processing">
+<meta name="keywords" content="hudi, incremental, batch, stream, processing,
Hive, ETL, Spark SQL">
<title>Incremental Processing | Hudi</title>
<link rel="stylesheet" href="css/syntax.css">
@@ -149,6 +149,10 @@
<li><a
href="https://cwiki.apache.org/confluence/pages/viewrecentblogposts.action?key=HUDI";
target="_blank">Blog</a></li>
+
+ <li><a
href="https://projects.apache.org/project.html?incubator-hudi";
target="_blank">Team</a></li>
+
+
</ul>
</li>
@@ -349,7 +353,7 @@ discusses a few tools that can be used to achieve these on
different contexts.</
<h2 id="incremental-ingestion">Incremental Ingestion</h2>
-<p>Following means can be used to apply a delta or an incremental change to a
Hudi dataset. For e.g, the incremental changes could be from a Kafka topic or
files uploaded to HDFS or
+<p>Following means can be used to apply a delta or an incremental change to a
Hudi dataset. For e.g, the incremental changes could be from a Kafka topic or
files uploaded to DFS or
even changes pulled from another Hudi dataset.</p>
<h4 id="deltastreamer-tool">DeltaStreamer Tool</h4>
@@ -360,9 +364,10 @@ from different sources such as DFS or Kafka.</p>
<p>The tool is a spark job (part of hoodie-utilities), that provides the
following functionality</p>
<ul>
- <li>Ability to consume new events from Kafka, incremental imports from Sqoop
or output of <code class="highlighter-rouge">HiveIncrementalPuller</code> or
files under a folder on HDFS</li>
+ <li>Ability to consume new events from Kafka, incremental imports from Sqoop
or output of <code class="highlighter-rouge">HiveIncrementalPuller</code> or
files under a folder on DFS</li>
<li>Support json, avro or a custom payload types for the incoming data</li>
- <li>New data is written to a Hudi dataset, with support for checkpointing
& schemas and registered onto Hive</li>
+ <li>Pick up avro schemas from DFS or Confluent <a
href="https://github.com/confluentinc/schema-registry";>schema registry</a>.</li>
+ <li>New data is written to a Hudi dataset, with support for checkpointing
and registered onto Hive</li>
</ul>
<p>Command line options describe capabilities in more detail (first build
hoodie-utilities using <code class="highlighter-rouge">mvn clean
package</code>).</p>
@@ -423,10 +428,10 @@ Usage: <main class> [options]
* --target-table
name of the target table in Hive
--transformer-class
- subclass of com.uber.hoodie.utilities.transform.Transformer. UDF to
- transform raw source dataset to a target dataset (conforming to target
- schema) before writing. Default : Not set. E:g -
- com.uber.hoodie.utilities.transform.SqlQueryBasedTransformer (which
+ subclass of com.uber.hoodie.utilities.transform.Transformer. UDF to
+ transform raw source dataset to a target dataset (conforming to target
+ schema) before writing. Default : Not set. E:g -
+ com.uber.hoodie.utilities.transform.SqlQueryBasedTransformer (which
allows a SQL query template to be passed as a transformation function)
</code></pre>
@@ -453,7 +458,7 @@ provided under <code
class="highlighter-rouge">hoodie-utilities/src/test/resourc
</code></pre>
</div>
-<p>In some cases, you may want to convert your existing dataset into Hoodie,
before you can begin ingesting new data. This can be accomplished using the
<code class="highlighter-rouge">hdfsparquetimport</code> command on the <code
class="highlighter-rouge">hoodie-cli</code>.
+<p>In some cases, you may want to convert your existing dataset into Hudi,
before you can begin ingesting new data. This can be accomplished using the
<code class="highlighter-rouge">hdfsparquetimport</code> command on the <code
class="highlighter-rouge">hoodie-cli</code>.
Currently, there is support for converting parquet datasets.</p>
<h4 id="via-custom-spark-job">Via Custom Spark Job</h4>
@@ -503,8 +508,6 @@ Usage: <main class> [options]
</code></pre>
</div>
-<div class="bs-callout bs-callout-info">Note that for now, due to jar
mismatches between Spark & Hive, its recommended to run this as a separate
Java task in your workflow manager/cron. This is getting fix <a
href="https://github.com/uber/hoodie/issues/123";>here</a></div>
-
<h2 id="incrementally-pulling">Incrementally Pulling</h2>
<p>Hudi datasets can be pulled incrementally, which means you can get ALL and
ONLY the updated & new rows since a specified commit timestamp.
@@ -530,7 +533,7 @@ This class can be used within existing Spark jobs and
offers the following funct
<p>Please refer to <a href="configurations.html">configurations</a> section,
to view all datasource options.</p>
-<p>Additionally, <code class="highlighter-rouge">HoodieReadClient</code>
offers the following functionality using Hoodie’s implicit indexing.</p>
+<p>Additionally, <code class="highlighter-rouge">HoodieReadClient</code>
offers the following functionality using Hudi’s implicit indexing.</p>
<table>
<tbody>
@@ -540,7 +543,7 @@ This class can be used within existing Spark jobs and
offers the following funct
</tr>
<tr>
<td>read(keys)</td>
- <td>Read out the data corresponding to the keys as a DataFrame, using
Hoodie’s own index for faster lookup</td>
+ <td>Read out the data corresponding to the keys as a DataFrame, using
Hudi’s own index for faster lookup</td>
</tr>
<tr>
<td>filterExists()</td>
@@ -590,7 +593,7 @@ e.g: <code
class="highlighter-rouge">/app/incremental-hql/intermediate/{source_t
</tr>
<tr>
<td>tmp</td>
- <td>Directory where the temporary delta data is stored in HDFS. The
directory structure will follow conventions. Please see the below section.</td>
+ <td>Directory where the temporary delta data is stored in DFS. The
directory structure will follow conventions. Please see the below section.</td>
<td> </td>
</tr>
<tr>
@@ -610,12 +613,12 @@ e.g: <code
class="highlighter-rouge">/app/incremental-hql/intermediate/{source_t
</tr>
<tr>
<td>sourceDataPath</td>
- <td>Source HDFS Base Path. This is where the Hudi metadata will be
read.</td>
+ <td>Source DFS Base Path. This is where the Hudi metadata will be
read.</td>
<td> </td>
</tr>
<tr>
<td>targetDataPath</td>
- <td>Target HDFS Base path. This is needed to compute the fromCommitTime.
This is not needed if fromCommitTime is specified explicitly.</td>
+ <td>Target DFS Base path. This is needed to compute the fromCommitTime.
This is not needed if fromCommitTime is specified explicitly.</td>
<td> </td>
</tr>
<tr>
@@ -647,7 +650,6 @@ it will automatically use the backfill configuration, since
applying the last 24
is the lack of support for self-joining the same table in mixed mode (normal
and incremental modes).</p>
-
<div class="tags">
</div>
diff --git a/content/index.html b/content/index.html
index bd31b4d..1a1c5ff 100644
--- a/content/index.html
+++ b/content/index.html
@@ -4,7 +4,7 @@
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="Hudi brings stream processing to big data,
providing fresh data while being an order of magnitude efficient over
traditional batch processing.">
-<meta name="keywords" content="getting_started, homepage">
+<meta name="keywords" content="big data, stream processing, cloud, hdfs,
storage, upserts, change capture">
<title>What is Hudi? | Hudi</title>
<link rel="stylesheet" href="css/syntax.css">
@@ -149,6 +149,10 @@
<li><a
href="https://cwiki.apache.org/confluence/pages/viewrecentblogposts.action?key=HUDI";
target="_blank">Blog</a></li>
+
+ <li><a
href="https://projects.apache.org/project.html?incubator-hudi";
target="_blank">Team</a></li>
+
+
</ul>
</li>
@@ -366,7 +370,7 @@ $('#toc').on('click', 'a', function() {
- <p>Hudi (pronounced “Hoodie”) ingests & manages storage of large
analytical datasets on <a
href="http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html";>HDFS</a>
or cloud stores and provides three logical views for query access.</p>
+ <p>Hudi (pronounced “Hoodie”) ingests & manages storage of large
analytical datasets over DFS (<a
href="http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html";>HDFS</a>
or cloud stores) and provides three logical views for query access.</p>
<ul>
<li><strong>Read Optimized View</strong> - Provides excellent query
performance on pure columnar storage, much like plain <a
href="https://parquet.apache.org/";>Parquet</a> tables.</li>
@@ -374,7 +378,7 @@ $('#toc').on('click', 'a', function() {
<li><strong>Near-Real time Table</strong> - Provides queries on real-time
data, using a combination of columnar & row based storage (e.g Parquet + <a
href="http://avro.apache.org/docs/current/mr.html";>Avro</a>)</li>
</ul>
-<figure><img class="docimage" src="images/hoodie_intro_1.png"
alt="hoodie_intro_1.png" /></figure>
+<figure><img class="docimage" src="images/hudi_intro_1.png"
alt="hudi_intro_1.png" /></figure>
<p>By carefully managing how data is laid out in storage & how it’s
exposed to queries, Hudi is able to power a rich data ecosystem where external
sources can be ingested in near real-time and made available for interactive
SQL Engines like <a href="https://prestodb.io";>Presto</a> & <a
href="https://spark.apache.org/sql/";>Spark</a>, while at the same time capable
of being consumed incrementally from processing/ETL frameworks like <a
href="https://hive.apache.org/";>Hive</a> & [...]
diff --git a/content/js/mydoc_scroll.html b/content/js/mydoc_scroll.html
index b23a6ad..ee70719 100644
--- a/content/js/mydoc_scroll.html
+++ b/content/js/mydoc_scroll.html
@@ -4,7 +4,7 @@
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="This page demonstrates how you the
integration of a script called ScrollTo, which is used here to link definitions
of a JSON code sample to a list of definit...">
-<meta name="keywords" content="special_layouts, json, scrolling, scrollto,
jquery plugin">
+<meta name="keywords" content="json, scrolling, scrollto, jquery plugin">
<title>Scroll layout | Hudi</title>
<link rel="stylesheet" href="css/syntax.css">
@@ -149,6 +149,10 @@
<li><a
href="https://cwiki.apache.org/confluence/pages/viewrecentblogposts.action?key=HUDI";
target="_blank">Blog</a></li>
+
+ <li><a
href="https://projects.apache.org/project.html?incubator-hudi";
target="_blank">Team</a></li>
+
+
</ul>
</li>
diff --git a/content/migration_guide.html b/content/migration_guide.html
index 7bcfa1d..03ea8a1 100644
--- a/content/migration_guide.html
+++ b/content/migration_guide.html
@@ -4,7 +4,7 @@
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="In this page, we will discuss some available
tools for migrating your existing dataset into a Hudi dataset">
-<meta name="keywords" content=" migration guide">
+<meta name="keywords" content="hudi, migration, use case">
<title>Migration Guide | Hudi</title>
<link rel="stylesheet" href="css/syntax.css">
@@ -149,6 +149,10 @@
<li><a
href="https://cwiki.apache.org/confluence/pages/viewrecentblogposts.action?key=HUDI";
target="_blank">Blog</a></li>
+
+ <li><a
href="https://projects.apache.org/project.html?incubator-hudi";
target="_blank">Team</a></li>
+
+
</ul>
</li>
@@ -362,7 +366,7 @@ Take this approach if your dataset is an append only type
of dataset and you do
<p>Import your existing dataset into a Hudi managed dataset. Since all the
data is Hudi managed, none of the limitations
of Approach 1 apply here. Updates spanning any partitions can be applied to
this dataset and Hudi will efficiently
- make the update available to queries. Note that not only do you get to use
all Hoodie primitives on this dataset,
+ make the update available to queries. Note that not only do you get to use
all Hudi primitives on this dataset,
there are other additional advantages of doing this. Hudi automatically
manages file sizes of a Hudi managed dataset
. You can define the desired file size when converting this dataset and Hudi
will ensure it writes out files
adhering to the config. It will also ensure that smaller files later get
corrected by routing some new inserts into
@@ -371,9 +375,8 @@ Take this approach if your dataset is an append only type
of dataset and you do
<p>There are a few options when choosing this approach.</p>
<h4 id="option-1">Option 1</h4>
-<p>Use the HDFSParquetImporter tool. As the name suggests, this only works if
your existing dataset is in
-parquet file
-format. This tool essentially starts a Spark Job to read the existing parquet
dataset and converts it into a HUDI managed dataset by re-writing all the
data.</p>
+<p>Use the HDFSParquetImporter tool. As the name suggests, this only works if
your existing dataset is in parquet file format.
+This tool essentially starts a Spark Job to read the existing parquet dataset
and converts it into a HUDI managed dataset by re-writing all the data.</p>
<h4 id="option-2">Option 2</h4>
<p>For huge datasets, this could be as simple as : for partition in [list of
partitions in source dataset] {
@@ -385,7 +388,7 @@ format. This tool essentially starts a Spark Job to read
the existing parquet da
<p>Write your own custom logic of how to load an existing dataset into a Hudi
managed one. Please read about the RDD API
<a href="quickstart.html">here</a>.</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>Using the
HDFSParquetImporter Tool. Once hoodie has been built via `mvn clean install
-DskipTests`, the shell can be
+<div class="highlighter-rouge"><pre class="highlight"><code>Using the
HDFSParquetImporter Tool. Once hudi has been built via `mvn clean install
-DskipTests`, the shell can be
fired by via `cd hoodie-cli && ./hoodie-cli.sh`.
hoodie->hdfsparquetimport
diff --git a/content/news.html b/content/news.html
index 645bae0..43d92a3 100644
--- a/content/news.html
+++ b/content/news.html
@@ -4,7 +4,7 @@
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="">
-<meta name="keywords" content=" news, blog, updates, release notes,
announcements">
+<meta name="keywords" content="apache, hudi, news, blog, updates, release
notes, announcements">
<title>News | Hudi</title>
<link rel="stylesheet" href="css/syntax.css">
@@ -149,6 +149,10 @@
<li><a
href="https://cwiki.apache.org/confluence/pages/viewrecentblogposts.action?key=HUDI";
target="_blank">Blog</a></li>
+
+ <li><a
href="https://projects.apache.org/project.html?incubator-hudi";
target="_blank">Team</a></li>
+
+
</ul>
</li>
@@ -266,7 +270,7 @@
<a href="tag_news.html">news</a>
</span>
- <p> We will be presenting Hoodie & general concepts around how
incremental processing works at Uber.
+ <p> We will be presenting Hudi & general concepts around how
incremental processing works at Uber.
Catch our talk “Incremental Processing on Hadoop At Uber”
</p>
diff --git a/content/news_archive.html b/content/news_archive.html
index 4d80715..d1986b5 100644
--- a/content/news_archive.html
+++ b/content/news_archive.html
@@ -4,7 +4,7 @@
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="">
-<meta name="keywords" content=" news, blog, updates, release notes,
announcements">
+<meta name="keywords" content="news, blog, updates, release notes,
announcements">
<title>News | Hudi</title>
<link rel="stylesheet" href="css/syntax.css">
@@ -149,6 +149,10 @@
<li><a
href="https://cwiki.apache.org/confluence/pages/viewrecentblogposts.action?key=HUDI";
target="_blank">Blog</a></li>
+
+ <li><a
href="https://projects.apache.org/project.html?incubator-hudi";
target="_blank">Team</a></li>
+
+
</ul>
</li>
diff --git a/content/powered_by.html b/content/powered_by.html
index 8f4b0d4..99991ca 100644
--- a/content/powered_by.html
+++ b/content/powered_by.html
@@ -4,7 +4,7 @@
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="">
-<meta name="keywords" content=" talks">
+<meta name="keywords" content="hudi, talks, presentation">
<title>Talks & Powered By | Hudi</title>
<link rel="stylesheet" href="css/syntax.css">
@@ -149,6 +149,10 @@
<li><a
href="https://cwiki.apache.org/confluence/pages/viewrecentblogposts.action?key=HUDI";
target="_blank">Blog</a></li>
+
+ <li><a
href="https://projects.apache.org/project.html?incubator-hudi";
target="_blank">Team</a></li>
+
+
</ul>
</li>
@@ -383,7 +387,6 @@ October 2018, Spark+AI Summit Europe, London, UK</p>
</ol>
-
<div class="tags">
</div>
diff --git a/content/privacy.html b/content/privacy.html
index 704bd3d..1804b9f 100644
--- a/content/privacy.html
+++ b/content/privacy.html
@@ -4,7 +4,7 @@
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="">
-<meta name="keywords" content=" privacy">
+<meta name="keywords" content="hudi, privacy">
<title>Privacy Policy | Hudi</title>
<link rel="stylesheet" href="css/syntax.css">
@@ -149,6 +149,10 @@
<li><a
href="https://cwiki.apache.org/confluence/pages/viewrecentblogposts.action?key=HUDI";
target="_blank">Blog</a></li>
+
+ <li><a
href="https://projects.apache.org/project.html?incubator-hudi";
target="_blank">Team</a></li>
+
+
</ul>
</li>
diff --git a/content/quickstart.html b/content/quickstart.html
index a73534d..b7781b3 100644
--- a/content/quickstart.html
+++ b/content/quickstart.html
@@ -4,7 +4,7 @@
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="">
-<meta name="keywords" content="quickstart, quickstart">
+<meta name="keywords" content="hudi, quickstart">
<title>Quickstart | Hudi</title>
<link rel="stylesheet" href="css/syntax.css">
@@ -149,6 +149,10 @@
<li><a
href="https://cwiki.apache.org/confluence/pages/viewrecentblogposts.action?key=HUDI";
target="_blank">Blog</a></li>
+
+ <li><a
href="https://projects.apache.org/project.html?incubator-hudi";
target="_blank">Team</a></li>
+
+
</ul>
</li>
@@ -362,7 +366,8 @@ $ mvn clean install -DskipTests -DskipITs -Dhive11
<h2 id="version-compatibility">Version Compatibility</h2>
-<p>Hudi requires Java 8 to be installed. Hudi works with Spark-2.x versions.
We have verified that Hudi works with the following combination of
Hadoop/Hive/Spark.</p>
+<p>Hudi requires Java 8 to be installed on a *nix system. Hudi works with
Spark-2.x versions.
+Further, we have verified that Hudi works with the following combination of
Hadoop/Hive/Spark.</p>
<table>
<thead>
@@ -395,8 +400,9 @@ $ mvn clean install -DskipTests -DskipITs -Dhive11
</tbody>
</table>
-<p>If your environment has other versions of hadoop/hive/spark, please try out
Hudi and let us know if there are any issues. We are limited by our bandwidth
to certify other combinations.
-It would be of great help if you can reach out to us with your setup and
experience with hoodie.</p>
+<p>If your environment has other versions of hadoop/hive/spark, please try out
Hudi and let us know if there are any issues.
+We are limited by our bandwidth to certify other combinations (e.g Docker on
Windows).
+It would be of great help if you can reach out to us with your setup and
experience with hudi.</p>
<h2 id="generate-a-hudi-dataset">Generate a Hudi Dataset</h2>
@@ -424,7 +430,7 @@ Use the RDD API to perform more involved actions on a Hudi
dataset</p>
<h4 id="datasource-api">DataSource API</h4>
-<p>Run <strong>hoodie-spark/src/test/java/HoodieJavaApp.java</strong> class,
to place a two commits (commit 1 => 100 inserts, commit 2 => 100 updates
to previously inserted 100 records) onto your HDFS/local filesystem. Use the
wrapper script
+<p>Run <strong>hoodie-spark/src/test/java/HoodieJavaApp.java</strong> class,
to place a two commits (commit 1 => 100 inserts, commit 2 => 100 updates
to previously inserted 100 records) onto your DFS/local filesystem. Use the
wrapper script
to run from command-line</p>
<div class="highlighter-rouge"><pre class="highlight"><code>cd hoodie-spark
@@ -679,9 +685,9 @@ data infrastructure is brought up in a local docker cluster
within your computer
<h3 id="setting-up-docker-cluster">Setting up Docker Cluster</h3>
-<h4 id="build-hoodie">Build Hoodie</h4>
+<h4 id="build-hudi">Build Hudi</h4>
-<p>The first step is to build hoodie
+<p>The first step is to build hudi
<code class="highlighter-rouge">
cd <HUDI_WORKSPACE>
mvn package -DskipTests
@@ -801,7 +807,7 @@ automatically initializes the datasets in the file-system
if they do not exist y
<div class="highlighter-rouge"><pre class="highlight"><code>docker exec -it
adhoc-2 /bin/bash
# Run the following spark-submit command to execute the delta-streamer and
ingest to stock_ticks_cow dataset in HDFS
-spark-submit --class
com.uber.hoodie.utilities.deltastreamer.HoodieDeltaStreamer
$HUDI_UTILITIES_BUNDLE --storage-type COPY_ON_WRITE --source-class
com.uber.hoodie.utilities.sources.JsonKafkaSource --source-ordering-field ts
--target-base-path /user/hive/warehouse/stock_ticks_cow --target-table
stock_ticks_cow --props /var/demo/config/kafka-source.properties
+spark-submit --class
com.uber.hoodie.utilities.deltastreamer.HoodieDeltaStreamer
$HUDI_UTILITIES_BUNDLE --storage-type COPY_ON_WRITE --source-class
com.uber.hoodie.utilities.sources.JsonKafkaSource --source-ordering-field ts
--target-base-path /user/hive/warehouse/stock_ticks_cow --target-table
stock_ticks_cow --props /var/demo/config/kafka-source.properties
--schemaprovider-class com.uber.hoodie.utilities.schema.FilebasedSchemaProvider
....
....
2018-09-24 22:20:00 INFO
OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:54 -
OutputCommitCoordinator stopped!
@@ -1329,7 +1335,7 @@ scala> spark.sql("select `_hoodie_commit_time`,
symbol, ts, volume, open, clo
Again, You can use Hudi CLI to manually schedule and run compaction</p>
<div class="highlighter-rouge"><pre class="highlight"><code>docker exec -it
adhoc-1 /bin/bash
-^[[Aroot@adhoc-1:/opt# /var/hoodie/ws/hoodie-cli/hoodie-cli.sh
+root@adhoc-1:/opt# /var/hoodie/ws/hoodie-cli/hoodie-cli.sh
============================================
* *
* _ _ _ _ *
@@ -1514,7 +1520,7 @@ scala> spark.sql("select `_hoodie_commit_time`,
symbol, ts, volume, open, clo
<h2 id="testing-hudi-in-local-docker-environment">Testing Hudi in Local Docker
environment</h2>
-<p>You can bring up a hadoop docker environment containing Hadoop, Hive and
Spark services with support for hoodie.
+<p>You can bring up a hadoop docker environment containing Hadoop, Hive and
Spark services with support for hudi.
<code class="highlighter-rouge">
$ mvn pre-integration-test -DskipTests
</code>
diff --git a/content/s3_hoodie.html b/content/s3_hoodie.html
index 217005c..0366721 100644
--- a/content/s3_hoodie.html
+++ b/content/s3_hoodie.html
@@ -4,8 +4,8 @@
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="In this page, we go over how to configure
Hudi with S3 filesystem.">
-<meta name="keywords" content=" sql hive s3 spark presto">
-<title>S3 Filesystem (experimental) | Hudi</title>
+<meta name="keywords" content="hudi, hive, aws, s3, spark, presto">
+<title>S3 Filesystem | Hudi</title>
<link rel="stylesheet" href="css/syntax.css">
@@ -149,6 +149,10 @@
<li><a
href="https://cwiki.apache.org/confluence/pages/viewrecentblogposts.action?key=HUDI";
target="_blank">Blog</a></li>
+
+ <li><a
href="https://projects.apache.org/project.html?incubator-hudi";
target="_blank">Team</a></li>
+
+
</ul>
</li>
@@ -158,7 +162,7 @@
- <a class="email" title="Submit feedback" href="#"
onclick="javascript:window.location='mailto:[email protected]?subject=Hudi
Documentation feedback&body=I have some feedback about the S3 Filesystem
(experimental) page: ' + window.location.href;"><i class="fa
fa-envelope-o"></i> Feedback</a>
+ <a class="email" title="Submit feedback" href="#"
onclick="javascript:window.location='mailto:[email protected]?subject=Hudi
Documentation feedback&body=I have some feedback about the S3 Filesystem page:
' + window.location.href;"><i class="fa fa-envelope-o"></i> Feedback</a>
<li>
@@ -176,7 +180,7 @@
searchInput:
document.getElementById('search-input'),
resultsContainer:
document.getElementById('results-container'),
dataSource: 'search.json',
- searchResultTemplate: '<li><a href="{url}"
title="S3 Filesystem (experimental)">{title}</a></li>',
+ searchResultTemplate: '<li><a href="{url}"
title="S3 Filesystem">{title}</a></li>',
noResultsText: 'No results found.',
limit: 10,
fuzzy: true,
@@ -327,7 +331,7 @@
<!-- Content Column -->
<div class="col-md-9">
<div class="post-header">
- <h1 class="post-title-main">S3 Filesystem (experimental)</h1>
+ <h1 class="post-title-main">S3 Filesystem</h1>
</div>
@@ -343,11 +347,11 @@
- <p>Hudi works with HDFS by default. There is an experimental work going on
Hoodie-S3 compatibility.</p>
+ <p>In this page, we explain how to get your Hudi spark job to store into AWS
S3.</p>
<h2 id="aws-configs">AWS configs</h2>
-<p>There are two configurations required for Hoodie-S3 compatibility:</p>
+<p>There are two configurations required for Hudi-S3 compatibility:</p>
<ul>
<li>Adding AWS Credentials for Hudi</li>
@@ -415,7 +419,6 @@ export
HOODIE_ENV_fs_DOT_s3n_DOT_impl=org.apache.hadoop.fs.s3a.S3AFileSystem
</ul>
-
<div class="tags">
</div>
diff --git a/content/search.json b/content/search.json
index 3f7eb15..0473b34 100644
--- a/content/search.json
+++ b/content/search.json
@@ -6,7 +6,7 @@
{
"title": "Admin Guide",
"tags": "",
-"keywords": "admin",
+"keywords": "hudi, administration, operation, devops",
"url": "admin_guide.html",
"summary": "This section offers an overview of tools available to operate an
ecosystem of Hudi datasets"
}
@@ -17,7 +17,7 @@
{
"title": "Community",
"tags": "",
-"keywords": "usecases",
+"keywords": "hudi, use cases, big data, apache",
"url": "community.html",
"summary": ""
}
@@ -28,7 +28,7 @@
{
"title": "Comparison",
"tags": "",
-"keywords": "usecases",
+"keywords": "apache, hudi, kafka, kudu, hive, hbase, stream processing",
"url": "comparison.html",
"summary": ""
}
@@ -39,7 +39,7 @@
{
"title": "Concepts",
"tags": "",
-"keywords": "concepts",
+"keywords": "hudi, design, storage, views, timeline",
"url": "concepts.html",
"summary": "Here we introduce some basic concepts & give a broad technical
overview of Hudi"
}
@@ -50,7 +50,7 @@
{
"title": "Configurations",
"tags": "",
-"keywords": "configurations",
+"keywords": "garbage collection, hudi, jvm, configs, tuning",
"url": "configurations.html",
"summary": "Here we list all possible configurations and what they mean"
}
@@ -61,7 +61,7 @@
{
"title": "Developer Setup",
"tags": "",
-"keywords": "developer setup",
+"keywords": "hudi, ide, developer, setup",
"url": "contributing.html",
"summary": ""
}
@@ -72,9 +72,9 @@
{
-"title": "GCS Filesystem (experimental)",
+"title": "GCS Filesystem",
"tags": "",
-"keywords": "sql hive gcs spark presto",
+"keywords": "hudi, hive, google cloud, storage, spark, presto",
"url": "gcs_hoodie.html",
"summary": "In this page, we go over how to configure hudi with Google Cloud
Storage."
}
@@ -85,7 +85,7 @@
{
"title": "Implementation",
"tags": "",
-"keywords": "implementation",
+"keywords": "hudi, index, storage, compaction, cleaning, implementation",
"url": "implementation.html",
"summary": ""
}
@@ -96,7 +96,7 @@
{
"title": "Incremental Processing",
"tags": "",
-"keywords": "incremental processing",
+"keywords": "hudi, incremental, batch, stream, processing, Hive, ETL, Spark
SQL",
"url": "incremental_processing.html",
"summary": "In this page, we will discuss some available tools for ingesting
data incrementally & consuming the changes."
}
@@ -107,7 +107,7 @@
{
"title": "What is Hudi?",
"tags": "getting_started",
-"keywords": "homepage",
+"keywords": "big data, stream processing, cloud, hdfs, storage, upserts,
change capture",
"url": "index.html",
"summary": "Hudi brings stream processing to big data, providing fresh data
while being an order of magnitude efficient over traditional batch processing."
}
@@ -118,7 +118,7 @@
{
"title": "Migration Guide",
"tags": "",
-"keywords": "migration guide",
+"keywords": "hudi, migration, use case",
"url": "migration_guide.html",
"summary": "In this page, we will discuss some available tools for migrating
your existing dataset into a Hudi dataset"
}
@@ -140,7 +140,7 @@
{
"title": "News",
"tags": "",
-"keywords": "news, blog, updates, release notes, announcements",
+"keywords": "apache, hudi, news, blog, updates, release notes, announcements",
"url": "news.html",
"summary": ""
}
@@ -162,7 +162,7 @@
{
"title": "Talks & Powered By",
"tags": "",
-"keywords": "talks",
+"keywords": "hudi, talks, presentation",
"url": "powered_by.html",
"summary": ""
}
@@ -173,7 +173,7 @@
{
"title": "Privacy Policy",
"tags": "",
-"keywords": "privacy",
+"keywords": "hudi, privacy",
"url": "privacy.html",
"summary": ""
}
@@ -184,7 +184,7 @@
{
"title": "Quickstart",
"tags": "quickstart",
-"keywords": "quickstart",
+"keywords": "hudi, quickstart",
"url": "quickstart.html",
"summary": ""
}
@@ -193,9 +193,9 @@
{
-"title": "S3 Filesystem (experimental)",
+"title": "S3 Filesystem",
"tags": "",
-"keywords": "sql hive s3 spark presto",
+"keywords": "hudi, hive, aws, s3, spark, presto",
"url": "s3_hoodie.html",
"summary": "In this page, we go over how to configure Hudi with S3 filesystem."
}
@@ -210,7 +210,7 @@
{
"title": "SQL Queries",
"tags": "",
-"keywords": "sql hive spark presto",
+"keywords": "hudi, hive, spark, sql, presto",
"url": "sql_queries.html",
"summary": "In this page, we go over how to enable SQL queries on Hudi built
tables."
}
@@ -221,7 +221,7 @@
{
"title": "Use Cases",
"tags": "",
-"keywords": "usecases",
+"keywords": "hudi, data ingestion, etl, real time, use cases",
"url": "use_cases.html",
"summary": "Following are some sample use-cases for Hudi, which illustrate the
benefits in terms of faster processing & increased efficiency"
}
diff --git a/content/sql_queries.html b/content/sql_queries.html
index 6936191..d7fa8cc 100644
--- a/content/sql_queries.html
+++ b/content/sql_queries.html
@@ -4,7 +4,7 @@
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="In this page, we go over how to enable SQL
queries on Hudi built tables.">
-<meta name="keywords" content=" sql hive spark presto">
+<meta name="keywords" content="hudi, hive, spark, sql, presto">
<title>SQL Queries | Hudi</title>
<link rel="stylesheet" href="css/syntax.css">
@@ -149,6 +149,10 @@
<li><a
href="https://cwiki.apache.org/confluence/pages/viewrecentblogposts.action?key=HUDI";
target="_blank">Blog</a></li>
+
+ <li><a
href="https://projects.apache.org/project.html?incubator-hudi";
target="_blank">Team</a></li>
+
+
</ul>
</li>
@@ -368,8 +372,6 @@ to using the Hive Serde to read the data
(planning/executions is still Spark). T
towards Parquet reading, which we will address in the next method based on
path filters.
However benchmarks have not revealed any real performance degradation with
Hudi & SparkSQL, compared to native support.</p>
-<div class="bs-callout bs-callout-info">Get involved to improve this
integration <a href="https://github.com/uber/hoodie/issues/7";>here</a> and <a
href="https://issues.apache.org/jira/browse/SPARK-19351";>here</a> </div>
-
<p>Sample command is provided below to spin up Spark Shell</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ spark-shell
--jars hoodie-spark-bundle-x.y.z-SNAPSHOT.jar --driver-class-path
/etc/hive/conf --packages com.databricks:spark-avro_2.11:4.0.0 --conf
spark.sql.hive.convertMetastoreParquet=false --num-executors 10 --driver-memory
7g --executor-memory 2g --master yarn-client
diff --git a/content/strata-talk.html b/content/strata-talk.html
index 13a8375..58b6f8a 100644
--- a/content/strata-talk.html
+++ b/content/strata-talk.html
@@ -4,7 +4,7 @@
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="">
-<meta name="keywords" content="news, ">
+<meta name="keywords" content="">
<title>Hudi entered Apache Incubator | Hudi</title>
<link rel="stylesheet" href="css/syntax.css">
@@ -149,6 +149,10 @@
<li><a
href="https://cwiki.apache.org/confluence/pages/viewrecentblogposts.action?key=HUDI";
target="_blank">Blog</a></li>
+
+ <li><a
href="https://projects.apache.org/project.html?incubator-hudi";
target="_blank">Team</a></li>
+
+
</ul>
</li>
diff --git a/content/use_cases.html b/content/use_cases.html
index 6df8c34..dcdf403 100644
--- a/content/use_cases.html
+++ b/content/use_cases.html
@@ -4,7 +4,7 @@
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="Following are some sample use-cases for
Hudi, which illustrate the benefits in terms of faster processing & increased
efficiency">
-<meta name="keywords" content=" usecases">
+<meta name="keywords" content="hudi, data ingestion, etl, real time, use
cases">
<title>Use Cases | Hudi</title>
<link rel="stylesheet" href="css/syntax.css">
@@ -149,6 +149,10 @@
<li><a
href="https://cwiki.apache.org/confluence/pages/viewrecentblogposts.action?key=HUDI";
target="_blank">Blog</a></li>
+
+ <li><a
href="https://projects.apache.org/project.html?incubator-hudi";
target="_blank">Team</a></li>
+
+
</ul>
</li>
@@ -350,7 +354,7 @@ In most (if not all) Hadoop deployments, it is
unfortunately solved in a pieceme
even though this data is arguably the most valuable for the entire
organization.</p>
<p>For RDBMS ingestion, Hudi provides <strong>faster loads via
Upserts</strong>, as opposed costly & inefficient bulk loads. For e.g, you
can read the MySQL BIN log or <a
href="https://sqoop.apache.org/docs/1.4.2/SqoopUserGuide.html#_incremental_imports";>Sqoop
Incremental Import</a> and apply them to an
-equivalent Hudi table on HDFS. This would be much faster/efficient than a <a
href="https://sqoop.apache.org/docs/1.4.0-incubating/SqoopUserGuide.html#id1770457";>bulk
merge job</a>
+equivalent Hudi table on DFS. This would be much faster/efficient than a <a
href="https://sqoop.apache.org/docs/1.4.0-incubating/SqoopUserGuide.html#id1770457";>bulk
merge job</a>
or <a
href="http://hortonworks.com/blog/four-step-strategy-incremental-updates-hive/";>complicated
handcrafted merge workflows</a></p>
<p>For NoSQL datastores like <a
href="http://cassandra.apache.org/";>Cassandra</a> / <a
href="http://www.project-voldemort.com/voldemort/";>Voldemort</a> / <a
href="https://hbase.apache.org/";>HBase</a>, even moderately big installations
store billions of rows.
@@ -367,13 +371,13 @@ This is absolutely perfect for lower scale (<a
href="https://blog.twitter.com/20
But, typically these systems end up getting abused for less interactive
queries also since data on Hadoop is intolerably stale. This leads to under
utilization & wasteful hardware/license costs.</p>
<p>On the other hand, interactive SQL solutions on Hadoop such as Presto &
SparkSQL excel in <strong>queries that finish within few seconds</strong>.
-By bringing <strong>data freshness to a few minutes</strong>, Hudi can provide
a much efficient alternative, as well unlock real-time analytics on
<strong>several magnitudes larger datasets</strong> stored in HDFS.
+By bringing <strong>data freshness to a few minutes</strong>, Hudi can provide
a much efficient alternative, as well unlock real-time analytics on
<strong>several magnitudes larger datasets</strong> stored in DFS.
Also, Hudi has no external dependencies (like a dedicated HBase cluster,
purely used for real-time analytics) and thus enables faster analytics on much
fresher analytics, without increasing the operational overhead.</p>
<h2 id="incremental-processing-pipelines">Incremental Processing Pipelines</h2>
<p>One fundamental ability Hadoop provides is to build a chain of datasets
derived from each other via DAGs expressed as workflows.
-Workflows often depend on new data being output by multiple upstream workflows
and traditionally, availability of new data is indicated by a new HDFS
Folder/Hive Partition.
+Workflows often depend on new data being output by multiple upstream workflows
and traditionally, availability of new data is indicated by a new DFS
Folder/Hive Partition.
Let’s take a concrete example to illustrate this. An upstream workflow <code
class="highlighter-rouge">U</code> can create a Hive partition for every hour,
with data for that hour (event_time) at the end of each hour (processing_time),
providing effective freshness of 1 hour.
Then, a downstream workflow <code class="highlighter-rouge">D</code>, kicks
off immediately after <code class="highlighter-rouge">U</code> finishes, and
does its own processing for the next hour, increasing the effective latency to
2 hours.</p>
@@ -388,19 +392,18 @@ like 15 mins, and providing an end-end latency of 30 mins
at <code class="highli
<div class="bs-callout bs-callout-info">To achieve this, Hudi has embraced
similar concepts from stream processing frameworks like <a
href="https://spark.apache.org/docs/latest/streaming-programming-guide.html#join-operations";>Spark
Streaming</a> , Pub/Sub systems like <a
href="http://kafka.apache.org/documentation/#theconsumer";>Kafka</a>
or database replication technologies like <a
href="https://docs.oracle.com/cd/E11882_01/server.112/e16545/xstrm_cncpt.htm#XSTRM187";>Oracle
XStream</a>.
-For the more curious, a more detailed explanation of the benefits of
Incremetal Processing (compared to Stream Processing & Batch Processing)
can be found <a
href="https://www.oreilly.com/ideas/ubers-case-for-incremental-processing-on-hadoop";>here</a></div>
+For the more curious, a more detailed explanation of the benefits of
Incremental Processing (compared to Stream Processing & Batch Processing)
can be found <a
href="https://www.oreilly.com/ideas/ubers-case-for-incremental-processing-on-hadoop";>here</a></div>
-<h2 id="data-dispersal-from-hadoop">Data Dispersal From Hadoop</h2>
+<h2 id="data-dispersal-from-dfs">Data Dispersal From DFS</h2>
<p>A popular use-case for Hadoop, is to crunch data and then disperse it back
to an online serving store, to be used by an application.
For e.g, a Spark Pipeline can <a
href="https://eng.uber.com/telematics/";>determine hard braking events on
Hadoop</a> and load them into a serving store like ElasticSearch, to be used by
the Uber application to increase safe driving. Typical architectures for this
employ a <code class="highlighter-rouge">queue</code> between Hadoop and
serving store, to prevent overwhelming the target serving store.
-A popular choice for this queue is Kafka and this model often results in
<strong>redundant storage of same data on HDFS (for offline analysis on
computed results) and Kafka (for dispersal)</strong></p>
+A popular choice for this queue is Kafka and this model often results in
<strong>redundant storage of same data on DFS (for offline analysis on computed
results) and Kafka (for dispersal)</strong></p>
<p>Once again Hudi can efficiently solve this problem, by having the Spark
Pipeline upsert output from
each run into a Hudi dataset, which can then be incrementally tailed (just
like a Kafka topic) for new data & written into the serving store.</p>
-
<div class="tags">
</div>
