This is an automated email from the ASF dual-hosted git repository.
yihua pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 32b1d361ed [HUDI-4884] Fixing faq for indexes with hudi and fixing
docker demo for hive sync (#6730)
32b1d361ed is described below
commit 32b1d361edc87dff7695557c45d3b0aef1b468b6
Author: Sivabalan Narayanan <[email protected]>
AuthorDate: Fri Sep 23 18:23:59 2022 -0700
[HUDI-4884] Fixing faq for indexes with hudi and fixing docker demo for
hive sync (#6730)
Co-authored-by: Y Ethan Guo <[email protected]>
---
website/docs/docker_demo.md | 6 ++++--
website/docs/faq.md | 8 ++++++--
website/versioned_docs/version-0.12.0/faq.md | 10 +++++++---
3 files changed, 17 insertions(+), 7 deletions(-)
diff --git a/website/docs/docker_demo.md b/website/docs/docker_demo.md
index 4a390506c3..698aec5439 100644
--- a/website/docs/docker_demo.md
+++ b/website/docs/docker_demo.md
@@ -247,7 +247,8 @@ docker exec -it adhoc-2 /bin/bash
--partitioned-by dt \
--base-path /user/hive/warehouse/stock_ticks_cow \
--database default \
- --table stock_ticks_cow
+ --table stock_ticks_cow \
+ --partition-value-extractor
org.apache.hudi.hive.SlashEncodedDayPartitionValueExtractor
.....
2020-01-25 19:51:28,953 INFO [main] hive.HiveSyncTool
(HiveSyncTool.java:syncHoodieTable(129)) - Sync complete for stock_ticks_cow
.....
@@ -260,7 +261,8 @@ docker exec -it adhoc-2 /bin/bash
--partitioned-by dt \
--base-path /user/hive/warehouse/stock_ticks_mor \
--database default \
- --table stock_ticks_mor
+ --table stock_ticks_mor \
+ --partition-value-extractor
org.apache.hudi.hive.SlashEncodedDayPartitionValueExtractor
...
2020-01-25 19:51:51,066 INFO [main] hive.HiveSyncTool
(HiveSyncTool.java:syncHoodieTable(129)) - Sync complete for stock_ticks_mor_ro
...
diff --git a/website/docs/faq.md b/website/docs/faq.md
index 40b2c0fca6..00ef95c2ec 100644
--- a/website/docs/faq.md
+++ b/website/docs/faq.md
@@ -245,9 +245,13 @@ The indexing component is a key part of the Hudi writing
and it maps a given rec
Hudi supports a few options for indexing as below
- - *HoodieBloomIndex (default)* : Uses a bloom filter and ranges information
placed in the footer of parquet/base files (and soon log files as well)
- - *HoodieGlobalBloomIndex* : The default indexing only enforces uniqueness of
a key inside a single partition i.e the user is expected to know the partition
under which a given record key is stored. This helps the indexing scale very
well for even [very large
datasets](https://eng.uber.com/uber-big-data-platform/). However, in some
cases, it might be necessary instead to do the de-duping/enforce uniqueness
across all partitions and the global bloom index does exactly that. If this is
us [...]
+ - *HoodieBloomIndex * : Uses a bloom filter and ranges information placed in
the footer of parquet/base files (and soon log files as well)
+ - *HoodieGlobalBloomIndex* : The non global indexing only enforces uniqueness
of a key inside a single partition i.e the user is expected to know the
partition under which a given record key is stored. This helps the indexing
scale very well for even [very large
datasets](https://eng.uber.com/uber-big-data-platform/). However, in some
cases, it might be necessary instead to do the de-duping/enforce uniqueness
across all partitions and the global bloom index does exactly that. If this is
[...]
- *HBaseIndex* : Apache HBase is a key value store, typically found in close
proximity to HDFS. You can also store the index inside HBase, which could be
handy if you are already operating HBase.
+ - *HoodieSimpleIndex (default)* : A simple index which reads interested
fields (record key and partition path) from base files and joins with incoming
records to find the tagged location.
+ - *HoodieGlobalSimpleIndex* : Global version of Simple Index, where in
uniqueness is on record key across entire table.
+ - *HoodieBucketIndex* : Each partition has statically defined buckets to
which records are tagged with. Since locations are tagged via hashing
mechanism, this index lookup will be very efficient.
+ - *HoodieSparkConsistentBucketIndex* : This is also similar to Bucket Index.
Only difference is that, data skews can be tackled by dynamically changing the
bucket number.
You can implement your own index if you'd like, by subclassing the
`HoodieIndex` class and configuring the index class name in configs.
diff --git a/website/versioned_docs/version-0.12.0/faq.md
b/website/versioned_docs/version-0.12.0/faq.md
index c06c58a612..43e80aea1e 100644
--- a/website/versioned_docs/version-0.12.0/faq.md
+++ b/website/versioned_docs/version-0.12.0/faq.md
@@ -245,9 +245,13 @@ The indexing component is a key part of the Hudi writing
and it maps a given rec
Hudi supports a few options for indexing as below
- - *HoodieBloomIndex (default)* : Uses a bloom filter and ranges information
placed in the footer of parquet/base files (and soon log files as well)
- - *HoodieGlobalBloomIndex* : The default indexing only enforces uniqueness of
a key inside a single partition i.e the user is expected to know the partition
under which a given record key is stored. This helps the indexing scale very
well for even [very large
datasets](https://eng.uber.com/uber-big-data-platform/). However, in some
cases, it might be necessary instead to do the de-duping/enforce uniqueness
across all partitions and the global bloom index does exactly that. If this is
us [...]
- - *HBaseIndex* : Apache HBase is a key value store, typically found in close
proximity to HDFS. You can also store the index inside HBase, which could be
handy if you are already operating HBase.
+- *HoodieBloomIndex * : Uses a bloom filter and ranges information placed in
the footer of parquet/base files (and soon log files as well)
+- *HoodieGlobalBloomIndex* : The non global indexing only enforces uniqueness
of a key inside a single partition i.e the user is expected to know the
partition under which a given record key is stored. This helps the indexing
scale very well for even [very large
datasets](https://eng.uber.com/uber-big-data-platform/). However, in some
cases, it might be necessary instead to do the de-duping/enforce uniqueness
across all partitions and the global bloom index does exactly that. If this is
[...]
+- *HBaseIndex* : Apache HBase is a key value store, typically found in close
proximity to HDFS. You can also store the index inside HBase, which could be
handy if you are already operating HBase.
+- *HoodieSimpleIndex (default)* : A simple index which reads interested fields
(record key and partition path) from base files and joins with incoming records
to find the tagged location.
+- *HoodieGlobalSimpleIndex* : Global version of Simple Index, where in
uniqueness is on record key across entire table.
+- *HoodieBucketIndex* : Each partition has statically defined buckets to which
records are tagged with. Since locations are tagged via hashing mechanism, this
index lookup will be very efficient.
+- *HoodieSparkConsistentBucketIndex* : This is also similar to Bucket Index.
Only difference is that, data skews can be tackled by dynamically changing the
bucket number.
You can implement your own index if you'd like, by subclassing the
`HoodieIndex` class and configuring the index class name in configs.
