This is an automated email from the ASF dual-hosted git repository. vinoth pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/incubator-hudi-site.git
commit 94f64a7c7e8c8ec0307fbd78c5ec13ec0a7b9175 Author: Vinoth Chandar <[email protected]> AuthorDate: Mon Feb 25 07:01:53 2019 -0800 Revised community, contributing pages - Community engagement instructions - Strawman contribution guide, to get us going - Fixed broken image urls from the hudi renames - Fixed broken code formatting on couple pages - Removed api_setup, roadmap pages and cleaned up structure --- .gitignore | 1 + docs/README.md | 5 + docs/_config.yml | 2 +- docs/_data/topnav.yml | 24 ++- docs/_includes/footer.html | 6 + docs/_posts/2019-01-18-asf-incubation.md | 10 ++ docs/admin_guide.md | 22 ++- docs/api_docs.md | 10 -- docs/code_and_design.md | 38 ----- docs/community.md | 38 +++-- docs/concepts.md | 28 ++-- docs/configurations.md | 38 +++-- docs/contributing.md | 101 +++++++++++++ docs/dev_setup.md | 13 -- docs/images/hoodie_cow.png | Bin 31136 -> 0 bytes docs/images/hoodie_mor.png | Bin 56002 -> 0 bytes docs/images/hudi_cow.png | Bin 0 -> 48994 bytes docs/images/hudi_mor.png | Bin 0 -> 92073 bytes .../{hoodie_timeline.png => hudi_timeline.png} | Bin docs/implementation.md | 165 +++++++++++---------- docs/index.md | 7 +- docs/migration_guide.md | 70 ++++----- docs/quickstart.md | 89 +++++------ docs/roadmap.md | 14 -- docs/sql_queries.md | 5 +- 25 files changed, 383 insertions(+), 303 deletions(-) diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..e43b0f9 --- /dev/null +++ b/.gitignore @@ -0,0 +1 @@ +.DS_Store diff --git a/docs/README.md b/docs/README.md index 0995250..8593206 100644 --- a/docs/README.md +++ b/docs/README.md @@ -11,6 +11,11 @@ The site is based on a [Jekyll](https://jekyllrb.com/) theme hosted [here](idrat Simply run `docker-compose build --no-cache && docker-compose up` from the `docs` folder and the site should be up & running at `http://localhost:4000` +To see edits reflect on the site, you may have to bounce the container + + - Stop existing container by `ctrl+c` the docker-compose program + - (or) alternatively via `docker stop docs_server_1` + - Bring up container again using `docker-compose up` #### Host OS diff --git a/docs/_config.yml b/docs/_config.yml index 781bdb6..9f0effd 100644 --- a/docs/_config.yml +++ b/docs/_config.yml @@ -77,7 +77,7 @@ defaults: sidebars: - mydoc_sidebar -description: "Apache Hudi (pronounced “Hoodie”) is a Spark Library, that provides upserts and incremental processing capaibilities on Hadoop datasets" +description: "Apache Hudi (pronounced “Hoodie”) provides upserts and incremental processing capaibilities on Big Data" # the description is used in the feed.xml file # needed for sitemap.xml file only diff --git a/docs/_data/topnav.yml b/docs/_data/topnav.yml index 190573a..0042feb 100644 --- a/docs/_data/topnav.yml +++ b/docs/_data/topnav.yml @@ -7,24 +7,22 @@ topnav: url: /news - title: Community url: /community.html - - title: Github + - title: Code external_url: https://github.com/uber/hoodie #Topnav dropdowns topnav_dropdowns: - title: Topnav dropdowns folders: - - title: Developer Resources + - title: Developers folderitems: - - title: Setup - url: /dev_setup.html - output: web - - title: API Docs - url: /api_docs.html - output: web - - title: Code Structure - url: /code_and_design.html - output: web - - title: Roadmap - url: /roadmap.html + - title: Contributing + url: /contributing.html output: web + - title: Wiki/Designs + external_url: https://cwiki.apache.org/confluence/display/HUDI + - title: Issues + external_url: https://issues.apache.org/jira/projects/HUDI/summary + - title: Blog + external_url: https://cwiki.apache.org/confluence/pages/viewrecentblogposts.action?key=HUDI + diff --git a/docs/_includes/footer.html b/docs/_includes/footer.html index 00605db..c920c5c 100755 --- a/docs/_includes/footer.html +++ b/docs/_includes/footer.html @@ -8,6 +8,12 @@ <a class="footer-link-img" href="https://apache.org";> <img src="images/asf_logo.svg" alt="The Apache Software Foundation" height="100px" widh="50px"></a> </p> + <p> + Apache Hudi is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the name of <a href="http://incubator.apache.org/";>Apache Incubator</a>. + Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have + stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a + reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF. + </p> </div> </div> </footer> diff --git a/docs/_posts/2019-01-18-asf-incubation.md b/docs/_posts/2019-01-18-asf-incubation.md new file mode 100644 index 0000000..79de37c --- /dev/null +++ b/docs/_posts/2019-01-18-asf-incubation.md @@ -0,0 +1,10 @@ +--- +title: "Hudi entered Apache Incubator" +categories: update +permalink: strata-talk.html +tags: [news] +--- + +In the coming weeks, we will be moving in our new home on the Apache Incubator. + +{% include links.html %} diff --git a/docs/admin_guide.md b/docs/admin_guide.md index 7f7e610..3d37d22 100644 --- a/docs/admin_guide.md +++ b/docs/admin_guide.md @@ -43,7 +43,9 @@ hoodie->create --path /user/hive/warehouse/table1 --tableName hoodie_table_1 --t ``` To see the description of hoodie table, use the command: + ``` + hoodie:hoodie_table_1->desc 18/09/06 15:57:19 INFO timeline.HoodieActiveTimeline: Loaded instants [] _________________________________________________________ @@ -55,6 +57,7 @@ hoodie:hoodie_table_1->desc | hoodie.table.name | hoodie_table_1 | | hoodie.table.type | COPY_ON_WRITE | | hoodie.archivelog.folder| | + ``` Following is a sample command to connect to a Hoodie dataset contains uber trips. @@ -183,7 +186,7 @@ order (See Concepts). The below commands allow users to view the file-slices for | Partition | FileId | Base-Instant | Data-File | Data-File Size| Num Delta Files| Total Delta Size| Delta Size - compaction scheduled| Delta Size - compaction unscheduled| Delta To Base Ratio - compaction scheduled| Delta To Base Ratio - compaction unscheduled| Delta Files - compaction scheduled | Delta Files - compaction unscheduled| |========================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================== [...] | 2018/08/31| 111415c3-f26d-4639-86c8-f9956f245ac3| 20181002180759| hdfs://namenode:8020/user/hive/warehouse/stock_ticks_mor/2018/08/31/111415c3-f26d-4639-86c8-f9956f245ac3_0_20181002180759.parquet| 432.5 KB | 1 | 20.8 KB | 20.8 KB | 0.0 B | 0.0 B | 0.0 B | [HoodieLogFile {hdfs://namenode:8020/user/hive/warehouse/stock_ticks_mor/2018/08/31/.111415c3-f26d-4639-86c8-f9956f245ac3_20181002180759.log.1}]| [] | - + hoodie:stock_ticks_mor-> ``` @@ -224,7 +227,7 @@ This is a sequence file that contains a mapping from commitNumber => json with r #### Compactions -To get an idea of the lag between compaction and writer applications, use the below command to list down all +To get an idea of the lag between compaction and writer applications, use the below command to list down all pending compactions. ``` @@ -316,7 +319,7 @@ hoodie:stock_ticks_mor->compaction validate --instant 20181005222611 ... COMPACTION PLAN VALID - + ___________________________________________________________________________________________________________________________________________________________________________________________________________________________ | File Id | Base Instant Time| Base Data File | Num Delta Files| Valid| Error| |==========================================================================================================================================================================================================================| @@ -340,14 +343,15 @@ hoodie:stock_ticks_mor->compaction validate --instant 20181005222601 The following commands must be executed without any other writer/ingestion application running. -Sometimes, it becomes necessary to remove a fileId from a compaction-plan inorder to speed-up or unblock compaction -operation. Any new log-files that happened on this file after the compaction got scheduled will be safely renamed +Sometimes, it becomes necessary to remove a fileId from a compaction-plan inorder to speed-up or unblock compaction +operation. Any new log-files that happened on this file after the compaction got scheduled will be safely renamed so that are preserved. Hudi provides the following CLI to support it ##### UnScheduling Compaction ``` + hoodie:trips->compaction unscheduleFileId --fileId <FileUUID> .... No File renames needed to unschedule file from pending compaction. Operation successful. @@ -356,24 +360,28 @@ No File renames needed to unschedule file from pending compaction. Operation suc In other cases, an entire compaction plan needs to be reverted. This is supported by the following CLI ``` + hoodie:trips->compaction unschedule --compactionInstant <compactionInstant> ..... No File renames needed to unschedule pending compaction. Operation successful. + ``` - + ##### Repair Compaction The above compaction unscheduling operations could sometimes fail partially (e:g -> HDFS temporarily unavailable). With -partial failures, the compaction operation could become inconsistent with the state of file-slices. When you run +partial failures, the compaction operation could become inconsistent with the state of file-slices. When you run `compaction validate`, you can notice invalid compaction operations if there is one. In these cases, the repair command comes to the rescue, it will rearrange the file-slices so that there is no loss and the file-slices are consistent with the compaction plan ``` + hoodie:stock_ticks_mor->compaction repair --instant 20181005222611 ...... Compaction successfully repaired ..... + ``` diff --git a/docs/api_docs.md b/docs/api_docs.md deleted file mode 100644 index 24bfd6b..0000000 --- a/docs/api_docs.md +++ /dev/null @@ -1,10 +0,0 @@ ---- -title: API Docs -keywords: usecases -sidebar: mydoc_sidebar -permalink: api_docs.html ---- - -Work In Progress - - diff --git a/docs/code_and_design.md b/docs/code_and_design.md deleted file mode 100644 index 3baaa97..0000000 --- a/docs/code_and_design.md +++ /dev/null @@ -1,38 +0,0 @@ ---- -title: Code Structure -keywords: usecases -sidebar: mydoc_sidebar -permalink: code_and_design.html ---- - -## Code & Project Structure - - * hoodie-client : Spark client library to take a bunch of inserts + updates and apply them to a Hoodie table - * hoodie-common : Common code shared between different artifacts of Hoodie - - ## HoodieLogFormat - - The following diagram depicts the LogFormat for Hoodie MergeOnRead. Each logfile consists of one or more log blocks. - Each logblock follows the format shown below. - - | Field | Description | - |-------------- |------------------| - | MAGIC | A magic header that marks the start of a block | - | VERSION | The version of the LogFormat, this helps define how to switch between different log format as it evolves | - | TYPE | The type of the log block | - | HEADER LENGTH | The length of the headers, 0 if no headers | - | HEADER | Metadata needed for a log block. For eg. INSTANT_TIME, TARGET_INSTANT_TIME, SCHEMA etc. | - | CONTENT LENGTH | The length of the content of the log block | - | CONTENT | The content of the log block, for example, for a DATA_BLOCK, the content is (number of records + actual records) in byte [] | - | FOOTER LENGTH | The length of the footers, 0 if no footers | - | FOOTER | Metadata needed for a log block. For eg. index entries, a bloom filter for records in a DATA_BLOCK etc. | - | LOGBLOCK LENGTH | The total number of bytes written for a log block, typically the SUM(everything_above). This is a LONG. This acts as a reverse pointer to be able to traverse the log in reverse.| - - - {% include image.html file="hoodie_log_format_v2.png" alt="hoodie_log_format_v2.png" %} - - - - - - diff --git a/docs/community.md b/docs/community.md index c508191..c16dc92 100644 --- a/docs/community.md +++ b/docs/community.md @@ -6,17 +6,35 @@ toc: false permalink: community.html --- +## Engage with us + +There are several ways to get in touch with the Hudi community. + +| When? | Channel to use | +|-------|--------| +| For any general questions, user support, development discussions | Dev Mailing list ([Subscribe](mailto:[email protected]), [Unsubscribe](mailto:[email protected]), [Archives](https://lists.apache.org/[email protected])). Empty email works for subscribe/unsubscribe | +| For reporting bugs or issues or discover known issues | Please use [ASF Hudi JIRA](https://issues.apache.org/jira/projects/HUDI/summary) | +| For quick pings & 1-1 chats | Join our [slack group](https://join.slack.com/t/apache-hudi/signup) | +| For proposing large features, changes | Start a Hudi Improvement Process (HIP). Instructions coming soon.| +| For stream of commits, pull requests etc | Commits Mailing list ([Subscribe](mailto:[email protected]), [Unsubscribe](mailto:[email protected]), [Archives](https://lists.apache.org/[email protected])) | + +If you wish to report a security vulnerability, please contact [[email protected]](mailto:[email protected]). +Apache Hudi follows the typical Apache vulnerability handling [process](https://apache.org/security/committers.html#vulnerability-handling). + ## Contributing -We :heart: contributions. If you find a bug in the library or would like to add new features, go ahead and open -issues or pull requests against this repo. Before you do so, please sign the -[Apache CLA](https://www.apache.org/licenses/icla.pdf). -Also, be sure to write unit tests for your bug fix or feature to show that it works as expected. -If the reviewer feels this contributions needs to be in the release notes, please add it to CHANGELOG.md as well. -If you want to participate in day-day conversations, please join our [slack group](https://join.slack.com/t/apache-hudi/signup). -If you are from select pre-listed email domains, you can self signup. Others, please subscribe to [email protected] +Apache Hudi community welcomes contributions from anyone! + +Here are few ways, you can get involved. + + - Ask (and/or) answer questions on our support channels listed above. + - Review code or HIPs + - Help improve documentation + - Testing; Improving out-of-box experience by reporting bugs + - Share new ideas/directions to pursue or propose a new HIP + - Contributing code to the project -## Becoming a Committer +#### Code Contributions -Hoodie has adopted a lot of guidelines set forth in [Google Chromium project](https://www.chromium.org/getting-involved/become-a-committer), to determine committership proposals. However, given this is a much younger project, we would have the contribution bar to be 10-15 non-trivial patches instead. -Additionally, we expect active engagement with the community over a few months, in terms of conference/meetup talks, helping out with issues/questions on slack/github. +Useful resources for contributing can be found under the "Developers" top menu. +Specifically, please refer to the detailed [contribution guide](contributing.html). diff --git a/docs/concepts.md b/docs/concepts.md index 5ce3fc6..845228a 100644 --- a/docs/concepts.md +++ b/docs/concepts.md @@ -20,7 +20,7 @@ Such key activities include * `COMMITS` - A single commit captures information about an **atomic write** of a batch of records into a dataset. Commits are identified by a monotonically increasing timestamp, denoting the start of the write operation. * `CLEANS` - Background activity that gets rid of older versions of files in the dataset, that are no longer needed. - * `DELTA_COMMITS` - A single commit captures information about an **atomic write** of a batch of records into a + * `DELTA_COMMITS` - A single commit captures information about an **atomic write** of a batch of records into a MergeOnRead storage type of dataset * `COMPACTIONS` - Background activity to reconcile differential data structures within Hudi e.g: moving updates from row based log files to columnar formats. @@ -37,15 +37,15 @@ only the changed files without say scanning all the time buckets > 07:00. ## Terminologies - * `Hudi Dataset` - A structured hive/spark dataset managed by Hudi. Hudi supports both partitioned and non-partitioned Hive tables. - * `Commit` - A commit marks a new batch of data applied to a dataset. Hudi maintains monotonically increasing timestamps to track commits and guarantees that a commit is atomically + * `Hudi Dataset` + A structured hive/spark dataset managed by Hudi. Hudi supports both partitioned and non-partitioned Hive tables. + * `Commit` + A commit marks a new batch of data applied to a dataset. Hudi maintains monotonically increasing timestamps to track commits and guarantees that a commit is atomically published. * `Commit Timeline` - Commit Timeline refers to the sequence of Commits that was applied in order on a dataset over its lifetime. - * `File Slice` - Hudi provides efficient handling of updates by having a fixed mapping between record key to a logical file Id. + Commit Timeline refers to the sequence of Commits that was applied in order on a dataset over its lifetime. + * `File Slice` + Hudi provides efficient handling of updates by having a fixed mapping between record key to a logical file Id. Hudi uses MVCC to provide atomicity and isolation of readers from a writer. This means that a logical fileId will have many physical versions of it. Each of these physical version of a file represents a complete view of the file as of a commit and is called File Slice @@ -69,8 +69,6 @@ Hudi (will) supports the following storage types. - Copy On Write : A heavily read optimized storage type, that simply creates new versions of files corresponding to the records that changed. - Merge On Read : Also provides a near-real time datasets in the order of 5 mins, by shifting some of the write cost, to the reads and merging incoming and on-disk data on-the-fly -{% include callout.html content="Hudi is a young project. merge-on-read is currently underway. Get involved [here](https://github.com/uber/Hudi/projects/1)" type="info" %} - Regardless of the storage type, Hudi organizes a datasets into a directory structure under a `basepath`, very similar to Hive tables. Dataset is broken up into partitions, which are folders containing files for that partition. Each partition uniquely identified by its `partitionpath`, which is relative to the basepath. @@ -92,12 +90,12 @@ commit, such that only columnar data exists. As a result, the write amplificatio Following illustrates how this works conceptually, when data written into copy-on-write storage and two queries running on top of it. -{% include image.html file="Hudi_cow.png" alt="Hudi_cow.png" %} +{% include image.html file="hudi_cow.png" alt="hudi_cow.png" %} As data gets written, updates to existing file ids, produce a new version for that file id stamped with the commit and inserts allocate a new file id and write its first version for that file id. These file versions and their commits are color coded above. -Normal SQL queries running against such dataset (eg: select count(*) counting the total records in that partition), first checks the timeline for latest commit +Normal SQL queries running against such dataset (eg: `select count(*)` counting the total records in that partition), first checks the timeline for latest commit and filters all but latest versions of each file id. As you can see, an old query does not see the current inflight commit's files colored in pink, but a new query starting after the commit picks up the new data. Thus queries are immune to any write failures/partial writes and only run on committed data. @@ -118,7 +116,7 @@ their columnar base data, to keep the query performance in check (larger append Following illustrates how the storage works, and shows queries on both near-real time table and read optimized table. -{% include image.html file="Hudi_mor.png" alt="Hudi_mor.png" max-width="1000" %} +{% include image.html file="hudi_mor.png" alt="hudi_mor.png" max-width="1000" %} There are lot of interesting things happening in this example, which bring out the subleties in the approach. @@ -135,8 +133,6 @@ There are lot of interesting things happening in this example, which bring out t strategy, where we aggressively compact the latest partitions compared to older partitions, we could ensure the RO Table sees data published within X minutes in a consistent fashion. -{% include callout.html content="Hudi is a young project. merge-on-read is currently underway. Get involved [here](https://github.com/uber/hoodie/projects/1)" type="info" %} - The intention of merge on read storage, is to enable near real-time processing directly on top of Hadoop, as opposed to copying data out to specialized systems, which may not be able to handle the data volume. @@ -156,4 +152,4 @@ data out to specialized systems, which may not be able to handle the data volume | Trade-off | ReadOptimized | RealTime | |-------------- |------------------| ------------------| | Data Latency | Higher | Lower | -| Query Latency | Lower (raw columnar performance) | Higher (merge columnar + row based delta) | \ No newline at end of file +| Query Latency | Lower (raw columnar performance) | Higher (merge columnar + row based delta) | diff --git a/docs/configurations.md b/docs/configurations.md index 50a7e5f..e6602e6 100644 --- a/docs/configurations.md +++ b/docs/configurations.md @@ -136,7 +136,7 @@ summary: "Here we list all possible configurations and what they mean" Actual value ontained by invoking .toString()</span> - [KEYGENERATOR_CLASS_OPT_KEY](#KEYGENERATOR_CLASS_OPT_KEY) (Default: com.uber.hoodie.SimpleKeyGenerator) <br/> <span style="color:grey">Key generator class, that implements will extract the key out of incoming `Row` object</span> - - [COMMIT_METADATA_KEYPREFIX_OPT_KEY](#COMMIT_METADATA_KEYPREFIX_OPT_KEY) (Default: _) <br/> + - [COMMIT_METADATA_KEYPREFIX_OPT_KEY](#COMMIT_METADATA_KEYPREFIX_OPT_KEY) (Default: `_`) <br/> <span style="color:grey">Option keys beginning with this prefix, are automatically added to the commit/deltacommit metadata. This is useful to store checkpointing information, in a consistent way with the hoodie timeline</span> @@ -160,22 +160,33 @@ summary: "Here we list all possible configurations and what they mean" Writing data via Hudi happens as a Spark job and thus general rules of spark debugging applies here too. Below is a list of things to keep in mind, if you are looking to improving performance or reliability. - - **Write operations** : Use `bulkinsert` to load new data into a table, and there on use `upsert`/`insert`. +**Write operations** : Use `bulkinsert` to load new data into a table, and there on use `upsert`/`insert`. Difference between them is that bulk insert uses a disk based write path to scale to load large inputs without need to cache it. - - **Input Parallelism** : By default, Hoodie tends to over-partition input (i.e `withParallelism(1500)`), to ensure each Spark partition stays within the 2GB limit for inputs upto 500GB. Bump this up accordingly if you have larger inputs. We recommend having shuffle parallelism `hoodie.[insert|upsert|bulkinsert].shuffle.parallelism` such that its atleast input_data_size/500MB - - **Off-heap memory** : Hoodie writes parquet files and that needs good amount of off-heap memory proportional to schema width. Consider setting something like `spark.yarn.executor.memoryOverhead` or `spark.yarn.driver.memoryOverhead`, if you are running into such failures. - - **Spark Memory** : Typically, hoodie needs to be able to read a single file into memory to perform merges or compactions and thus the executor memory should be sufficient to accomodate this. In addition, Hoodie caches the input to be able to intelligently place data and thus leaving some `spark.storage.memoryFraction` will generally help boost performance. - - **Sizing files** : Set `limitFileSize` above judiciously, to balance ingest/write latency vs number of files & consequently metadata overhead associated with it. - - **Timeseries/Log data** : Default configs are tuned for database/nosql changelogs where individual record sizes are large. Another very popular class of data is timeseries/event/log data that tends to be more volumnious with lot more records per partition. In such cases + +**Input Parallelism** : By default, Hoodie tends to over-partition input (i.e `withParallelism(1500)`), to ensure each Spark partition stays within the 2GB limit for inputs upto 500GB. Bump this up accordingly if you have larger inputs. We recommend having shuffle parallelism `hoodie.[insert|upsert|bulkinsert].shuffle.parallelism` such that its atleast input_data_size/500MB + +**Off-heap memory** : Hoodie writes parquet files and that needs good amount of off-heap memory proportional to schema width. Consider setting something like `spark.yarn.executor.memoryOverhead` or `spark.yarn.driver.memoryOverhead`, if you are running into such failures. + +**Spark Memory** : Typically, hoodie needs to be able to read a single file into memory to perform merges or compactions and thus the executor memory should be sufficient to accomodate this. In addition, Hoodie caches the input to be able to intelligently place data and thus leaving some `spark.storage.memoryFraction` will generally help boost performance. + +**Sizing files** : Set `limitFileSize` above judiciously, to balance ingest/write latency vs number of files & consequently metadata overhead associated with it. + +**Timeseries/Log data** : Default configs are tuned for database/nosql changelogs where individual record sizes are large. Another very popular class of data is timeseries/event/log data that tends to be more volumnious with lot more records per partition. In such cases - Consider tuning the bloom filter accuracy via `.bloomFilterFPP()/bloomFilterNumEntries()` to achieve your target index look up time - Consider making a key that is prefixed with time of the event, which will enable range pruning & significantly speeding up index lookup. - - **GC Tuning** : Please be sure to follow garbage collection tuning tips from Spark tuning guide to avoid OutOfMemory errors - - [Must] Use G1/CMS Collector. Sample CMS Flags to add to spark.executor.extraJavaOptions : ``-XX:NewSize=1g -XX:SurvivorRatio=2 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+PrintTenuringDistribution -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -XX:+PrintTenuringDistribution -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/ho [...] - - If it keeps OOMing still, reduce spark memory conservatively: `spark.memory.fraction=0.2, spark.memory.storageFraction=0.2` allowing it to spill rather than OOM. (reliably slow vs crashing intermittently) - Below is a full working production config +**GC Tuning** : Please be sure to follow garbage collection tuning tips from Spark tuning guide to avoid OutOfMemory errors +[Must] Use G1/CMS Collector. Sample CMS Flags to add to spark.executor.extraJavaOptions : - ``` +``` +-XX:NewSize=1g -XX:SurvivorRatio=2 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -XX:+PrintTenuringDistribution -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/hoodie-heapdump.hprof +```` + +If it keeps OOMing still, reduce spark memory conservatively: `spark.memory.fraction=0.2, spark.memory.storageFraction=0.2` allowing it to spill rather than OOM. (reliably slow vs crashing intermittently) + +Below is a full working production config + +``` spark.driver.extraClassPath /etc/hive/conf spark.driver.extraJavaOptions -XX:+PrintTenuringDistribution -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCTimeStamps -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/hoodie-heapdump.hprof spark.driver.maxResultSize 2g @@ -200,4 +211,5 @@ Writing data via Hudi happens as a Spark job and thus general rules of spark deb spark.yarn.driver.memoryOverhead 1024 spark.yarn.executor.memoryOverhead 3072 spark.yarn.max.executor.failures 100 - ``` + +```` diff --git a/docs/contributing.md b/docs/contributing.md new file mode 100644 index 0000000..a93ba54 --- /dev/null +++ b/docs/contributing.md @@ -0,0 +1,101 @@ +--- +title: Developer Setup +keywords: developer setup +sidebar: mydoc_sidebar +toc: false +permalink: contributing.html +--- +## Pre-requisites + +To contribute code, you need + + - a GitHub account + - a Linux (or) macOS development environment with Java JDK 8, Apache Maven (3.x+) installed + - [Docker](https://www.docker.com/) installed for running demo, integ tests or building website + - for large contributions, a signed [Individual Contributor License + Agreement](https://www.apache.org/licenses/icla.pdf) (ICLA) to the Apache + Software Foundation (ASF). + - (Recommended) Create an account on [JIRA](https://issues.apache.org/jira/projects/HUDI/summary) to open issues/find similar issues. + - (Recommended) Join our dev mailing list & slack channel, listed on [community](community.html) page. + + +## IDE Setup + +To contribute, you would need to fork the Hudi code on Github & then clone your own fork locally. Once cloned, we recommend building as per instructions on [quickstart](quickstart.html) + +We have embraced the code style largely based on [google format](https://google.github.io/styleguide/javaguide.html). Please setup your IDE with style files from [here](../style/). +These instructions have been tested on IntelliJ. We also recommend setting up the [Save Action Plugin](https://plugins.jetbrains.com/plugin/7642-save-actions) to auto format & organize imports on save. The Maven Compilation life-cycle will fail if there are checkstyle violations. + + +## Lifecycle + +Here's a typical lifecycle of events to contribute to Hudi. + + - [Recommended] Share your intent on the mailing list, so that community can provide early feedback, point out any similar JIRAs or HIPs. + - [Optional] If you want to get involved, but don't have a project in mind, please check JIRA for small, quick-starters. + - [Optional] Familiarize yourself with internals of Hudi using content on this page, as well as [wiki](https://cwiki.apache.org/confluence/display/HUDI) + - Once you finalize on a project/task, please open a new JIRA or assign an existing one to yourself. (If you don't have perms to do this, please email the dev mailing list with your JIRA id and a small intro for yourself. We'd be happy to add you as a contributor) + - Make your code change + - Every source file needs to include the Apache license header. Every new dependency needs to + have an open source license [compatible](https://www.apache.org/legal/resolved.html#criteria) with Apache. + - Get existing tests to pass using `mvn clean install -DskipITs` + - Add adequate tests for your new functionality + - [Optional] For involved changes, its best to also run the entire integration test suite using `mvn clean install` + - For website changes, please build the site locally & test navigation, formatting & links thoroughly + - Format commit messages and the pull request title like `[HUDI-XXX] Fixes bug in Spark Datasource`, + where you replace HUDI-XXX with the appropriate JIRA issue. + - Push your commit to your own fork/branch & create a pull request (PR) against the Hudi repo. + - If you don't hear back within 3 days on the PR, please send an email to dev @ mailing list. + - Address code review comments & keep pushing changes to your fork/branch, which automatically updates the PR + - Before your change can be merged, it should be squashed into a single commit for cleaner commit history. + + +## Releases + + - Apache Hudi community plans to do minor version releases every 6 weeks or so. + - If your contribution merged onto `master` branch after the last release, it will become part of next release. + - Website changes are regenerated once a week (until automation in place to reflect immediately) + + +## Accounts and Permissions + + - [Hudi issue tracker (JIRA)](https://issues.apache.org/jira/projects/HUDI/issues): + Anyone can access it and browse issues. Anyone can register an account and login + to create issues or add comments. Only contributors can be assigned issues. If + you want to be assigned issues, a PMC member can add you to the project contributor + group. Email the dev mailing list to ask to be added as a contributor, and include your ASF Jira username. + + - [Hudi Wiki Space](https://cwiki.apache.org/confluence/display/HUDI): + Anyone has read access. If you wish to contribute changes, please create an account and + request edit access on the dev@ mailing list (include your Wiki account user ID). + + - Pull requests can only be merged by a HUDI committer, listed [here](https://incubator.apache.org/projects/hudi.html) + + - [Voting on a release](https://www.apache.org/foundation/voting.html): Everyone can vote. + Only Hudi PMC members should mark their votes as binding. + +## Communication + +All communication is expected to align with the [Code of Conduct](https://www.apache.org/foundation/policies/conduct). +Discussion about contributing code to Hudi happens on the [dev@ mailing list](community.html). Introduce yourself! + + +## Code & Project Structure + + * `docker` : Docker containers used by demo and integration tests. Brings up a mini data ecosystem locally + * `hoodie-cli` : CLI to inspect, manage and administer datasets + * `hoodie-client` : Spark client library to take a bunch of inserts + updates and apply them to a Hoodie table + * `hoodie-common` : Common classes used across modules + * `hoodie-hadoop-mr` : InputFormat implementations for ReadOptimized, Incremental, Realtime views + * `hoodie-hive` : Manage hive tables off Hudi datasets and houses the HiveSyncTool + * `hoodie-integ-test` : Longer running integration test processes + * `hoodie-spark` : Spark datasource for writing and reading Hudi datasets. Streaming sink. + * `hoodie-utilities` : Houses tools like DeltaStreamer, SnapshotCopier + * `packaging` : Poms for building out bundles for easier drop in to Spark, Hive, Presto + * `style` : Code formatting, checkstyle files + + +## Website + +[Apache Hudi site](https://hudi.apache.org) is hosted on a special `asf-site` branch. Please follow the `README` file under `docs` on that branch for +instructions on making changes to the website. diff --git a/docs/dev_setup.md b/docs/dev_setup.md deleted file mode 100644 index 1bdeec7..0000000 --- a/docs/dev_setup.md +++ /dev/null @@ -1,13 +0,0 @@ ---- -title: Developer Setup -keywords: developer setup -sidebar: mydoc_sidebar -permalink: dev_setup.html ---- - -### Code Style - - We have embraced the code style largely based on [google format](https://google.github.io/styleguide/javaguide.html). - Please setup your IDE with style files from [here](../style/) - We also recommend setting up the [Save Action Plugin](https://plugins.jetbrains.com/plugin/7642-save-actions) to auto format & organize imports on save. - The Maven Compilation life-cycle will fail if there are checkstyle violations. diff --git a/docs/images/hoodie_cow.png b/docs/images/hoodie_cow.png deleted file mode 100644 index bad15a8..0000000 Binary files a/docs/images/hoodie_cow.png and /dev/null differ diff --git a/docs/images/hoodie_mor.png b/docs/images/hoodie_mor.png deleted file mode 100644 index 8d7d902..0000000 Binary files a/docs/images/hoodie_mor.png and /dev/null differ diff --git a/docs/images/hudi_cow.png b/docs/images/hudi_cow.png new file mode 100644 index 0000000..40aca71 Binary files /dev/null and b/docs/images/hudi_cow.png differ diff --git a/docs/images/hudi_mor.png b/docs/images/hudi_mor.png new file mode 100644 index 0000000..100b8f0 Binary files /dev/null and b/docs/images/hudi_mor.png differ diff --git a/docs/images/hoodie_timeline.png b/docs/images/hudi_timeline.png similarity index 100% rename from docs/images/hoodie_timeline.png rename to docs/images/hudi_timeline.png diff --git a/docs/implementation.md b/docs/implementation.md index 6215155..e87a541 100644 --- a/docs/implementation.md +++ b/docs/implementation.md @@ -23,7 +23,7 @@ Hudi upsert/insert is merely a Spark DAG, that can be broken into two big pieces Hudi currently provides two choices for indexes : `BloomIndex` and `HBaseIndex` to map a record key into the file id to which it belongs to. This enables us to speed up upserts significantly, without scanning over every record in the dataset. Hudi Indices can be classified based on -their ability to lookup records across partition. A `global` index does not need partition information for finding the file-id for a record key +their ability to lookup records across partition. A `global` index does not need partition information for finding the file-id for a record key but a `non-global` does. #### HBase Index (global) @@ -63,8 +63,8 @@ records such that In this storage, index updation is a no-op, since the bloom filters are already written as a part of committing data. -In the case of Copy-On-Write, a single parquet file constitutes one `file slice` which contains one complete version of -the file +In the case of Copy-On-Write, a single parquet file constitutes one `file slice` which contains one complete version of +the file {% include image.html file="hoodie_log_format_v2.png" alt="hoodie_log_format_v2.png" max-width="1000" %} @@ -73,27 +73,27 @@ the file In the Merge-On-Read storage model, there are 2 logical components - one for ingesting data (both inserts/updates) into the dataset and another for creating compacted views. The former is hereby referred to as `Writer` while the later is referred as `Compactor`. - + ##### Merge On Read Writer - + At a high level, Merge-On-Read Writer goes through same stages as Copy-On-Write writer in ingesting data. - The key difference here is that updates are appended to latest log (delta) file belonging to the latest file slice + The key difference here is that updates are appended to latest log (delta) file belonging to the latest file slice without merging. For inserts, Hudi supports 2 modes: 1. Inserts to Log Files - This is done for datasets that have an indexable log files (for eg global index) 2. Inserts to parquet files - This is done for datasets that do not have indexable log files, for eg bloom index embedded in parquer files. Hudi treats writing new records in the same way as inserting to Copy-On-Write files. -As in the case of Copy-On-Write, the input tagged records are partitioned such that all upserts destined to +As in the case of Copy-On-Write, the input tagged records are partitioned such that all upserts destined to a `file id` are grouped together. This upsert-batch is written as one or more log-blocks written to log-files. Hudi allows clients to control log file sizes (See [Storage Configs](../configurations)) The WriteClient API is same for both Copy-On-Write and Merge-On-Read writers. - + With Merge-On-Read, several rounds of data-writes would have resulted in accumulation of one or more log-files. All these log-files along with base-parquet (if exists) constitute a `file slice` which represents one complete version -of the file. - +of the file. + #### Compactor Realtime Readers will perform in-situ merge of these delta log-files to provide the most recent (committed) view of @@ -106,48 +106,52 @@ Asynchronous Compaction involves 2 steps: to be compacted atomically in a single compaction commit. Hudi allows pluggable strategies for choosing file slices for each compaction runs. This step is typically done inline by Writer process as Hudi expects only one schedule is being generated at a time which allows Hudi to enforce the constraint that pending compaction - plans do not step on each other file-slices. This constraint allows for multiple concurrent `Compactors` to run at + plans do not step on each other file-slices. This constraint allows for multiple concurrent `Compactors` to run at the same time. Some of the common strategies used for choosing `file slice` for compaction are: - * BoundedIO - Limit the number of file slices chosen for a compaction plan by expected total IO (read + write) - needed to complete compaction run + * BoundedIO - Limit the number of file slices chosen for a compaction plan by expected total IO (read + write) + needed to complete compaction run * Log File Size - Prefer file-slices with larger amounts of delta log data to be merged * Day Based - Prefer file slice belonging to latest day partitions - ``` - API for scheduling compaction - /** - * Schedules a new compaction instant - * @param extraMetadata - * @return Compaction Instant timestamp if a new compaction plan is scheduled - */ - Optional<String> scheduleCompaction(Optional<Map<String, String>> extraMetadata) throws IOException; - ``` + * `Compactor` : Hudi provides a separate API in Write Client to execute a compaction plan. The compaction plan (just like a commit) is identified by a timestamp. Most of the design and implementation complexities for Async Compaction is for guaranteeing snapshot isolation to readers and writer when multiple concurrent compactors are running. Typical compactor deployment involves launching a separate spark application which executes pending compactions when they become available. The core logic of compacting file slices in the Compactor is very similar to that of merging updates in a Copy-On-Write table. The only - difference being in the case of compaction, there is an additional step of merging the records in delta log-files. - - Here are the main API to lookup and execute a compaction plan. - ``` - Main API in HoodieWriteClient for running Compaction: - /** - * Performs Compaction corresponding to instant-time - * @param compactionInstantTime Compaction Instant Time - * @return - * @throws IOException - */ - public JavaRDD<WriteStatus> compact(String compactionInstantTime) throws IOException; - - To lookup all pending compactions, use the API defined in HoodieReadClient - - /** - * Return all pending compactions with instant time for clients to decide what to compact next. - * @return - */ - public List<Pair<String, HoodieCompactionPlan>> getPendingCompactions(); - ``` + difference being in the case of compaction, there is an additional step of merging the records in delta log-files. + +Here are the main API to lookup and execute a compaction plan. + +``` + Main API in HoodieWriteClient for running Compaction: + /** + * Performs Compaction corresponding to instant-time + * @param compactionInstantTime Compaction Instant Time + * @return + * @throws IOException + */ + public JavaRDD<WriteStatus> compact(String compactionInstantTime) throws IOException; + + To lookup all pending compactions, use the API defined in HoodieReadClient + + /** + * Return all pending compactions with instant time for clients to decide what to compact next. + * @return + */ + public List<Pair<String, HoodieCompactionPlan>> getPendingCompactions(); +``` +API for scheduling compaction + +``` + + /** + * Schedules a new compaction instant + * @param extraMetadata + * @return Compaction Instant timestamp if a new compaction plan is scheduled + */ + Optional<String> scheduleCompaction(Optional<Map<String, String>> extraMetadata) throws IOException; +``` Refer to __hoodie-client/src/test/java/HoodieClientExample.java__ class for an example of how compaction is scheduled and executed. @@ -172,65 +176,65 @@ plan to be run to figure out the number of file slices being compacted and choos ## Async Compaction Design Deep-Dive (Optional) -For the purpose of this section, it is important to distinguish between 2 types of commits as pertaining to the file-group: +For the purpose of this section, it is important to distinguish between 2 types of commits as pertaining to the file-group: A commit which generates a merged and read-optimized file-slice is called `snapshot commit` (SC) with respect to that file-group. -A commit which merely appended the new/updated records assigned to the file-group into a new log block is called `delta commit` (DC) +A commit which merely appended the new/updated records assigned to the file-group into a new log block is called `delta commit` (DC) with respect to that file-group. ### Algorithm The algorithm is described with an illustration. Let us assume a scenario where there are commits SC1, DC2, DC3 that have -already completed on a data-set. Commit DC4 is currently ongoing with the writer (ingestion) process using it to upsert data. -Let us also imagine there are a set of file-groups (FG1 … FGn) in the data-set whose latest version (`File-Slice`) -contains the base file created by commit SC1 (snapshot-commit in columnar format) and a log file containing row-based -log blocks of 2 delta-commits (DC2 and DC3). +already completed on a data-set. Commit DC4 is currently ongoing with the writer (ingestion) process using it to upsert data. +Let us also imagine there are a set of file-groups (FG1 … FGn) in the data-set whose latest version (`File-Slice`) +contains the base file created by commit SC1 (snapshot-commit in columnar format) and a log file containing row-based +log blocks of 2 delta-commits (DC2 and DC3). {% include image.html file="async_compac_1.png" alt="async_compac_1.png" max-width="1000" %} - * Writer (Ingestion) that is going to commit "DC4" starts. The record updates in this batch are grouped by file-groups - and appended in row formats to the corresponding log file as delta commit. Let us imagine a subset of file-groups has + * Writer (Ingestion) that is going to commit "DC4" starts. The record updates in this batch are grouped by file-groups + and appended in row formats to the corresponding log file as delta commit. Let us imagine a subset of file-groups has this new log block (delta commit) DC4 added. - * Before the writer job completes, it runs the compaction strategy to decide which file-group to compact by compactor - and creates a new compaction-request commit SC5. This commit file is marked as “requested” with metadata denoting - which fileIds to compact (based on selection policy). Writer completes without running compaction (will be run async). - + * Before the writer job completes, it runs the compaction strategy to decide which file-group to compact by compactor + and creates a new compaction-request commit SC5. This commit file is marked as “requested” with metadata denoting + which fileIds to compact (based on selection policy). Writer completes without running compaction (will be run async). + {% include image.html file="async_compac_2.png" alt="async_compac_2.png" max-width="1000" %} - - * Writer job runs again ingesting next batch. It starts with commit DC6. It reads the earliest inflight compaction - request marker commit in timeline order and collects the (fileId, Compaction Commit Id “CcId” ) pairs from meta-data. - Ingestion DC6 ensures a new file-slice with base-commit “CcId” gets allocated for the file-group. - The Writer will simply append records in row-format to the first log-file (as delta-commit) assuming the + + * Writer job runs again ingesting next batch. It starts with commit DC6. It reads the earliest inflight compaction + request marker commit in timeline order and collects the (fileId, Compaction Commit Id “CcId” ) pairs from meta-data. + Ingestion DC6 ensures a new file-slice with base-commit “CcId” gets allocated for the file-group. + The Writer will simply append records in row-format to the first log-file (as delta-commit) assuming the base-file (“Phantom-Base-File”) will be created eventually by the compactor. - + {% include image.html file="async_compac_3.png" alt="async_compac_3.png" max-width="1000" %} - - * Compactor runs at some time and commits at “Tc” (concurrently or before/after Ingestion DC6). It reads the commit-timeline - and finds the first unprocessed compaction request marker commit. Compactor reads the commit’s metadata finding the - file-slices to be compacted. It compacts the file-slice and creates the missing base-file (“Phantom-Base-File”) - with “CCId” as the commit-timestamp. Compactor then marks the compaction commit timestamp as completed. - It is important to realize that at data-set level, there could be different file-groups requesting compaction at + + * Compactor runs at some time and commits at “Tc” (concurrently or before/after Ingestion DC6). It reads the commit-timeline + and finds the first unprocessed compaction request marker commit. Compactor reads the commit’s metadata finding the + file-slices to be compacted. It compacts the file-slice and creates the missing base-file (“Phantom-Base-File”) + with “CCId” as the commit-timestamp. Compactor then marks the compaction commit timestamp as completed. + It is important to realize that at data-set level, there could be different file-groups requesting compaction at different commit timestamps. - + {% include image.html file="async_compac_4.png" alt="async_compac_4.png" max-width="1000" %} - * Near Real-time reader interested in getting the latest snapshot will have 2 cases. Let us assume that the + * Near Real-time reader interested in getting the latest snapshot will have 2 cases. Let us assume that the incremental ingestion (writer at DC6) happened before the compaction (some time “Tc”’). - The below description is with regards to compaction from file-group perspective. - * `Reader querying at time between ingestion completion time for DC6 and compaction finish “Tc”`: - Hoodie’s implementation will be changed to become aware of file-groups currently waiting for compaction and - merge log-files corresponding to DC2-DC6 with the base-file corresponding to SC1. In essence, Hudi will create - a pseudo file-slice by combining the 2 file-slices starting at base-commits SC1 and SC5 to one. - For file-groups not waiting for compaction, the reader behavior is essentially the same - read latest file-slice + The below description is with regards to compaction from file-group perspective. + * `Reader querying at time between ingestion completion time for DC6 and compaction finish “Tc”`: + Hoodie’s implementation will be changed to become aware of file-groups currently waiting for compaction and + merge log-files corresponding to DC2-DC6 with the base-file corresponding to SC1. In essence, Hudi will create + a pseudo file-slice by combining the 2 file-slices starting at base-commits SC1 and SC5 to one. + For file-groups not waiting for compaction, the reader behavior is essentially the same - read latest file-slice and merge on the fly. - * `Reader querying at time after compaction finished (> “Tc”)` : In this case, reader will not find any pending - compactions in the timeline and will simply have the current behavior of reading the latest file-slice and + * `Reader querying at time after compaction finished (> “Tc”)` : In this case, reader will not find any pending + compactions in the timeline and will simply have the current behavior of reading the latest file-slice and merging on-the-fly. - - * Read-Optimized View readers will query against the latest columnar base-file for each file-groups. + + * Read-Optimized View readers will query against the latest columnar base-file for each file-groups. The above algorithm explains Async compaction w.r.t a single compaction run on a single file-group. It is important -to note that multiple compaction plans can be run concurrently as they are essentially operating on different +to note that multiple compaction plans can be run concurrently as they are essentially operating on different file-groups. ## Performance @@ -272,4 +276,3 @@ with no impact on queries. Following charts compare the Hudi vs non-Hudi dataset **Presto** {% include image.html file="hoodie_query_perf_presto.png" alt="hoodie_query_perf_presto.png" max-width="1000" %} - diff --git a/docs/index.md b/docs/index.md index b5b9da7..ad87933 100644 --- a/docs/index.md +++ b/docs/index.md @@ -4,12 +4,9 @@ keywords: homepage tags: [getting_started] sidebar: mydoc_sidebar permalink: index.html -summary: "Hudi lowers data latency across the board, while simultaneously achieving orders of magnitude of efficiency over traditional batch processing." +summary: "Hudi brings stream processing to big data, providing fresh data while being an order of magnitude efficient over traditional batch processing." --- - - - Hudi (pronounced “Hoodie”) ingests & manages storage of large analytical datasets on [HDFS](http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html) or cloud stores and provides three logical views for query access. * **Read Optimized View** - Provides excellent query performance on pure columnar storage, much like plain [Parquet](https://parquet.apache.org/) tables. @@ -21,4 +18,4 @@ Hudi (pronounced “Hoodie”) ingests & manages storage of large analytical dat By carefully managing how data is laid out in storage & how it’s exposed to queries, Hudi is able to power a rich data ecosystem where external sources can be ingested in near real-time and made available for interactive SQL Engines like [Presto](https://prestodb.io) & [Spark](https://spark.apache.org/sql/), while at the same time capable of being consumed incrementally from processing/ETL frameworks like [Hive](https://hive.apache.org/) & [Spark](https://spark.apache.org/docs/latest/) t [...] -Hudi broadly consists of a self contained Spark library to build datasets and integrations with existing query engines for data access. +Hudi broadly consists of a self contained Spark library to build datasets and integrations with existing query engines for data access. See [quickstart](quickstart.html) for a demo. diff --git a/docs/migration_guide.md b/docs/migration_guide.md index a5d5506..13c27ac 100644 --- a/docs/migration_guide.md +++ b/docs/migration_guide.md @@ -4,9 +4,8 @@ keywords: migration guide sidebar: mydoc_sidebar permalink: migration_guide.html toc: false -summary: In this page, we will discuss some available tools for migrating your existing dataset into a Hudi managed -dataset - +summary: In this page, we will discuss some available tools for migrating your existing dataset into a Hudi dataset +--- Hudi maintains metadata such as commit timeline and indexes to manage a dataset. The commit timelines helps to understand the actions happening on a dataset as well as the current state of a dataset. Indexes are used by Hudi to maintain a record key to file id mapping to efficiently locate a record. At the moment, Hudi supports writing only parquet columnar formats. To be able to start using Hudi for your existing dataset, you will need to migrate your existing dataset into a Hudi managed dataset. There are a couple of ways to achieve this. @@ -15,57 +14,60 @@ To be able to start using Hudi for your existing dataset, you will need to migra ## Approaches -### Approach 1 +#### Use Hudi for new partitions alone -Hudi can be used to manage an existing dataset without affecting/altering the historical data already present in the -dataset. Hudi has been implemented to be compatible with such a mixed dataset with a caveat that either the complete -Hive partition is Hudi managed or not. Thus the lowest granularity at which Hudi manages a dataset is a Hive -partition. Start using the datasource API or the WriteClient to write to the dataset and make sure you start writing +Hudi can be used to manage an existing dataset without affecting/altering the historical data already present in the +dataset. Hudi has been implemented to be compatible with such a mixed dataset with a caveat that either the complete +Hive partition is Hudi managed or not. Thus the lowest granularity at which Hudi manages a dataset is a Hive +partition. Start using the datasource API or the WriteClient to write to the dataset and make sure you start writing to a new partition or convert your last N partitions into Hudi instead of the entire table. Note, since the historical - partitions are not managed by HUDI, none of the primitives provided by HUDI work on the data in those partitions. More concretely, one cannot perform upserts or incremental pull on such older partitions not managed by the HUDI dataset. + partitions are not managed by HUDI, none of the primitives provided by HUDI work on the data in those partitions. More concretely, one cannot perform upserts or incremental pull on such older partitions not managed by the HUDI dataset. Take this approach if your dataset is an append only type of dataset and you do not expect to perform any updates to existing (or non Hudi managed) partitions. -### Approach 2 +#### Convert existing dataset to Hudi Import your existing dataset into a Hudi managed dataset. Since all the data is Hudi managed, none of the limitations - of Approach 1 apply here. Updates spanning any partitions can be applied to this dataset and Hudi will efficiently - make the update available to queries. Note that not only do you get to use all Hoodie primitives on this dataset, + of Approach 1 apply here. Updates spanning any partitions can be applied to this dataset and Hudi will efficiently + make the update available to queries. Note that not only do you get to use all Hoodie primitives on this dataset, there are other additional advantages of doing this. Hudi automatically manages file sizes of a Hudi managed dataset - . You can define the desired file size when converting this dataset and Hudi will ensure it writes out files - adhering to the config. It will also ensure that smaller files later get corrected by routing some new inserts into + . You can define the desired file size when converting this dataset and Hudi will ensure it writes out files + adhering to the config. It will also ensure that smaller files later get corrected by routing some new inserts into small files rather than writing new small ones thus maintaining the health of your cluster. There are a few options when choosing this approach. + #### Option 1 -Use the HDFSParquetImporter tool. As the name suggests, this only works if your existing dataset is in -parquet file -format. This tool essentially starts a Spark Job to read the existing parquet dataset and converts it into a HUDI managed dataset by re-writing all the data. -#### Option 2 +Use the HDFSParquetImporter tool. As the name suggests, this only works if your existing dataset is in +parquet file +format. This tool essentially starts a Spark Job to read the existing parquet dataset and converts it into a HUDI managed dataset by re-writing all the data. + +#### Option 2 For huge datasets, this could be as simple as : for partition in [list of partitions in source dataset] { val inputDF = spark.read.format("any_input_format").load("partition_path") inputDF.write.format("com.uber.hoodie").option()....save("basePath") } + #### Option 3 Write your own custom logic of how to load an existing dataset into a Hudi managed one. Please read about the RDD API - [here](quickstart.md). + [here](quickstart.html). ``` -Using the HDFSParquetImporter Tool. Once hoodie has been built via `mvn clean install -DskipTests`, the shell can be +Using the HDFSParquetImporter Tool. Once hoodie has been built via `mvn clean install -DskipTests`, the shell can be fired by via `cd hoodie-cli && ./hoodie-cli.sh`. -hoodie->hdfsparquetimport - --upsert false - --srcPath /user/parquet/dataset/basepath - --targetPath - /user/hoodie/dataset/basepath - --tableName hoodie_table - --tableType COPY_ON_WRITE - --rowKeyField _row_key - --partitionPathField partitionStr - --parallelism 1500 - --schemaFilePath /user/table/schema - --format parquet - --sparkMemory 6g +hoodie->hdfsparquetimport + --upsert false + --srcPath /user/parquet/dataset/basepath + --targetPath + /user/hoodie/dataset/basepath + --tableName hoodie_table + --tableType COPY_ON_WRITE + --rowKeyField _row_key + --partitionPathField partitionStr + --parallelism 1500 + --schemaFilePath /user/table/schema + --format parquet + --sparkMemory 6g --retry 2 -``` \ No newline at end of file +``` diff --git a/docs/quickstart.md b/docs/quickstart.md index f1516ae..1e6fa49 100644 --- a/docs/quickstart.md +++ b/docs/quickstart.md @@ -13,13 +13,14 @@ permalink: quickstart.html Check out code and pull it into Intellij as a normal maven project. Normally build the maven project, from command line + ``` $ mvn clean install -DskipTests -DskipITs +``` To work with older version of Hive (pre Hive-1.2.1), use - +``` $ mvn clean install -DskipTests -DskipITs -Dhive11 - ``` {% include callout.html content="You might want to add your spark jars folder to project dependencies under 'Module Setttings', to be able to run Spark from IDE" type="info" %} @@ -31,13 +32,13 @@ $ mvn clean install -DskipTests -DskipITs -Dhive11 Hudi requires Java 8 to be installed. Hudi works with Spark-2.x versions. We have verified that Hudi works with the following combination of Hadoop/Hive/Spark. -| Hadoop | Hive | Spark | Instructions to Build Hudi | +| Hadoop | Hive | Spark | Instructions to Build Hudi | | ---- | ----- | ---- | ---- | | 2.6.0-cdh5.7.2 | 1.1.0-cdh5.7.2 | spark-2.[1-3].x | Use “mvn clean install -DskipTests -Dhadoop.version=2.6.0-cdh5.7.2 -Dhive.version=1.1.0-cdh5.7.2” | | Apache hadoop-2.8.4 | Apache hive-2.3.3 | spark-2.[1-3].x | Use "mvn clean install -DskipTests" | | Apache hadoop-2.7.3 | Apache hive-1.2.1 | spark-2.[1-3].x | Use "mvn clean install -DskipTests" | -If your environment has other versions of hadoop/hive/spark, please try out Hudi and let us know if there are any issues. We are limited by our bandwidth to certify other combinations. +If your environment has other versions of hadoop/hive/spark, please try out Hudi and let us know if there are any issues. We are limited by our bandwidth to certify other combinations. It would be of great help if you can reach out to us with your setup and experience with hoodie. ## Generate a Hudi Dataset @@ -60,7 +61,7 @@ export PATH=$JAVA_HOME/bin:$HIVE_HOME/bin:$HADOOP_HOME/bin:$SPARK_INSTALL/bin:$P ### Supported API's -Use the DataSource API to quickly start reading or writing Hudi datasets in few lines of code. Ideal for most +Use the DataSource API to quickly start reading or writing Hudi datasets in few lines of code. Ideal for most ingestion use-cases. Use the RDD API to perform more involved actions on a Hudi dataset @@ -132,11 +133,11 @@ This can be run as frequently as the ingestion pipeline to make sure new partiti cd hoodie-hive ./run_sync_tool.sh --user hive - --pass hive - --database default - --jdbc-url "jdbc:hive2://localhost:10010/" - --base-path tmp/hoodie/sample-table/ - --table hoodie_test + --pass hive + --database default + --jdbc-url "jdbc:hive2://localhost:10010/" + --base-path tmp/hoodie/sample-table/ + --table hoodie_test --partitioned-by field1,field2 ``` @@ -304,7 +305,7 @@ hive> ## A Demo using docker containers Lets use a real world example to see how hudi works end to end. For this purpose, a self contained -data infrastructure is brought up in a local docker cluster within your computer. +data infrastructure is brought up in a local docker cluster within your computer. The steps assume you are using Mac laptop @@ -313,7 +314,7 @@ The steps assume you are using Mac laptop * Docker Setup : For Mac, Please follow the steps as defined in [https://docs.docker.com/v17.12/docker-for-mac/install/]. For running Spark-SQL queries, please ensure atleast 6 GB and 4 CPUs are allocated to Docker (See Docker -> Preferences -> Advanced). Otherwise, spark-SQL queries could be killed because of memory issues. * kafkacat : A command-line utility to publish/consume from kafka topics. Use `brew install kafkacat` to install kafkacat * /etc/hosts : The demo references many services running in container by the hostname. Add the following settings to /etc/hosts - + ``` 127.0.0.1 adhoc-1 127.0.0.1 adhoc-2 @@ -378,15 +379,15 @@ At this point, the docker cluster will be up and running. The demo cluster bring * HDFS Services (NameNode, DataNode) * Spark Master and Worker * Hive Services (Metastore, HiveServer2 along with PostgresDB) - * Kafka Broker and a Zookeeper Node (Kakfa will be used as upstream source for the demo) + * Kafka Broker and a Zookeeper Node (Kakfa will be used as upstream source for the demo) * Adhoc containers to run Hudi/Hive CLI commands ### Demo -Stock Tracker data will be used to showcase both different Hudi Views and the effects of Compaction. +Stock Tracker data will be used to showcase both different Hudi Views and the effects of Compaction. -Take a look at the directory `docker/demo/data`. There are 2 batches of stock data - each at 1 minute granularity. -The first batch contains stocker tracker data for some stock symbols during the first hour of trading window +Take a look at the directory `docker/demo/data`. There are 2 batches of stock data - each at 1 minute granularity. +The first batch contains stocker tracker data for some stock symbols during the first hour of trading window (9:30 a.m to 10:30 a.m). The second batch contains tracker data for next 30 mins (10:30 - 11 a.m). Hudi will be used to ingest these batches to a dataset which will contain the latest stock tracker data at hour level granularity. The batches are windowed intentionally so that the second batch contains updates to some of the rows in the first batch. @@ -396,7 +397,7 @@ The batches are windowed intentionally so that the second batch contains updates Upload the first batch to Kafka topic 'stock ticks' ``` -cat docker/demo/data/batch_1.json | kafkacat -b kafkabroker -t stock_ticks -P +cat docker/demo/data/batch_1.json | kafkacat -b kafkabroker -t stock_ticks -P To check if the new topic shows up, use kafkacat -b kafkabroker -L -J | jq . @@ -443,7 +444,7 @@ kafkacat -b kafkabroker -L -J | jq . Hudi comes with a tool named DeltaStreamer. This tool can connect to variety of data sources (including Kafka) to pull changes and apply to Hudi dataset using upsert/insert primitives. Here, we will use the tool to download -json data from kafka topic and ingest to both COW and MOR tables we initialized in the previous step. This tool +json data from kafka topic and ingest to both COW and MOR tables we initialized in the previous step. This tool automatically initializes the datasets in the file-system if they do not exist yet. ``` @@ -468,8 +469,8 @@ spark-submit --class com.uber.hoodie.utilities.deltastreamer.HoodieDeltaStreamer exit ``` -You can use HDFS web-browser to look at the datasets -`http://namenode:50070/explorer.html#/user/hive/warehouse/stock_ticks_cow`. +You can use HDFS web-browser to look at the datasets +`http://namenode:50070/explorer.html#/user/hive/warehouse/stock_ticks_cow`. You can explore the new partition folder created in the dataset along with a "deltacommit" file under .hoodie which signals a successful commit. @@ -501,7 +502,7 @@ docker exec -it adhoc-2 /bin/bash .... exit ``` -After executing the above command, you will notice +After executing the above command, you will notice 1. A hive table named `stock_ticks_cow` created which provides Read-Optimized view for the Copy On Write dataset. 2. Two new tables `stock_ticks_mor` and `stock_ticks_mor_rt` created for the Merge On Read dataset. The former @@ -511,7 +512,7 @@ provides the ReadOptimized view for the Hudi dataset and the later provides the #### Step 4 (a): Run Hive Queries Run a hive query to find the latest timestamp ingested for stock symbol 'GOOG'. You will notice that both read-optimized -(for both COW and MOR dataset)and realtime views (for MOR dataset)give the same value "10:29 a.m" as Hudi create a +(for both COW and MOR dataset)and realtime views (for MOR dataset)give the same value "10:29 a.m" as Hudi create a parquet file for the first batch of data. ``` @@ -565,7 +566,7 @@ Now, run a projection query: # Merge-On-Read Queries: ========================== -Lets run similar queries against M-O-R dataset. Lets look at both +Lets run similar queries against M-O-R dataset. Lets look at both ReadOptimized and Realtime views supported by M-O-R dataset # Run against ReadOptimized View. Notice that the latest timestamp is 10:29 @@ -670,7 +671,7 @@ scala> spark.sql("select `_hoodie_commit_time`, symbol, ts, volume, open, close # Merge-On-Read Queries: ========================== -Lets run similar queries against M-O-R dataset. Lets look at both +Lets run similar queries against M-O-R dataset. Lets look at both ReadOptimized and Realtime views supported by M-O-R dataset # Run against ReadOptimized View. Notice that the latest timestamp is 10:29 @@ -718,7 +719,7 @@ Upload the second batch of data and ingest this batch using delta-streamer. As t partitions, there is no need to run hive-sync ``` -cat docker/demo/data/batch_2.json | kafkacat -b kafkabroker -t stock_ticks -P +cat docker/demo/data/batch_2.json | kafkacat -b kafkabroker -t stock_ticks -P # Within Docker container, run the ingestion command docker exec -it adhoc-2 /bin/bash @@ -734,15 +735,15 @@ exit With Copy-On-Write table, the second ingestion by DeltaStreamer resulted in a new version of Parquet file getting created. See `http://namenode:50070/explorer.html#/user/hive/warehouse/stock_ticks_cow/2018/08/31` -With Merge-On-Read table, the second ingestion merely appended the batch to an unmerged delta (log) file. +With Merge-On-Read table, the second ingestion merely appended the batch to an unmerged delta (log) file. Take a look at the HDFS filesystem to get an idea: `http://namenode:50070/explorer.html#/user/hive/warehouse/stock_ticks_mor/2018/08/31` #### Step 6(a): Run Hive Queries -With Copy-On-Write table, the read-optimized view immediately sees the changes as part of second batch once the batch -got committed as each ingestion creates newer versions of parquet files. +With Copy-On-Write table, the read-optimized view immediately sees the changes as part of second batch once the batch +got committed as each ingestion creates newer versions of parquet files. -With Merge-On-Read table, the second ingestion merely appended the batch to an unmerged delta (log) file. +With Merge-On-Read table, the second ingestion merely appended the batch to an unmerged delta (log) file. This is the time, when ReadOptimized and Realtime views will provide different results. ReadOptimized view will still return "10:29 am" as it will only read from the Parquet file. Realtime View will do on-the-fly merge and return latest committed data which is "10:59 a.m". @@ -773,7 +774,7 @@ WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the futu As you can notice, the above queries now reflect the changes that came as part of ingesting second batch. -# Merge On Read Table: +# Merge On Read Table: # Read Optimized View 0: jdbc:hive2://hiveserver:10000> select symbol, max(ts) from stock_ticks_mor group by symbol HAVING symbol = 'GOOG'; @@ -843,7 +844,7 @@ scala> spark.sql("select `_hoodie_commit_time`, symbol, ts, volume, open, close As you can notice, the above queries now reflect the changes that came as part of ingesting second batch. -# Merge On Read Table: +# Merge On Read Table: # Read Optimized View scala> spark.sql("select symbol, max(ts) from stock_ticks_mor group by symbol HAVING symbol = 'GOOG'").show(100, false) @@ -909,8 +910,8 @@ To show the effects of incremental-query, let us assume that a reader has alread ingesting first batch. Now, for the reader to see effect of the second batch, he/she has to keep the start timestamp to the commit time of the first batch (20180924064621) and run incremental query -`Hudi incremental mode` provides efficient scanning for incremental queries by filtering out files that do not have any -candidate rows using hudi-managed metadata. +`Hudi incremental mode` provides efficient scanning for incremental queries by filtering out files that do not have any +candidate rows using hudi-managed metadata. ``` docker exec -it adhoc-2 /bin/bash @@ -1008,7 +1009,7 @@ hoodie:stock_ticks_mor->compactions show all ___________________________________________________________________ | Compaction Instant Time| State | Total FileIds to be Compacted| |==================================================================| - + # Schedule a compaction. This will use Spark Launcher to schedule compaction hoodie:stock_ticks_mor->compaction schedule .... @@ -1028,7 +1029,7 @@ hoodie:stock_ticks_mor->compactions show all ___________________________________________________________________ | Compaction Instant Time| State | Total FileIds to be Compacted| |==================================================================| - | 20180924070031 | REQUESTED| 1 | + | 20180924070031 | REQUESTED| 1 | # Execute the compaction. The compaction instant value passed below must be the one displayed in the above "compactions show all" query hoodie:stock_ticks_mor->compaction run --compactionInstant 20180924070031 --parallelism 2 --sparkMemory 1G --schemaFilePath /var/demo/config/schema.avsc --retry 1 @@ -1052,7 +1053,7 @@ hoodie:stock_ticks->compactions show all |==================================================================| | 20180924070031 | COMPLETED| 1 | -``` +``` #### Step 9: Run Hive Queries including incremental queries @@ -1169,9 +1170,9 @@ You can bring up a hadoop docker environment containing Hadoop, Hive and Spark s ``` $ mvn pre-integration-test -DskipTests ``` -The above command builds docker images for all the services with -current Hudi source installed at /var/hoodie/ws and also brings up the services using a compose file. We -currently use Hadoop (v2.8.4), Hive (v2.3.3) and Spark (v2.3.1) in docker images. +The above command builds docker images for all the services with +current Hudi source installed at /var/hoodie/ws and also brings up the services using a compose file. We +currently use Hadoop (v2.8.4), Hive (v2.3.3) and Spark (v2.3.1) in docker images. To bring down the containers ``` @@ -1185,9 +1186,9 @@ $ cd hoodie-integ-test $ mvn docker-compose:up -DdetachedMode=true ``` -Hudi is a library that is operated in a broader data analytics/ingestion environment +Hudi is a library that is operated in a broader data analytics/ingestion environment involving Hadoop, Hive and Spark. Interoperability with all these systems is a key objective for us. We are -actively adding integration-tests under __hoodie-integ-test/src/test/java__ that makes use of this +actively adding integration-tests under __hoodie-integ-test/src/test/java__ that makes use of this docker environment (See __hoodie-integ-test/src/test/java/com/uber/hoodie/integ/ITTestHoodieSanity.java__ ) @@ -1202,10 +1203,10 @@ and compose scripts are carefully implemented so that they serve dual-purpose inbuilt jars by mounting local HUDI workspace over the docker location This helps avoid maintaining separate docker images and avoids the costly step of building HUDI docker images locally. -But if users want to test hudi from locations with lower network bandwidth, they can still build local images -run the script +But if users want to test hudi from locations with lower network bandwidth, they can still build local images +run the script `docker/build_local_docker_images.sh` to build local docker images before running `docker/setup_demo.sh` - + Here are the commands: ``` diff --git a/docs/roadmap.md b/docs/roadmap.md deleted file mode 100644 index c65c3a9..0000000 --- a/docs/roadmap.md +++ /dev/null @@ -1,14 +0,0 @@ ---- -title: Roadmap -keywords: usecases -sidebar: mydoc_sidebar -permalink: roadmap.html ---- - -## Planned Features - -* Support for Self Joins - As of now, you cannot incrementally consume the same table more than once, since the InputFormat does not understand the QueryPlan. -* Hudi Spark Datasource - Allows for reading and writing data back using Apache Spark natively (without falling back to InputFormat), which can be more performant -* Hudi Presto Connector - Allows for querying data managed by Hudi using Presto natively, which can again boost [performance](https://prestodb.io/docs/current/release/release-0.138.html) - - diff --git a/docs/sql_queries.md b/docs/sql_queries.md index 955e794..44848eb 100644 --- a/docs/sql_queries.md +++ b/docs/sql_queries.md @@ -62,7 +62,4 @@ spark.sparkContext.hadoopConfiguration.setClass("mapreduce.input.pathFilter.clas ## Presto -Presto requires a [patch](https://github.com/prestodb/presto/pull/7002) (until the PR is merged) and the hoodie-hadoop-mr-bundle jar to be placed -into `<presto_install>/plugin/hive-hadoop2/`. - -{% include callout.html content="Get involved to improve this integration [here](https://github.com/uber/hoodie/issues/81)" type="info" %} +Presto requires the `hoodie-presto-bundle` jar to be placed into `<presto_install>/plugin/hive-hadoop2/`, across the installation.
