yanghua commented on a change in pull request #1191: [HUDI-503] Add hudi test suite documentation into the README file of the test suite module URL: https://github.com/apache/incubator-hudi/pull/1191#discussion_r363690393
########## File path: hudi-test-suite/README.md ########## @@ -0,0 +1,291 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +This page describes in detail how to run end to end tests on a hudi dataset that helps in improving our confidence +in a release as well as perform large scale performance benchmarks. + +# Objectives + +1. Test with different versions of core libraries and components such as `hdfs`, `parquet`, `spark`, +`hive` and `avro`. +2. Generate different types of workloads across different dimensions such as `payload size`, `number of updates`, +`number of inserts`, `number of partitions` +3. Perform multiple types of operations such as `insert`, `bulk_insert`, `upsert`, `compact`, `query` +4. Support custom post process actions and validations + +# High Level Design + +The Hudi test suite runs as a long running spark job. The suite is divided into the following high level components : + +## Workload Generation + +This component does the work of generating the workload; `inserts`, `upserts` etc. + +## Workload Scheduling + +Depending on the type of workload generated, data is either ingested into the target hudi +dataset or the corresponding workload operation is executed. For example compaction does not necessarily need a workload +to be generated/ingested but can require an execution. + +## Other actions/operatons + +The test suite supports different types of operations besides ingestion such as Hive Query execution, Clean action etc. + +# Usage instructions + + +## Entry class to the test suite + +``` +org.apache.hudi.bench.job.HudiTestSuiteJob.java - Entry Point of the hudi test suite job. This +class wraps all the functionalities required to run a configurable integration suite. +``` + +## Configurations required to run the job +``` +org.apache.hudi.bench.job.HudiTestSuiteConfig - Config class that drives the behavior of the +integration test suite. This class extends from com.uber.hoodie.utilities.DeltaStreamerConfig. Look at +link#HudiDeltaStreamer page to learn about all the available configs applicable to your test suite. +``` + +## Generating a custom Workload Pattern +``` +There are 2 ways to generate a workload pattern +1. Programatically +Choose to write up the entire DAG of operations programatically, take a look at WorkflowDagGenerator class. +Once you're ready with the DAG you want to execute, simply pass the class name as follows +spark-submit +... +... +--class org.apache.hudi.bench.job.HudiTestSuiteJob +--workload-generator-classname org.apache.hudi.bench.dag.scheduler.<your_workflowdaggenerator> +... +2. YAML file +Choose to write up the entire DAG of operations in YAML, take a look at complex-workload-dag-cow.yaml or +complex-workload-dag-mor.yaml. +Once you're ready with the DAG you want to execute, simply pass the yaml file path as follows +spark-submit +... +... +--class org.apache.hudi.bench.job.HudiTestSuiteJob +--workload-yaml-path /path/to/your-workflow-dag.yaml +... +``` Review comment: Thanks for your suggestion. I have addressed it. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
