data-storyteller commented on a change in pull request #3482: URL: https://github.com/apache/hudi/pull/3482#discussion_r691588664
########## File path: hudi-integ-test/README.md ########## @@ -253,23 +254,119 @@ spark-submit \ --conf spark.network.timeout=600s \ --conf spark.yarn.max.executor.failures=10 \ --conf spark.sql.catalogImplementation=hive \ +--conf spark.driver.extraClassPath=/var/demo/jars/* \ +--conf spark.executor.extraClassPath=/var/demo/jars/* \ --class org.apache.hudi.integ.testsuite.HoodieTestSuiteJob \ -/opt/hudi-integ-test-bundle-0.8.0-SNAPSHOT.jar \ +/opt/hudi-integ-test-bundle-0.9.0-SNAPSHOT.jar \ --source-ordering-field test_suite_source_ordering_field \ --use-deltastreamer \ --target-base-path /user/hive/warehouse/hudi-integ-test-suite/output \ --input-base-path /user/hive/warehouse/hudi-integ-test-suite/input \ --target-table table1 \ --props file:/var/hoodie/ws/docker/demo/config/test-suite/test.properties \ ---schemaprovider-class org.apache.hudi.utilities.schema.FilebasedSchemaProvider \ +--schemaprovider-class org.apache.hudi.integ.testsuite.schema.TestSuiteFileBasedSchemaProvider \ --source-class org.apache.hudi.utilities.sources.AvroDFSSource \ --input-file-size 125829120 \ --workload-yaml-path file:/var/hoodie/ws/docker/demo/config/test-suite/complex-dag-mor.yaml \ --workload-generator-classname org.apache.hudi.integ.testsuite.dag.WorkflowDagGenerator \ --table-type MERGE_ON_READ \ ---compact-scheduling-minshare 1 +--compact-scheduling-minshare 1 \ +--hoodie-conf hoodie.metrics.on=true \ +--hoodie-conf hoodie.metrics.reporter.type=GRAPHITE \ +--hoodie-conf hoodie.metrics.graphite.host=graphite \ +--hoodie-conf hoodie.metrics.graphite.port=2003 \ +--clean-input \ +--clean-output ``` +## Visualize and inspect the hoodie metrics and performance (local) +Graphite server is already setup (and up) in ```docker/setup_demo.sh```. + +Open browser and access metrics at +``` +http://localhost:80 +``` +Dashboard +``` +http://localhost/dashboard + +``` + +## Running test suite on an EMR cluster +- Copy over the necessary files and jars that are required to your cluster. Review comment: Done. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
