[GitHub] flink pull request #6172: [FLINK-9594][E2E tests] Improve docs for new test ...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/6172 ---
[GitHub] flink pull request #6172: [FLINK-9594][E2E tests] Improve docs for new test ...
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/6172#discussion_r195992732 --- Diff: flink-end-to-end-tests/README.md --- @@ -33,6 +33,53 @@ $ FLINK_DIR= flink-end-to-end-tests/test-scripts/test_batch_wordcount ## Writing Tests -Have a look at test_batch_wordcount.sh for a very basic test and -test_streaming_kafka010.sh for a more involved example. Whenever possible, try -to put new functionality in common.sh so that it can be reused by other tests. +### Examples +Have a look at `test_batch_wordcount.sh` for a very basic test and +`test_streaming_kafka010.sh` for a more involved example. Whenever possible, try +to put new functionality in `common.sh` so that it can be reused by other tests. + +### Adding a test case +In order to add a new test case you need a new line to either `test-scripts/run-nightly-tests.sh` and / or `test-scripts/run-pre-commit-tests.sh` + +Adding a new test case generally follows the following pattern + +```sh +run_test "simple end-to-end test" "$END_TO_END_DIR/test-scripts/simple.sh arg1 arg2" +``` + +_Note: If you want to parameterize your tests please do so by adding multiple test cases with parameters as arguments to the nightly / pre-commit test suites. This allows the test runner to do a cleanup in between each individual test and also to fail those tests individually._ + +### Passing your test +A test is considered to have passed if it: +- has exit code 0 +- there are no non-empty .out files (nothing was written to stdout / stderr by your flink program) +- there are no exceptions in the log files +- there are no errors in the log files + +### Failing your test +A test is considered to have failed if it: +- exited with non-zero exit code +- has non-empty *.out files (something was written to stdout / stderr by your flink program) +- there are errors in the log files +- there are exceptions in the log files + +_There is a whitelist for exceptions and errors that do not lead to failure, they can be found in the `check_logs_for_errors` and `check_logs_for_exceptions` in `test-scripts/common.sh`_ + +Please note that a previously supported pattern where you could assign a value the global variable `PASS` to have your tests fail **is not supported anymore** --- End diff -- Missing period at the end. ---
[GitHub] flink pull request #6172: [FLINK-9594][E2E tests] Improve docs for new test ...
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/6172#discussion_r195992679 --- Diff: flink-end-to-end-tests/README.md --- @@ -33,6 +33,53 @@ $ FLINK_DIR= flink-end-to-end-tests/test-scripts/test_batch_wordcount ## Writing Tests -Have a look at test_batch_wordcount.sh for a very basic test and -test_streaming_kafka010.sh for a more involved example. Whenever possible, try -to put new functionality in common.sh so that it can be reused by other tests. +### Examples +Have a look at `test_batch_wordcount.sh` for a very basic test and +`test_streaming_kafka010.sh` for a more involved example. Whenever possible, try +to put new functionality in `common.sh` so that it can be reused by other tests. + +### Adding a test case +In order to add a new test case you need a new line to either `test-scripts/run-nightly-tests.sh` and / or `test-scripts/run-pre-commit-tests.sh` + +Adding a new test case generally follows the following pattern + +```sh +run_test "simple end-to-end test" "$END_TO_END_DIR/test-scripts/simple.sh arg1 arg2" +``` + +_Note: If you want to parameterize your tests please do so by adding multiple test cases with parameters as arguments to the nightly / pre-commit test suites. This allows the test runner to do a cleanup in between each individual test and also to fail those tests individually._ + +### Passing your test +A test is considered to have passed if it: +- has exit code 0 +- there are no non-empty .out files (nothing was written to stdout / stderr by your flink program) +- there are no exceptions in the log files +- there are no errors in the log files + +### Failing your test +A test is considered to have failed if it: +- exited with non-zero exit code +- has non-empty *.out files (something was written to stdout / stderr by your flink program) +- there are errors in the log files +- there are exceptions in the log files + +_There is a whitelist for exceptions and errors that do not lead to failure, they can be found in the `check_logs_for_errors` and `check_logs_for_exceptions` in `test-scripts/common.sh`_ --- End diff -- `... not lead to failure, they can be found ...` --> `.. not lead to failure, "which" can be found` ---
[GitHub] flink pull request #6172: [FLINK-9594][E2E tests] Improve docs for new test ...
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/6172#discussion_r195992920 --- Diff: flink-end-to-end-tests/README.md --- @@ -33,6 +33,53 @@ $ FLINK_DIR= flink-end-to-end-tests/test-scripts/test_batch_wordcount ## Writing Tests -Have a look at test_batch_wordcount.sh for a very basic test and -test_streaming_kafka010.sh for a more involved example. Whenever possible, try -to put new functionality in common.sh so that it can be reused by other tests. +### Examples +Have a look at `test_batch_wordcount.sh` for a very basic test and +`test_streaming_kafka010.sh` for a more involved example. Whenever possible, try +to put new functionality in `common.sh` so that it can be reused by other tests. + +### Adding a test case +In order to add a new test case you need a new line to either `test-scripts/run-nightly-tests.sh` and / or `test-scripts/run-pre-commit-tests.sh` + +Adding a new test case generally follows the following pattern + +```sh +run_test "simple end-to-end test" "$END_TO_END_DIR/test-scripts/simple.sh arg1 arg2" +``` + +_Note: If you want to parameterize your tests please do so by adding multiple test cases with parameters as arguments to the nightly / pre-commit test suites. This allows the test runner to do a cleanup in between each individual test and also to fail those tests individually._ + +### Passing your test +A test is considered to have passed if it: +- has exit code 0 +- there are no non-empty .out files (nothing was written to stdout / stderr by your flink program) +- there are no exceptions in the log files +- there are no errors in the log files + +### Failing your test +A test is considered to have failed if it: +- exited with non-zero exit code +- has non-empty *.out files (something was written to stdout / stderr by your flink program) +- there are errors in the log files +- there are exceptions in the log files + +_There is a whitelist for exceptions and errors that do not lead to failure, they can be found in the `check_logs_for_errors` and `check_logs_for_exceptions` in `test-scripts/common.sh`_ + +Please note that a previously supported pattern where you could assign a value the global variable `PASS` to have your tests fail **is not supported anymore** + +### Cleanup +The test runner performs a cleanup after each test case, which includes: +- Stopping the cluster +- Killing all task and job managers +- Reverting config to default (if changed before) +- Cleaning up log and temp directories + +If your test case should do some *additional* cleanup, a common pattern is to trap a `test_cleanup` function to `EXIT` in your test case like so: --- End diff -- I would explicitly state that this typically consists of shutting down external systems used by the e2e test, e.g. Kafka / Elasticsearch. ---
[GitHub] flink pull request #6172: [FLINK-9594][E2E tests] Improve docs for new test ...
Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/6172#discussion_r195991753 --- Diff: flink-end-to-end-tests/README.md --- @@ -33,6 +33,53 @@ $ FLINK_DIR= flink-end-to-end-tests/test-scripts/test_batch_wordcount ## Writing Tests -Have a look at test_batch_wordcount.sh for a very basic test and -test_streaming_kafka010.sh for a more involved example. Whenever possible, try -to put new functionality in common.sh so that it can be reused by other tests. +### Examples +Have a look at `test_batch_wordcount.sh` for a very basic test and +`test_streaming_kafka010.sh` for a more involved example. Whenever possible, try +to put new functionality in `common.sh` so that it can be reused by other tests. + +### Adding a test case +In order to add a new test case you need a new line to either `test-scripts/run-nightly-tests.sh` and / or `test-scripts/run-pre-commit-tests.sh` + +Adding a new test case generally follows the following pattern + +```sh +run_test "simple end-to-end test" "$END_TO_END_DIR/test-scripts/simple.sh arg1 arg2" +``` + +_Note: If you want to parameterize your tests please do so by adding multiple test cases with parameters as arguments to the nightly / pre-commit test suites. This allows the test runner to do a cleanup in between each individual test and also to fail those tests individually._ + +### Passing your test +A test is considered to have passed if it: +- has exit code 0 +- there are no non-empty .out files (nothing was written to stdout / stderr by your flink program) --- End diff -- Capital `f` for `Flink` ---
[GitHub] flink pull request #6172: [FLINK-9594][E2E tests] Improve docs for new test ...
Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/6172#discussion_r195823386 --- Diff: flink-end-to-end-tests/README.md --- @@ -33,6 +33,53 @@ $ FLINK_DIR= flink-end-to-end-tests/test-scripts/test_batch_wordcount ## Writing Tests -Have a look at test_batch_wordcount.sh for a very basic test and -test_streaming_kafka010.sh for a more involved example. Whenever possible, try -to put new functionality in common.sh so that it can be reused by other tests. +### Examples +Have a look at `test_batch_wordcount.sh` for a very basic test and +`test_streaming_kafka010.sh` for a more involved example. Whenever possible, try +to put new functionality in `common.sh` so that it can be reused by other tests. + +### Adding a test case +In order to add a new test case you need a new line to either `test-scripts/run-nightly-tests.sh` and / or `test-scripts/run-pre-commit-tests.sh` + +Adding a new test case generally follows the following pattern + +```sh +run_test "simple end-to-end test" "$END_TO_END_DIR/test-scripts/simple.sh arg1 arg2" +``` + +_Note: If you want to parameterize your tests please do so by adding multiple test cases with parameters as arguments to the nightly / pre-commit test suites. This allows the test runner to do a cleanup in between each individual test and also to fail those tests individually._ + +### Passing your test +A test is considered to have passed if it: +- has exit code 0 +- there are no non-empty .out files (nothing was written to stdout / stderr by your flink program) +- there are no exceptions in the log files +- there are no errors in the log files + +### Failing your test --- End diff -- this section is just the inverse of `Passing your test`, we can omit it. ---
[GitHub] flink pull request #6172: [FLINK-9594][E2E tests] Improve docs for new test ...
Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/6172#discussion_r195823151 --- Diff: flink-end-to-end-tests/README.md --- @@ -33,6 +33,53 @@ $ FLINK_DIR= flink-end-to-end-tests/test-scripts/test_batch_wordcount ## Writing Tests -Have a look at test_batch_wordcount.sh for a very basic test and -test_streaming_kafka010.sh for a more involved example. Whenever possible, try -to put new functionality in common.sh so that it can be reused by other tests. +### Examples +Have a look at `test_batch_wordcount.sh` for a very basic test and +`test_streaming_kafka010.sh` for a more involved example. Whenever possible, try +to put new functionality in `common.sh` so that it can be reused by other tests. + +### Adding a test case +In order to add a new test case you need a new line to either `test-scripts/run-nightly-tests.sh` and / or `test-scripts/run-pre-commit-tests.sh` + +Adding a new test case generally follows the following pattern + +```sh +run_test "simple end-to-end test" "$END_TO_END_DIR/test-scripts/simple.sh arg1 arg2" --- End diff -- refer to the templates in the respective files instead ---
[GitHub] flink pull request #6172: [FLINK-9594][E2E tests] Improve docs for new test ...
GitHub user florianschmidt1994 opened a pull request: https://github.com/apache/flink/pull/6172 [FLINK-9594][E2E tests] Improve docs for new test runner] ## What is the purpose of the change Improve documentation for end-to-end tests with the new test runner pattern introduced in FLINK-9257 ## Brief change log - Add more docs ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not applicable You can merge this pull request into a Git repository by running: $ git pull https://github.com/florianschmidt1994/flink FLINK-9594-add-docs-for-e2e-tests Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/6172.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #6172 commit 7a0761ac27d5b876425616cc9433daa0be949cc9 Author: Florian Schmidt Date: 2018-06-15T09:13:39Z [FLINK-9594][E2E tests] Improve docs for new test runner] ---
