Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/6172#discussion_r195823386
--- Diff: flink-end-to-end-tests/README.md ---
@@ -33,6 +33,53 @@ $ FLINK_DIR=<flink dir>
flink-end-to-end-tests/test-scripts/test_batch_wordcount
## Writing Tests
-Have a look at test_batch_wordcount.sh for a very basic test and
-test_streaming_kafka010.sh for a more involved example. Whenever possible,
try
-to put new functionality in common.sh so that it can be reused by other
tests.
+### Examples
+Have a look at `test_batch_wordcount.sh` for a very basic test and
+`test_streaming_kafka010.sh` for a more involved example. Whenever
possible, try
+to put new functionality in `common.sh` so that it can be reused by other
tests.
+
+### Adding a test case
+In order to add a new test case you need a new line to either
`test-scripts/run-nightly-tests.sh` and / or
`test-scripts/run-pre-commit-tests.sh`
+
+Adding a new test case generally follows the following pattern
+
+```sh
+run_test "simple end-to-end test" "$END_TO_END_DIR/test-scripts/simple.sh
arg1 arg2"
+```
+
+_Note: If you want to parameterize your tests please do so by adding
multiple test cases with parameters as arguments to the nightly / pre-commit
test suites. This allows the test runner to do a cleanup in between each
individual test and also to fail those tests individually._
+
+### Passing your test
+A test is considered to have passed if it:
+- has exit code 0
+- there are no non-empty .out files (nothing was written to stdout /
stderr by your flink program)
+- there are no exceptions in the log files
+- there are no errors in the log files
+
+### Failing your test
--- End diff --
this section is just the inverse of `Passing your test`, we can omit it.
---
