[
https://issues.apache.org/jira/browse/FLINK-9594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16515474#comment-16515474
]
ASF GitHub Bot commented on FLINK-9594:
---------------------------------------
Github user tzulitai commented on a diff in the pull request:
https://github.com/apache/flink/pull/6172#discussion_r195992732
--- Diff: flink-end-to-end-tests/README.md ---
@@ -33,6 +33,53 @@ $ FLINK_DIR=<flink dir>
flink-end-to-end-tests/test-scripts/test_batch_wordcount
## Writing Tests
-Have a look at test_batch_wordcount.sh for a very basic test and
-test_streaming_kafka010.sh for a more involved example. Whenever possible,
try
-to put new functionality in common.sh so that it can be reused by other
tests.
+### Examples
+Have a look at `test_batch_wordcount.sh` for a very basic test and
+`test_streaming_kafka010.sh` for a more involved example. Whenever
possible, try
+to put new functionality in `common.sh` so that it can be reused by other
tests.
+
+### Adding a test case
+In order to add a new test case you need a new line to either
`test-scripts/run-nightly-tests.sh` and / or
`test-scripts/run-pre-commit-tests.sh`
+
+Adding a new test case generally follows the following pattern
+
+```sh
+run_test "simple end-to-end test" "$END_TO_END_DIR/test-scripts/simple.sh
arg1 arg2"
+```
+
+_Note: If you want to parameterize your tests please do so by adding
multiple test cases with parameters as arguments to the nightly / pre-commit
test suites. This allows the test runner to do a cleanup in between each
individual test and also to fail those tests individually._
+
+### Passing your test
+A test is considered to have passed if it:
+- has exit code 0
+- there are no non-empty .out files (nothing was written to stdout /
stderr by your flink program)
+- there are no exceptions in the log files
+- there are no errors in the log files
+
+### Failing your test
+A test is considered to have failed if it:
+- exited with non-zero exit code
+- has non-empty *.out files (something was written to stdout / stderr by
your flink program)
+- there are errors in the log files
+- there are exceptions in the log files
+
+_There is a whitelist for exceptions and errors that do not lead to
failure, they can be found in the `check_logs_for_errors` and
`check_logs_for_exceptions` in `test-scripts/common.sh`_
+
+Please note that a previously supported pattern where you could assign a
value the global variable `PASS` to have your tests fail **is not supported
anymore**
--- End diff --
Missing period at the end.
> Add documentation for e2e test changes introduced with FLINK-9257
> -----------------------------------------------------------------
>
> Key: FLINK-9594
> URL: https://issues.apache.org/jira/browse/FLINK-9594
> Project: Flink
> Issue Type: Improvement
> Reporter: Florian Schmidt
> Assignee: Florian Schmidt
> Priority: Minor
>
> There were some changes introduced with FLINK-9257 in how end-to-end tests
> are structured and how we handle triggering failure from a test case. This
> should be documented for future developers writing e2e tests
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
