fgksgf commented on code in PR #119:
URL:
https://github.com/apache/skywalking-infra-e2e/pull/119#discussion_r1337357390
##########
test/e2e/README.md:
##########
@@ -0,0 +1,105 @@
+# Use E2E to test itself
+
+After updating the features of e2e, you can use the files in the test/e2e/
directory to perform testing for both new and old features of e2e.
+You can perform testing locally. And when you submit a pull request (PR),
GitHub Actions will automatically run the tests.
+
+## Files introduction
+- `test/e2e`
+ - `concurrency`
+ - `fail-fast` (concurrency & fail-fast mode)
+ - `internal`
+ - `expected.yaml`
+ - `verify.yaml` (configuration file for the inner infra
E2E(concurrency & fail-fast))
+ - `expected.yaml`
+ - `non-fail-fast` (concurrency & non-fail-fast mode)
+ - `internal`
+ - `expected.yaml`
+ - `verify.yaml` (configuration file for the inner infra
E2E(concurrency & non-fail-fast))
+ - `expected.yaml`
+ - `non-concurrency`
+ - `fail-fast` (non-concurrency & fail-fast mode)
+ - `internal`
+ - `expected.yaml`
+ - `verify.yaml` (configuration file for the inner infra
E2E(non-concurrency & fail-fast))
+ - `expected.yaml`
+ - `non-fail-fast` (non-concurrency & non-fail-fast mode)
+ - `internal`
+ - `expected.yaml`
+ - `verify.yaml` (configuration file for the inner infra
E2E(non-concurrency & non-fail-fast))
+ - `expected.yaml`
+ - `docker-compose.yaml` (run a httpbin container, which can return YAML
data)
+ - `e2e.yaml` (configuration file for the outer infra E2E)
+
+## Tips
+### concurrency & fail-fast mode
+- We add `&& sleep=5` in `query`of the last few cases, to make sure that there
will be cases to be skipped.
+examples: `'curl -s 127.0.0.1:8080/get?case=failure -H "accept:
application/json" && sleep 5'`.
+- In the `expected.yaml`, we use yaml grammar in it:
+```yaml
+# It means that the `passed` field in the `expected.yaml` will be populated
+# with the elements (which contains a prefix of "passed") of `passed` field in
the summary generated by infra E2E (concurrency & fail-fast mode).
+passed:
+{{range .passed}}
+- {{ if hasPrefix . "passed" }}{{.}}{{ end }}
+{{end}}
+# It means that the `failed` field in the `expected.yaml` will be populated
+# with the elements (which contain a prefix of "failed") of `failed` field in
the summary generated by infra E2E (concurrency & fail-fast mode).
+failed:
+ {{range .failed}}
+- {{ if hasPrefix . "failed" }}{{.}}{{ end }}
+ {{end}}
+skipped:
+ {{range .skipped}}
+- {{.}}
+ {{end}}
+
+# if A <= B, the value of {{le A B}} is A.
+# you need to replace the `totalPassedCount` with your real total count of
passed cases.
+passedCount: {{le .passedCount totalPassedCount}}
+# you need to replace the `totalFailedCount` with your real total count of
failed cases.
+failedCount: {{le .failedCount totalFailedCount}}
+# the subtrtactor is a function. The value of {{subtrator A B C D ...}} is
(A-B-C-D-...)
+# you need to replace the totalCount with your real count of cases.
+skippedCount: {{subtractor totalCount .passedCount .failedCount}}
+```
+
+### concurrency & non-fail-fast mode
+- In the `expected.yaml`, we use yaml grammar in it:
+```yaml
+# unlike in concurrency & fail-fast mode, the summary of infra E2E
(concurrency & non-fail-fast) is predictable.
+# but the sequence of the elements of `passed` and `failed` field of the
summary isn't predictable, so we use `contains`.
+# `contains` selects the elements from the collection of `contains` that
belong to the collection of `passed` (the collection of `passed` field of
summary),
+# and arranges them in the order they appear in collection of `passed`. Then,
it populates the `passed` field in `expected.yaml` with these elements.
+passed:
+{{- contains .passed}}
+- passed-case-1
+- passed-case-2
+- passed-case-3
+...
+{{- end}}
+# `contains` is the same in `failed`.
+failed:
+{{- contains .failed }}
+- failed-case-1
+- failed-case-2
+- failed-case-3
+...
+{{- end }}
+skipped: []
+# you need to replace the `totalPassedCount` with your real total count of
passed cases.
+passedCount: totalPassedCount
+# you need to replace the `totalFailedCount` with your real total count of
failed cases.
+failedCount: totalFailedCount
+skippedCount: 0
+```
+
+## Local Test
+After `make build`, you can use `skywalking-infra-e2e/test/e2e/e2e.yaml` as
the configuration file to run `e2e run`.
+
+
+
+
+
+
+
+
Review Comment:
Remove redundant empty lines
##########
test/e2e/README.md:
##########
@@ -0,0 +1,105 @@
+# Use E2E to test itself
+
+After updating the features of e2e, you can use the files in the test/e2e/
directory to perform testing for both new and old features of e2e.
+You can perform testing locally. And when you submit a pull request (PR),
GitHub Actions will automatically run the tests.
+
+## Files introduction
+- `test/e2e`
+ - `concurrency`
+ - `fail-fast` (concurrency & fail-fast mode)
+ - `internal`
+ - `expected.yaml`
+ - `verify.yaml` (configuration file for the inner infra
E2E(concurrency & fail-fast))
+ - `expected.yaml`
+ - `non-fail-fast` (concurrency & non-fail-fast mode)
+ - `internal`
+ - `expected.yaml`
+ - `verify.yaml` (configuration file for the inner infra
E2E(concurrency & non-fail-fast))
+ - `expected.yaml`
+ - `non-concurrency`
+ - `fail-fast` (non-concurrency & fail-fast mode)
+ - `internal`
+ - `expected.yaml`
+ - `verify.yaml` (configuration file for the inner infra
E2E(non-concurrency & fail-fast))
+ - `expected.yaml`
+ - `non-fail-fast` (non-concurrency & non-fail-fast mode)
+ - `internal`
+ - `expected.yaml`
+ - `verify.yaml` (configuration file for the inner infra
E2E(non-concurrency & non-fail-fast))
+ - `expected.yaml`
+ - `docker-compose.yaml` (run a httpbin container, which can return YAML
data)
+ - `e2e.yaml` (configuration file for the outer infra E2E)
+
+## Tips
+### concurrency & fail-fast mode
+- We add `&& sleep=5` in `query`of the last few cases, to make sure that there
will be cases to be skipped.
+examples: `'curl -s 127.0.0.1:8080/get?case=failure -H "accept:
application/json" && sleep 5'`.
+- In the `expected.yaml`, we use yaml grammar in it:
+```yaml
+# It means that the `passed` field in the `expected.yaml` will be populated
+# with the elements (which contains a prefix of "passed") of `passed` field in
the summary generated by infra E2E (concurrency & fail-fast mode).
+passed:
+{{range .passed}}
+- {{ if hasPrefix . "passed" }}{{.}}{{ end }}
+{{end}}
+# It means that the `failed` field in the `expected.yaml` will be populated
+# with the elements (which contain a prefix of "failed") of `failed` field in
the summary generated by infra E2E (concurrency & fail-fast mode).
+failed:
+ {{range .failed}}
+- {{ if hasPrefix . "failed" }}{{.}}{{ end }}
+ {{end}}
+skipped:
+ {{range .skipped}}
+- {{.}}
+ {{end}}
+
+# if A <= B, the value of {{le A B}} is A.
+# you need to replace the `totalPassedCount` with your real total count of
passed cases.
+passedCount: {{le .passedCount totalPassedCount}}
+# you need to replace the `totalFailedCount` with your real total count of
failed cases.
+failedCount: {{le .failedCount totalFailedCount}}
+# the subtrtactor is a function. The value of {{subtrator A B C D ...}} is
(A-B-C-D-...)
+# you need to replace the totalCount with your real count of cases.
+skippedCount: {{subtractor totalCount .passedCount .failedCount}}
+```
+
+### concurrency & non-fail-fast mode
+- In the `expected.yaml`, we use yaml grammar in it:
+```yaml
+# unlike in concurrency & fail-fast mode, the summary of infra E2E
(concurrency & non-fail-fast) is predictable.
+# but the sequence of the elements of `passed` and `failed` field of the
summary isn't predictable, so we use `contains`.
+# `contains` selects the elements from the collection of `contains` that
belong to the collection of `passed` (the collection of `passed` field of
summary),
+# and arranges them in the order they appear in collection of `passed`. Then,
it populates the `passed` field in `expected.yaml` with these elements.
+passed:
+{{- contains .passed}}
+- passed-case-1
+- passed-case-2
+- passed-case-3
+...
+{{- end}}
+# `contains` is the same in `failed`.
+failed:
+{{- contains .failed }}
+- failed-case-1
+- failed-case-2
+- failed-case-3
+...
+{{- end }}
+skipped: []
+# you need to replace the `totalPassedCount` with your real total count of
passed cases.
+passedCount: totalPassedCount
+# you need to replace the `totalFailedCount` with your real total count of
failed cases.
+failedCount: totalFailedCount
+skippedCount: 0
+```
+
+## Local Test
+After `make build`, you can use `skywalking-infra-e2e/test/e2e/e2e.yaml` as
the configuration file to run `e2e run`.
Review Comment:
Can we just use one single make command to run e2e locally?
##########
test/e2e/README.md:
##########
@@ -0,0 +1,105 @@
+# Use E2E to test itself
+
+After updating the features of e2e, you can use the files in the test/e2e/
directory to perform testing for both new and old features of e2e.
+You can perform testing locally. And when you submit a pull request (PR),
GitHub Actions will automatically run the tests.
Review Comment:
You need also introduce how this work, such as the outer e2e and the inner
e2e.
##########
test/e2e/README.md:
##########
@@ -0,0 +1,105 @@
+# Use E2E to test itself
+
+After updating the features of e2e, you can use the files in the test/e2e/
directory to perform testing for both new and old features of e2e.
+You can perform testing locally. And when you submit a pull request (PR),
GitHub Actions will automatically run the tests.
+
+## Files introduction
+- `test/e2e`
+ - `concurrency`
+ - `fail-fast` (concurrency & fail-fast mode)
+ - `internal`
+ - `expected.yaml`
+ - `verify.yaml` (configuration file for the inner infra
E2E(concurrency & fail-fast))
+ - `expected.yaml`
+ - `non-fail-fast` (concurrency & non-fail-fast mode)
+ - `internal`
+ - `expected.yaml`
+ - `verify.yaml` (configuration file for the inner infra
E2E(concurrency & non-fail-fast))
+ - `expected.yaml`
+ - `non-concurrency`
+ - `fail-fast` (non-concurrency & fail-fast mode)
+ - `internal`
+ - `expected.yaml`
+ - `verify.yaml` (configuration file for the inner infra
E2E(non-concurrency & fail-fast))
+ - `expected.yaml`
+ - `non-fail-fast` (non-concurrency & non-fail-fast mode)
+ - `internal`
+ - `expected.yaml`
+ - `verify.yaml` (configuration file for the inner infra
E2E(non-concurrency & non-fail-fast))
+ - `expected.yaml`
+ - `docker-compose.yaml` (run a httpbin container, which can return YAML
data)
+ - `e2e.yaml` (configuration file for the outer infra E2E)
+
+## Tips
Review Comment:
One detailed example is enough, the purpose of the readme is to tell new
contributors how to add new cases for their changes.
##########
test/e2e/e2e.yaml:
##########
@@ -32,9 +33,9 @@ verify:
# the interval between two attempts, e.g. 10s, 1m.
interval: 1s
- # when a case fails, whether to stop verifying other cases. This property
defaults to true.
+ # when a case fails, whether to stop verifying other cases. This property is
recommended to false.
fail-fast: false
- # Whether to verify cases concurrently. This property defaults to false.
Review Comment:
Why it's recommended?
##########
test/e2e/README.md:
##########
@@ -0,0 +1,105 @@
+# Use E2E to test itself
+
+After updating the features of e2e, you can use the files in the test/e2e/
directory to perform testing for both new and old features of e2e.
+You can perform testing locally. And when you submit a pull request (PR),
GitHub Actions will automatically run the tests.
+
+## Files introduction
Review Comment:
It would be better to use a code block to introduce the file structure.
##########
test/e2e/README.md:
##########
@@ -0,0 +1,105 @@
+# Use E2E to test itself
+
+After updating the features of e2e, you can use the files in the test/e2e/
directory to perform testing for both new and old features of e2e.
+You can perform testing locally. And when you submit a pull request (PR),
GitHub Actions will automatically run the tests.
+
+## Files introduction
+- `test/e2e`
+ - `concurrency`
+ - `fail-fast` (concurrency & fail-fast mode)
+ - `internal`
+ - `expected.yaml`
+ - `verify.yaml` (configuration file for the inner infra
E2E(concurrency & fail-fast))
+ - `expected.yaml`
+ - `non-fail-fast` (concurrency & non-fail-fast mode)
+ - `internal`
+ - `expected.yaml`
+ - `verify.yaml` (configuration file for the inner infra
E2E(concurrency & non-fail-fast))
+ - `expected.yaml`
+ - `non-concurrency`
+ - `fail-fast` (non-concurrency & fail-fast mode)
+ - `internal`
+ - `expected.yaml`
+ - `verify.yaml` (configuration file for the inner infra
E2E(non-concurrency & fail-fast))
+ - `expected.yaml`
+ - `non-fail-fast` (non-concurrency & non-fail-fast mode)
+ - `internal`
+ - `expected.yaml`
+ - `verify.yaml` (configuration file for the inner infra
E2E(non-concurrency & non-fail-fast))
+ - `expected.yaml`
+ - `docker-compose.yaml` (run a httpbin container, which can return YAML
data)
+ - `e2e.yaml` (configuration file for the outer infra E2E)
+
+## Tips
+### concurrency & fail-fast mode
+- We add `&& sleep=5` in `query`of the last few cases, to make sure that there
will be cases to be skipped.
Review Comment:
This can be added as comments to related yaml files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
