GitHub user Yicong-Huang edited a comment on the discussion: Manual testing expectations for PRs
Hey guys, I find myself could not sleep on this question. Let me mention a few points about manual tests before I layout my opinion on how should we expect manual tests. 1. First of all, manual test can only verify at the **exact moment when you tested**, on the **exact environment you used**, the code was working as expected, exactly **once**. However, how do we know a code is working as expected, after 3 months? How do I know the feature will still work when any OS, env, deployment method changed? Next week a new dependency upgraded, the feature may just stopped working because the lib API changed. For some flaky cases caused by race condition, which may not trigger every time, how do you know if you are not just lucky to not encounter it during your manual test? On the other hand, automated test in CI, once you write it out, it will be executed every time when another PR is raised, another commit is submitted, the same code path will be verified thousands of times. Regression can be detected when time passes, dependencies are updated, and flakiness can be exposed by repeated testing. 2. Secondly, there are many aspects of "expectations", and many of them are **not feasible to be manually tested**, or easily eyeballed. For instance, success path needs to be verified, error cases needs to be verified, code performance is important for hot path, and stability is key for anything connecting to external resources like db, network, etc. Many of those cannot be possibly done manually: how do you expect a human to differentiate a debounce time of 300ms, vs 500ms? how do you expect a human to feel the difference between a network request which had 3 retries and 5 retries? how do you expect a human to read a json output from an API and check if any of the fields are incorrect? what about a feature that is not exposed easily on the UI or user-facing API, how to "manually" test it? What's more, as project goes bigger, there are too many features connected with each other. How do you expect a human to manually verify introducing a new feature did not silently killed some feat ures he/she was not even aware of existing? Automated tests can put those expectations programmatically, if an API lowers than 300ms, timeout error can happen, if a method becomes slow, it ran report, if network retries more than 5 times to succeed, it can alert. It's much easier to quantify those expectations in automated tests. 3. Thirdly, human is extremely slow and developers' time is precious. For me, when I manually test a feature, I need to switch to the branch, start the deployment, click my mouse on browser, try for a particular sequence of operations, and eye ball the result. This could easily cost me 10+ minutes. With the same 10 minutes, automation test can run thousands of cases, with different OS, env, different inputs, scenarios. It is not scalable with manual testing, and it eats up developers' time, which we could have saved for many many other aspects. So as a conclusion, manual test 1) is single time verification, 2) can cover very little expectations, and 3) can be costly and waste developers' time. If your impression is "automated tests cannot cover all cases, I always find more issues when I manually test it," it is because you did not have enough automated tests written, you should go back write more automated tests. Remember human cannot beat a machine's speed to verify things, if you find something machine forget to check, just ask the machine to check it for you in the future, instead of doing it by yourself manually. Of course, you may need to first manually test it, to discover the gap that machine forget to test. **But all of those stated, I am not saying manual test is not important.** My view is, we should still do manual testing, but we need to 1) do it after we have enough automated tests, as additional verifications. 2) do it in a smarter way to save time. So here is my proposal: 1. **Not every single PR needs manual tests.** Not worth the time. It is a balance of risk and efficiency. If you think some change is risky, and you think automated tests may have a gap, yes go ahead deploy everything and verify it. If you know some automation tests are verifying it, it is find to minimize manual tests, just to save time. After a PR is merged, other developers while developing their features, if they encounter issues related to your change, can raise issues, this servers as post merge manual test as well. If so, go ahead fix it with new automated tests. 2. **Go easy on development and be harsh on release.** Main branch serves as the "developing branch", any code change goes onto it, will not affect users immediately, until a release is made. So, it is fine if main branch has bugs and issues to be fixed. It is fine if main branch contains half-baked features. It is fine if a merged PR is found problematic later: just revert it or fix forward, no big deal. When we cut a release, we can be more strict. In addition to automation tests, we can start to do more manual tests to verify the release, that's why we have release candidates for you to test, verify and vote, and people vote -1 in case any issues are found during RC preparation phase. We could also organize bug bash events during release windows, and use a dedicated 1 hour to manually verify things together, and we only need to do this before release, not every day. 3. **PR authors do manual testing, reviewers optional.** By default, PR authors should write automated tests, and in addition do some manual tests when preparing a PR, so that you know your code works as expected. PR reviewers should verify "evidence", instead of verify from manual tests. For instance, reviewer should check if automated tests are written, from the tests, are the source code behaving as described? If author did some manual tests, what are the evidence (e.g., urls for testing conducted, screenshots/gifs of the feature in action)? Reviewers review the produced artifacts, evaluate the risk of the change vs the benefit of the change. Reviewers can push back to ask authors provide more evidences, including do more manual tests or write more automated tests. Again, I want to reduce reviewers' burden, and encourage more ppl to help review each other's code. If we require reviewer to manually verify every PR, then it is a discouragement, a waste of reviewer's time, and it is too slow to review. The process is just too heavy to afford, and eventually there will be no one willing to review any code, and the community will stop moving. 4. **Automation tests should cover more env, and closer to production.** We need to make automated tests more similar or even equal to production environment, so that there will be less scenarios that "in my local environment but failed after deployment". Align with this purpose, we need staging server, which mimics the setting (as close as possible) of production env, and write some tests against staging environment, or conduct manual bug bash on the staging server. 5. **Improve our automation tests/CI.** We need to increase unit test coverage, add more integration tests, performance test, pressure tests, etc. We need to make things automated, instead of fallback to manual effort. I know we are far away from it, which is a fundamental reason why we are still having a lot of manual test need. Again, if you find a gap, write a script to verify it repeatedly. Every line of test you write, can save other developers hours to verify manually. All in all, I hope we can elevate our overall productivity, and I believe manual testing is one giant burden we have had in the past that prevent us from doing things quicker. I know we had been doing only manual tests for a long time, and we are eating our own debt by moving ahead very very slowly: with the complexity of the project goes up, more features and more environments to support, it is harder and harder to ship new features and manually verify it works everywhere, every time, and did not break any existing features. Frankly speaking, with the old pace, this project will not last long and will soon be outdated. It's 5 am in the morning, I typed all of those word by word, 0 AI used. I really mean it with my heart. GitHub link: https://github.com/apache/texera/discussions/6028#discussioncomment-17523083 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
