GitHub user Yicong-Huang edited a comment on the discussion: Manual testing expectations for PRs
> 1. What should the manual testing expectations be for different types of PRs? This is a very big question. But generally speaking, the amount of testing is related to the risk of the change. > I prefer all of the PRs should be manually tested, 1 setting at least should > be good enough. hmm, how to manually test a PR with doc only change? how to manually test a performance change? how to manually test a race condition that only happens 1 out of 1000 times? If I tested on windows, local dev mode, is it sufficient to say my change works on ubuntu with k8s as well? > 2. What should PR authors be expected to include? I think current template is > good but the points you mentioned will be helpful. I think we can improve our template though. for instance, we should ask authors to provide more explicit evidences, for example, screenshots/gifs for UI change. > 3. What should reviewers be expected to check? I prefer reviewer test the PR > manually in 1 setting at least. I really think that would be a huge burden on the reviewer. How many PRs can a reviewer work on in a day if we **require** a reviewer to manually verify a PR? 1, 2, or 10? Taking myself as an example, I have so far reviewed 998 PRs in apache/texera in total, if every PR needs manual tests, say let's 10 minutes, that's 9980 minutes, or about 166 hours, more than 20 full workdays spent on manual testing alone without a break. Not to mention I need time to write comments, discuss with author, and many PRs needs multiple rounds of reviews and testing. The burden to review is too much to afford. With today's setup, only I and @aglinxinyuan reviewed more than 200 PRs and still being active on reviews. with more burden on reviewer, I doubt anyone would spend more time to review. At least I won't be able to help given this is only an open source project that I contribute to voluntarily. I truly think reviewers should check the evidence provided by author, PR should be self-contained to explain the change, risk, and evidence why it is good to be merged. Of course, I don't want to prevent reviewers from manually verify the change, just to be safer, and I do test manually a lot during review. but requiring reviewers to manually test is too harsh. We need more help on review, and I want to encourage more ppl to help on review. > 4. How can we make testing across different environments more practical? I > have not proposal for this point. This can only be done with CI/CD. it is not feasible for every developer to own/maintain multiple envs. GitHub link: https://github.com/apache/texera/discussions/6028#discussioncomment-17523552 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
