GitHub user xuang7 edited a discussion: Manual testing expectations for PRs
Hi community, I would like to start a discussion about our expectations for manual testing in PRs, especially around what should be tested and how testing responsibilities should be shared. Currently, we rely heavily on automated checks, which are helpful and should remain the main baseline. We are also continuing to improve test coverage. At the same time, some issues may only show up in specific manual workflows or environments, such as local setup, Docker, or Kubernetes. For example, during RC2 validation, we found an environment-specific issue related to the new PVE feature (#5963). We have also seen cases during workshops or demos where some issues were only discovered during actual usage. We probably do not need to require every PR to be tested in every environment, since that would add a lot of overhead. Instead, it may be helpful to define a lightweight standard for manual testing based on the type and risk of the PR. Some questions I would like to discuss: 1. What should the manual testing expectations be for different types of PRs? Examples: * Frontend behavior changes * Backend/API changes * Database or schema changes * Authentication/authorization changes * Deployment, Docker, Kubernetes, or CI changes * Large feature PRs 2. What should PR authors be expected to include? Examples: * What environment was tested * What manual workflow was tested * Whether screenshots, logs, or videos are useful * Known limitations or untested areas 3. What should reviewers be expected to check? Examples: * Whether the testing section is clear enough * Whether the testing level matches the scope/risk of the change * Whether additional manual testing should be requested before merge * Whether reviewers are expected to manually test the PR as well, and under what circumstances 4. How can we make testing across different environments more practical? Examples: * Are there ways to provide easier setup instructions for Docker or Kubernetes testing? * For environments that are time-consuming to set up, can maintainers or reviewers help verify them when needed? * Should certain high-risk PRs request help from someone who already has the relevant environment available? The goal is not to increase overhead unnecessarily, but to develop a shared expectation for when manual testing is needed beyond automated checks. This can help clarify what information authors should provide and what reviewers should check, while improving feature reliability in real-world usage. Please feel free to share your thoughts on what would be practical and reasonable for different types of PRs. Thanks! GitHub link: https://github.com/apache/texera/discussions/6028 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
