nevzheng opened a new issue, #14165: URL: https://github.com/apache/gravitino/issues/14165
### Describe the proposal One `presubmit` workflow replaces today's 15 build and test workflows: lint → build → small tests → medium tests → integration tests. Each stage fans out in parallel, and a stage starts only when the one before it passes. **Why it matters:** Every push starts all 15 build and test workflows at once. A typo still costs a full set of Docker IT runs, and the first red check on a lint or compile error lands 30–57 minutes in (`build (17)` on recent green main runs). Some suites (CLI IT, web-v2's 338 unit tests, web v1 ITs) run in no PR job at all. **North star:** CI runs what the build declares, cheapest first, with the same task names you run locally. It fails fast, and never silently skips or loses a test. **Targets (today → goal):** - Time to the first red check on a lint or compile error: 30–57 min → minutes - Build and test workflows started per push: 15 → 1 (`presubmit`) - Test mode flags (`skipTests`, `skipITs`, `skipDockerTests`, `testMode`, ...): 6 → 0 - backend-it `-x` exclusions: 28 → 0 - Silent skips (suites run by no PR job): CLI IT, web-v2 unit tests, web v1 ITs → 0 **How it works:** - Gradle owns *what* runs. GitHub Actions owns order and fan-out. No shell scripts. - `ciVerifyShards` fails the build when a module is unlisted, listed twice, or the catch-all grows. - Build once: a run-scoped Gradle build cache now, a remote cache (ASF Develocity) later. - Cheaper medium and integration stages will come from separate container-sharing work. **The big picture:** "What the build declares" is the lasting rule. `gradle/ci-stages.gradle.kts` is interim scaffolding. A later test-organization effort (Gradle JVM test suites, the flags retired, `check` hermetic) swaps the commands without changing the stage shape. The design doc maps each interim command to its later form. **Design:** `design-docs/ci/presubmit/SPIP-presubmit.md` (#01). **Related:** #3834 (earlier "Optimize CI" epic) and #12545 (one required CI check). ### Task list Independent (any order): - [ ] #A Fix shellcheck errors in release and docker scripts - [ ] #B Drop libssl1.1 and inline shell from the build workflows Stack (in order): - [ ] #01 Add the presubmit SPIP - [ ] #02 Run the build workflow in stages, cheapest first - [ ] #03 Rename build to presubmit and run the integration tests as its last stage - [ ] #04 Add lint jobs for workflows, python, shell, docs and web - [ ] #05 Run the CLI integration tests in a cli-it job - [ ] #06 Derive the backend-it module list in Gradle - [ ] #07 Build once and reuse the build cache across presubmit stages - [ ] #08 Run the frontend, contrib catalog and chart tests as presubmit stages - [ ] #09 Run the web UI v1 ITs and both UIs' unit tests in presubmit -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
