dididy opened a new pull request, #5072: URL: https://github.com/apache/zeppelin/pull/5072
### What is this PR for? #### [Summary] - **Proposal:** Migrate the existing Protractor (and Karma/Jasmine-based E2E/testing system) to a **Playwright**-based testing framework. - **Objective:** Achieve a more stable and faster testing process while supporting modern E2E requirements such as cross-browser testing, multi-tab scenarios, and automatic waits. Additionally, leverage **Playwright MCP / code generator** to automate and accelerate test creation. #### [Background] 1. **Protractor** has long been the standard for E2E testing in the Angular ecosystem. However, with the [official deprecation/termination of support by the Angular team](https://blog.angular.dev/protractor-deprecation-update-august-2023-2beac7402ce0), long-term maintenance can no longer be relied upon. 2. Existing Protractor tests are WebDriver-based (control method, wait strategies, etc.), which often led to flaky test cases and increased maintenance costs. 3. The **Karma + Jasmine**-based unit testing environment, widely used as the default in Angular CLI, also has limitations in terms of build/compile costs, speed, and developer experience (configuration, parallel execution, etc.). Modern alternatives offer a better DX. - Since there were no test files, the related environment was removed from this branch, and any future unit tests will be migrated to Jest. #### [Why Playwright?] > Playwright provides a robust ecosystem with **stability, speed, standardization, and AI-assisted automation capabilities**. - **[Test Generator (Codegen)](https://playwright.dev/docs/codegen)** - Records user interactions in the browser and generates scaffolding for tests—useful for quickly setting up E2E scenarios. - **[MCP (Model Context Protocol) Integration](https://playwright.dev/agents/playwright-mcp-generating-tests)** - Playwright has extensive documentation and examples for workflows that generate and validate tests via MCP (LLM ↔ test runner interface), enabling automated test creation and LLM-assisted refactoring. This reduces the need for fully manual test writing. - In the MCP era, Playwright is the most widely used tool. - According to [State of JavaScript 2024](https://2024.stateofjs.com/en-US/libraries/testing/#testing_work), Playwright is the most widely used E2E tool at work. - **Functional Advantages** - Multi-browser support: Chromium / Firefox / WebKit - Multi-tab & cross-domain (cross-origin) testing - Robust auto-waiting, parallel execution - Built-in reporters, traces, videos, snapshots - **Why Not Cypress?** - Parallel execution requires paid [Cypress Cloud](https://docs.cypress.io/cloud/features/smart-orchestration/parallelization) - Cannot handle [iframes](https://docs.cypress.io/app/guides/cross-origin-testing#Origin), [new tabs](https://reflect.run/articles/accessing-a-new-window-in-cypress-tests), [multi-tab](https://momentic.ai/resources/the-definitive-guide-to-cypress-multi-tab-and-window-handling?utm_source=chatgpt.com), or [popup scenarios](https://reflect.run/articles/accessing-a-new-window-in-cypress-tests)without tricks - MCP support is community-driven and not standardized - **Why Not Puppeteer?** - [Chrome-dependent](https://pptr.dev/supported-browsers), [unsuitable for multi-environment E2E testing](https://www.browserstack.com/guide/cross-browser-testing-in-puppeteer) #### [Project Structure Overview] - **fixtures/**: Environment setup tools for test execution (e.g., `tearup` / `teardown`). - **models/**: Contains POM (Page Object Model) classes for reusable page actions. - **tests/**: Contains spec files for the actual test cases. - **helper.ts**: Zeppelin-specific helper classes. - **reporter.coverage.*.ts**: Custom reporters for checking test coverage. - **utils.ts**: Test utilities and page constants. [Node 16 is **no longer supported** in `@playwright/test` versions **after v1.53.2**](https://playwright.dev/docs/release-notes#miscellaneous-1). As a result, this project uses **v1.53.2** instead of the latest version (v1.55.0) to maintain compatibility. #### [Coverage Rule] > **Goal:** Ensure that for each `*.component.ts` file, **at least one test passes**. In other words, verify that E2E tests cover the functionality of each component. <img width="1130" height="888" alt="스크린샷 2025-09-11 오후 9 09 57" src="https://github.com/user-attachments/assets/7307d9e5-66b4-45d8-a8aa-6691fdbe6558"; /> **Approach (Proposed Flow)** 1. **Enable frontend code coverage collection** by adding a custom reporter. 2. At the start of each test (or when entering a page), register coverage using a helper like `testPageBeforeEach({TEST-PATH})`. Example usage shown below. 3. During the test execution, if additional coverage is generated, register it via the `testPage({TEST-PATH}, testInfo)` helper. 4. **In the Playwright custom reporter**, after all tests have finished, parse the collected coverage data: - Read the list of files matching `src/app/**/*.component.ts`. - Verify that each component has test coverage - If any component is not sufficiently covered, fail the CI or generate a separate report. **Example: Using `testPageBeforeEach` and `testPage`** ```ts test.describe('TEST EXAMPLE', () => { testPageBeforeEach('src/app/app.component'); ... test('TEST FOR LOGIN', async ({ page }, testInfo) => { testPage('src/app/pages/login/login.component', testInfo); ... }); ... }); ``` #### [Leveraging MCP / Test Generators] - **Automated Initial Scaffolding** - Developers no longer need to manually write each scenario. Using Playwright Codegen / MCP, initial test templates can be generated automatically and then refined by humans, dramatically increasing productivity. - **[LLM-Assisted Test Generation](https://www.checklyhq.com/blog/generate-end-to-end-tests-with-ai-and-playwright)** - Through MCP, LLMs (e.g., Copilot or ChatGPT agents) can be provided with the page state to suggest or generate test cases, enabling rapid coverage expansion. - Developed by Microsoft, MCP integrates well with IDEs like VS Code, making AI-assisted features (test code auto-generation, scenario suggestions, etc.) seamless. - Example workflow connecting Playwright test runner with MCP server: - Read the webpage state → LLM generates test cases automatically - Page Object Model is auto-created/recommended - Scenario execution and result feedback can be integrated - **[Using Test Generator Options](https://playwright.dev/docs/codegen)** - You can specify viewport, device, color scheme, geolocation, locale, timezone, `auth.json` file, etc. - `npm run e2e:codegen` - **Playwright MCP Integrations** - [Copilot](https://dev.to/debs_obrien/letting-playwright-mcp-explore-your-site-and-write-your-tests-mf1) / [Claude](https://kailash-pathak.medium.com/api-testing-with-llm-claude-and-playwright-mcp-model-context-protocol-a08d6ab979dd) / [Cursor](https://medium.com/@jagdalebr/supercharge-testing-with-playwright-mcp-server-and-cursor-ai-0e66f2430d11) / [Chrome Extension](https://kailash-pathak.medium.com/streamline-web-automation-with-the-playwright-mcp-chrome-extension-4ff9e43469cd) #### [Reference] - [apache/apisix-dashboard](https://github.com/apache/apisix-dashboard) - [Protractor deprecation: Angular Blog](https://blog.angular.dev/protractor-deprecation-update-august-2023-2beac7402ce0) - [Playwright Codegen](https://playwright.dev/docs/codegen) - [Playwright MCP / Generative test examples](https://playwright.dev/agents/playwright-mcp-generating-tests) - [Playwright Migration Guide](https://playwright.dev/docs/protractor) - [Playwright Reporter API / Custom reporters](https://playwright.dev/docs/api/class-reporter) ### What type of PR is it? Improvement Feature ### Todos - [ ] The account currently logs in as **anonymous** in the local environment, but I’m not sure how to perform a tear-up with a specific account. I’ll need to look into this further. If anyone is familiar with this, I would appreciate advice. - [ ] Regarding coverage measurement, we could consider adopting a better approach if one exists. From what I’ve found so far, there doesn’t seem to be a tool that can automatically detect and measure coverage. - [ ] The GitHub Action is running too slowly, and it seems we could improve performance by using a caching strategy. - [ ] We should discuss the approach of moving the E2E tests to a separate repository. - [ ] We need to consider how to contribute to E2E tests. Currently, the plan is as follows: - After this PR is merged, either use [ZEPPELIN-6314](https://issues.apache.org/jira/browse/ZEPPELIN-6314) or create a. new parent issue, then manage contributions as sub-issues. - To avoid duplicate work, contributors can: 1. Check the `PAGES` variable in `utils` to find areas they want to work on. 2. Leave a comment and open a new sub-issue. - Example comment: - "I would like to work on testing the header area." - Path: `src/app/share/header/header.component` ### What is the Jira issue? * [[ZEPPELIN-6314](https://issues.apache.org/jira/browse/ZEPPELIN-6314)] ### How should this be tested? ### Screenshots (if appropriate) ### Questions: * Does the license files need to update? N * Is there breaking changes for older versions? N * Does this needs documentation? N -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
