DanielLeens opened a new pull request, #11338: URL: https://github.com/apache/seatunnel/pull/11338
### Purpose of this pull request This PR revives the Python transform request discussed in [#7006](https://github.com/apache/seatunnel/issues/7006). I also reviewed the old draft PR [#8854](https://github.com/apache/seatunnel/pull/8854) for reference. The previous draft was useful for option scope and E2E placement, but the runtime implementation in this PR is a fresh design based on a lightweight stdin/stdout Python worker instead of a `py4j` bridge. ### Does this PR introduce _any_ user-facing change? Yes. Users can now define a `Python` transform in SeaTunnel jobs and choose either inline `source_code` or runtime `source_code_path` to execute Python logic during row transformation. This PR adds: - a new `Python` transform plugin for row-level scripted enrichment - `script_config`, typed output `columns`, and `row_error_handle_way` support - unit tests and E2E test resources for inline and path-based scripts - English and Chinese user documentation ### How was this patch tested? Executed locally: - `./mvnw -pl seatunnel-transforms-v2,seatunnel-e2e/seatunnel-transforms-v2-e2e/seatunnel-transforms-v2-e2e-part-2 spotless:apply` - `./mvnw -pl seatunnel-transforms-v2 -DskipITs -DskipUTs=false -Dtest=PythonTransformTest test` - `./mvnw -pl seatunnel-transforms-v2 -DskipTests install` E2E status on this machine: - `./mvnw -pl seatunnel-e2e/seatunnel-transforms-v2-e2e/seatunnel-transforms-v2-e2e-part-2 -DskipUTs=false -DskipITs=false -Dtest=TestPythonTransformIT -DfailIfNoTests=false test` - The E2E suite could not complete locally because the required starter jars were not built in this checkout, and Docker was not available on this machine for Testcontainers. Local verification note: - On this machine, `seatunnel-transforms-v2` currently has an unrelated `ScalaClassParser` constructor mismatch in existing dynamic-compile code during compilation. I used a local-only compile compatibility workaround to run the Python transform unit test and install verification, and then reverted that unrelated file before creating this PR. That workaround is not part of this PR. ### Check list * [ ] If any new Jar binary package adding in your PR, please add License Notice according [New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/contribution/new-license.md) * [x] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs * [ ] If you are contributing the connector code, please check that the following files are updated: 1. Update [plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it 2. Update the pom file of [seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml) 3. Add ci label in [label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml) 4. Add e2e testcase in [seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/) 5. Update connector [plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config) * [ ] Update the [`release-note`](https://github.com/apache/seatunnel/blob/dev/release-note.md). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
