rmannibucau opened a new pull request, #9481:
URL: https://github.com/apache/hop/pull/9481
## Summary
Introduce a generic, opt-in execution observation contract for local
pipelines and workflows. Execution lifetime is separate from thread-local
activation, allowing integrations to observe complete invocations without
introducing a telemetry SDK dependency into Hop.
- Add execution handles, kinds, terminal states, observers and closeable
activation scopes.
- Isolate observer failures from business execution; preserve existing
extension-point semantics.
- Observe actual transform copies and action clones, including
initialization, processing and disposal.
- Capture the actual launching action/transform before asynchronous child
dispatch in ActionPipeline, ActionWorkflow, PipelineExecutor and
WorkflowExecutor.
- Support local multithreaded and single-threaded execution, with no per-row
execution identities or notifications. Without observers, no execution registry
entries or generated identities are created.
- Defer pipeline terminal publication until startup and running workers have
settled; handle fast completion, callback failures and partial worker launch.
- Add lifecycle, restoration, child-parentage, asynchronous action and
runner-completion regression tests.
There is no OpenTelemetry, MDC or exporter dependency in the engine changes.
A separate integration was used to exercise the contract; its plugin code and
dependencies are not included here.
## Scope and review points
This is a proposed public API, based on Hop
`2d995a647a223a53630ef958462628fe00c6c513` (2.21.0-SNAPSHOT, Java 21).
Maintainer agreement on the API and a complete independent lifecycle review are
still outstanding.
Execution lifetime and activation are intentionally distinct. A stop request
is not completion, asynchronous children may outlive their launcher, and legacy
workflow-runner postludes remain separate from the engine invocation. See
`docs/otel-execution-design.md`.
Remote execution, HopServer, Beam/Spark workers and arbitrary third-party
threads are not qualified by this change. Long-duration/production-scale leak
and performance qualification is also outside the validation performed here.
## Validation
### Fresh pre-PR checks
Passed:
```sh
mvn -q -pl engine jacoco:prepare-agent surefire:test \
-Dtest=ExecutionLifecycleTest,PipelineExecutionLifecycleTest,PipelineTest,WorkflowLifecycleTest,ExecutionFailurePipelineTest,ExecutionFailureWorkflowTest
\
-Dspotless.skip=true -Drat.skip=true -DargLine= \
-DforkedProcessTimeoutInSeconds=180
mvn -q -pl
engine,plugins/actions/pipeline,plugins/actions/workflow,plugins/transforms/pipelineexecutor,plugins/transforms/workflowexecutor,plugins/engines/single-threaded
\
-Dspotless.skip=false spotless:check
git diff --cached --check
```
**Known current test failure (reason for draft status):** the launcher-suite
command below was rerun twice and timed out in
`ActionWorkflowAsyncLifecycleTest.realNoWaitLauncherCapturesActionBeforeParentEndsAndDoesNotWaitForChild`
at its 10-second JUnit timeout. The root cause has not been established; this
is not being claimed as an environment-only failure. The reactor stops in the
workflow-action module, so subsequent executor modules were not freshly
revalidated by these attempts.
```sh
mvn -q -pl
plugins/actions/pipeline,plugins/actions/workflow,plugins/transforms/pipelineexecutor,plugins/transforms/workflowexecutor,plugins/engines/single-threaded
\
jacoco:prepare-agent surefire:test \
-Dspotless.skip=true -Drat.skip=true -DargLine= \
-DforkedProcessTimeoutInSeconds=180
```
### Earlier bounded qualification (historical, not a replacement for the
failure above)
A previous isolated campaign passed in 281.929 seconds, excluding
compilation, packaging and downloads. Hop-side results were 73 engine tests, 9
pipeline-action tests, 5 workflow-action tests, 56 PipelineExecutor tests, 9
WorkflowExecutor tests and 4 single-threaded-engine tests, with no
failures/errors/skips in that campaign.
The external integration campaign also exercised distribution ZIPs with an
OTLP Collector, exact child parentage, 520 repeated real pipeline runs and
2,600 pipeline/copy invocations, and warmed multithreaded/single-threaded
benchmarks. These checks do not constitute exhaustive qualification or a stable
performance guarantee. Their harness is outside this PR.
## Contribution checklist
- [ ] Run `mvn clean install apache-rat:check` (full reactor/license check
not run for this PR).
- [x] Keep the change in one commit.
- [ ] Mention a corresponding issue, if applicable (none linked).
- [ ] I hereby declare this contribution to be licensed under the Apache
License Version 2.0, January 2004 (left for the contributor to acknowledge).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]