rokadepiyush49-rgb commented on issue #18011:
URL: https://github.com/apache/beam/issues/18011#issuecomment-6565278374
Checking where this stands today: most runners seem to support
`ProfilingOptions` already.
- `FnApiRunner` builds a profiler with `Profile.factory_from_options`
(`fn_runner.py`)
- the Python SDK harness reads `ProfilingOptions` in `sdk_worker_main.py`,
which covers Prism and the portable runners (Dataflow, Flink, Spark), since
they all run the harness
- `SwitchingDirectRunner` now tries Prism first, then `FnApiRunner`
The one place I couldn't find a profiling hook is `BundleBasedDirectRunner`
in `runners/direct/`, which the direct runner only falls back to for pipelines
neither Prism nor `FnApiRunner` can run.
So this could either be closed as mostly done, or narrowed to wiring
`Profile.factory_from_options` into `BundleBasedDirectRunner`. Which would you
prefer? I'm happy to do the second if it's still useful.
If you'd rather narrow it than close it, this is what I'd do:
<details>
<summary>Proposed implementation plan (diagram)</summary>
```mermaid
flowchart TD
subgraph P1["1. Before coding"]
A1["Post the status comment"] --> A2{"Maintainer decision"}
A2 -- "close as mostly done" --> A3["Done: the issue is
closed,<br/>no code needed"]
A2 -- "narrow to BundleBasedDirectRunner" --> A4["Comment
.take-issue, branch from master"]
end
subgraph P2["2. Implement in runners/direct"]
B1["Read how fn_runner.py wires<br/>Profile.factory_from_options"]
--> B2["BundleBasedDirectRunner.run_pipeline:<br/>build the profiler factory
from ProfilingOptions"]
B2 --> B3["Wrap pipeline execution with the profiler,<br/>following
the FnApiRunner pattern"]
end
subgraph P3["3. Test in direct_runner_test.py"]
C1["Use BundleBasedDirectRunner directly,<br/>so Prism and
FnApiRunner are not picked"] --> C2["Run with profile_cpu and a temp
profile_location"]
C2 --> C3["Assert a profile file was written"]
C3 --> C4["Without profiling options: nothing is written"]
end
subgraph P4["4. Pull request"]
D1["gh pr create --repo apache/beam<br/>body: Fixes 18011"]
end
A4 --> B1
B3 --> C1
C4 --> D1
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
