Superskyyy opened a new issue, #9939: URL: https://github.com/apache/skywalking/issues/9939
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/skywalking/issues?q=is%3Aissue) and found no similar feature requirement. ### Description Making the Python agent work in a multiprocessing-enabled environment has been the main center of concern based on my observation of user feedback. This should be addressed before we release v1.0.0. To make it easier for our users and given the new meter reporter capability contributed by @jiang1997, we need this issue to track several important changes: The core issue to address: Separate instances when a new process is spawned/forked from the original process that initialized the Python agent, so that we don't report PVM metrics from multiple Python processes to a single instance. Cases to provide solution for: 1. Common case 1: Normal Python code with fork/spawn, we reinitialize Python agent in the child process and keep the parent process agent. 2. Common case 2: Normal Python web application with pre-fork servers (Uwsgi, Gunicorn etc.), we may give an additional `sw-python -bla` CLI option to spawn only in child process (parent process does not take requests) 3. Edge case: Multiprocessing pool /ProcessPoolExecutor: If we give each new process a new instance ID, we need to be careful not spawn new agents in pool-based child processes. Otherwise, we may end up with hundreds of thousands of instances (for example, in a data science application, we may frequently use pool-based processes to process data) ### Use case Users could run Python agent smoothly in a multiprocessing Python project (which is rather common given the limitations of GIL) ### Related issues After the above changes, the following issues should all be addressed along the way. I will work with the OSPP student @jiang1997 to tackle these problems one by one. https://github.com/apache/skywalking/issues/9721 https://github.com/apache/skywalking/issues/9071 https://github.com/apache/skywalking/issues/8854 https://github.com/apache/skywalking/issues/7319 ### Are you willing to submit a PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
