det101 commented on issue #17752: URL: https://github.com/apache/dolphinscheduler/issues/17752#issuecomment-3588253428
> Thanks for bringing this proposal. Before PR, there are some details needs to be discuss. > > 1. How to verify the task type is `Yarn` or `K8S` if the task run through a shell or python or java? > 2. Yarn task stores applicationIds in DS, and a DS task may correspond to multiple application IDs. How to correspond to each other one by one in this case? > 3. Yarn performs the operation of task kill through applicationId. If we only save the appName set by DS, how can we ensure that Yarn kill is normal? [@det101](https://github.com/det101) These are all great questions, This is what I think about 1.For Shell/Python tasks we can parse the command line (e.g., detect spark-submit --master yarn or k8s://) to infer the target platform. Java tasks are much more of a black box—if the code doesn’t log where it’s submitting, we simply can’t tell. I’d rather not support auto-detection there; users should handle shutdown via the hooks in their own code. 2.As far as I know, each submit generates exactly one Yarn application entry point; the multiple “jobs” you see are internal and don’t matter. I haven’t encountered a scenario where a single submit produces multiple application IDs. 3.On Yarn, the workflow is: use the REST API to filter by appName, find the matching applicationId, then kill that applicationId. @SbloodyS -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
