geosmart commented on issue #5648: URL: https://github.com/apache/dolphinscheduler/issues/5648#issuecomment-868401293
when use spark on k8s(spark 2.4.8 ), I make a list what we need to do: 1. change dolphinscheudler from hdfs to minio; 2. change kill yarn job by applicationId(both in worker and master) to stop spark-driver-pod. 3. without yarn log aggregate, have to manual aggregate spark executor log by ELK or some other tool. 4. change the spark job monitor way from yarn-rest-api to k8s-api; 5... @blackberrier anything else please do some complement ? I'm doing a POC to migrate from yarn to k8s -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
