[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #11857: [Improvement][Spark] Remove the spark version of spark task

[GitBox]

github-actions[bot] commented on issue #11857:
URL: 
https://github.com/apache/dolphinscheduler/issues/11857#issuecomment-1240406077

   ### Search before asking
   
   - [X] I had searched in the 
[issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and 
found no similar feature requirement.
   
   
   ### Description
   
   Currently, the spark version is misleading.
   <img width="577" alt="截屏2022-09-08 15 24 34" 
src="https://user-images.githubusercontent.com/38122586/189061020-9198f272-9239-4f32-a671-530c5b66a05e.png";>
   
   This is not the spark version that DS currently supports. E.g., DS can also 
run the spark task of `SPARK-3.x.x`.
   
   The spark version selected by the user only determines the environment 
variables used and the final command executed as below:
   
   <img width="640" alt="image" 
src="https://user-images.githubusercontent.com/38122586/189061737-4674445c-072f-4705-a891-f96f9ea70d61.png";>
   
   And also there is a bug that `spark-sql` can only be executed by `SPARK2`, 
see https://github.com/apache/dolphinscheduler/pull/11721
   
   So why not just remove the spark version, like other task (e.g., `Flink`)
   And use `{SPARK_HOME}/spark_submit` and `{SPARK_HOME}/spark-sql`
   
   ### Are you willing to submit a PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org