[GitHub] [dolphinscheduler] rickchengx opened a new pull request, #11860: [Improvement-11857][Spark] Remove the spark version of spark task

[GitBox]

rickchengx opened a new pull request, #11860:
URL: https://github.com/apache/dolphinscheduler/pull/11860

   <!--Thanks very much for contributing to Apache DolphinScheduler. Please 
review 
https://dolphinscheduler.apache.org/en-us/community/development/pull-request.html
 before opening a pull request.-->
   
   
   ## Purpose of the pull request
   
   related: #11857 
   
   Currently, the spark version is misleading.
   <img width="577" alt="截屏2022-09-08 15 24 34" 
src="https://user-images.githubusercontent.com/38122586/189061020-9198f272-9239-4f32-a671-530c5b66a05e.png";>
   
   This is not the spark version that DS currently supports. E.g., DS can also 
run the spark task of `SPARK-3.x.x`.
   
   The spark version selected by the user only determines the environment 
variables used and the final command executed as below:
   
   <img width="640" alt="image" 
src="https://user-images.githubusercontent.com/38122586/189061737-4674445c-072f-4705-a891-f96f9ea70d61.png";>
   
   And also there is a bug that `spark-sql` can only be executed by `SPARK2`, 
see https://github.com/apache/dolphinscheduler/pull/11721
   
   So why not just remove the spark version, like other task
   And use `{SPARK_HOME}/bin/spark_submit` and `{SPARK_HOME}/bin/spark-sql`
   
   ## Brief change log
   
    use `{SPARK_HOME}/bin/spark_submit` and `{SPARK_HOME}/bin/spark-sql`
   
   ## Verify this pull request
   
   <!--*(Please pick either of the following options)*-->
   
   manually tested
   
   <img width="1021" alt="截屏2022-09-08 16 51 41" 
src="https://user-images.githubusercontent.com/38122586/189083321-837ca066-7f29-43a6-a5a9-abe723feee2f.png";>
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org