pegasas commented on code in PR #15589:
URL:
https://github.com/apache/dolphinscheduler/pull/15589#discussion_r1493948848
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-spark/src/main/java/org/apache/dolphinscheduler/plugin/task/spark/SparkTask.java:
##########
@@ -124,22 +124,30 @@ protected Map<String, String> getProperties() {
*/
private List<String> populateSparkOptions() {
List<String> args = new ArrayList<>();
- args.add(SparkConstants.MASTER);
+ // see
https://spark.apache.org/docs/latest/submitting-applications.html
String deployMode =
StringUtils.isNotEmpty(sparkParameters.getDeployMode()) ?
sparkParameters.getDeployMode()
: SparkConstants.DEPLOY_MODE_LOCAL;
+ boolean onLocal = SparkConstants.DEPLOY_MODE_LOCAL.equals(deployMode);
boolean onNativeKubernetes =
StringUtils.isNotEmpty(sparkParameters.getNamespace());
- String masterUrl = onNativeKubernetes ? SPARK_ON_K8S_MASTER_PREFIX +
-
Config.fromKubeconfig(taskExecutionContext.getK8sTaskExecutionContext().getConfigYaml()).getMasterUrl()
- : SparkConstants.SPARK_ON_YARN;
+ String masterUrl = StringUtils.isNotEmpty(sparkParameters.getMaster())
? sparkParameters.getMaster()
Review Comment:
> @pegasas please add UT for this PR. Also, I think the code here is less
readable, WDYT?
Sure. I will add more UTs for this function.
Maybe we can build a function for this to make it clear?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
