LuciferYang commented on code in PR #38079:
URL: https://github.com/apache/spark/pull/38079#discussion_r986060446
##########
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:
##########
@@ -1513,7 +1520,9 @@ private[spark] object Client extends Logging {
}
sys.env.get(ENV_DIST_CLASSPATH).foreach { cp =>
- addClasspathEntry(getClusterPath(sparkConf, cp), env)
+ val newCp = cp.split(CLASS_PATH_SEPARATOR)
+ .filterNot(cpSet.contains).mkString(CLASS_PATH_SEPARATOR)
+ addClasspathEntry(getClusterPath(sparkConf, newCp), env)
Review Comment:
Is it necessary to change it to
```
val newCp = if (Utils.isTesting) {
cp.split(File.pathSeparator)
.filterNot(cpSet.contains).mkString(File.pathSeparator)
} else cp
```
##########
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:
##########
@@ -1513,7 +1520,9 @@ private[spark] object Client extends Logging {
}
sys.env.get(ENV_DIST_CLASSPATH).foreach { cp =>
- addClasspathEntry(getClusterPath(sparkConf, cp), env)
+ val newCp = cp.split(CLASS_PATH_SEPARATOR)
+ .filterNot(cpSet.contains).mkString(CLASS_PATH_SEPARATOR)
+ addClasspathEntry(getClusterPath(sparkConf, newCp), env)
Review Comment:
Is it necessary to change it to
```
val newCp = if (Utils.isTesting) {
cp.split(File.pathSeparator)
.filterNot(cpSet.contains).mkString(File.pathSeparator)
} else cp
```
but I think de duplication may also be useful for the production environment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
