[GitHub] [spark] LuciferYang commented on a diff in pull request #37938: [SPARK-40490][YARN][TESTS] Ensure `YarnShuffleIntegrationSuite` tests registeredExecFile reload scenarios
LuciferYang commented on code in PR #37938:
URL: https://github.com/apache/spark/pull/37938#discussion_r975408239
##
common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java:
##
@@ -129,6 +130,9 @@ public class YarnShuffleService extends AuxiliaryService {
// Whether failure during service initialization should stop the NM.
@VisibleForTesting
static final String STOP_ON_FAILURE_KEY = "spark.yarn.shuffle.stopOnFailure";
+
+ @VisibleForTesting
+ static final String INTEGRATION_TESTING =
"spark.yarn.shuffle.integrationTesting";
Review Comment:
[c5f658a](https://github.com/apache/spark/pull/37938/commits/c5f658ace1d05153fcc189c0c1e3968663f7b6e7)
fix this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #37938: [SPARK-40490][YARN][TESTS] Ensure `YarnShuffleIntegrationSuite` tests registeredExecFile reload scenarios
LuciferYang commented on code in PR #37938:
URL: https://github.com/apache/spark/pull/37938#discussion_r974854247
##
common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java:
##
@@ -237,6 +241,10 @@ protected void serviceInit(Configuration externalConf)
throws Exception {
boolean stopOnFailure = _conf.getBoolean(STOP_ON_FAILURE_KEY,
DEFAULT_STOP_ON_FAILURE);
+if (_recoveryPath == null && _conf.getBoolean(INTEGRATION_TESTING, false))
{
Review Comment:
Use Utils.isTesting directly need fix some other cases, for example:
```
test("recovery db should not be created if NM recovery is not enabled") {
s1 = new YarnShuffleService
s1.init(yarnConfig)
s1._recoveryPath should be (null)
s1.registeredExecutorFile should be (null)
s1.secretsFile should be (null)
}
```
Is it better to use a separate configuration as this pr?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #37938: [SPARK-40490][YARN][TESTS] Ensure `YarnShuffleIntegrationSuite` tests registeredExecFile reload scenarios
LuciferYang commented on code in PR #37938:
URL: https://github.com/apache/spark/pull/37938#discussion_r974846876
##
common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java:
##
@@ -237,6 +241,10 @@ protected void serviceInit(Configuration externalConf)
throws Exception {
boolean stopOnFailure = _conf.getBoolean(STOP_ON_FAILURE_KEY,
DEFAULT_STOP_ON_FAILURE);
+if (_recoveryPath == null && _conf.getBoolean(INTEGRATION_TESTING, false))
{
Review Comment:
It is not sure whether it is possible to use `Utils.isTesting` directly, but
it requires a refactor work because `Utils.isTesting` cannot be used in this
module.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
