This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new e86d90b [SPARK-32824][CORE] Improve the error message when the user
forgets the .amount in a resource config
e86d90b is described below
commit e86d90b21d4b3d6658b3cb6dd30daafb32b0c1bd
Author: Thomas Graves <[email protected]>
AuthorDate: Wed Sep 9 10:28:40 2020 +0900
[SPARK-32824][CORE] Improve the error message when the user forgets the
.amount in a resource config
### What changes were proposed in this pull request?
If the user forgets to specify .amount on a resource config like
spark.executor.resource.gpu, the error message thrown is very confusing:
```
ERROR SparkContext: Error initializing
SparkContext.java.lang.StringIndexOutOfBoundsException: String index out of
range:
-1 at java.lang.String.substring(String.java:1967) at
```
This makes it so we have a readable error thrown
### Why are the changes needed?
confusing error for users
### Does this PR introduce _any_ user-facing change?
just error message
### How was this patch tested?
Tested manually on standalone cluster
Closes #29685 from tgravescs/SPARK-32824.
Authored-by: Thomas Graves <[email protected]>
Signed-off-by: HyukjinKwon <[email protected]>
(cherry picked from commit e8634d8f6f8548852a284a32c1b7da24bedd8ff7)
Signed-off-by: HyukjinKwon <[email protected]>
---
core/src/main/scala/org/apache/spark/resource/ResourceUtils.scala | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/core/src/main/scala/org/apache/spark/resource/ResourceUtils.scala
b/core/src/main/scala/org/apache/spark/resource/ResourceUtils.scala
index 994b363..16fe897 100644
--- a/core/src/main/scala/org/apache/spark/resource/ResourceUtils.scala
+++ b/core/src/main/scala/org/apache/spark/resource/ResourceUtils.scala
@@ -148,7 +148,12 @@ private[spark] object ResourceUtils extends Logging {
def listResourceIds(sparkConf: SparkConf, componentName: String):
Seq[ResourceID] = {
sparkConf.getAllWithPrefix(s"$componentName.$RESOURCE_PREFIX.").map { case
(key, _) =>
- key.substring(0, key.indexOf('.'))
+ val index = key.indexOf('.')
+ if (index < 0) {
+ throw new SparkException(s"You must specify an amount config for
resource: $key " +
+ s"config: $componentName.$RESOURCE_PREFIX.$key")
+ }
+ key.substring(0, index)
}.distinct.map(name => new ResourceID(componentName, name))
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
