Jacek Laskowski created SPARK-34264:
---------------------------------------
Summary: Prevent incomplete master URLs for Spark on Kubernetes
early
Key: SPARK-34264
URL: https://issues.apache.org/jira/browse/SPARK-34264
Project: Spark
Issue Type: Improvement
Components: Kubernetes, Spark Submit
Affects Versions: 3.0.1, 3.1.1
Reporter: Jacek Laskowski
It turns out that {{--master k8s://}} is accepted and although leads to
termination displays stacktraces that don't really tell what the real cause is.
This may happen when the Kubernetes API server(s) are described by an
environment variable that's not initialized in the current terminal.
{code}
$ ./bin/spark-shell --master k8s:// --verbose
...
Spark config:
(spark.jars,)
(spark.app.name,Spark shell)
(spark.submit.pyFiles,)
(spark.ui.showConsoleProgress,true)
(spark.submit.deployMode,client)
(spark.master,k8s://https://)
...
21/01/27 14:29:44 ERROR Main: Failed to initialize Spark session.
io.fabric8.kubernetes.client.KubernetesClientException: Failed to start
websocket
at
io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onFailure(WatchConnectionManager.java:208)
...
Caused by: java.net.UnknownHostException: api: nodename nor servname provided,
or not known
at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at
java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:929)
at
java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1519)
at
java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:848)
at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1509)
at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1368)
at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1302)
at okhttp3.Dns$1.lookup(Dns.java:40)
at
okhttp3.internal.connection.RouteSelector.resetNextInetSocketAddress(RouteSelector.java:185)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
