[
https://issues.apache.org/jira/browse/SPARK-22077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16183117#comment-16183117
]
Eric Vandenberg commented on SPARK-22077:
-----------------------------------------
We are using a Facebook internal cluster scheduler. I don't think the issue is
the cluster manager, the actual java.net.URI call returns null for host, name
etc (see description above) so it is immediately throwing at this point. Are
you saying the example above *does* parse correctly for you? If so, then I
wonder if the discrepancy is due to different JDK versions.
According to java.net.URI the URL supports IPv6
(https://docs.oracle.com/javase/7/docs/api/java/net/URI.html) as defined in the
RFC 2732 (http://www.ietf.org/rfc/rfc2732.txt) which includes IPv6 address
enclosed in square braces [], for example:
http://[FEDC:BA98:7654:3210:FEDC:BA98:7654:3210]:80/index.html
Thanks,
Eric
> RpcEndpointAddress fails to parse spark URL if it is an ipv6 address.
> ---------------------------------------------------------------------
>
> Key: SPARK-22077
> URL: https://issues.apache.org/jira/browse/SPARK-22077
> Project: Spark
> Issue Type: Bug
> Components: Deploy
> Affects Versions: 2.0.0
> Reporter: Eric Vandenberg
> Priority: Minor
>
> RpcEndpointAddress fails to parse spark URL if it is an ipv6 address.
> For example,
> sparkUrl = "spark://HeartbeatReceiver@2401:db00:2111:40a1:face:0:21:0:35243"
> is parsed as:
> host = null
> port = -1
> name = null
> While sparkUrl = spark://HeartbeatReceiver@localhost:55691 is parsed properly.
> This is happening on our production machines and causing spark to not start
> up.
> org.apache.spark.SparkException: Invalid Spark URL:
> spark://HeartbeatReceiver@2401:db00:2111:40a1:face:0:21:0:35243
> at
> org.apache.spark.rpc.RpcEndpointAddress$.apply(RpcEndpointAddress.scala:65)
> at
> org.apache.spark.rpc.netty.NettyRpcEnv.asyncSetupEndpointRefByURI(NettyRpcEnv.scala:133)
> at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:88)
> at org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEnv.scala:96)
> at org.apache.spark.util.RpcUtils$.makeDriverRef(RpcUtils.scala:32)
> at org.apache.spark.executor.Executor.<init>(Executor.scala:121)
> at
> org.apache.spark.scheduler.local.LocalEndpoint.<init>(LocalSchedulerBackend.scala:59)
> at
> org.apache.spark.scheduler.local.LocalSchedulerBackend.start(LocalSchedulerBackend.scala:126)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:173)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:507)
> at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2283)
> at
> org.apache.spark.sql.SparkSession$Builder$$anonfun$8.apply(SparkSession.scala:833)
> at
> org.apache.spark.sql.SparkSession$Builder$$anonfun$8.apply(SparkSession.scala:825)
> at scala.Option.getOrElse(Option.scala:121)
> at
> org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:825)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
