[
https://issues.apache.org/jira/browse/SPARK-24192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16825616#comment-16825616
]
Aseem Patni commented on SPARK-24192:
-------------------------------------
export SPARK_LOCAL_HOSTNAME=localhost should also fix this.
> Invalid Spark URL in local spark session since upgrading from
> org.apache.spark:spark-sql_2.11:2.2.1 to org.apache.spark:spark-sql_2.11:2.3.0
> --------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-24192
> URL: https://issues.apache.org/jira/browse/SPARK-24192
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, SQL
> Affects Versions: 2.3.0
> Reporter: Tal Barda
> Priority: Major
> Labels: HeartbeatReceiver, config, session, spark, spark-conf,
> spark-session
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> since updating to Spark 2.3.0, tests which are run in my CI (Codeship) fail
> due to a allegedly invalid spark url when creating the (local) spark context.
> Here's a log from my _*mvn clean install*_ command execution:
> {quote}{{2018-05-03 13:18:47.668 ERROR 5533 --- [ main]
> org.apache.spark.SparkContext : Error initializing SparkContext.
> org.apache.spark.SparkException: Invalid Spark URL:
> spark://HeartbeatReceiver@railsonfire_61eb1c99-232b-49d0-abb5-a1eb9693516b_52bcc09bb48b:44284
> at
> org.apache.spark.rpc.RpcEndpointAddress$.apply(RpcEndpointAddress.scala:66)
> ~[spark-core_2.11-2.3.0.jar:2.3.0] at
> org.apache.spark.rpc.netty.NettyRpcEnv.asyncSetupEndpointRefByURI(NettyRpcEnv.scala:134)
> ~[spark-core_2.11-2.3.0.jar:2.3.0] at
> org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:101)
> ~[spark-core_2.11-2.3.0.jar:2.3.0] at
> org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEnv.scala:109)
> ~[spark-core_2.11-2.3.0.jar:2.3.0] at
> org.apache.spark.util.RpcUtils$.makeDriverRef(RpcUtils.scala:32)
> ~[spark-core_2.11-2.3.0.jar:2.3.0] at
> org.apache.spark.executor.Executor.<init>(Executor.scala:155)
> ~[spark-core_2.11-2.3.0.jar:2.3.0] at
> org.apache.spark.scheduler.local.LocalEndpoint.<init>(LocalSchedulerBackend.scala:59)
> ~[spark-core_2.11-2.3.0.jar:2.3.0] at
> org.apache.spark.scheduler.local.LocalSchedulerBackend.start(LocalSchedulerBackend.scala:126)
> ~[spark-core_2.11-2.3.0.jar:2.3.0] at
> org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:164)
> ~[spark-core_2.11-2.3.0.jar:2.3.0] at
> org.apache.spark.SparkContext.<init>(SparkContext.scala:500)
> ~[spark-core_2.11-2.3.0.jar:2.3.0] at
> org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2486)
> [spark-core_2.11-2.3.0.jar:2.3.0] at
> org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:930)
> [spark-sql_2.11-2.3.0.jar:2.3.0] at
> org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:921)
> [spark-sql_2.11-2.3.0.jar:2.3.0] at scala.Option.getOrElse(Option.scala:121)
> [scala-library-2.11.8.jar:na] at
> org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:921)
> [spark-sql_2.11-2.3.0.jar:2.3.0] at
> com.planck.spark_data_features_extractors.utils.SparkConfig.sparkSession(SparkConfig.java:43)
> [classes/:na] at
> com.planck.spark_data_features_extractors.utils.SparkConfig$$EnhancerBySpringCGLIB$$66dd1f72.CGLIB$sparkSession$0(<generated>)
> [classes/:na] at
> com.planck.spark_data_features_extractors.utils.SparkConfig$$EnhancerBySpringCGLIB$$66dd1f72$$FastClassBySpringCGLIB$$a213b647.invoke(<generated>)
> [classes/:na] at
> org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
> [spring-core-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:361)
> [spring-context-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> com.planck.spark_data_features_extractors.utils.SparkConfig$$EnhancerBySpringCGLIB$$66dd1f72.sparkSession(<generated>)
> [classes/:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[na:1.8.0_171] at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[na:1.8.0_171] at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[na:1.8.0_171] at java.lang.reflect.Method.invoke(Method.java:498)
> ~[na:1.8.0_171] at
> org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
> [spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:579)
> [spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1250)
> [spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1099)
> [spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:545)
> [spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:502)
> [spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:312)
> [spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228)
> ~[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:310)
> [spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200)
> [spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:251)
> ~[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1135)
> ~[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1062)
> ~[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:583)
> ~[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:91)
> ~[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:372)
> ~[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1344)
> [spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582)
> [spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:502)
> [spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:312)
> [spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228)
> ~[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:310)
> [spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200)
> [spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:251)
> ~[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1135)
> ~[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1062)
> ~[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:583)
> ~[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:91)
> ~[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:372)
> ~[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1344)
> [spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582)
> [spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:502)
> [spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:312)
> [spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228)
> ~[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:310)
> [spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200)
> [spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:758)
> ~[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:868)
> ~[spring-context-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:549)
> ~[spring-context-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.springframework.boot.SpringApplication.refresh(SpringApplication.java:751)
> ~[spring-boot-2.0.0.M7.jar:2.0.0.M7] at
> org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:387)
> ~[spring-boot-2.0.0.M7.jar:2.0.0.M7] at
> org.springframework.boot.SpringApplication.run(SpringApplication.java:327)
> ~[spring-boot-2.0.0.M7.jar:2.0.0.M7] at
> org.springframework.boot.test.context.SpringBootContextLoader.loadContext(SpringBootContextLoader.java:138)
> ~[spring-boot-test-2.0.0.M7.jar:2.0.0.M7] at
> org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContextInternal(DefaultCacheAwareContextLoaderDelegate.java:99)
> ~[spring-test-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContext(DefaultCacheAwareContextLoaderDelegate.java:117)
> ~[spring-test-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.springframework.test.context.support.DefaultTestContext.getApplicationContext(DefaultTestContext.java:107)
> ~[spring-test-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.springframework.test.context.web.ServletTestExecutionListener.setUpRequestContextIfNecessary(ServletTestExecutionListener.java:190)
> ~[spring-test-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.springframework.test.context.web.ServletTestExecutionListener.prepareTestInstance(ServletTestExecutionListener.java:132)
> ~[spring-test-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.springframework.test.context.TestContextManager.prepareTestInstance(TestContextManager.java:242)
> ~[spring-test-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.springframework.test.context.junit4.SpringJUnit4ClassRunner.createTest(SpringJUnit4ClassRunner.java:227)
> ~[spring-test-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.springframework.test.context.junit4.SpringJUnit4ClassRunner$1.runReflectiveCall(SpringJUnit4ClassRunner.java:289)
> ~[spring-test-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> ~[junit-4.12.jar:4.12] at
> org.springframework.test.context.junit4.SpringJUnit4ClassRunner.methodBlock(SpringJUnit4ClassRunner.java:291)
> ~[spring-test-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:246)
> ~[spring-test-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:97)
> ~[spring-test-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> ~[junit-4.12.jar:4.12] at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> ~[junit-4.12.jar:4.12] at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> ~[junit-4.12.jar:4.12] at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> ~[junit-4.12.jar:4.12] at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> ~[junit-4.12.jar:4.12] at
> org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
> ~[spring-test-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)
> ~[spring-test-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> ~[junit-4.12.jar:4.12] at
> org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:190)
> ~[spring-test-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369)
> ~[surefire-junit4-2.20.1.jar:2.20.1] at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275)
> ~[surefire-junit4-2.20.1.jar:2.20.1] at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239)
> ~[surefire-junit4-2.20.1.jar:2.20.1] at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160)
> ~[surefire-junit4-2.20.1.jar:2.20.1] at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373)
> ~[surefire-booter-2.20.1.jar:2.20.1] at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334)
> ~[surefire-booter-2.20.1.jar:2.20.1] at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119)
> ~[surefire-booter-2.20.1.jar:2.20.1] at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407)
> ~[surefire-booter-2.20.1.jar:2.20.1]}}
> {quote}
>
> my SparkConf object is initialized this way:
> {quote}new SparkConf().setAppName("SparkForFeaturesExtraction")
> .setMaster("local)
> .set("spark.driver.memory", "512m")
> .set("spark.executor.memory", "512m")
> .set("spark.cores.max", 1)
> .set("spark.default.parallelism", 2)
> .set("spark.sql.shuffle.partitions", 2)
> {quote}
>
> also see the issue in
> https://stackoverflow.com/questions/49143271/invalid-spark-url-in-local-spark-session
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
