Tal Barda created SPARK-24192:
---------------------------------
Summary: Invalid Spark URL in local spark session since upgrading
from org.apache.spark:spark-sql_2.11:2.2.1 to
org.apache.spark:spark-sql_2.11:2.3.0
Key: SPARK-24192
URL: https://issues.apache.org/jira/browse/SPARK-24192
Project: Spark
Issue Type: Bug
Components: Spark Core, SQL
Affects Versions: 2.3.0
Reporter: Tal Barda
since updating to Spark 2.3.0, tests which are run in my CI (Codeship) fail due
to a allegedly invalid spark url when creating the (local) spark context.
Here's a log from my _*mvn clean install*_ command execution:
{quote}{{2018-05-03 13:18:47.668 ERROR 5533 --- [ main]
org.apache.spark.SparkContext : Error initializing SparkContext.
org.apache.spark.SparkException: Invalid Spark URL:
spark://HeartbeatReceiver@railsonfire_61eb1c99-232b-49d0-abb5-a1eb9693516b_52bcc09bb48b:44284
at org.apache.spark.rpc.RpcEndpointAddress$.apply(RpcEndpointAddress.scala:66)
~[spark-core_2.11-2.3.0.jar:2.3.0] at
org.apache.spark.rpc.netty.NettyRpcEnv.asyncSetupEndpointRefByURI(NettyRpcEnv.scala:134)
~[spark-core_2.11-2.3.0.jar:2.3.0] at
org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:101)
~[spark-core_2.11-2.3.0.jar:2.3.0] at
org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEnv.scala:109)
~[spark-core_2.11-2.3.0.jar:2.3.0] at
org.apache.spark.util.RpcUtils$.makeDriverRef(RpcUtils.scala:32)
~[spark-core_2.11-2.3.0.jar:2.3.0] at
org.apache.spark.executor.Executor.<init>(Executor.scala:155)
~[spark-core_2.11-2.3.0.jar:2.3.0] at
org.apache.spark.scheduler.local.LocalEndpoint.<init>(LocalSchedulerBackend.scala:59)
~[spark-core_2.11-2.3.0.jar:2.3.0] at
org.apache.spark.scheduler.local.LocalSchedulerBackend.start(LocalSchedulerBackend.scala:126)
~[spark-core_2.11-2.3.0.jar:2.3.0] at
org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:164)
~[spark-core_2.11-2.3.0.jar:2.3.0] at
org.apache.spark.SparkContext.<init>(SparkContext.scala:500)
~[spark-core_2.11-2.3.0.jar:2.3.0] at
org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2486)
[spark-core_2.11-2.3.0.jar:2.3.0] at
org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:930)
[spark-sql_2.11-2.3.0.jar:2.3.0] at
org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:921)
[spark-sql_2.11-2.3.0.jar:2.3.0] at scala.Option.getOrElse(Option.scala:121)
[scala-library-2.11.8.jar:na] at
org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:921)
[spark-sql_2.11-2.3.0.jar:2.3.0] at
com.planck.spark_data_features_extractors.utils.SparkConfig.sparkSession(SparkConfig.java:43)
[classes/:na] at
com.planck.spark_data_features_extractors.utils.SparkConfig$$EnhancerBySpringCGLIB$$66dd1f72.CGLIB$sparkSession$0(<generated>)
[classes/:na] at
com.planck.spark_data_features_extractors.utils.SparkConfig$$EnhancerBySpringCGLIB$$66dd1f72$$FastClassBySpringCGLIB$$a213b647.invoke(<generated>)
[classes/:na] at
org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
[spring-core-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:361)
[spring-context-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
com.planck.spark_data_features_extractors.utils.SparkConfig$$EnhancerBySpringCGLIB$$66dd1f72.sparkSession(<generated>)
[classes/:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[na:1.8.0_171] at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[na:1.8.0_171] at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[na:1.8.0_171] at java.lang.reflect.Method.invoke(Method.java:498)
~[na:1.8.0_171] at
org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:579)
[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1250)
[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1099)
[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:545)
[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:502)
[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:312)
[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228)
~[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:310)
[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200)
[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:251)
~[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1135)
~[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1062)
~[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:583)
~[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:91)
~[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:372)
~[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1344)
[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582)
[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:502)
[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:312)
[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228)
~[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:310)
[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200)
[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:251)
~[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1135)
~[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1062)
~[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:583)
~[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:91)
~[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:372)
~[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1344)
[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582)
[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:502)
[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:312)
[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228)
~[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:310)
[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200)
[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:758)
~[spring-beans-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:868)
~[spring-context-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:549)
~[spring-context-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.springframework.boot.SpringApplication.refresh(SpringApplication.java:751)
~[spring-boot-2.0.0.M7.jar:2.0.0.M7] at
org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:387)
~[spring-boot-2.0.0.M7.jar:2.0.0.M7] at
org.springframework.boot.SpringApplication.run(SpringApplication.java:327)
~[spring-boot-2.0.0.M7.jar:2.0.0.M7] at
org.springframework.boot.test.context.SpringBootContextLoader.loadContext(SpringBootContextLoader.java:138)
~[spring-boot-test-2.0.0.M7.jar:2.0.0.M7] at
org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContextInternal(DefaultCacheAwareContextLoaderDelegate.java:99)
~[spring-test-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContext(DefaultCacheAwareContextLoaderDelegate.java:117)
~[spring-test-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.springframework.test.context.support.DefaultTestContext.getApplicationContext(DefaultTestContext.java:107)
~[spring-test-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.springframework.test.context.web.ServletTestExecutionListener.setUpRequestContextIfNecessary(ServletTestExecutionListener.java:190)
~[spring-test-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.springframework.test.context.web.ServletTestExecutionListener.prepareTestInstance(ServletTestExecutionListener.java:132)
~[spring-test-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.springframework.test.context.TestContextManager.prepareTestInstance(TestContextManager.java:242)
~[spring-test-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.springframework.test.context.junit4.SpringJUnit4ClassRunner.createTest(SpringJUnit4ClassRunner.java:227)
~[spring-test-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.springframework.test.context.junit4.SpringJUnit4ClassRunner$1.runReflectiveCall(SpringJUnit4ClassRunner.java:289)
~[spring-test-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
~[junit-4.12.jar:4.12] at
org.springframework.test.context.junit4.SpringJUnit4ClassRunner.methodBlock(SpringJUnit4ClassRunner.java:291)
~[spring-test-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:246)
~[spring-test-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:97)
~[spring-test-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
~[junit-4.12.jar:4.12] at
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
~[junit-4.12.jar:4.12] at
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
~[junit-4.12.jar:4.12] at
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
~[junit-4.12.jar:4.12] at
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
~[junit-4.12.jar:4.12] at
org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
~[spring-test-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)
~[spring-test-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.junit.runners.ParentRunner.run(ParentRunner.java:363)
~[junit-4.12.jar:4.12] at
org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:190)
~[spring-test-5.0.2.RELEASE.jar:5.0.2.RELEASE] at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369)
~[surefire-junit4-2.20.1.jar:2.20.1] at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275)
~[surefire-junit4-2.20.1.jar:2.20.1] at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239)
~[surefire-junit4-2.20.1.jar:2.20.1] at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160)
~[surefire-junit4-2.20.1.jar:2.20.1] at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373)
~[surefire-booter-2.20.1.jar:2.20.1] at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334)
~[surefire-booter-2.20.1.jar:2.20.1] at
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119)
~[surefire-booter-2.20.1.jar:2.20.1] at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407)
~[surefire-booter-2.20.1.jar:2.20.1]}}
{quote}
my SparkConf object is initialized this way:
{quote}new SparkConf().setAppName("SparkForFeaturesExtraction")
.setMaster("local)
.set("spark.driver.memory", "512m")
.set("spark.executor.memory", "512m")
.set("spark.cores.max", 1)
.set("spark.default.parallelism", 2)
.set("spark.sql.shuffle.partitions", 2)
{quote}
also see the issue in
https://stackoverflow.com/questions/49143271/invalid-spark-url-in-local-spark-session
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
