[
https://issues.apache.org/jira/browse/BEAM-944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Davor Bonaci updated BEAM-944:
------------------------------
Description:
Running any example with *any* runner throws an exception as soon as it tries
to construct PipelineOptions as long as SparkRunner is on the class path:
{code}
mvn compile exec:java
-Dexec.mainClass=org.apache.beam.examples.complete.game.LeaderBoard
-Dexec.args="--runner=DirectRunner"
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Tutorial 0.0.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ Tutorial
---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources,
i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory
/Users/fjp/dev/mobile/src/main/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ Tutorial ---
[INFO] No sources to compile
[INFO]
[INFO] --- exec-maven-plugin:1.5.0:java (default-cli) @ Tutorial ---
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-jdk14/1.7.5/slf4j-jdk14-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
[WARNING]
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:294)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ExceptionInInitializerError
at
org.apache.beam.examples.complete.game.LeaderBoard.main(LeaderBoard.java:179)
... 6 more
Caused by: java.lang.TypeNotPresentException: Type
org.apache.spark.streaming.api.java.JavaStreamingListener not present
at
sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:117)
at
sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:125)
at
sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
at
sun.reflect.generics.visitor.Reifier.reifyTypeArguments(Reifier.java:68)
at
sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:138)
at
sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
at
sun.reflect.generics.repository.ConstructorRepository.getParameterTypes(ConstructorRepository.java:94)
at
java.lang.reflect.Executable.getGenericParameterTypes(Executable.java:284)
at java.lang.reflect.Method.getGenericParameterTypes(Method.java:282)
at
java.beans.FeatureDescriptor.getParameterTypes(FeatureDescriptor.java:387)
at
java.beans.PropertyDescriptor.findPropertyType(PropertyDescriptor.java:653)
at
java.beans.PropertyDescriptor.setWriteMethod(PropertyDescriptor.java:326)
at java.beans.PropertyDescriptor.<init>(PropertyDescriptor.java:138)
at
org.apache.beam.sdk.options.PipelineOptionsFactory.getPropertyDescriptors(PipelineOptionsFactory.java:896)
at
org.apache.beam.sdk.options.PipelineOptionsFactory.validateClass(PipelineOptionsFactory.java:992)
at
org.apache.beam.sdk.options.PipelineOptionsFactory.validateWellFormed(PipelineOptionsFactory.java:627)
at
org.apache.beam.sdk.options.PipelineOptionsFactory.register(PipelineOptionsFactory.java:561)
at
org.apache.beam.sdk.options.PipelineOptionsFactory.initializeRegistry(PipelineOptionsFactory.java:587)
at
org.apache.beam.sdk.options.PipelineOptionsFactory.<clinit>(PipelineOptionsFactory.java:536)
... 7 more
Caused by: java.lang.ClassNotFoundException:
org.apache.spark.streaming.api.java.JavaStreamingListener
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at
sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:114)
... 25 more
{code}
It is introduced by https://github.com/apache/incubator-beam/pull/1072 that was
added in mid-October, so it reproduces starting with 0.3.0-incubating, but not
earlier.
A possible workaround for users is to add this explicit dependency, which is
marked as provided in the Spark runner:
{code}
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.10</artifactId>
<version>${spark.version}</version>
</dependency>
{code}
On a quick glance, JavaStreamingListener is not a real pipeline option. Perhaps
it can be just removed from pipeline options, and passed around differently.
Separately, we should perhaps re-think how this works with respect to
dependencies in general.
Marking blocking for 0.4.0-incubating since it affects Spark quickstart.
[~amitsela], what do you think?
was:
Running any example with *any* runner throws an exception as soon as it tries
to construct PipelineOptions as long as SparkRunner is on the class path:
{code}
mvn compile exec:java
-Dexec.mainClass=org.apache.beam.examples.complete.game.LeaderBoard
-Dexec.args="--runner=DirectRunner"
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Tutorial 0.0.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ Tutorial
---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources,
i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory
/Users/fjp/dev/mobile/src/main/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ Tutorial ---
[INFO] No sources to compile
[INFO]
[INFO] --- exec-maven-plugin:1.5.0:java (default-cli) @ Tutorial ---
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-jdk14/1.7.5/slf4j-jdk14-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
[WARNING]
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:294)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ExceptionInInitializerError
at
org.apache.beam.examples.complete.game.LeaderBoard.main(LeaderBoard.java:179)
... 6 more
Caused by: java.lang.TypeNotPresentException: Type
org.apache.spark.streaming.api.java.JavaStreamingListener not present
at
sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:117)
at
sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:125)
at
sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
at
sun.reflect.generics.visitor.Reifier.reifyTypeArguments(Reifier.java:68)
at
sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:138)
at
sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
at
sun.reflect.generics.repository.ConstructorRepository.getParameterTypes(ConstructorRepository.java:94)
at
java.lang.reflect.Executable.getGenericParameterTypes(Executable.java:284)
at java.lang.reflect.Method.getGenericParameterTypes(Method.java:282)
at
java.beans.FeatureDescriptor.getParameterTypes(FeatureDescriptor.java:387)
at
java.beans.PropertyDescriptor.findPropertyType(PropertyDescriptor.java:653)
at
java.beans.PropertyDescriptor.setWriteMethod(PropertyDescriptor.java:326)
at java.beans.PropertyDescriptor.<init>(PropertyDescriptor.java:138)
at
org.apache.beam.sdk.options.PipelineOptionsFactory.getPropertyDescriptors(PipelineOptionsFactory.java:896)
at
org.apache.beam.sdk.options.PipelineOptionsFactory.validateClass(PipelineOptionsFactory.java:992)
at
org.apache.beam.sdk.options.PipelineOptionsFactory.validateWellFormed(PipelineOptionsFactory.java:627)
at
org.apache.beam.sdk.options.PipelineOptionsFactory.register(PipelineOptionsFactory.java:561)
at
org.apache.beam.sdk.options.PipelineOptionsFactory.initializeRegistry(PipelineOptionsFactory.java:587)
at
org.apache.beam.sdk.options.PipelineOptionsFactory.<clinit>(PipelineOptionsFactory.java:536)
... 7 more
Caused by: java.lang.ClassNotFoundException:
org.apache.spark.streaming.api.java.JavaStreamingListener
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at
sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:114)
... 25 more
{code}
It is caused by https://github.com/apache/incubator-beam/pull/1072 that was
added in mid-October, so it reproduces starting with 0.3.0-incubating, but not
earlier.
A possible workaround for users is to add this explicit dependency, which is
marked as provided in the Spark runner:
{code}
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.10</artifactId>
<version>${spark.version}</version>
</dependency>
{code}
On a quick glance, JavaStreamingListener is not a real pipeline option. Perhaps
it can be just removed from pipeline options, and passed around differently.
Separately, we should perhaps re-think how this works with respect to
dependencies in general.
Marking blocking for 0.4.0-incubating since it affects Spark quickstart.
[~amitsela], what do you think?
> Spark runner causes an exception when creating pipeline options
> ---------------------------------------------------------------
>
> Key: BEAM-944
> URL: https://issues.apache.org/jira/browse/BEAM-944
> Project: Beam
> Issue Type: Bug
> Components: runner-spark
> Affects Versions: 0.3.0-incubating
> Reporter: Davor Bonaci
> Assignee: Amit Sela
> Fix For: 0.4.0-incubating
>
>
> Running any example with *any* runner throws an exception as soon as it tries
> to construct PipelineOptions as long as SparkRunner is on the class path:
> {code}
> mvn compile exec:java
> -Dexec.mainClass=org.apache.beam.examples.complete.game.LeaderBoard
> -Dexec.args="--runner=DirectRunner"
> [INFO] Scanning for projects...
> [INFO]
>
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Building Tutorial 0.0.1-SNAPSHOT
> [INFO]
> ------------------------------------------------------------------------
> [INFO]
> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
> Tutorial ---
> [WARNING] Using platform encoding (UTF-8 actually) to copy filtered
> resources, i.e. build is platform dependent!
> [INFO] skip non existing resourceDirectory
> /Users/fjp/dev/mobile/src/main/resources
> [INFO]
> [INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ Tutorial ---
> [INFO] No sources to compile
> [INFO]
> [INFO] --- exec-maven-plugin:1.5.0:java (default-cli) @ Tutorial ---
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-jdk14/1.7.5/slf4j-jdk14-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> [WARNING]
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:294)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ExceptionInInitializerError
> at
> org.apache.beam.examples.complete.game.LeaderBoard.main(LeaderBoard.java:179)
> ... 6 more
> Caused by: java.lang.TypeNotPresentException: Type
> org.apache.spark.streaming.api.java.JavaStreamingListener not present
> at
> sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:117)
> at
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:125)
> at
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
> at
> sun.reflect.generics.visitor.Reifier.reifyTypeArguments(Reifier.java:68)
> at
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:138)
> at
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
> at
> sun.reflect.generics.repository.ConstructorRepository.getParameterTypes(ConstructorRepository.java:94)
> at
> java.lang.reflect.Executable.getGenericParameterTypes(Executable.java:284)
> at java.lang.reflect.Method.getGenericParameterTypes(Method.java:282)
> at
> java.beans.FeatureDescriptor.getParameterTypes(FeatureDescriptor.java:387)
> at
> java.beans.PropertyDescriptor.findPropertyType(PropertyDescriptor.java:653)
> at
> java.beans.PropertyDescriptor.setWriteMethod(PropertyDescriptor.java:326)
> at java.beans.PropertyDescriptor.<init>(PropertyDescriptor.java:138)
> at
> org.apache.beam.sdk.options.PipelineOptionsFactory.getPropertyDescriptors(PipelineOptionsFactory.java:896)
> at
> org.apache.beam.sdk.options.PipelineOptionsFactory.validateClass(PipelineOptionsFactory.java:992)
> at
> org.apache.beam.sdk.options.PipelineOptionsFactory.validateWellFormed(PipelineOptionsFactory.java:627)
> at
> org.apache.beam.sdk.options.PipelineOptionsFactory.register(PipelineOptionsFactory.java:561)
> at
> org.apache.beam.sdk.options.PipelineOptionsFactory.initializeRegistry(PipelineOptionsFactory.java:587)
> at
> org.apache.beam.sdk.options.PipelineOptionsFactory.<clinit>(PipelineOptionsFactory.java:536)
> ... 7 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.spark.streaming.api.java.JavaStreamingListener
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at
> sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:114)
> ... 25 more
> {code}
> It is introduced by https://github.com/apache/incubator-beam/pull/1072 that
> was added in mid-October, so it reproduces starting with 0.3.0-incubating,
> but not earlier.
> A possible workaround for users is to add this explicit dependency, which is
> marked as provided in the Spark runner:
> {code}
> <dependency>
> <groupId>org.apache.spark</groupId>
> <artifactId>spark-streaming_2.10</artifactId>
> <version>${spark.version}</version>
> </dependency>
> {code}
> On a quick glance, JavaStreamingListener is not a real pipeline option.
> Perhaps it can be just removed from pipeline options, and passed around
> differently.
> Separately, we should perhaps re-think how this works with respect to
> dependencies in general.
> Marking blocking for 0.4.0-incubating since it affects Spark quickstart.
> [~amitsela], what do you think?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
