James Z.M. Gao created SPARK-2507:
-------------------------------------
Summary: Compile error of streaming project with 2.0.0-cdh4.6.0
Key: SPARK-2507
URL: https://issues.apache.org/jira/browse/SPARK-2507
Project: Spark
Issue Type: Bug
Components: Streaming
Affects Versions: 1.0.0, 0.9.1, 0.9.0
Environment: RedHat 5.3
2.0.0-cdh4.6.0
enable yarn
java version "1.6.0_45"
Reporter: James Z.M. Gao
Priority: Minor
Hi,
When compiling with
{quote}
./make-distribution.sh --hadoop 2.0.0-cdh4.6.0 --with-yarn --tgz
{quote}
I have the following errors on streaming java api:
{quote}
Version is 0.9.0-incubating
Making spark-0.9.0-incubating-hadoop_2.0.0-cdh4.6.0-bin.tar.gz
Hadoop version set to 2.0.0-cdh4.6.0
YARN enabled
[info] Loading project definition from /root/spark-source/project/project
[info] Loading project definition from /root/spark-source/project
[info] Set current project to root (in build file:/root/spark-source/)
[info] Compiling 1 Scala source to
/root/spark-source/streaming/target/scala-2.10/classes...
[error]
/root/spark-source/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:57:
type mismatch;
[error] found : org.apache.spark.streaming.dstream.DStream[(K, V)]
[error] required: org.apache.spark.streaming.api.java.JavaPairDStream[K,V]
[error] Note: implicit method fromPairDStream is not applicable here because
it comes after the application point and it lacks an explicit result type
[error] dstream.filter((x => f(x).booleanValue()))
[error] ^
[error]
/root/spark-source/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:60:
type mismatch;
[error] found : org.apache.spark.streaming.dstream.DStream[(K, V)]
[error] required: org.apache.spark.streaming.api.java.JavaPairDStream[K,V]
[error] Note: implicit method fromPairDStream is not applicable here because
it comes after the application point and it lacks an explicit result type
[error] def cache(): JavaPairDStream[K, V] = dstream.cache()
[error] ^
[error]
/root/spark-source/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:63:
type mismatch;
[error] found : org.apache.spark.streaming.dstream.DStream[(K, V)]
[error] required: org.apache.spark.streaming.api.java.JavaPairDStream[K,V]
[error] Note: implicit method fromPairDStream is not applicable here because
it comes after the application point and it lacks an explicit result type
[error] def persist(): JavaPairDStream[K, V] = dstream.persist()
[error] ^
......
[error]
/root/spark-source/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:669:
type mismatch;
[error] found : org.apache.spark.streaming.dstream.DStream[(K,
(com.google.common.base.Optional[V], W))]
[error] required:
org.apache.spark.streaming.api.java.JavaPairDStream[K,(com.google.common.base.Optional[V],
W)]
[error] Note: implicit method fromPairDStream is not applicable here because
it comes after the application point and it lacks an explicit result type
[error] joinResult.mapValues{case (v, w) => (JavaUtils.optionToOptional(v),
w)}
[error] ^
[error] 44 errors found
[error] (streaming/compile:compile) Compilation failed
{quote}
Here is a simple PR fix this problem: https://github.com/apache/spark/pull/153
--
This message was sent by Atlassian JIRA
(v6.2#6252)
