Re: reference to dstream in package org.apache.spark.streaming which is not available

[Tathagata Das]

The real fix is that the spark sink suite does not really need to use to
the spark-streaming test jars. Removing that dependency altogether, and
submitting a PR.

TD


On Fri, Aug 22, 2014 at 6:34 PM, Tathagata Das 
wrote:

> Figured it out. Fixing this ASAP.
>
> TD
>
>
> On Fri, Aug 22, 2014 at 5:51 PM, Patrick Wendell 
> wrote:
>
>> Hey All,
>>
>> We can sort this out ASAP. Many of the Spark committers were at a company
>> offsite for the last 72 hours, so sorry that it is broken.
>>
>> - Patrick
>>
>>
>> On Fri, Aug 22, 2014 at 4:07 PM, Hari Shreedharan <
>> hshreedha...@cloudera.com
>> > wrote:
>>
>> > Sean - I think only the ones in 1726 are enough. It is weird that any
>> > class that uses the test-jar actually requires the streaming jar to be
>> > added explicitly. Shouldn't maven take care of this?
>> >
>> > I posted some comments on the PR.
>> >
>> > --
>> >
>> > Thanks,
>> > Hari
>> >
>> >
>> >  Sean Owen 
>> >> August 22, 2014 at 3:58 PM
>> >>
>> >> Yes, master hasn't compiled for me for a few days. It's fixed in:
>> >>
>> >> https://github.com/apache/spark/pull/1726
>> >> https://github.com/apache/spark/pull/2075
>> >>
>> >> Could a committer sort this out?
>> >>
>> >> Sean
>> >>
>> >>
>> >> -
>> >> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
>> >> For additional commands, e-mail: dev-h...@spark.apache.org
>> >>
>> >> Ted Yu 
>> >> August 22, 2014 at 1:55 PM
>> >>
>> >> Hi,
>> >> Using the following command on (refreshed) master branch:
>> >> mvn clean package -DskipTests
>> >>
>> >> I got:
>> >>
>> >> constituent[36]: file:/homes/hortonzy/apache-maven-3.1.1/conf/logging/
>> >> ---
>> >> java.lang.reflect.InvocationTargetException
>> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> >> at
>> >> sun.reflect.NativeMethodAccessorImpl.invoke(
>> >> NativeMethodAccessorImpl.java:57)
>> >> at
>> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(
>> >> DelegatingMethodAccessorImpl.java:43)
>> >> at java.lang.reflect.Method.invoke(Method.java:606)
>> >> at
>> >> org.codehaus.plexus.classworlds.launcher.Launcher.
>> >> launchEnhanced(Launcher.java:289)
>> >> at
>> >> org.codehaus.plexus.classworlds.launcher.Launcher.
>> >> launch(Launcher.java:229)
>> >> at
>> >> org.codehaus.plexus.classworlds.launcher.Launcher.
>> >> mainWithExitCode(Launcher.java:415)
>> >> at org.codehaus.plexus.classworlds.launcher.Launcher.
>> >> main(Launcher.java:356)
>> >> Caused by: scala.reflect.internal.Types$TypeError: bad symbolic
>> >> reference.
>> >> A signature in TestSuiteBase.class refers to term dstream
>> >> in package org.apache.spark.streaming which is not available.
>> >> It may be completely missing from the current classpath, or the
>> version on
>> >> the classpath might be incompatible with the version used when
>> compiling
>> >> TestSuiteBase.class.
>> >> at
>> >> scala.reflect.internal.pickling.UnPickler$Scan.
>> >> toTypeError(UnPickler.scala:847)
>> >> at
>> >> scala.reflect.internal.pickling.UnPickler$Scan$LazyTypeRef.complete(
>> >> UnPickler.scala:854)
>> >> at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1231)
>> >> at
>> >>
>> scala.reflect.internal.Types$TypeMap$$anonfun$noChangeToSymbols$1.apply(
>> >> Types.scala:4280)
>> >> at
>> >>
>> scala.reflect.internal.Types$TypeMap$$anonfun$noChangeToSymbols$1.apply(
>> >> Types.scala:4280)
>> >> at
>> >> scala.collection.LinearSeqOptimized$class.forall(LinearSeqOptimized.
>> >> scala:70)
>> >> at scala.collection.immutable.List.forall(List.scala:84)
>> >> at scala.reflect.internal.Types$TypeMap.noChangeToSymbols(
>> >> Types.scala:4280)
>> >> at scala.reflect.internal.Types$TypeMap.mapOver(Types.scala:4293)
>> >> at scala.reflect.internal.Types$TypeMap.mapOver(Types.scala:4196)
>> >> at scala.reflect.internal.Types$AsSeenFromMap.apply(Types.scala:4638)
>> >> at scala.reflect.internal.Types$TypeMap.mapOver(Types.scala:4202)
>> >> at scala.reflect.internal.Types$AsSeenFromMap.apply(Types.scala:4638)
>> >> at scala.reflect.internal.Types$Type.asSeenFrom(Types.scala:754)
>> >> at scala.reflect.internal.Types$Type.memberInfo(Types.scala:773)
>> >> at xsbt.ExtractAPI.defDef(ExtractAPI.scala:224)
>> >> at xsbt.ExtractAPI.xsbt$ExtractAPI$$definition(ExtractAPI.scala:315)
>> >> at
>> >> xsbt.ExtractAPI$$anonfun$xsbt$ExtractAPI$$processDefinitions$1.apply(
>> >> ExtractAPI.scala:296)
>> >> at
>> >> xsbt.ExtractAPI$$anonfun$xsbt$ExtractAPI$$processDefinitions$1.apply(
>> >> ExtractAPI.scala:296)
>> >> at
>> >> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(
>> >> TraversableLike.scala:251)
>> >> at
>> >> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(
>> >> TraversableLike.scala:251)
>> >> at
>> >> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.
>> >> scala:33)
>> >> at scala.collection.mutable.ArrayOps$ofR

Re: reference to dstream in package org.apache.spark.streaming which is not available

[Tathagata Das]

Figured it out. Fixing this ASAP.

TD


On Fri, Aug 22, 2014 at 5:51 PM, Patrick Wendell  wrote:

> Hey All,
>
> We can sort this out ASAP. Many of the Spark committers were at a company
> offsite for the last 72 hours, so sorry that it is broken.
>
> - Patrick
>
>
> On Fri, Aug 22, 2014 at 4:07 PM, Hari Shreedharan <
> hshreedha...@cloudera.com
> > wrote:
>
> > Sean - I think only the ones in 1726 are enough. It is weird that any
> > class that uses the test-jar actually requires the streaming jar to be
> > added explicitly. Shouldn't maven take care of this?
> >
> > I posted some comments on the PR.
> >
> > --
> >
> > Thanks,
> > Hari
> >
> >
> >  Sean Owen 
> >> August 22, 2014 at 3:58 PM
> >>
> >> Yes, master hasn't compiled for me for a few days. It's fixed in:
> >>
> >> https://github.com/apache/spark/pull/1726
> >> https://github.com/apache/spark/pull/2075
> >>
> >> Could a committer sort this out?
> >>
> >> Sean
> >>
> >>
> >> -
> >> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
> >> For additional commands, e-mail: dev-h...@spark.apache.org
> >>
> >> Ted Yu 
> >> August 22, 2014 at 1:55 PM
> >>
> >> Hi,
> >> Using the following command on (refreshed) master branch:
> >> mvn clean package -DskipTests
> >>
> >> I got:
> >>
> >> constituent[36]: file:/homes/hortonzy/apache-maven-3.1.1/conf/logging/
> >> ---
> >> java.lang.reflect.InvocationTargetException
> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> at
> >> sun.reflect.NativeMethodAccessorImpl.invoke(
> >> NativeMethodAccessorImpl.java:57)
> >> at
> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(
> >> DelegatingMethodAccessorImpl.java:43)
> >> at java.lang.reflect.Method.invoke(Method.java:606)
> >> at
> >> org.codehaus.plexus.classworlds.launcher.Launcher.
> >> launchEnhanced(Launcher.java:289)
> >> at
> >> org.codehaus.plexus.classworlds.launcher.Launcher.
> >> launch(Launcher.java:229)
> >> at
> >> org.codehaus.plexus.classworlds.launcher.Launcher.
> >> mainWithExitCode(Launcher.java:415)
> >> at org.codehaus.plexus.classworlds.launcher.Launcher.
> >> main(Launcher.java:356)
> >> Caused by: scala.reflect.internal.Types$TypeError: bad symbolic
> >> reference.
> >> A signature in TestSuiteBase.class refers to term dstream
> >> in package org.apache.spark.streaming which is not available.
> >> It may be completely missing from the current classpath, or the version
> on
> >> the classpath might be incompatible with the version used when compiling
> >> TestSuiteBase.class.
> >> at
> >> scala.reflect.internal.pickling.UnPickler$Scan.
> >> toTypeError(UnPickler.scala:847)
> >> at
> >> scala.reflect.internal.pickling.UnPickler$Scan$LazyTypeRef.complete(
> >> UnPickler.scala:854)
> >> at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1231)
> >> at
> >> scala.reflect.internal.Types$TypeMap$$anonfun$noChangeToSymbols$1.apply(
> >> Types.scala:4280)
> >> at
> >> scala.reflect.internal.Types$TypeMap$$anonfun$noChangeToSymbols$1.apply(
> >> Types.scala:4280)
> >> at
> >> scala.collection.LinearSeqOptimized$class.forall(LinearSeqOptimized.
> >> scala:70)
> >> at scala.collection.immutable.List.forall(List.scala:84)
> >> at scala.reflect.internal.Types$TypeMap.noChangeToSymbols(
> >> Types.scala:4280)
> >> at scala.reflect.internal.Types$TypeMap.mapOver(Types.scala:4293)
> >> at scala.reflect.internal.Types$TypeMap.mapOver(Types.scala:4196)
> >> at scala.reflect.internal.Types$AsSeenFromMap.apply(Types.scala:4638)
> >> at scala.reflect.internal.Types$TypeMap.mapOver(Types.scala:4202)
> >> at scala.reflect.internal.Types$AsSeenFromMap.apply(Types.scala:4638)
> >> at scala.reflect.internal.Types$Type.asSeenFrom(Types.scala:754)
> >> at scala.reflect.internal.Types$Type.memberInfo(Types.scala:773)
> >> at xsbt.ExtractAPI.defDef(ExtractAPI.scala:224)
> >> at xsbt.ExtractAPI.xsbt$ExtractAPI$$definition(ExtractAPI.scala:315)
> >> at
> >> xsbt.ExtractAPI$$anonfun$xsbt$ExtractAPI$$processDefinitions$1.apply(
> >> ExtractAPI.scala:296)
> >> at
> >> xsbt.ExtractAPI$$anonfun$xsbt$ExtractAPI$$processDefinitions$1.apply(
> >> ExtractAPI.scala:296)
> >> at
> >> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(
> >> TraversableLike.scala:251)
> >> at
> >> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(
> >> TraversableLike.scala:251)
> >> at
> >> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.
> >> scala:33)
> >> at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
> >> at scala.collection.TraversableLike$class.flatMap(
> >> TraversableLike.scala:251)
> >> at scala.collection.mutable.ArrayOps$ofRef.flatMap(ArrayOps.scala:108)
> >> at xsbt.ExtractAPI.xsbt$ExtractAPI$$processDefinitions(ExtractAPI.
> >> scala:296)
> >> at xsbt.ExtractAPI$$anonfun$mkStructure$4.apply(ExtractAPI.scala:293)
> >> at xsb

Re: reference to dstream in package org.apache.spark.streaming which is not available

[Patrick Wendell]

Hey All,

We can sort this out ASAP. Many of the Spark committers were at a company
offsite for the last 72 hours, so sorry that it is broken.

- Patrick


On Fri, Aug 22, 2014 at 4:07 PM, Hari Shreedharan  wrote:

> Sean - I think only the ones in 1726 are enough. It is weird that any
> class that uses the test-jar actually requires the streaming jar to be
> added explicitly. Shouldn't maven take care of this?
>
> I posted some comments on the PR.
>
> --
>
> Thanks,
> Hari
>
>
>  Sean Owen 
>> August 22, 2014 at 3:58 PM
>>
>> Yes, master hasn't compiled for me for a few days. It's fixed in:
>>
>> https://github.com/apache/spark/pull/1726
>> https://github.com/apache/spark/pull/2075
>>
>> Could a committer sort this out?
>>
>> Sean
>>
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
>> For additional commands, e-mail: dev-h...@spark.apache.org
>>
>> Ted Yu 
>> August 22, 2014 at 1:55 PM
>>
>> Hi,
>> Using the following command on (refreshed) master branch:
>> mvn clean package -DskipTests
>>
>> I got:
>>
>> constituent[36]: file:/homes/hortonzy/apache-maven-3.1.1/conf/logging/
>> ---
>> java.lang.reflect.InvocationTargetException
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(
>> NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(
>> DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at
>> org.codehaus.plexus.classworlds.launcher.Launcher.
>> launchEnhanced(Launcher.java:289)
>> at
>> org.codehaus.plexus.classworlds.launcher.Launcher.
>> launch(Launcher.java:229)
>> at
>> org.codehaus.plexus.classworlds.launcher.Launcher.
>> mainWithExitCode(Launcher.java:415)
>> at org.codehaus.plexus.classworlds.launcher.Launcher.
>> main(Launcher.java:356)
>> Caused by: scala.reflect.internal.Types$TypeError: bad symbolic
>> reference.
>> A signature in TestSuiteBase.class refers to term dstream
>> in package org.apache.spark.streaming which is not available.
>> It may be completely missing from the current classpath, or the version on
>> the classpath might be incompatible with the version used when compiling
>> TestSuiteBase.class.
>> at
>> scala.reflect.internal.pickling.UnPickler$Scan.
>> toTypeError(UnPickler.scala:847)
>> at
>> scala.reflect.internal.pickling.UnPickler$Scan$LazyTypeRef.complete(
>> UnPickler.scala:854)
>> at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1231)
>> at
>> scala.reflect.internal.Types$TypeMap$$anonfun$noChangeToSymbols$1.apply(
>> Types.scala:4280)
>> at
>> scala.reflect.internal.Types$TypeMap$$anonfun$noChangeToSymbols$1.apply(
>> Types.scala:4280)
>> at
>> scala.collection.LinearSeqOptimized$class.forall(LinearSeqOptimized.
>> scala:70)
>> at scala.collection.immutable.List.forall(List.scala:84)
>> at scala.reflect.internal.Types$TypeMap.noChangeToSymbols(
>> Types.scala:4280)
>> at scala.reflect.internal.Types$TypeMap.mapOver(Types.scala:4293)
>> at scala.reflect.internal.Types$TypeMap.mapOver(Types.scala:4196)
>> at scala.reflect.internal.Types$AsSeenFromMap.apply(Types.scala:4638)
>> at scala.reflect.internal.Types$TypeMap.mapOver(Types.scala:4202)
>> at scala.reflect.internal.Types$AsSeenFromMap.apply(Types.scala:4638)
>> at scala.reflect.internal.Types$Type.asSeenFrom(Types.scala:754)
>> at scala.reflect.internal.Types$Type.memberInfo(Types.scala:773)
>> at xsbt.ExtractAPI.defDef(ExtractAPI.scala:224)
>> at xsbt.ExtractAPI.xsbt$ExtractAPI$$definition(ExtractAPI.scala:315)
>> at
>> xsbt.ExtractAPI$$anonfun$xsbt$ExtractAPI$$processDefinitions$1.apply(
>> ExtractAPI.scala:296)
>> at
>> xsbt.ExtractAPI$$anonfun$xsbt$ExtractAPI$$processDefinitions$1.apply(
>> ExtractAPI.scala:296)
>> at
>> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(
>> TraversableLike.scala:251)
>> at
>> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(
>> TraversableLike.scala:251)
>> at
>> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.
>> scala:33)
>> at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
>> at scala.collection.TraversableLike$class.flatMap(
>> TraversableLike.scala:251)
>> at scala.collection.mutable.ArrayOps$ofRef.flatMap(ArrayOps.scala:108)
>> at xsbt.ExtractAPI.xsbt$ExtractAPI$$processDefinitions(ExtractAPI.
>> scala:296)
>> at xsbt.ExtractAPI$$anonfun$mkStructure$4.apply(ExtractAPI.scala:293)
>> at xsbt.ExtractAPI$$anonfun$mkStructure$4.apply(ExtractAPI.scala:293)
>> at xsbt.Message$$anon$1.apply(Message.scala:8)
>> at xsbti.SafeLazy$$anonfun$apply$1.apply(SafeLazy.scala:8)
>> at xsbti.SafeLazy$Impl._t$lzycompute(SafeLazy.scala:20)
>> at xsbti.SafeLazy$Impl._t(SafeLazy.scala:18)
>> at xsbti.SafeLazy$Impl.get(SafeLazy.scala:24)
>> at xsbt.ExtractAPI$$anonfun$forceStructures$1.app

Re: reference to dstream in package org.apache.spark.streaming which is not available

[Hari Shreedharan]

Sean - I think only the ones in 1726 are enough. It is weird that any 
class that uses the test-jar actually requires the streaming jar to be 
added explicitly. Shouldn't maven take care of this?


I posted some comments on the PR.

--

Thanks,
Hari



Sean Owen 
August 22, 2014 at 3:58 PM
Yes, master hasn't compiled for me for a few days. It's fixed in:

https://github.com/apache/spark/pull/1726
https://github.com/apache/spark/pull/2075

Could a committer sort this out?

Sean


-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Ted Yu 
August 22, 2014 at 1:55 PM
Hi,
Using the following command on (refreshed) master branch:
mvn clean package -DskipTests

I got:

constituent[36]: file:/homes/hortonzy/apache-maven-3.1.1/conf/logging/
---
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
at
org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
at
org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at 
org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)

Caused by: scala.reflect.internal.Types$TypeError: bad symbolic reference.
A signature in TestSuiteBase.class refers to term dstream
in package org.apache.spark.streaming which is not available.
It may be completely missing from the current classpath, or the version on
the classpath might be incompatible with the version used when compiling
TestSuiteBase.class.
at
scala.reflect.internal.pickling.UnPickler$Scan.toTypeError(UnPickler.scala:847)
at
scala.reflect.internal.pickling.UnPickler$Scan$LazyTypeRef.complete(UnPickler.scala:854)
at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1231)
at
scala.reflect.internal.Types$TypeMap$$anonfun$noChangeToSymbols$1.apply(Types.scala:4280)
at
scala.reflect.internal.Types$TypeMap$$anonfun$noChangeToSymbols$1.apply(Types.scala:4280)
at
scala.collection.LinearSeqOptimized$class.forall(LinearSeqOptimized.scala:70)
at scala.collection.immutable.List.forall(List.scala:84)
at 
scala.reflect.internal.Types$TypeMap.noChangeToSymbols(Types.scala:4280)

at scala.reflect.internal.Types$TypeMap.mapOver(Types.scala:4293)
at scala.reflect.internal.Types$TypeMap.mapOver(Types.scala:4196)
at scala.reflect.internal.Types$AsSeenFromMap.apply(Types.scala:4638)
at scala.reflect.internal.Types$TypeMap.mapOver(Types.scala:4202)
at scala.reflect.internal.Types$AsSeenFromMap.apply(Types.scala:4638)
at scala.reflect.internal.Types$Type.asSeenFrom(Types.scala:754)
at scala.reflect.internal.Types$Type.memberInfo(Types.scala:773)
at xsbt.ExtractAPI.defDef(ExtractAPI.scala:224)
at xsbt.ExtractAPI.xsbt$ExtractAPI$$definition(ExtractAPI.scala:315)
at
xsbt.ExtractAPI$$anonfun$xsbt$ExtractAPI$$processDefinitions$1.apply(ExtractAPI.scala:296)
at
xsbt.ExtractAPI$$anonfun$xsbt$ExtractAPI$$processDefinitions$1.apply(ExtractAPI.scala:296)
at
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
at 
scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)

at scala.collection.mutable.ArrayOps$ofRef.flatMap(ArrayOps.scala:108)
at 
xsbt.ExtractAPI.xsbt$ExtractAPI$$processDefinitions(ExtractAPI.scala:296)

at xsbt.ExtractAPI$$anonfun$mkStructure$4.apply(ExtractAPI.scala:293)
at xsbt.ExtractAPI$$anonfun$mkStructure$4.apply(ExtractAPI.scala:293)
at xsbt.Message$$anon$1.apply(Message.scala:8)
at xsbti.SafeLazy$$anonfun$apply$1.apply(SafeLazy.scala:8)
at xsbti.SafeLazy$Impl._t$lzycompute(SafeLazy.scala:20)
at xsbti.SafeLazy$Impl._t(SafeLazy.scala:18)
at xsbti.SafeLazy$Impl.get(SafeLazy.scala:24)
at xsbt.ExtractAPI$$anonfun$forceStructures$1.apply(ExtractAPI.scala:138)
at xsbt.ExtractAPI$$anonfun$forceStructures$1.apply(ExtractAPI.scala:138)
at scala.collection.immutable.List.foreach(List.scala:318)
at xsbt.ExtractAPI.forceStructures(ExtractAPI.scala:138)
at xsbt.ExtractAPI.forceStructures(ExtractAPI.scala:139)
at xsbt.API$ApiPhase.processScalaUnit(API.scala:54)
at xsbt.API$ApiPhase.processUnit(API.scala:38)
at xsbt.API$ApiPhase$$anonfun$run$1.apply(API.scala:34)
at xsbt.API$ApiPhase$$anonfun$run$1.apply(API.scala:34)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.Abs

Re: reference to dstream in package org.apache.spark.streaming which is not available

[Sean Owen]

Yes, master hasn't compiled for me for a few days. It's fixed in:

https://github.com/apache/spark/pull/1726
https://github.com/apache/spark/pull/2075

Could a committer sort this out?

Sean


On Fri, Aug 22, 2014 at 9:55 PM, Ted Yu  wrote:
> Hi,
> Using the following command on (refreshed) master branch:
> mvn clean package -DskipTests
>
> I got:
>
> constituent[36]: file:/homes/hortonzy/apache-maven-3.1.1/conf/logging/
> ---
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
> at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
> Caused by: scala.reflect.internal.Types$TypeError: bad symbolic reference.
> A signature in TestSuiteBase.class refers to term dstream
> in package org.apache.spark.streaming which is not available.
> It may be completely missing from the current classpath, or the version on
> the classpath might be incompatible with the version used when compiling
> TestSuiteBase.class.
> at
> scala.reflect.internal.pickling.UnPickler$Scan.toTypeError(UnPickler.scala:847)
> at
> scala.reflect.internal.pickling.UnPickler$Scan$LazyTypeRef.complete(UnPickler.scala:854)
> at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1231)
> at
> scala.reflect.internal.Types$TypeMap$$anonfun$noChangeToSymbols$1.apply(Types.scala:4280)
> at
> scala.reflect.internal.Types$TypeMap$$anonfun$noChangeToSymbols$1.apply(Types.scala:4280)
> at
> scala.collection.LinearSeqOptimized$class.forall(LinearSeqOptimized.scala:70)
> at scala.collection.immutable.List.forall(List.scala:84)
> at scala.reflect.internal.Types$TypeMap.noChangeToSymbols(Types.scala:4280)
> at scala.reflect.internal.Types$TypeMap.mapOver(Types.scala:4293)
> at scala.reflect.internal.Types$TypeMap.mapOver(Types.scala:4196)
> at scala.reflect.internal.Types$AsSeenFromMap.apply(Types.scala:4638)
> at scala.reflect.internal.Types$TypeMap.mapOver(Types.scala:4202)
> at scala.reflect.internal.Types$AsSeenFromMap.apply(Types.scala:4638)
> at scala.reflect.internal.Types$Type.asSeenFrom(Types.scala:754)
> at scala.reflect.internal.Types$Type.memberInfo(Types.scala:773)
> at xsbt.ExtractAPI.defDef(ExtractAPI.scala:224)
> at xsbt.ExtractAPI.xsbt$ExtractAPI$$definition(ExtractAPI.scala:315)
> at
> xsbt.ExtractAPI$$anonfun$xsbt$ExtractAPI$$processDefinitions$1.apply(ExtractAPI.scala:296)
> at
> xsbt.ExtractAPI$$anonfun$xsbt$ExtractAPI$$processDefinitions$1.apply(ExtractAPI.scala:296)
> at
> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
> at
> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
> at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
> at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
> at scala.collection.mutable.ArrayOps$ofRef.flatMap(ArrayOps.scala:108)
> at xsbt.ExtractAPI.xsbt$ExtractAPI$$processDefinitions(ExtractAPI.scala:296)
> at xsbt.ExtractAPI$$anonfun$mkStructure$4.apply(ExtractAPI.scala:293)
> at xsbt.ExtractAPI$$anonfun$mkStructure$4.apply(ExtractAPI.scala:293)
> at xsbt.Message$$anon$1.apply(Message.scala:8)
> at xsbti.SafeLazy$$anonfun$apply$1.apply(SafeLazy.scala:8)
> at xsbti.SafeLazy$Impl._t$lzycompute(SafeLazy.scala:20)
> at xsbti.SafeLazy$Impl._t(SafeLazy.scala:18)
> at xsbti.SafeLazy$Impl.get(SafeLazy.scala:24)
> at xsbt.ExtractAPI$$anonfun$forceStructures$1.apply(ExtractAPI.scala:138)
> at xsbt.ExtractAPI$$anonfun$forceStructures$1.apply(ExtractAPI.scala:138)
> at scala.collection.immutable.List.foreach(List.scala:318)
> at xsbt.ExtractAPI.forceStructures(ExtractAPI.scala:138)
> at xsbt.ExtractAPI.forceStructures(ExtractAPI.scala:139)
> at xsbt.API$ApiPhase.processScalaUnit(API.scala:54)
> at xsbt.API$ApiPhase.processUnit(API.scala:38)
> at xsbt.API$ApiPhase$$anonfun$run$1.apply(API.scala:34)
> at xsbt.API$ApiPhase$$anonfun$run$1.apply(API.scala:34)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at xsbt.API$ApiPhase.run(API.scala:34)
> at scala.tools.nsc.Global$Run.compileUnitsInternal(Global.scala:1583)
> at scala.tools.nsc.Global$Run.compileUnits(Global.scala:1557)
> at scala.tools.nsc.Global$Run.compileSources(Global.scala:1553)
> at scala.tools.nsc.Global$Run.compile(Global.scala:1662)
> at xsbt.CachedCompi

reference to dstream in package org.apache.spark.streaming which is not available

[Ted Yu]

Hi,
Using the following command on (refreshed) master branch:
mvn clean package -DskipTests

I got:

constituent[36]: file:/homes/hortonzy/apache-maven-3.1.1/conf/logging/
---
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
at
org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
at
org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: scala.reflect.internal.Types$TypeError: bad symbolic reference.
A signature in TestSuiteBase.class refers to term dstream
in package org.apache.spark.streaming which is not available.
It may be completely missing from the current classpath, or the version on
the classpath might be incompatible with the version used when compiling
TestSuiteBase.class.
at
scala.reflect.internal.pickling.UnPickler$Scan.toTypeError(UnPickler.scala:847)
at
scala.reflect.internal.pickling.UnPickler$Scan$LazyTypeRef.complete(UnPickler.scala:854)
at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1231)
at
scala.reflect.internal.Types$TypeMap$$anonfun$noChangeToSymbols$1.apply(Types.scala:4280)
at
scala.reflect.internal.Types$TypeMap$$anonfun$noChangeToSymbols$1.apply(Types.scala:4280)
at
scala.collection.LinearSeqOptimized$class.forall(LinearSeqOptimized.scala:70)
at scala.collection.immutable.List.forall(List.scala:84)
at scala.reflect.internal.Types$TypeMap.noChangeToSymbols(Types.scala:4280)
at scala.reflect.internal.Types$TypeMap.mapOver(Types.scala:4293)
at scala.reflect.internal.Types$TypeMap.mapOver(Types.scala:4196)
at scala.reflect.internal.Types$AsSeenFromMap.apply(Types.scala:4638)
at scala.reflect.internal.Types$TypeMap.mapOver(Types.scala:4202)
at scala.reflect.internal.Types$AsSeenFromMap.apply(Types.scala:4638)
at scala.reflect.internal.Types$Type.asSeenFrom(Types.scala:754)
at scala.reflect.internal.Types$Type.memberInfo(Types.scala:773)
at xsbt.ExtractAPI.defDef(ExtractAPI.scala:224)
at xsbt.ExtractAPI.xsbt$ExtractAPI$$definition(ExtractAPI.scala:315)
at
xsbt.ExtractAPI$$anonfun$xsbt$ExtractAPI$$processDefinitions$1.apply(ExtractAPI.scala:296)
at
xsbt.ExtractAPI$$anonfun$xsbt$ExtractAPI$$processDefinitions$1.apply(ExtractAPI.scala:296)
at
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
at scala.collection.mutable.ArrayOps$ofRef.flatMap(ArrayOps.scala:108)
at xsbt.ExtractAPI.xsbt$ExtractAPI$$processDefinitions(ExtractAPI.scala:296)
at xsbt.ExtractAPI$$anonfun$mkStructure$4.apply(ExtractAPI.scala:293)
at xsbt.ExtractAPI$$anonfun$mkStructure$4.apply(ExtractAPI.scala:293)
at xsbt.Message$$anon$1.apply(Message.scala:8)
at xsbti.SafeLazy$$anonfun$apply$1.apply(SafeLazy.scala:8)
at xsbti.SafeLazy$Impl._t$lzycompute(SafeLazy.scala:20)
at xsbti.SafeLazy$Impl._t(SafeLazy.scala:18)
at xsbti.SafeLazy$Impl.get(SafeLazy.scala:24)
at xsbt.ExtractAPI$$anonfun$forceStructures$1.apply(ExtractAPI.scala:138)
at xsbt.ExtractAPI$$anonfun$forceStructures$1.apply(ExtractAPI.scala:138)
at scala.collection.immutable.List.foreach(List.scala:318)
at xsbt.ExtractAPI.forceStructures(ExtractAPI.scala:138)
at xsbt.ExtractAPI.forceStructures(ExtractAPI.scala:139)
at xsbt.API$ApiPhase.processScalaUnit(API.scala:54)
at xsbt.API$ApiPhase.processUnit(API.scala:38)
at xsbt.API$ApiPhase$$anonfun$run$1.apply(API.scala:34)
at xsbt.API$ApiPhase$$anonfun$run$1.apply(API.scala:34)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at xsbt.API$ApiPhase.run(API.scala:34)
at scala.tools.nsc.Global$Run.compileUnitsInternal(Global.scala:1583)
at scala.tools.nsc.Global$Run.compileUnits(Global.scala:1557)
at scala.tools.nsc.Global$Run.compileSources(Global.scala:1553)
at scala.tools.nsc.Global$Run.compile(Global.scala:1662)
at xsbt.CachedCompiler0.run(CompilerInterface.scala:123)
at xsbt.CachedCompiler0.run(CompilerInterface.scala:99)
at xsbt.CompilerInterface.run(CompilerInterface.scala:27)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.in