date:20151107

Build fails due to...multiple overloaded alternatives of constructor RDDInfo define default arguments?

2015-11-07 Thread Jacek Laskowski

Hi,

Checked out the latest sources and the build failed:

[error] 
/Users/jacek/dev/oss/spark/core/src/main/scala/org/apache/spark/storage/RDDInfo.scala:25:
in class RDDInfo, multiple overloaded alternatives of constructor
RDDInfo define default arguments.
[error] class RDDInfo(
[error]   ^

The build commands:

➜  spark git:(master) ✗ git rev-parse --short HEAD
2ff0e79

➜  spark git:(master) ./dev/change-scala-version.sh 2.11

➜  spark git:(master) ✗ ./build/mvn -Pyarn -Phadoop-2.6
-Dhadoop.version=2.7.1 -Dscala-2.11 -Phive -Phive-thriftserver
-DskipTests clean install

Pozdrawiam,
Jacek

--
Jacek Laskowski | http://blog.japila.pl | http://blog.jaceklaskowski.pl
Follow me at https://twitter.com/jaceklaskowski
Upvote at http://stackoverflow.com/users/1305344/jacek-laskowski

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: Build fails due to...multiple overloaded alternatives of constructor RDDInfo define default arguments?

2015-11-07 Thread Jacek Laskowski

Worked for me. Thanks!

Pozdrawiam,
Jacek

--
Jacek Laskowski | http://blog.japila.pl | http://blog.jaceklaskowski.pl
Follow me at https://twitter.com/jaceklaskowski
Upvote at http://stackoverflow.com/users/1305344/jacek-laskowski


On Sat, Nov 7, 2015 at 1:56 PM, Ted Yu  wrote:
> Created a PR for the compilation error:
> https://github.com/apache/spark/pull/9538
>
> Cheers
>
> On Sat, Nov 7, 2015 at 4:41 AM, Jacek Laskowski  wrote:
>>
>> Hi,
>>
>> Checked out the latest sources and the build failed:
>>
>> [error]
>> /Users/jacek/dev/oss/spark/core/src/main/scala/org/apache/spark/storage/RDDInfo.scala:25:
>> in class RDDInfo, multiple overloaded alternatives of constructor
>> RDDInfo define default arguments.
>> [error] class RDDInfo(
>> [error]   ^
>>
>> The build commands:
>>
>> ➜  spark git:(master) ✗ git rev-parse --short HEAD
>> 2ff0e79
>>
>> ➜  spark git:(master) ./dev/change-scala-version.sh 2.11
>>
>> ➜  spark git:(master) ✗ ./build/mvn -Pyarn -Phadoop-2.6
>> -Dhadoop.version=2.7.1 -Dscala-2.11 -Phive -Phive-thriftserver
>> -DskipTests clean install
>>
>> Pozdrawiam,
>> Jacek
>>
>> --
>> Jacek Laskowski | http://blog.japila.pl | http://blog.jaceklaskowski.pl
>> Follow me at https://twitter.com/jaceklaskowski
>> Upvote at http://stackoverflow.com/users/1305344/jacek-laskowski
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
>> For additional commands, e-mail: dev-h...@spark.apache.org
>>
>

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: Build fails due to...multiple overloaded alternatives of constructor RDDInfo define default arguments?

2015-11-07 Thread Ted Yu

Created a PR for the compilation error:
https://github.com/apache/spark/pull/9538

Cheers

On Sat, Nov 7, 2015 at 4:41 AM, Jacek Laskowski  wrote:

> Hi,
>
> Checked out the latest sources and the build failed:
>
> [error]
> /Users/jacek/dev/oss/spark/core/src/main/scala/org/apache/spark/storage/RDDInfo.scala:25:
> in class RDDInfo, multiple overloaded alternatives of constructor
> RDDInfo define default arguments.
> [error] class RDDInfo(
> [error]   ^
>
> The build commands:
>
> ➜  spark git:(master) ✗ git rev-parse --short HEAD
> 2ff0e79
>
> ➜  spark git:(master) ./dev/change-scala-version.sh 2.11
>
> ➜  spark git:(master) ✗ ./build/mvn -Pyarn -Phadoop-2.6
> -Dhadoop.version=2.7.1 -Dscala-2.11 -Phive -Phive-thriftserver
> -DskipTests clean install
>
> Pozdrawiam,
> Jacek
>
> --
> Jacek Laskowski | http://blog.japila.pl | http://blog.jaceklaskowski.pl
> Follow me at https://twitter.com/jaceklaskowski
> Upvote at http://stackoverflow.com/users/1305344/jacek-laskowski
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
> For additional commands, e-mail: dev-h...@spark.apache.org
>
>

Re: [VOTE] Release Apache Spark 1.5.2 (RC2)

2015-11-07 Thread Joseph Bradley

+1 tested on OS X

On Sat, Nov 7, 2015 at 10:25 AM, Reynold Xin  wrote:

> +1 myself too
>
> On Sat, Nov 7, 2015 at 12:01 AM, Robin East 
> wrote:
>
>> +1
>> Mac OS X 10.10.5 Yosemite
>>
>> mvn clean package -DskipTests (13min)
>>
>> Basic graph tests
>>   Load graph using edgeListFile...SUCCESS
>>   Run PageRank...SUCCESS
>> Connected Components tests
>>   Kaggle social circles competition...SUCCESS
>> Minimum Spanning Tree Algorithm
>>   Run basic Minimum Spanning Tree algorithmSUCCESS
>>   Run Minimum Spanning Tree taxonomy creation...SUCCESS
>>
>>
>> ---
>> Robin East
>> *Spark GraphX in Action* Michael Malak and Robin East
>> Manning Publications Co.
>> http://www.manning.com/books/spark-graphx-in-action
>>
>>
>>
>>
>>
>> On 6 Nov 2015, at 17:27, Chester Chen  wrote:
>>
>> +1
>> Test against CDH5.4.2 with hadoop 2.6.0 version using yesterday's code,
>> build locally.
>>
>> Regression running in Yarn Cluster mode against few internal ML (
>> logistic regression, linear regression, random forest and statistic
>> summary) as well Mlib KMeans. all seems to work fine.
>>
>> Chester
>>
>>
>> On Tue, Nov 3, 2015 at 3:22 PM, Reynold Xin  wrote:
>>
>>> Please vote on releasing the following candidate as Apache Spark version
>>> 1.5.2. The vote is open until Sat Nov 7, 2015 at 00:00 UTC and passes if a
>>> majority of at least 3 +1 PMC votes are cast.
>>>
>>> [ ] +1 Release this package as Apache Spark 1.5.2
>>> [ ] -1 Do not release this package because ...
>>>
>>>
>>> The release fixes 59 known issues in Spark 1.5.1, listed here:
>>> http://s.apache.org/spark-1.5.2
>>>
>>> The tag to be voted on is v1.5.2-rc2:
>>> https://github.com/apache/spark/releases/tag/v1.5.2-rc2
>>>
>>> The release files, including signatures, digests, etc. can be found at:
>>> http://people.apache.org/~pwendell/spark-releases/spark-1.5.2-rc2-bin/
>>>
>>> Release artifacts are signed with the following key:
>>> https://people.apache.org/keys/committer/pwendell.asc
>>>
>>> The staging repository for this release can be found at:
>>> - as version 1.5.2-rc2:
>>> https://repository.apache.org/content/repositories/orgapachespark-1153
>>> - as version 1.5.2:
>>> https://repository.apache.org/content/repositories/orgapachespark-1152
>>>
>>> The documentation corresponding to this release can be found at:
>>> http://people.apache.org/~pwendell/spark-releases/spark-1.5.2-rc2-docs/
>>>
>>>
>>> ===
>>> How can I help test this release?
>>> ===
>>> If you are a Spark user, you can help us test this release by taking an
>>> existing Spark workload and running on this release candidate, then
>>> reporting any regressions.
>>>
>>> 
>>> What justifies a -1 vote for this release?
>>> 
>>> -1 vote should occur for regressions from Spark 1.5.1. Bugs already
>>> present in 1.5.1 will not block this release.
>>>
>>>
>>>
>>
>>
>

Re: Calling stop on StreamingContext locks up

2015-11-07 Thread Ted Yu

Would the following change work for you ?

diff --git
a/core/src/main/scala/org/apache/spark/util/AsynchronousListenerBus.scala
b/core/src/main/scala/org/apache/spark/util/AsynchronousListenerBus.scala
index 61b5a4c..c330d25 100644
---
a/core/src/main/scala/org/apache/spark/util/AsynchronousListenerBus.scala
+++
b/core/src/main/scala/org/apache/spark/util/AsynchronousListenerBus.scala
@@ -66,6 +66,7 @@ private[spark] abstract class AsynchronousListenerBus[L
<: AnyRef, E](name: Stri
 self.synchronized {
   processingEvent = true
 }
+if (stopped.get()) return
 try {
   val event = eventQueue.poll
   if (event == null) {

On Sat, Nov 7, 2015 at 12:17 PM, vonnagy  wrote:

> If I have a streaming job (Spark 1.5.1) and attempt to stop the stream
> after
> the first batch, the system locks up and never completes. The pseudo code
> below shows that after the batch complete notification is called the stream
> is stopped. I have traced the lockup to the call `listener.stop()`in
> JobScheduler (line 114) which attempts to join the thread in
> AsynchronousListenerBus. That thread never ends because it is still getting
> messages `SparkListenerExecutorMetricsUpdate` from the DAGScheduler. The
> thread never ends because the events continue to come in.
>
> Any thoughts/ideas on how I can effectively stop the stream after the first
> batch would greatly appreciated.
>
> Psuedo Example:
>
> class SomeJob {
>
> val ssc = createStreamingContext()
> val listener = new MyListener(ssc)
> ssc.addStreamingListener(listener)
>
> val stream = getStream
>
> stream.foreachRDD { rdd =>
> // Do something with the data
> }
> }
>
> class MyListener(ctx: StreamingContext) extends StreamingListener {
> override def onBatchCompleted(batchCompleted:
> StreamingListenerBatchCompleted) = synchronized {
> ctx.stop(false, false)
> // NOTE: I get the same results with ctx.stop(), ctx.stop(true),
> ctx.stop(true, true), or ctx.stop(false, false)
> }
> }
>
>
>
> --
> View this message in context:
> http://apache-spark-developers-list.1001551.n3.nabble.com/Calling-stop-on-StreamingContext-locks-up-tp15063.html
> Sent from the Apache Spark Developers List mailing list archive at
> Nabble.com.
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
> For additional commands, e-mail: dev-h...@spark.apache.org
>
>

Re: [VOTE] Release Apache Spark 1.5.2 (RC2)

2015-11-07 Thread Mark Hamstra

+1

On Tue, Nov 3, 2015 at 3:22 PM, Reynold Xin  wrote:

> Please vote on releasing the following candidate as Apache Spark version
> 1.5.2. The vote is open until Sat Nov 7, 2015 at 00:00 UTC and passes if a
> majority of at least 3 +1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache Spark 1.5.2
> [ ] -1 Do not release this package because ...
>
>
> The release fixes 59 known issues in Spark 1.5.1, listed here:
> http://s.apache.org/spark-1.5.2
>
> The tag to be voted on is v1.5.2-rc2:
> https://github.com/apache/spark/releases/tag/v1.5.2-rc2
>
> The release files, including signatures, digests, etc. can be found at:
> http://people.apache.org/~pwendell/spark-releases/spark-1.5.2-rc2-bin/
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/pwendell.asc
>
> The staging repository for this release can be found at:
> - as version 1.5.2-rc2:
> https://repository.apache.org/content/repositories/orgapachespark-1153
> - as version 1.5.2:
> https://repository.apache.org/content/repositories/orgapachespark-1152
>
> The documentation corresponding to this release can be found at:
> http://people.apache.org/~pwendell/spark-releases/spark-1.5.2-rc2-docs/
>
>
> ===
> How can I help test this release?
> ===
> If you are a Spark user, you can help us test this release by taking an
> existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> 
> What justifies a -1 vote for this release?
> 
> -1 vote should occur for regressions from Spark 1.5.1. Bugs already
> present in 1.5.1 will not block this release.
>
>
>

Calling stop on StreamingContext locks up

2015-11-07 Thread vonnagy

If I have a streaming job (Spark 1.5.1) and attempt to stop the stream after
the first batch, the system locks up and never completes. The pseudo code
below shows that after the batch complete notification is called the stream
is stopped. I have traced the lockup to the call `listener.stop()`in
JobScheduler (line 114) which attempts to join the thread in
AsynchronousListenerBus. That thread never ends because it is still getting
messages `SparkListenerExecutorMetricsUpdate` from the DAGScheduler. The
thread never ends because the events continue to come in.

Any thoughts/ideas on how I can effectively stop the stream after the first
batch would greatly appreciated.

Psuedo Example:

class SomeJob {

val ssc = createStreamingContext()
val listener = new MyListener(ssc)
ssc.addStreamingListener(listener)

val stream = getStream

stream.foreachRDD { rdd =>
// Do something with the data
}
}

class MyListener(ctx: StreamingContext) extends StreamingListener {
override def onBatchCompleted(batchCompleted:
StreamingListenerBatchCompleted) = synchronized {
ctx.stop(false, false)
// NOTE: I get the same results with ctx.stop(), ctx.stop(true),
ctx.stop(true, true), or ctx.stop(false, false)
}
}



--
View this message in context: 
http://apache-spark-developers-list.1001551.n3.nabble.com/Calling-stop-on-StreamingContext-locks-up-tp15063.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: [VOTE] Release Apache Spark 1.5.2 (RC2)

2015-11-07 Thread Reynold Xin

+1 myself too

On Sat, Nov 7, 2015 at 12:01 AM, Robin East  wrote:

> +1
> Mac OS X 10.10.5 Yosemite
>
> mvn clean package -DskipTests (13min)
>
> Basic graph tests
>   Load graph using edgeListFile...SUCCESS
>   Run PageRank...SUCCESS
> Connected Components tests
>   Kaggle social circles competition...SUCCESS
> Minimum Spanning Tree Algorithm
>   Run basic Minimum Spanning Tree algorithmSUCCESS
>   Run Minimum Spanning Tree taxonomy creation...SUCCESS
>
>
> ---
> Robin East
> *Spark GraphX in Action* Michael Malak and Robin East
> Manning Publications Co.
> http://www.manning.com/books/spark-graphx-in-action
>
>
>
>
>
> On 6 Nov 2015, at 17:27, Chester Chen  wrote:
>
> +1
> Test against CDH5.4.2 with hadoop 2.6.0 version using yesterday's code,
> build locally.
>
> Regression running in Yarn Cluster mode against few internal ML ( logistic
> regression, linear regression, random forest and statistic summary) as well
> Mlib KMeans. all seems to work fine.
>
> Chester
>
>
> On Tue, Nov 3, 2015 at 3:22 PM, Reynold Xin  wrote:
>
>> Please vote on releasing the following candidate as Apache Spark version
>> 1.5.2. The vote is open until Sat Nov 7, 2015 at 00:00 UTC and passes if a
>> majority of at least 3 +1 PMC votes are cast.
>>
>> [ ] +1 Release this package as Apache Spark 1.5.2
>> [ ] -1 Do not release this package because ...
>>
>>
>> The release fixes 59 known issues in Spark 1.5.1, listed here:
>> http://s.apache.org/spark-1.5.2
>>
>> The tag to be voted on is v1.5.2-rc2:
>> https://github.com/apache/spark/releases/tag/v1.5.2-rc2
>>
>> The release files, including signatures, digests, etc. can be found at:
>> http://people.apache.org/~pwendell/spark-releases/spark-1.5.2-rc2-bin/
>>
>> Release artifacts are signed with the following key:
>> https://people.apache.org/keys/committer/pwendell.asc
>>
>> The staging repository for this release can be found at:
>> - as version 1.5.2-rc2:
>> https://repository.apache.org/content/repositories/orgapachespark-1153
>> - as version 1.5.2:
>> https://repository.apache.org/content/repositories/orgapachespark-1152
>>
>> The documentation corresponding to this release can be found at:
>> http://people.apache.org/~pwendell/spark-releases/spark-1.5.2-rc2-docs/
>>
>>
>> ===
>> How can I help test this release?
>> ===
>> If you are a Spark user, you can help us test this release by taking an
>> existing Spark workload and running on this release candidate, then
>> reporting any regressions.
>>
>> 
>> What justifies a -1 vote for this release?
>> 
>> -1 vote should occur for regressions from Spark 1.5.1. Bugs already
>> present in 1.5.1 will not block this release.
>>
>>
>>
>
>

Re: [VOTE] Release Apache Spark 1.5.2 (RC2)

2015-11-07 Thread Denny Lee

+1


On Sat, Nov 7, 2015 at 12:01 PM Mark Hamstra 
wrote:

> +1
>
> On Tue, Nov 3, 2015 at 3:22 PM, Reynold Xin  wrote:
>
>> Please vote on releasing the following candidate as Apache Spark version
>> 1.5.2. The vote is open until Sat Nov 7, 2015 at 00:00 UTC and passes if a
>> majority of at least 3 +1 PMC votes are cast.
>>
>> [ ] +1 Release this package as Apache Spark 1.5.2
>> [ ] -1 Do not release this package because ...
>>
>>
>> The release fixes 59 known issues in Spark 1.5.1, listed here:
>> http://s.apache.org/spark-1.5.2
>>
>> The tag to be voted on is v1.5.2-rc2:
>> https://github.com/apache/spark/releases/tag/v1.5.2-rc2
>>
>> The release files, including signatures, digests, etc. can be found at:
>> http://people.apache.org/~pwendell/spark-releases/spark-1.5.2-rc2-bin/
>>
>> Release artifacts are signed with the following key:
>> https://people.apache.org/keys/committer/pwendell.asc
>>
>> The staging repository for this release can be found at:
>> - as version 1.5.2-rc2:
>> https://repository.apache.org/content/repositories/orgapachespark-1153
>> - as version 1.5.2:
>> https://repository.apache.org/content/repositories/orgapachespark-1152
>>
>> The documentation corresponding to this release can be found at:
>> http://people.apache.org/~pwendell/spark-releases/spark-1.5.2-rc2-docs/
>>
>>
>> ===
>> How can I help test this release?
>> ===
>> If you are a Spark user, you can help us test this release by taking an
>> existing Spark workload and running on this release candidate, then
>> reporting any regressions.
>>
>> 
>> What justifies a -1 vote for this release?
>> 
>> -1 vote should occur for regressions from Spark 1.5.1. Bugs already
>> present in 1.5.1 will not block this release.
>>
>>
>>

Build fails due to...multiple overloaded alternatives of constructor RDDInfo define default arguments?

Re: Build fails due to...multiple overloaded alternatives of constructor RDDInfo define default arguments?

Re: Build fails due to...multiple overloaded alternatives of constructor RDDInfo define default arguments?

Re: [VOTE] Release Apache Spark 1.5.2 (RC2)

Re: Calling stop on StreamingContext locks up

Re: [VOTE] Release Apache Spark 1.5.2 (RC2)

Calling stop on StreamingContext locks up

Re: [VOTE] Release Apache Spark 1.5.2 (RC2)

Re: [VOTE] Release Apache Spark 1.5.2 (RC2)

9 matches

Site Navigation

Mail list logo

Footer information