Re: Spark Avro Generation

2014-08-12 Thread Devl Devel
Thanks very much that helps, not having to generate the entire build. On Mon, Aug 11, 2014 at 6:09 PM, Ron Gonzalez zlgonza...@yahoo.com wrote: If you don't want to build the entire thing, you can also do mvn generate-sources in externals/flume-sink Thanks, Ron Sent from my iPhone On

Re: [SPARK-2878] Kryo serialisation with custom Kryo registrator failing

2014-08-12 Thread Graham Dennis
I've submitted a work-in-progress pull request for this issue that I'd like feedback on. See https://github.com/apache/spark/pull/1890 . I've also submitted a pull request for the related issue that the exceptions hit when trying to use a custom kryo registrator are being swallowed:

Re: fair scheduler

2014-08-12 Thread fireflyc
@Crystal You can use spark on yarn. Yarn have fair scheduler,modified yarn-site.xml. 发自我的 iPad 在 2014年8月11日,6:49,Matei Zaharia matei.zaha...@gmail.com 写道: Hi Crystal, The fair scheduler is only for jobs running concurrently within the same SparkContext (i.e. within an application), not

Re: Contributing to MLlib: Proposal for Clustering Algorithms

2014-08-12 Thread RJ Nowling
Hi all, I wanted to follow up. I have a prototype for an optimized version of hierarchical k-means. I wanted to get some feedback on my apporach. Jeremy's implementation splits the largest cluster in each round. Is it better to do it that way or to split each cluster in half? Are there are

Re: Spark testsuite error for hive 0.13.

2014-08-12 Thread Zhan Zhang
Problem solved by a walkaround with create database and use database. -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Spark-testsuite-error-for-hive-0-13-tp7807p7819.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com.

Re: Using mllib-1.1.0-SNAPSHOT on Spark 1.0.1

2014-08-12 Thread Debasish Das
I figured out the issuethe driver memory was at 512 MB and for our datasets, the following code needed more memory... // Materialize usersOut and productsOut. usersOut.count() productsOut.count() Thanks. Deb On Sat, Aug 9, 2014 at 6:12 PM, Debasish Das debasish.da...@gmail.com wrote:

FileNotFoundException with _temporary in the name

2014-08-12 Thread Andrew Ash
Hi Spark devs, Several people on the mailing list have seen issues with FileNotFoundExceptions related to _temporary in the name. I've personally observed this several times, as have a few of my coworkers on various Spark clusters. Any ideas what might be going on? I've collected the various