Re: Welcoming three new committers

Congratulations! On 4 February 2015 at 02:34, Matei Zaharia wrote: > Hi all, > > The PMC recently voted to add three new committers: Cheng Lian, Joseph > Bradley and Sean Owen. All three have been major contributors to Spark in > the past year: Cheng on Spark SQL, Joseph on MLlib, and Sean on ML

Re: ASF Git / GitHub sync is down

I filed an INFRA ticket: https://issues.apache.org/jira/browse/INFRA-9115 I wish ASF can reconsider requests like this in order to handle downtime gracefully https://issues.apache.org/jira/browse/INFRA-8738 On Tue, Feb 3, 2015 at 9:09 PM, Reynold Xin wrote: > Haven't sync-ed anything for the

Re: Welcoming three new committers

Congratulations Cheng, Joseph and Sean. On Tuesday, February 3, 2015, Zhan Zhang wrote: > Congratulations! > > On Feb 3, 2015, at 2:34 PM, Matei Zaharia > wrote: > > > Hi all, > > > > The PMC recently voted to add three new committers: Cheng Lian, Joseph > Bradley and Sean Owen. All three have

ASF Git / GitHub sync is down

Haven't sync-ed anything for the last 4 hours. Seems like this little piece of infrastructure always stops working around our own code freeze time ...

Re: Welcoming three new committers

Congratulations! On Feb 3, 2015, at 2:34 PM, Matei Zaharia wrote: > Hi all, > > The PMC recently voted to add three new committers: Cheng Lian, Joseph > Bradley and Sean Owen. All three have been major contributors to Spark in the > past year: Cheng on Spark SQL, Joseph on MLlib, and Sean on

Re: Welcoming three new committers

Thanks to everyone in the community for past collaborations, and I look forward to continuing in the future! Joseph On Tue, Feb 3, 2015 at 6:23 PM, Shixiong Zhu wrote: > Congrats guys! > > Best Regards, > Shixiong Zhu > > 2015-02-04 6:34 GMT+08:00 Matei Zaharia : > >> Hi all, >> >> The PMC recen

Re: Welcoming three new committers

Congratulations! -- Ye Xianjin Sent with Sparrow (http://www.sparrowmailapp.com/?sig) On Wednesday, February 4, 2015 at 6:34 AM, Matei Zaharia wrote: > Hi all, > > The PMC recently voted to add three new committers: Cheng Lian, Joseph > Bradley and Sean Owen. All three have been major contri

Re: Welcoming three new committers

Congrats guys! Best Regards, Shixiong Zhu 2015-02-04 6:34 GMT+08:00 Matei Zaharia : > Hi all, > > The PMC recently voted to add three new committers: Cheng Lian, Joseph > Bradley and Sean Owen. All three have been major contributors to Spark in > the past year: Cheng on Spark SQL, Joseph on MLli

Re: Welcoming three new committers

Congratulations ! Keep helping the community :-) On Tue, Feb 3, 2015 at 5:34 PM, Denny Lee wrote: > Awesome stuff - congratulations! :) > > On Tue Feb 03 2015 at 5:34:06 PM Chao Chen wrote: > > > Congratulations guys, well done! > > > > 在 15-2-4 上午9:26, Nan Zhu 写道: > > > Congratulations! > > >

Re: Welcoming three new committers

Awesome stuff - congratulations! :) On Tue Feb 03 2015 at 5:34:06 PM Chao Chen wrote: > Congratulations guys, well done! > > 在 15-2-4 上午9:26, Nan Zhu 写道: > > Congratulations! > > > > -- > > Nan Zhu > > http://codingcat.me > > > > > > On Tuesday, February 3, 2015 at 8:08 PM, Xuefeng Wu wrote: > >

Re: Welcoming three new committers

Congratulations guys, well done! 在 15-2-4 上午9:26, Nan Zhu 写道: Congratulations! -- Nan Zhu http://codingcat.me On Tuesday, February 3, 2015 at 8:08 PM, Xuefeng Wu wrote: Congratulations！well done. Yours, Xuefeng Wu 吴雪峰 敬上 On 2015年2月4日, at 上午6:34, Matei Zaharia mailto:matei.zaha...@gma

Re: 2GB limit for partitions?

That is fairly out of date (we used to run some of our jobs on it ... But that is forked off 1.1 actually). Regards Mridul On Tuesday, February 3, 2015, Imran Rashid wrote: > Thanks for the explanations, makes sense. For the record looks like this > was worked on a while back (and maybe the wo

Re: Welcoming three new committers

Congratulations ! Keep up the good work :-) Regards Mridul On Tuesday, February 3, 2015, Matei Zaharia wrote: > Hi all, > > The PMC recently voted to add three new committers: Cheng Lian, Joseph > Bradley and Sean Owen. All three have been major contributors to Spark in > the past year: Cheng

Re: Welcoming three new committers

Congratulations! -- Nan Zhu http://codingcat.me On Tuesday, February 3, 2015 at 8:08 PM, Xuefeng Wu wrote: > Congratulations！well done. > > Yours, Xuefeng Wu 吴雪峰 敬上 > > > On 2015年2月4日, at 上午6:34, Matei Zaharia > (mailto:matei.zaha...@gmail.com)> wrote: > > > > Hi all, > > > > The P

Re: Welcoming three new committers

Congratulations！well done. Yours, Xuefeng Wu 吴雪峰 敬上 > On 2015年2月4日, at 上午6:34, Matei Zaharia wrote: > > Hi all, > > The PMC recently voted to add three new committers: Cheng Lian, Joseph > Bradley and Sean Owen. All three have been major contributors to Spark in the > past year: Cheng on Sp

Re: Welcoming three new committers

Congrats guys! On Tue, Feb 3, 2015 at 7:01 PM, Evan Chan wrote: > Congrats everyone!!! > > On Tue, Feb 3, 2015 at 3:17 PM, Timothy Chen wrote: > > Congrats all! > > > > Tim > > > > > >> On Feb 4, 2015, at 7:10 AM, Pritish Nawlakhe < > prit...@nirvana-international.com> wrote: > >> > >> Congrats

Re: IDF for ml pipeline

Thank you for your reply. I will do it. — Mailbox から送信 On Tue, Feb 3, 2015 at 6:12 PM, Xiangrui Meng wrote: > Yes, we need a wrapper under spark.ml. Feel free to create a JIRA for > it. -Xiangrui > On Mon, Feb 2, 2015 at 8:56 PM, masaki rikitoku wrote: >> Hi all >> >> I am trying the ml pipe

Re: SparkSubmit.scala and stderr

Why not just use SLF4J? On Tue, Feb 3, 2015 at 2:22 PM, Reynold Xin wrote: > We can use ScalaTest's privateMethodTester also instead of exposing that. > > On Tue, Feb 3, 2015 at 2:18 PM, Marcelo Vanzin wrote: > >> Hi Jay, >> >> On Tue, Feb 3, 2015 at 6:28 AM, jayhutfles wrote: >> > // Expos

Re: Welcoming three new committers

Congrats everyone!!! On Tue, Feb 3, 2015 at 3:17 PM, Timothy Chen wrote: > Congrats all! > > Tim > > >> On Feb 4, 2015, at 7:10 AM, Pritish Nawlakhe >> wrote: >> >> Congrats and welcome back!! >> >> >> >> Thank you!! >> >> Regards >> Pritish >> Nirvana International Inc. >> >> Big Data, Hadoop,

Re: 2GB limit for partitions?

Thanks for the explanations, makes sense. For the record looks like this was worked on a while back (and maybe the work is even close to a solution?) https://issues.apache.org/jira/browse/SPARK-1476 and perhaps an independent solution was worked on here? https://issues.apache.org/jira/browse/SP

Re: 2GB limit for partitions?

cc dev list How are you saving the data? There are two relevant 2GB limits: 1. Caching 2. Shuffle For caching, a partition is turned into a single block. For shuffle, each map partition is partitioned into R blocks, where R = number of reduce tasks. It is unlikely a shuffle block > 2G, altho

Re: Welcoming three new committers

Congrats all! Tim > On Feb 4, 2015, at 7:10 AM, Pritish Nawlakhe > wrote: > > Congrats and welcome back!! > > > > Thank you!! > > Regards > Pritish > Nirvana International Inc. > > Big Data, Hadoop, Oracle EBS and IT Solutions > VA - SWaM, MD - MBE Certified Company > prit...@nirvana-int

RE: Welcoming three new committers

Congrats and welcome back!! Thank you!! Regards Pritish Nirvana International Inc. Big Data, Hadoop, Oracle EBS and IT Solutions VA - SWaM, MD - MBE Certified Company prit...@nirvana-international.com http://www.nirvana-international.com Twitter: @nirvanainternat -Original Message-

Re: Welcoming three new committers

Congrats Cheng, Joseph and Owen! Well done! Thanks, Hari On Tue, Feb 3, 2015 at 2:55 PM, Ted Yu wrote: > Congratulations, Cheng, Joseph and Sean. > On Tue, Feb 3, 2015 at 2:53 PM, Nicholas Chammas > wrote: >> Congratulations guys! >> >> On Tue Feb 03 2015 at 2:36:12 PM Matei Zaharia >> wrot

Re: Welcoming three new committers

Congratulations, Cheng, Joseph and Sean. On Tue, Feb 3, 2015 at 2:53 PM, Nicholas Chammas wrote: > Congratulations guys! > > On Tue Feb 03 2015 at 2:36:12 PM Matei Zaharia > wrote: > > > Hi all, > > > > The PMC recently voted to add three new committers: Cheng Lian, Joseph > > Bradley and Sean

Re: Welcoming three new committers

Congratulations guys! On Tue Feb 03 2015 at 2:36:12 PM Matei Zaharia wrote: > Hi all, > > The PMC recently voted to add three new committers: Cheng Lian, Joseph > Bradley and Sean Owen. All three have been major contributors to Spark in > the past year: Cheng on Spark SQL, Joseph on MLlib, and S

Welcoming three new committers

Hi all, The PMC recently voted to add three new committers: Cheng Lian, Joseph Bradley and Sean Owen. All three have been major contributors to Spark in the past year: Cheng on Spark SQL, Joseph on MLlib, and Sean on ML and many pieces throughout Spark Core. Join me in welcoming them as committ

Re: SparkSubmit.scala and stderr

We can use ScalaTest's privateMethodTester also instead of exposing that. On Tue, Feb 3, 2015 at 2:18 PM, Marcelo Vanzin wrote: > Hi Jay, > > On Tue, Feb 3, 2015 at 6:28 AM, jayhutfles wrote: > > // Exposed for testing > > private[spark] var printStream: PrintStream = System.err > > > B

Re: SparkSubmit.scala and stderr

Hi Jay, On Tue, Feb 3, 2015 at 6:28 AM, jayhutfles wrote: > // Exposed for testing > private[spark] var printStream: PrintStream = System.err > But as the comment states that it's for testing, maybe I'm > misunderstanding its intent... The comment is there to tell someone reading the co

Re: SparkSubmit.scala and stderr

Despite its name, stderr is frequently used as the destination for anything that's not the output of the program, which includes log messages. That way, for example, you can redirect the output of such a program to capture its result without also capturing log or error messages, which will still ju

Re: [spark-sql] JsonRDD

We probably will extract general purpose functions from JsonRDD and also do the renaming through https://issues.apache.org/jira/browse/SPARK-5260. On Tue, Feb 3, 2015 at 9:15 AM, Daniil Osipov wrote: > Thanks Reynold, > > Case sensitivity issues are definitely orthogonal. I'll submit a bug or PR

[ANNOUNCE] branch-1.3 has been cut

Hey All, Just wanted to announce that we've cut the 1.3 branch which will become the 1.3 release after community testing. There are still some features that will go in (in higher level libraries, and some stragglers in spark core), but overall this indicates the end of major feature development f

Re: [VOTE] Release Apache Spark 1.2.1 (RC3)

+1 Tested the REPL release against the Spark Kernel project (compilation/testing/manual execution). Everything still checks out fine. Signed, Chip Senkbeil IBM Emerging Technologies Software Engineer On Tue Feb 03 2015 at 12:50:12 PM Nicholas Chammas < nicholas.cham...@gmail.com> wrote: > I beli

Re: [VOTE] Release Apache Spark 1.2.1 (RC3)

I believe this was changed for 1.2.1. Here are the relevant JIRA issues . On Tue Feb 03 2015 at 10:43:59 AM Dirceu Semighini Filho

SparkSubmit.scala and stderr

Hi all, I just saw that the SparkSubmit.scala class has the following lines: object SparkSubmit { ... // Exposed for testing private[spark] var printStream: PrintStream = System.err ... } This causes all verbose logging messages elsewhere in SparkSubmit to go to stderr, not s

Re: [VOTE] Release Apache Spark 1.2.1 (RC3)

Hi Patrick, I work in an Startup and we want make one of our projects as open source. This project is based on Spark, and it will help users to instantiate spark clusters in a cloud environment. But for that project we need to use the repl, hive and thrift-server. Can the decision of not publishing

Re: Can spark provide an option to start reduce stage early?

There's a JIRA tracking this here: https://issues.apache.org/jira/browse/SPARK-2387 On Mon, Feb 2, 2015 at 9:48 PM, Xuelin Cao wrote: > In hadoop MR, there is an option *mapred.reduce.slowstart.completed.maps* > > which can be used to start reducer stage when X% mappers are completed. By > doing

Re: [spark-sql] JsonRDD

Thanks Reynold, Case sensitivity issues are definitely orthogonal. I'll submit a bug or PR. Is there a way to rename the object to eliminate the confusion? Not sure how locked down the API is at this time, but it seems like a potential confusion point for developers. On Mon, Feb 2, 2015 at 4:30

Re: Jenkins install reference

here's the wiki describing the system setup: https://cwiki.apache.org/confluence/display/SPARK/Spark+QA+Infrastructure we have 1 master and 8 worker nodes, 12 executors per worker (we'd be better off w/more and smaller worker nodes however). you don't need to install sbt -- it's in the build/ dir

Re: Accessing indices and values in SparseVector

Alright, thanks for the quick clarification.

Re: Jenkins install reference

Here my question is: 1 How to set jenkins to make it build for multi PR parallel?. or one machine only support one PR building? 2 do we need install sbt on the CI machine since the script dev/run-tests will auto fetch the sbt jar ? - Fei On 2015/2/3 15:53, scwf wrote: Hi, all we w

Re: Accessing indices and values in SparseVector

When you are describing an error, you should say what the error is. Here I'm pretty sure it says there is no such member of Vector, right? You explicitly made the type of sv2 Vector and not SparseVector, and the trait does not have any indices member. No it's not a problem, and I think the compiler

Re: [VOTE] Release Apache Spark 1.2.1 (RC3)

+1 The signatures are still fine. Building for Hadoop 2.6 with YARN works; tests pass, except that MQTTStreamSuite, which we established is a test problem and already fixed in master. On Tue, Feb 3, 2015 at 12:34 AM, Krishna Sankar wrote: > +1 (non-binding, of course) > > 1. Compiled OSX 10.10 (

Accessing indices and values in SparseVector

Hello, This is related to one of the issues that I'm working on. I am not sure if this is expected behavior or not. This works fine. val sv2 = new SparseVector(3, Array(0, 2), Array(1.1, 3.0)) sv2.indices But when I do this val sv2: Vector = Vectors.sparse(3, Array(0, 2), Array(1.1, 3.0)) sv2.in

Re: IDF for ml pipeline

Yes, we need a wrapper under spark.ml. Feel free to create a JIRA for it. -Xiangrui On Mon, Feb 2, 2015 at 8:56 PM, masaki rikitoku wrote: > Hi all > > I am trying the ml pipeline for text classfication now. > > recently, i succeed to execute the pipeline processing in ml packages, > which consis

Jenkins install reference

Hi, all we want to set up a CI env for spark in our team, is there any reference of how to install jenkins over spark? Thanks Fei - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: