Re: [SparkR] is toDF() necessary

2015-05-17 Thread Davies Liu
toDF() is first introduced in Scala and Python (because createDataFrame is too long), is used in lots places, I think it's useful. On Fri, May 8, 2015 at 11:03 AM, Shivaram Venkataraman shiva...@eecs.berkeley.edu wrote: Agree that toDF is not very useful. In fact it was removed from the

Re: [build system] scheduled datacenter downtime, sunday may 17th

2015-05-17 Thread Josh Rosen
Reminder: the network migration has started this morning, so Jenkins is currently down. Status updates on the migration are being published at http://ucbsystems.org/ On Wed, May 13, 2015 at 5:12 PM, shane knapp skn...@berkeley.edu wrote: our datacenter is rejiggering our network (read: fully

Re: Resource usage of a spark application

2015-05-17 Thread Akhil Das
You can either pull the high level information from your resource manager, or if you want more control/specific information you can write a script and pull the resource usage information from the OS. Something like this

Re: Tentative due dates for Spark 1.3.2 release

2015-05-17 Thread Reynold Xin
You can just look at this branch, can't you? https://github.com/apache/spark/tree/branch-1.3 On Sun, May 17, 2015 at 9:20 PM, Niranda Perera niranda.per...@gmail.com wrote: Hi Patrick, Is there a separate location where I could download all the patches of each branch to this date, so that i

Re: [build system] scheduled datacenter downtime, sunday may 17th

2015-05-17 Thread shane knapp
our sysadmins fixed the auth issue about an hour ago... /etc/shadow's perms got borked somehow and that was breaking logins for local (non-ldap) accounts. we're all green. On Sun, May 17, 2015 at 2:46 PM, shane knapp skn...@berkeley.edu wrote: ...and we're back up. it looks like things are

Re: Tentative due dates for Spark 1.3.2 release

2015-05-17 Thread Niranda Perera
Hi Reynold, sorry, my mistake. can do that. thanks On Mon, May 18, 2015 at 9:51 AM, Reynold Xin r...@databricks.com wrote: You can just look at this branch, can't you? https://github.com/apache/spark/tree/branch-1.3 On Sun, May 17, 2015 at 9:20 PM, Niranda Perera niranda.per...@gmail.com

Re: Tentative due dates for Spark 1.3.2 release

2015-05-17 Thread Niranda Perera
Hi Patrick, Is there a separate location where I could download all the patches of each branch to this date, so that i could apply it locally? rgds On Fri, May 15, 2015 at 11:52 PM, Patrick Wendell pwend...@gmail.com wrote: Hi Niranda, Maintenance releases are not done on a predetermined

Re: [build system] scheduled datacenter downtime, sunday may 17th

2015-05-17 Thread shane knapp
jenkins is being a little recalcitrant and i'm looking at logs to see why it won't start.

Re: [build system] scheduled datacenter downtime, sunday may 17th

2015-05-17 Thread shane knapp
ok, i think it's time to reboot the jenkins master. On Sun, May 17, 2015 at 1:44 PM, shane knapp skn...@berkeley.edu wrote: jenkins is being a little recalcitrant and i'm looking at logs to see why it won't start.

Re: [build system] scheduled datacenter downtime, sunday may 17th

2015-05-17 Thread shane knapp
machine rebooted, but auth is completely broken (web and CLI on the server). i'm trying to fix this now. On Sun, May 17, 2015 at 1:51 PM, shane knapp skn...@berkeley.edu wrote: ok, i think it's time to reboot the jenkins master. On Sun, May 17, 2015 at 1:44 PM, shane knapp

Re: [build system] scheduled datacenter downtime, sunday may 17th

2015-05-17 Thread shane knapp
auth is fixed, and jenkins is out of quiet mode and now building. sorry for the delay! On Sun, May 17, 2015 at 2:06 PM, shane knapp skn...@berkeley.edu wrote: machine rebooted, but auth is completely broken (web and CLI on the server). i'm trying to fix this now. On Sun, May 17, 2015 at

Re: [build system] scheduled datacenter downtime, sunday may 17th

2015-05-17 Thread shane knapp
...and we've lost network connectivity again. things are still very flaky. more updates as they come. On Sun, May 17, 2015 at 2:32 PM, shane knapp skn...@berkeley.edu wrote: actually, LDAP auth is fixed, but if you have a local account that i've created for you, it's not letting you log in

Re: [build system] scheduled datacenter downtime, sunday may 17th

2015-05-17 Thread shane knapp
actually, LDAP auth is fixed, but if you have a local account that i've created for you, it's not letting you log in to jenkins' UI. looking at this now. On Sun, May 17, 2015 at 2:13 PM, shane knapp skn...@berkeley.edu wrote: auth is fixed, and jenkins is out of quiet mode and now building.

Resource usage of a spark application

2015-05-17 Thread Peter Prettenhofer
Hi all, I'm looking for a way to measure the current memory / cpu usage of a spark application to provide users feedback how much resources are actually being used. It seems that the metric system provides this information to some extend. It logs metrics on application level (nr of cores granted)