[jira] [Commented] (SPARK-3849) Automate remaining Spark Code Style Guide rules

2015-03-25 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380603#comment-14380603 ] Nicholas Chammas commented on SPARK-3849: - Sounds good. My quick summary (which

[jira] [Commented] (SPARK-6481) Set In Progress when a PR is opened for an issue

2015-03-25 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380464#comment-14380464 ] Nicholas Chammas commented on SPARK-6481: - The Spark user can initiate state

[jira] [Commented] (SPARK-6481) Set In Progress when a PR is opened for an issue

2015-03-25 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380585#comment-14380585 ] Nicholas Chammas commented on SPARK-6481: - PR for this: https://github.com

[jira] [Commented] (SPARK-6481) Set In Progress when a PR is opened for an issue

2015-03-24 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14378838#comment-14378838 ] Nicholas Chammas commented on SPARK-6481: - Since there is no guaranteed way to map

[jira] [Commented] (SPARK-6481) Set In Progress when a PR is opened for an issue

2015-03-24 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14378114#comment-14378114 ] Nicholas Chammas commented on SPARK-6481: - [~pwendell] - Where is the GitHub JIRA

[jira] [Comment Edited] (SPARK-6481) Set In Progress when a PR is opened for an issue

2015-03-24 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14378393#comment-14378393 ] Nicholas Chammas edited comment on SPARK-6481 at 3/24/15 7:07 PM

[jira] [Commented] (SPARK-6481) Set In Progress when a PR is opened for an issue

2015-03-24 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14378393#comment-14378393 ] Nicholas Chammas commented on SPARK-6481: - So should that script be removed from

[jira] [Commented] (SPARK-6481) Set In Progress when a PR is opened for an issue

2015-03-24 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14378436#comment-14378436 ] Nicholas Chammas commented on SPARK-6481: - The change Michael/Patrick want

[jira] [Commented] (SPARK-2394) Make it easier to read LZO-compressed files from EC2 clusters

2015-03-23 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14375929#comment-14375929 ] Nicholas Chammas commented on SPARK-2394: - Thank you for posting this information

[jira] [Updated] (SPARK-6474) Replace image.run with connection.run_instances in spark_ec2.py

2015-03-23 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-6474: Issue Type: Improvement (was: Bug) Replace image.run with connection.run_instances

[jira] [Commented] (SPARK-6474) Replace image.run with connection.run_instances in spark_ec2.py

2015-03-23 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14376584#comment-14376584 ] Nicholas Chammas commented on SPARK-6474: - This change also fits the pattern

[jira] [Updated] (SPARK-6474) Replace image.run with connection.run_instances in spark_ec2.py

2015-03-23 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-6474: Priority: Minor (was: Major) Replace image.run with connection.run_instances in spark_ec2

[jira] [Comment Edited] (SPARK-6474) Replace image.run with connection.run_instances in spark_ec2.py

2015-03-23 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14376572#comment-14376572 ] Nicholas Chammas edited comment on SPARK-6474 at 3/23/15 8:29 PM

[jira] [Commented] (SPARK-6474) Replace image.run with connection.run_instances in spark_ec2.py

2015-03-23 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14376572#comment-14376572 ] Nicholas Chammas commented on SPARK-6474: - LGTM. Replace image.run

[jira] [Commented] (SPARK-6481) Set In Progress when a PR is opened for an issue

2015-03-23 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14377034#comment-14377034 ] Nicholas Chammas commented on SPARK-6481: - I'm guessing this will be done via

[issue21423] concurrent.futures.ThreadPoolExecutor/ProcessPoolExecutor should accept an initializer argument

2015-03-20 Thread Nicholas Chammas
Changes by Nicholas Chammas nicholas.cham...@gmail.com: -- nosy: +Nicholas Chammas ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21423

Re: Apache Spark User List: people's responses not showing in the browser view

2015-03-19 Thread Nicholas Chammas
Nabble is a third-party site that tries its best to archive mail sent out over the list. Nothing guarantees it will be in sync with the real mailing list. To get the truth on what was sent over this, Apache-managed list, you unfortunately need to go the Apache archives:

Re: Apache Spark User List: people's responses not showing in the browser view

2015-03-19 Thread Nicholas Chammas
-hadoop.com which provides better search capability. Cheers On Thu, Mar 19, 2015 at 6:48 AM, Nicholas Chammas nicholas.cham...@gmail.com wrote: Nabble is a third-party site that tries its best to archive mail sent out over the list. Nothing guarantees it will be in sync with the real mailing list

Re: Apache Spark User List: people's responses not showing in the browser view

2015-03-19 Thread Nicholas Chammas
to find stuff in. Is there a search engine on top of them? so as to find e.g. your own posts easily? On Thu, Mar 19, 2015 at 10:34 AM, Nicholas Chammas nicholas.cham...@gmail.com wrote: Sure, you can use Nabble or search-hadoop or whatever you prefer. My point is just that the source of truth

Re: Processing of text file in large gzip archive

2015-03-16 Thread Nicholas Chammas
You probably want to update this line as follows: lines = sc.textFile('file.gz').repartition(sc.defaultParallelism * 3) For more details on why, see this answer http://stackoverflow.com/a/27631722/877069. Nick ​ On Mon, Mar 16, 2015 at 6:50 AM Marius Soutier mps@gmail.com wrote: 1. I

[jira] [Updated] (SPARK-6342) Leverage cfncluster in spark_ec2

2015-03-15 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-6342: Component/s: EC2 Leverage cfncluster in spark_ec2

[jira] [Commented] (SPARK-6282) Strange Python import error when using random() in a lambda function

2015-03-13 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14360534#comment-14360534 ] Nicholas Chammas commented on SPARK-6282: - [~joshrosen], [~davies]: Does

[jira] [Commented] (SPARK-6282) Strange Python import error when using random() in a lambda function

2015-03-12 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359404#comment-14359404 ] Nicholas Chammas commented on SPARK-6282: - Shouldn't be related to boto. _winreg

[jira] [Updated] (SPARK-5189) Reorganize EC2 scripts so that nodes can be provisioned independent of Spark master

2015-03-12 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-5189: Description: As of 1.2.0, we launch Spark clusters on EC2 by setting up the master first

[jira] [Commented] (SPARK-5189) Reorganize EC2 scripts so that nodes can be provisioned independent of Spark master

2015-03-12 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359665#comment-14359665 ] Nicholas Chammas commented on SPARK-5189: - For the record, this is the script I

[jira] [Commented] (SPARK-4325) Improve spark-ec2 cluster launch times

2015-03-10 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354956#comment-14354956 ] Nicholas Chammas commented on SPARK-4325: - At this point it's more an umbrella

[jira] [Commented] (SPARK-4325) Improve spark-ec2 cluster launch times

2015-03-10 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354939#comment-14354939 ] Nicholas Chammas commented on SPARK-4325: - [~srowen] - I should perhaps change

[jira] [Created] (SPARK-6246) spark-ec2 can't handle clusters with 100 nodes

2015-03-10 Thread Nicholas Chammas (JIRA)
Nicholas Chammas created SPARK-6246: --- Summary: spark-ec2 can't handle clusters with 100 nodes Key: SPARK-6246 URL: https://issues.apache.org/jira/browse/SPARK-6246 Project: Spark Issue

[jira] [Commented] (SPARK-6246) spark-ec2 can't handle clusters with 100 nodes

2015-03-10 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354969#comment-14354969 ] Nicholas Chammas commented on SPARK-6246: - FYI [~shivaram]. spark-ec2 can't

[jira] [Reopened] (SPARK-4325) Improve spark-ec2 cluster launch times

2015-03-10 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas reopened SPARK-4325: - Reopening after updating contains issue links. Improve spark-ec2 cluster launch times

[jira] [Commented] (SPARK-6220) Allow extended EC2 options to be passed through spark-ec2

2015-03-10 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354991#comment-14354991 ] Nicholas Chammas commented on SPARK-6220: - Another thought to add

[jira] [Updated] (SPARK-5312) Use sbt to detect new or changed public classes in PRs

2015-03-10 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-5312: Description: We currently use an [unwieldy grep/sed contraption|https://github.com/apache

[jira] [Commented] (SPARK-5312) Use sbt to detect new or changed public classes in PRs

2015-03-10 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355622#comment-14355622 ] Nicholas Chammas commented on SPARK-5312: - Thanks for looking into this [~boyork

[jira] [Commented] (SPARK-6246) spark-ec2 can't handle clusters with 100 nodes

2015-03-10 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355642#comment-14355642 ] Nicholas Chammas commented on SPARK-6246: - I dunno, I haven't looked

[jira] [Commented] (SPARK-5313) Create simple framework for highlighting changes introduced in a PR

2015-03-10 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355819#comment-14355819 ] Nicholas Chammas commented on SPARK-5313: - I had an idea to generalize the process

[jira] [Updated] (SPARK-4325) Improve spark-ec2 cluster launch times

2015-03-10 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-4325: Description: This is an umbrella task to capture several pieces of work related

[jira] [Commented] (SPARK-6219) Expand Python lint checks to check for compilation errors

2015-03-09 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353325#comment-14353325 ] Nicholas Chammas commented on SPARK-6219: - That's a good point, I haven't checked

[jira] [Commented] (SPARK-6220) Allow extended EC2 options to be passed through spark-ec2

2015-03-09 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354217#comment-14354217 ] Nicholas Chammas commented on SPARK-6220: - I took another look at the 2 boto

[jira] [Commented] (SPARK-6206) spark-ec2 script reporting SSL error?

2015-03-08 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14352481#comment-14352481 ] Nicholas Chammas commented on SPARK-6206: - OK, let us know what you find

[jira] [Updated] (SPARK-6220) Allow extended EC2 options to be passed through spark-ec2

2015-03-08 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-6220: Description: There are many EC2 options exposed by the boto library that spark-ec2 uses

[jira] [Commented] (SPARK-6220) Allow extended EC2 options to be passed through spark-ec2

2015-03-08 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14352489#comment-14352489 ] Nicholas Chammas commented on SPARK-6220: - cc [~joshrosen] and [~shivaram

[jira] [Updated] (SPARK-6220) Allow extended EC2 options to be passed through spark-ec2

2015-03-08 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-6220: Description: There are many EC2 options exposed by the boto library that spark-ec2 uses

[jira] [Updated] (SPARK-6220) Allow extended EC2 options to be passed through spark-ec2

2015-03-08 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-6220: Description: There are many EC2 options exposed by the boto library that spark-ec2 uses

[jira] [Updated] (SPARK-6220) Allow extended EC2 options to be passed through spark-ec2

2015-03-08 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-6220: Description: There are many EC2 options exposed by the boto library that spark-ec2 uses

[jira] [Created] (SPARK-6218) Upgrade spark-ec2 from optparse to argparse

2015-03-08 Thread Nicholas Chammas (JIRA)
Nicholas Chammas created SPARK-6218: --- Summary: Upgrade spark-ec2 from optparse to argparse Key: SPARK-6218 URL: https://issues.apache.org/jira/browse/SPARK-6218 Project: Spark Issue Type

[jira] [Commented] (SPARK-6218) Upgrade spark-ec2 from optparse to argparse

2015-03-08 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14352331#comment-14352331 ] Nicholas Chammas commented on SPARK-6218: - [~shivaram], [~joshrosen]: What do you

[jira] [Updated] (SPARK-6218) Upgrade spark-ec2 from optparse to argparse

2015-03-08 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-6218: Description: spark-ec2 [currently uses optparse|https://github.com/apache/spark/blob

[jira] [Commented] (SPARK-6220) Allow extended EC2 options to be passed through spark-ec2

2015-03-08 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14352524#comment-14352524 ] Nicholas Chammas commented on SPARK-6220: - As far as places where we create

[jira] [Created] (SPARK-6219) Expand Python lint checks to check for compilation errors

2015-03-08 Thread Nicholas Chammas (JIRA)
Nicholas Chammas created SPARK-6219: --- Summary: Expand Python lint checks to check for compilation errors Key: SPARK-6219 URL: https://issues.apache.org/jira/browse/SPARK-6219 Project: Spark

[jira] [Updated] (SPARK-6191) Generalize spark-ec2's ability to download libraries from PyPI

2015-03-05 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-6191: Description: Right now we have a method to specifically download boto. Let's generalize

[jira] [Created] (SPARK-6191) Generalize spark-ec2's ability to download libraries from PyPI

2015-03-05 Thread Nicholas Chammas (JIRA)
Nicholas Chammas created SPARK-6191: --- Summary: Generalize spark-ec2's ability to download libraries from PyPI Key: SPARK-6191 URL: https://issues.apache.org/jira/browse/SPARK-6191 Project: Spark

[jira] [Updated] (SPARK-6191) Generalize spark-ec2's ability to download libraries from PyPI

2015-03-05 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-6191: Description: Right now we have a method to specifically download boto. Let's generalize

[jira] [Commented] (SPARK-3369) Java mapPartitions Iterator-Iterable is inconsistent with Scala's Iterator-Iterator

2015-03-05 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14349577#comment-14349577 ] Nicholas Chammas commented on SPARK-3369: - {quote} How about breaking backward

[jira] [Created] (SPARK-6193) Speed up how spark-ec2 searches for clusters

2015-03-05 Thread Nicholas Chammas (JIRA)
Nicholas Chammas created SPARK-6193: --- Summary: Speed up how spark-ec2 searches for clusters Key: SPARK-6193 URL: https://issues.apache.org/jira/browse/SPARK-6193 Project: Spark Issue Type

[jira] [Updated] (SPARK-5473) Expose SSH failures after status checks pass

2015-03-04 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-5473: Description: If there is some fatal problem with launching a cluster, `spark-ec2` just

[jira] [Updated] (SPARK-3533) Add saveAsTextFileByKey() method to RDDs

2015-03-04 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-3533: Target Version/s: 1.4.0 Add saveAsTextFileByKey() method to RDDs

Re: spark-ec2 default to Hadoop 2

2015-03-02 Thread Nicholas Chammas
shift towards 2.x at least as defaults. On Sun, Mar 1, 2015 at 10:59 PM, Nicholas Chammas nicholas.cham...@gmail.com wrote: https://github.com/apache/spark/blob/fd8d283eeb98e310b1e85ef8c3a8af 9e547ab5e0/ec2/spark_ec2.py#L162-L164 Is there any reason we shouldn't update the default Hadoop

[jira] [Commented] (SPARK-882) Have link for feedback/suggestions in docs

2015-03-02 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14344475#comment-14344475 ] Nicholas Chammas commented on SPARK-882: Is the intended use here that users could

[jira] [Commented] (SPARK-2545) Add a diagnosis mode for closures to figure out what they're bringing in

2015-03-02 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14344482#comment-14344482 ] Nicholas Chammas commented on SPARK-2545: - [~adav] - Would this potentially also

[jira] [Commented] (SPARK-2545) Add a diagnosis mode for closures to figure out what they're bringing in

2015-03-02 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14344504#comment-14344504 ] Nicholas Chammas commented on SPARK-2545: - cc [~tobias.schlatter] Add

[jira] [Commented] (SPARK-2095) sc.getExecutorCPUCounts()

2015-03-02 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14344480#comment-14344480 ] Nicholas Chammas commented on SPARK-2095: - cc [~pwendell], [~joshrosen

spark-ec2 default to Hadoop 2

2015-03-01 Thread Nicholas Chammas
https://github.com/apache/spark/blob/fd8d283eeb98e310b1e85ef8c3a8af9e547ab5e0/ec2/spark_ec2.py#L162-L164 Is there any reason we shouldn't update the default Hadoop major version in spark-ec2 to 2? Nick

[jira] [Commented] (SPARK-6077) Multiple spark streaming tabs on UI when reuse the same sparkcontext

2015-03-01 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14342704#comment-14342704 ] Nicholas Chammas commented on SPARK-6077: - Please disregard the comments on SPARK

[jira] [Commented] (SPARK-2463) Creating then stopping StreamingContext multiple times from shell generates duplicate Streaming tabs in UI

2015-03-01 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14342714#comment-14342714 ] Nicholas Chammas commented on SPARK-2463: - For people reading through

[jira] [Updated] (SPARK-2463) Creating then stopping StreamingContext multiple times from shell generates duplicate Streaming tabs in UI

2015-03-01 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-2463: Description: Start a {{StreamingContext}} from the interactive shell and then stop it. Go

[jira] [Updated] (SPARK-2463) Creating then stopping StreamingContext multiple times from shell generates duplicate Streaming tabs in UI

2015-03-01 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-2463: Description: Start a {{StreamingContext}} from the interactive shell and then stop it. Go

[jira] [Commented] (SPARK-6084) spark-shell broken on Windows

2015-02-28 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341787#comment-14341787 ] Nicholas Chammas commented on SPARK-6084: - Ah, there's also SPARK-5396, though

[jira] [Commented] (SPARK-5389) spark-shell.cmd does not run from DOS Windows 7

2015-02-28 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341789#comment-14341789 ] Nicholas Chammas commented on SPARK-5389: - Yeah, I think we found another instance

[jira] [Reopened] (SPARK-6084) spark-shell broken on Windows

2015-02-28 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas reopened SPARK-6084: - Don't see how this is a dup of SPARK-4833. spark-shell broken on Windows

[jira] [Resolved] (SPARK-6084) spark-shell broken on Windows

2015-02-28 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas resolved SPARK-6084. - Resolution: Duplicate Resolving as duplicate of SPARK-5389. That seems a more likely

[jira] [Commented] (SPARK-5396) Syntax error in spark scripts on windows.

2015-02-28 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341788#comment-14341788 ] Nicholas Chammas commented on SPARK-5396: - What does that error message say

[jira] [Updated] (SPARK-5389) spark-shell.cmd does not run from DOS Windows 7

2015-02-28 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-5389: Description: spark-shell.cmd crashes in DOS prompt Windows 7. Works fine under PowerShell

[jira] [Reopened] (SPARK-5389) spark-shell.cmd does not run from DOS Windows 7

2015-02-28 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas reopened SPARK-5389: - spark-shell.cmd does not run from DOS Windows 7

[jira] [Comment Edited] (SPARK-5389) spark-shell.cmd does not run from DOS Windows 7

2015-02-28 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341790#comment-14341790 ] Nicholas Chammas edited comment on SPARK-5389 at 2/28/15 9:48 PM

[jira] [Commented] (SPARK-6084) spark-shell broken on Windows

2015-02-28 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341776#comment-14341776 ] Nicholas Chammas commented on SPARK-6084: - I took a look at the linked issue

[jira] [Created] (SPARK-6084) spark-shell broken on Windows

2015-02-28 Thread Nicholas Chammas (JIRA)
Nicholas Chammas created SPARK-6084: --- Summary: spark-shell broken on Windows Key: SPARK-6084 URL: https://issues.apache.org/jira/browse/SPARK-6084 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-6084) spark-shell broken on Windows

2015-02-28 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341746#comment-14341746 ] Nicholas Chammas commented on SPARK-6084: - cc [~pwendell], [~andrewor14] I

[jira] [Updated] (SPARK-5971) Add Mesos support to spark-ec2

2015-02-24 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-5971: Description: Right now, spark-ec2 can only launch Spark clusters that use the standalone

[jira] [Commented] (SPARK-3850) Scala style: disallow trailing spaces

2015-02-24 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14335074#comment-14335074 ] Nicholas Chammas commented on SPARK-3850: - Ah I see. I'm fine with closing

[jira] [Updated] (SPARK-3850) Scala style: disallow trailing spaces

2015-02-24 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-3850: Description: Background discussions: * https://github.com/apache/spark/pull/2619 * http

[jira] [Commented] (SPARK-3850) Scala style: disallow trailing spaces

2015-02-24 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14335045#comment-14335045 ] Nicholas Chammas commented on SPARK-3850: - I guess the root is the [Style Guide

[jira] [Updated] (SPARK-5971) Add Mesos support to spark-ec2

2015-02-24 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-5971: Summary: Add Mesos support to spark-ec2 (was: Add support for launching Spark-on-Mesos

[jira] [Created] (SPARK-5971) Add support for launching Spark-on-Mesos clusters to spark-ec2

2015-02-24 Thread Nicholas Chammas (JIRA)
Nicholas Chammas created SPARK-5971: --- Summary: Add support for launching Spark-on-Mesos clusters to spark-ec2 Key: SPARK-5971 URL: https://issues.apache.org/jira/browse/SPARK-5971 Project: Spark

[jira] [Commented] (SPARK-3674) Add support for launching YARN clusters in spark-ec2

2015-02-24 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14335199#comment-14335199 ] Nicholas Chammas commented on SPARK-3674: - There is an open PR for this here

[jira] [Commented] (SPARK-5312) Use sbt to detect new or changed public classes in PRs

2015-02-24 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14335573#comment-14335573 ] Nicholas Chammas commented on SPARK-5312: - It's something to consider I guess

Re: Posting to the list

2015-02-23 Thread Nicholas Chammas
Nabble is a third-party site. If you send stuff through Nabble, Nabble has to forward it along to the Apache mailing list. If something goes wrong with that, you will have a message show up on Nabble that no-one saw. The reverse can also happen, where something actually goes out on the list and

Re: Launching Spark cluster on EC2 with Ubuntu AMI

2015-02-23 Thread Nicholas Chammas
I know that Spark EC2 scripts are not guaranteed to work with custom AMIs but still, it should work… Nope, it shouldn’t, unfortunately. The Spark base AMIs are custom-built for spark-ec2. No other AMI will work unless it was built with that goal in mind. Using a random AMI from the Amazon

[jira] [Commented] (SPARK-5944) Python release docs say SNAPSHOT + Author is missing

2015-02-23 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14333496#comment-14333496 ] Nicholas Chammas commented on SPARK-5944: - I'm not sure, but I think [here

[jira] [Updated] (SPARK-5944) Python release docs say SNAPSHOT + Author is missing

2015-02-23 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-5944: Target Version/s: 1.2.2 Python release docs say SNAPSHOT + Author is missing

Re: [jenkins infra -- pls read ] installing anaconda, moving default python from 2.6 - 2.7

2015-02-23 Thread Nicholas Chammas
The first concern for Spark will probably be to ensure that we still build and test against Python 2.6, since that's the minimum version of Python we support. Otherwise this seems OK. We use numpy and other Python packages in PySpark, but I don't think we're pinned to any particular version of

[jira] [Commented] (SPARK-4123) Show new dependencies added in pull requests

2015-02-23 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14334352#comment-14334352 ] Nicholas Chammas commented on SPARK-4123: - Go ahead! I haven't done anything

[jira] [Comment Edited] (SPARK-3850) Scala style: disallow trailing spaces

2015-02-23 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14334351#comment-14334351 ] Nicholas Chammas edited comment on SPARK-3850 at 2/24/15 5:16 AM

[jira] [Commented] (SPARK-5312) Use sbt to detect new or changed public classes in PRs

2015-02-23 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14334355#comment-14334355 ] Nicholas Chammas commented on SPARK-5312: - Yeah, this is not a priority really. I

[jira] [Resolved] (SPARK-4958) Bake common tools like ganglia into Spark AMI

2015-02-22 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas resolved SPARK-4958. - Resolution: Duplicate Fix Version/s: (was: 1.3.0) Closing this as a duplicate

Re: Improving metadata in Spark JIRA

2015-02-22 Thread Nicholas Chammas
it advance the house-cleaning a bit more, but I'm sure we'd rediscover some important work and issues that need attention. On Sun, Feb 22, 2015 at 7:54 AM, Nicholas Chammas nicholas.cham...@gmail.com wrote: As of right now, there are no more open JIRA issues without an assigned component

[jira] [Commented] (SPARK-3821) Develop an automated way of creating Spark images (AMI, Docker, and others)

2015-02-22 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14332303#comment-14332303 ] Nicholas Chammas commented on SPARK-3821: - For those wanting to use the work being

Git Achievements

2015-02-22 Thread Nicholas Chammas
For fun: http://acha-acha.co/#/repo/https://github.com/apache/spark I just added Spark to this site. Some of these “achievements” are hilarious. Leo Tolstoy: More than 10 lines in a commit message Dangerous Game: Commit after 6PM friday Nick ​

[jira] [Commented] (SPARK-5944) Python release docs say SNAPSHOT + Author is missing

2015-02-22 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14332394#comment-14332394 ] Nicholas Chammas commented on SPARK-5944: - cc [~davies], [~joshrosen] Python

[jira] [Created] (SPARK-5944) Python release docs say SNAPSHOT + Author is missing

2015-02-22 Thread Nicholas Chammas (JIRA)
Nicholas Chammas created SPARK-5944: --- Summary: Python release docs say SNAPSHOT + Author is missing Key: SPARK-5944 URL: https://issues.apache.org/jira/browse/SPARK-5944 Project: Spark

[jira] [Commented] (SPARK-765) Test suite should run Spark example programs

2015-02-22 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14332438#comment-14332438 ] Nicholas Chammas commented on SPARK-765: Seems like a good idea. [~joshrosen] I

<    7   8   9   10   11   12   13   14   15   16   >