Re: How to create spark AMI in AWS

Guodong > > On Tue, Feb 10, 2015 at 3:59 AM, Nicholas Chammas < > nicholas.cham...@gmail.com> wrote: > >> Guodong, >> >> spark-ec2 does not currently support the cn-north-1 region, but you can >> follow [SPARK-4241](https://issues.apache.org/jira/brows

[jira] [Commented] (SPARK-5676) License missing from spark-ec2 repo

[ https://issues.apache.org/jira/browse/SPARK-5676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14313271#comment-14313271 ] Nicholas Chammas commented on SPARK-5676: - Yeah, AFAIK it has nothing to do

[jira] [Commented] (SPARK-5676) License missing from spark-ec2 repo

[ https://issues.apache.org/jira/browse/SPARK-5676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14313254#comment-14313254 ] Nicholas Chammas commented on SPARK-5676: - It ended up in Mesos because [S

[jira] [Commented] (SPARK-5676) License missing from spark-ec2 repo

[ https://issues.apache.org/jira/browse/SPARK-5676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14313214#comment-14313214 ] Nicholas Chammas commented on SPARK-5676: - [~srowen] - I don't t

[jira] [Commented] (SPARK-1805) Error launching cluster when master and slave machines are of different virtualization types

[ https://issues.apache.org/jira/browse/SPARK-1805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14313079#comment-14313079 ] Nicholas Chammas commented on SPARK-1805: - I've created a PR to catch t

[jira] [Commented] (SPARK-3044) Create RSS feed for Spark News

[ https://issues.apache.org/jira/browse/SPARK-3044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14312878#comment-14312878 ] Nicholas Chammas commented on SPARK-3044: - I use RSS plenty to track compa

[jira] [Commented] (SPARK-3044) Create RSS feed for Spark News

[ https://issues.apache.org/jira/browse/SPARK-3044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14312854#comment-14312854 ] Nicholas Chammas commented on SPARK-3044: - [~rxin] / [~pwendell] Is there an

Re: How to create spark AMI in AWS

Guodong, spark-ec2 does not currently support the cn-north-1 region, but you can follow [SPARK-4241](https://issues.apache.org/jira/browse/SPARK-4241) to find out when it does. The base AMI used to generate the current Spark AMIs is very old. I'm not sure anyone knows what it is anymore. What I k

Re: Keep or remove Debian packaging in Spark?

+1 to an "official" deprecation + redirecting users to some other project that will or already is taking this on. Nate? On Mon Feb 09 2015 at 10:08:27 AM Patrick Wendell wrote: > I have wondered whether we should sort of deprecated it more > officially, since otherwise I think people have the

[jira] [Commented] (SPARK-5685) Show warning when users open text files compressed with non-splittable algorithms like gzip

[ https://issues.apache.org/jira/browse/SPARK-5685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14311931#comment-14311931 ] Nicholas Chammas commented on SPARK-5685: - [~joshrosen] - What do you thin

[jira] [Created] (SPARK-5685) Show warning when users open text files compressed with non-splittable algorithms like gzip

Nicholas Chammas created SPARK-5685: --- Summary: Show warning when users open text files compressed with non-splittable algorithms like gzip Key: SPARK-5685 URL: https://issues.apache.org/jira/browse/SPARK-5685

Re: Using CUDA within Spark / boosting linear algebra

Lemme butt in randomly here and say there is an interesting discussion on this Spark PR about netlib-java, JBLAS, Breeze, and other things I know nothing of, that y'all may find interesting. Among the participants is the author of netlib-java. On Sun Feb

Re: Improving metadata in Spark JIRA

y merge them into Spark Core. > > On Fri, Feb 6, 2015 at 11:53 AM, Nicholas Chammas > wrote: > > Do we need some new components to be added to the JIRA project? > > > > Like: > > > >- > > > >scheduler > > - > >

Re: Improving metadata in Spark JIRA

we already have a YARN component. > > https://issues.apache.org/jira/issues/?jql=project%20% > 3D%20SPARK%20AND%20component%20%3D%20YARN > > I don't think JIRA allows it to be mandatory, but if it does, that > would be useful. > > On Sat, Feb 7, 2015 at 5:08 PM, Nicholas

[jira] [Commented] (SPARK-3431) Parallelize Scala/Java test execution

[ https://issues.apache.org/jira/browse/SPARK-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14311135#comment-14311135 ] Nicholas Chammas commented on SPARK-3431: - [~srowen] - Have you tried anyt

[jira] [Issue Comment Deleted] (SPARK-5524) Remove messy dependencies to log4j

[ https://issues.apache.org/jira/browse/SPARK-5524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-5524: Comment: was deleted (was: Oh my bad. Thanks for the correction.) > Remove me

[jira] [Commented] (SPARK-5524) Remove messy dependencies to log4j

[ https://issues.apache.org/jira/browse/SPARK-5524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14311121#comment-14311121 ] Nicholas Chammas commented on SPARK-5524: - Oh my bad. Thanks for the correc

[jira] [Commented] (SPARK-5524) Remove messy dependencies to log4j

[ https://issues.apache.org/jira/browse/SPARK-5524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14311122#comment-14311122 ] Nicholas Chammas commented on SPARK-5524: - Oh my bad. Thanks for the correc

[jira] [Updated] (SPARK-5668) spark_ec2.py region parameter could be either mandatory or its value displayed

[ https://issues.apache.org/jira/browse/SPARK-5668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-5668: Labels: starter (was: ) > spark_ec2.py region parameter could be either mandatory or

[jira] [Commented] (SPARK-5628) Add option to return spark-ec2 version

[ https://issues.apache.org/jira/browse/SPARK-5628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14311055#comment-14311055 ] Nicholas Chammas commented on SPARK-5628: - We still need a backport to 1.2.2

[jira] [Commented] (SPARK-5668) spark_ec2.py region parameter could be either mandatory or its value displayed

[ https://issues.apache.org/jira/browse/SPARK-5668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14311054#comment-14311054 ] Nicholas Chammas commented on SPARK-5668: - This sounds good to me, Miguel.

[jira] [Updated] (SPARK-4383) Delay scheduling doesn't work right when jobs have tasks with different locality levels

[ https://issues.apache.org/jira/browse/SPARK-4383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-4383: Component/s: Scheduler > Delay scheduling doesn't work right when jobs have ta

[jira] [Updated] (SPARK-1142) Allow adding jars on app submission, outside of code

[ https://issues.apache.org/jira/browse/SPARK-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-1142: Component/s: Spark Submit > Allow adding jars on app submission, outside of c

[jira] [Updated] (SPARK-5156) Priority queue for cross application scheduling

[ https://issues.apache.org/jira/browse/SPARK-5156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-5156: Component/s: Scheduler > Priority queue for cross application schedul

[jira] [Updated] (SPARK-5080) Expose more cluster resource information to user

[ https://issues.apache.org/jira/browse/SPARK-5080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-5080: Component/s: Spark Core > Expose more cluster resource information to u

[jira] [Updated] (SPARK-5524) Remove messy dependencies to log4j

[ https://issues.apache.org/jira/browse/SPARK-5524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-5524: Component/s: Build > Remove messy dependencies to lo

[jira] [Updated] (SPARK-4808) Spark fails to spill with small number of large objects

[ https://issues.apache.org/jira/browse/SPARK-4808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-4808: Component/s: Spark Core > Spark fails to spill with small number of large obje

[jira] [Commented] (SPARK-5363) Spark 1.2 freeze without error notification

[ https://issues.apache.org/jira/browse/SPARK-5363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14311050#comment-14311050 ] Nicholas Chammas commented on SPARK-5363: - [~TJKlein] - Can you provide

[jira] [Updated] (SPARK-5175) bug in updating counters when starting multiple workers/supervisors in actor-based receiver

[ https://issues.apache.org/jira/browse/SPARK-5175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-5175: Component/s: (was: Spark Core) Streaming > bug in updating count

[jira] [Updated] (SPARK-5363) Spark 1.2 freeze without error notification

[ https://issues.apache.org/jira/browse/SPARK-5363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-5363: Component/s: PySpark > Spark 1.2 freeze without error notificat

[jira] [Updated] (SPARK-5175) bug in updating counters when starting multiple workers/supervisors in actor-based receiver

[ https://issues.apache.org/jira/browse/SPARK-5175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-5175: Component/s: Spark Core > bug in updating counters when starting multiple work

[jira] [Updated] (SPARK-5259) Fix endless retry stage by add task equal() and hashcode() to avoid stage.pendingTasks not empty while stage map output is available

[ https://issues.apache.org/jira/browse/SPARK-5259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-5259: Component/s: Spark Core > Fix endless retry stage by add task equal() and hashcode()

[jira] [Updated] (SPARK-1061) allow Hadoop RDDs to be read w/ a partitioner

[ https://issues.apache.org/jira/browse/SPARK-1061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-1061: Component/s: Spark Core > allow Hadoop RDDs to be read w/ a partitio

[jira] [Updated] (SPARK-5664) Restore stty settings when exiting for launching spark-shell from SBT

[ https://issues.apache.org/jira/browse/SPARK-5664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-5664: Component/s: Build > Restore stty settings when exiting for launching spark-shell from

[jira] [Updated] (SPARK-5628) Add option to return spark-ec2 version

[ https://issues.apache.org/jira/browse/SPARK-5628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-5628: Labels: backport-needed (was: ) > Add option to return spark-ec2 vers

[jira] [Updated] (SPARK-5628) Add option to return spark-ec2 version

[ https://issues.apache.org/jira/browse/SPARK-5628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-5628: Fix Version/s: 1.2.2 > Add option to return spark-ec2 vers

[jira] [Updated] (SPARK-3956) Python API for Distributed Matrix

[ https://issues.apache.org/jira/browse/SPARK-3956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-3956: Component/s: PySpark > Python API for Distributed Mat

[jira] [Updated] (SPARK-3600) RDD[Double] doesn't use primitive arrays for caching

[ https://issues.apache.org/jira/browse/SPARK-3600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-3600: Component/s: Spark Core > RDD[Double] doesn't use primitive arrays for

[jira] [Updated] (SPARK-4024) Remember user preferences for metrics to show in the UI

[ https://issues.apache.org/jira/browse/SPARK-4024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-4024: Component/s: Web UI > Remember user preferences for metrics to show in the

[jira] [Updated] (SPARK-3246) Support weighted SVMWithSGD for classification of unbalanced dataset

[ https://issues.apache.org/jira/browse/SPARK-3246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-3246: Component/s: MLlib > Support weighted SVMWithSGD for classification of unbalanced data

Re: Improving metadata in Spark JIRA

Do we need some new components to be added to the JIRA project? Like: - scheduler - YARN - spark-submit - …? Nick ​ On Fri Feb 06 2015 at 10:50:41 AM Nicholas Chammas < nicholas.cham...@gmail.com> wrote: > +9000 on cleaning up JIRA. > > Thank you Sean for

[jira] [Updated] (SPARK-2064) web ui should not remove executors if they are dead

[ https://issues.apache.org/jira/browse/SPARK-2064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-2064: Component/s: Web UI > web ui should not remove executors if they are d

[jira] [Updated] (SPARK-2654) Leveled logging in PySpark

[ https://issues.apache.org/jira/browse/SPARK-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-2654: Component/s: PySpark > Leveled logging in PySp

[jira] [Updated] (SPARK-1346) Backport SPARK-1210 into 0.9 branch

[ https://issues.apache.org/jira/browse/SPARK-1346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-1346: Labels: backport-needed (was: ) > Backport SPARK-1210 into 0.9 bra

[jira] [Updated] (SPARK-1927) Implicits declared in companion objects not found in Spark shell

[ https://issues.apache.org/jira/browse/SPARK-1927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-1927: Component/s: Spark Shell > Implicits declared in companion objects not found in Spark sh

[jira] [Commented] (SPARK-1927) Implicits declared in companion objects not found in Spark shell

[ https://issues.apache.org/jira/browse/SPARK-1927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309755#comment-14309755 ] Nicholas Chammas commented on SPARK-1927: - cc [~tobias.schlatter] > Im

[jira] [Updated] (SPARK-729) Closures not always serialized at capture time

[ https://issues.apache.org/jira/browse/SPARK-729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-729: --- Component/s: Spark Core > Closures not always serialized at capture t

[jira] [Commented] (SPARK-1799) Add init script to the debian packaging

[ https://issues.apache.org/jira/browse/SPARK-1799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309753#comment-14309753 ] Nicholas Chammas commented on SPARK-1799: - cc [~markhamstra], [~sr

[jira] [Updated] (SPARK-2326) DiskBlockManager could add DiskChecker function for kicking off bad directories

[ https://issues.apache.org/jira/browse/SPARK-2326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-2326: Component/s: Block Manager > DiskBlockManager could add DiskChecker function for kick

[jira] [Updated] (SPARK-1425) PySpark can crash Executors if worker.py fails while serializing data

[ https://issues.apache.org/jira/browse/SPARK-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-1425: Component/s: PySpark > PySpark can crash Executors if worker.py fails while serializ

[jira] [Commented] (SPARK-761) Print a nicer error message when incompatible Spark binaries try to talk

[ https://issues.apache.org/jira/browse/SPARK-761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309741#comment-14309741 ] Nicholas Chammas commented on SPARK-761: Not sure what component this falls u

[jira] [Commented] (SPARK-706) Failures in block manager put leads to task hanging

[ https://issues.apache.org/jira/browse/SPARK-706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309733#comment-14309733 ] Nicholas Chammas commented on SPARK-706: [~rxin] Is this issue still v

[jira] [Updated] (SPARK-706) Failures in block manager put leads to task hanging

[ https://issues.apache.org/jira/browse/SPARK-706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-706: --- Component/s: Block Manager > Failures in block manager put leads to task hang

[jira] [Commented] (SPARK-540) Add API to customize in-memory representation of RDDs

[ https://issues.apache.org/jira/browse/SPARK-540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309728#comment-14309728 ] Nicholas Chammas commented on SPARK-540: [~matei], [~rxin]: Is this issue s

[jira] [Updated] (SPARK-540) Add API to customize in-memory representation of RDDs

[ https://issues.apache.org/jira/browse/SPARK-540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-540: --- Component/s: Spark Core > Add API to customize in-memory representation of R

[jira] [Commented] (SPARK-560) Specialize RDDs / iterators

[ https://issues.apache.org/jira/browse/SPARK-560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309709#comment-14309709 ] Nicholas Chammas commented on SPARK-560: [~matei], [~pwendell], [~rxin]: Is

[jira] [Updated] (SPARK-560) Specialize RDDs / iterators

[ https://issues.apache.org/jira/browse/SPARK-560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-560: --- Component/s: Spark Core > Specialize RDDs / iterat

[jira] [Commented] (SPARK-4983) Tag EC2 instances in the same call that launches them

[ https://issues.apache.org/jira/browse/SPARK-4983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309649#comment-14309649 ] Nicholas Chammas commented on SPARK-4983: - To summarize the discussion we ha

Re: Improving metadata in Spark JIRA

+9000 on cleaning up JIRA. Thank you Sean for laying out some specific things to tackle. I will assist with this. Regarding email, I think Sandy is right. I only get JIRA email for issues I'm watching. Nick On Fri Feb 06 2015 at 9:52:58 AM Sandy Ryza wrote: > JIRA updates don't go to this lis

[jira] [Comment Edited] (SPARK-5637) Expose spark_ec2 as as StarCluster Plugin

[ https://issues.apache.org/jira/browse/SPARK-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14308394#comment-14308394 ] Nicholas Chammas edited comment on SPARK-5637 at 2/6/15 1:1

[jira] [Comment Edited] (SPARK-5335) Destroying cluster in VPC with "--delete-groups" fails to remove security groups

[ https://issues.apache.org/jira/browse/SPARK-5335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14308133#comment-14308133 ] Nicholas Chammas edited comment on SPARK-5335 at 2/6/15 1:1

[jira] [Commented] (SPARK-5637) Expose spark_ec2 as as StarCluster Plugin

[ https://issues.apache.org/jira/browse/SPARK-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14308394#comment-14308394 ] Nicholas Chammas commented on SPARK-5637: - [~agrothberg] - Can you expand on

[jira] [Updated] (SPARK-5637) Expose spark_ec2 as as StarCluster Plugin

[ https://issues.apache.org/jira/browse/SPARK-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-5637: Component/s: EC2 > Expose spark_ec2 as as StarCluster Plu

PSA: Maven supports parallel builds

Y’all may already know this, but I haven’t seen it mentioned anywhere in our docs on here and it’s a pretty easy win. Maven supports parallel builds with the -T command line option. For example: ./build/mvn -T 1C -Dha

[jira] [Commented] (SPARK-5335) Destroying cluster in VPC with "--delete-groups" fails to remove security groups

[ https://issues.apache.org/jira/browse/SPARK-5335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14308133#comment-14308133 ] Nicholas Chammas commented on SPARK-5335: - For the record: [AWS says|h

[jira] [Created] (SPARK-5629) Add spark-ec2 option to return info about cluster for programmatic consumption

Nicholas Chammas created SPARK-5629: --- Summary: Add spark-ec2 option to return info about cluster for programmatic consumption Key: SPARK-5629 URL: https://issues.apache.org/jira/browse/SPARK-5629

[jira] [Updated] (SPARK-5629) Add spark-ec2 option to return info about cluster for programmatic consumption

[ https://issues.apache.org/jira/browse/SPARK-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-5629: Description: If someone is programmatically launching a cluster with {{spark-ec2}}, they

[jira] [Created] (SPARK-5628) Add option to return spark-ec2 version

Nicholas Chammas created SPARK-5628: --- Summary: Add option to return spark-ec2 version Key: SPARK-5628 URL: https://issues.apache.org/jira/browse/SPARK-5628 Project: Spark Issue Type

[jira] [Created] (SPARK-5627) Enhance spark-ec2 for some programmatic use cases

Nicholas Chammas created SPARK-5627: --- Summary: Enhance spark-ec2 for some programmatic use cases Key: SPARK-5627 URL: https://issues.apache.org/jira/browse/SPARK-5627 Project: Spark Issue

[jira] [Commented] (SPARK-5403) Ignore UserKnownHostsFile in SSH calls

[ https://issues.apache.org/jira/browse/SPARK-5403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14307904#comment-14307904 ] Nicholas Chammas commented on SPARK-5403: - I believe this issue duplicate

[jira] [Updated] (SPARK-3185) SPARK launch on Hadoop 2 in EC2 throws Tachyon exception when Formatting JOURNAL_FOLDER

[ https://issues.apache.org/jira/browse/SPARK-3185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-3185: Component/s: EC2 > SPARK launch on Hadoop 2 in EC2 throws Tachyon exception when Formatt

[jira] [Commented] (SPARK-4868) Twitter DStream.map() throws "Task not serializable"

[ https://issues.apache.org/jira/browse/SPARK-4868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14305798#comment-14305798 ] Nicholas Chammas commented on SPARK-4868: - [~tobias.schlatter] - Here's

Re: Welcoming three new committers

Congratulations guys! On Tue Feb 03 2015 at 2:36:12 PM Matei Zaharia wrote: > Hi all, > > The PMC recently voted to add three new committers: Cheng Lian, Joseph > Bradley and Sean Owen. All three have been major contributors to Spark in > the past year: Cheng on Spark SQL, Joseph on MLlib, and S

Re: [VOTE] Release Apache Spark 1.2.1 (RC3)

I believe this was changed for 1.2.1. Here are the relevant JIRA issues . On Tue Feb 03 2015 at 10:43:59 AM Dirceu Semighini Filho

Re: Building Spark with Pants

hare. On Mon Feb 02 2015 at 4:40:45 PM Nicholas Chammas < nicholas.cham...@gmail.com> wrote: > I'm asking from an experimental standpoint; this is not happening anytime > soon. > > Of course, if the experiment turns out very well, Pants would replace both > sbt and Mave

Re: Building Spark with Pants

sently > for sbt and with a little bit of tweaking with maven as well. > > 2015-02-02 16:25 GMT-08:00 Nicholas Chammas : > >> Does anyone here have experience with Pants >> > <http://pantsbuild.github.io/index.html> or interest in trying to build > > >> Sp

[jira] [Commented] (SPARK-2005) Investigate linux container-based solution

[ https://issues.apache.org/jira/browse/SPARK-2005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14302554#comment-14302554 ] Nicholas Chammas commented on SPARK-2005: - [~mengxr] - Do you mind if I ren

[jira] [Commented] (SPARK-2004) QA Automation

[ https://issues.apache.org/jira/browse/SPARK-2004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14302549#comment-14302549 ] Nicholas Chammas commented on SPARK-2004: - [~mengxr] - Do you mind if I ren

Building Spark with Pants

Does anyone here have experience with Pants or interest in trying to build Spark with it? Pants has an interesting story. It was born at Twitter to help them build their Scala, Java, and Python projects as several independent components in one monolithic re

[jira] [Commented] (SPARK-5541) Allow running Maven or SBT in run-tests

[ https://issues.apache.org/jira/browse/SPARK-5541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14302525#comment-14302525 ] Nicholas Chammas commented on SPARK-5541: - Dup of SPARK-3355? > Allow

Spark Master Maven with YARN build is broken

https://amplab.cs.berkeley.edu/jenkins/view/Spark/job/Spark-Master-Maven-with-YARN/HADOOP_PROFILE=hadoop-2.4,label=centos/ Is this is a known issue? It seems to have been broken since last night. Here’s a snippet from the build output of one of the builds

Re: [VOTE] Release Apache Spark 1.2.1 (RC2)

Do we have any open JIRA issues to add automated testing on Windows to Jenkins? I assume that's something we want to do. On Sat Jan 31 2015 at 10:37:42 PM Matei Zaharia wrote: > This looks like a pretty serious problem, thanks! Glad people are testing > on Windows. > > Matei > > > On Jan 31, 201

[jira] [Created] (SPARK-5473) Expose SSH failures after status checks pass

Nicholas Chammas created SPARK-5473: --- Summary: Expose SSH failures after status checks pass Key: SPARK-5473 URL: https://issues.apache.org/jira/browse/SPARK-5473 Project: Spark Issue Type

Re: spark 1.2 ec2 launch script hang

eah, I agree ~ should work. And it could have been [read: probably was] >> the fact that one of the EC2 hosts was in my known_hosts (don't know, never >> saw an error message, but the behavior is no error message for that state), >> which I had fixed later with Pete's pa

Re: spark 1.2 ec2 launch script hang

If that was indeed the problem, I suggest updating your answer on SO <http://stackoverflow.com/a/28005151/877069> to help others who may run into this same problem. ​ On Wed Jan 28 2015 at 9:40:39 PM Nicholas Chammas < nicholas.cham...@gmail.com> wrote: > Thanks for sending t

Re: spark 1.2 ec2 launch script hang

and 2 production clusters on EC2 > since with no problems.) > > On Wed Jan 28 2015 at 12:05:43 PM Nicholas Chammas < > nicholas.cham...@gmail.com> wrote: > >> Ey-chih, >> >> That makes more sense. This is a known issue that will be fixed as part >> of

Re: spark 1.2 ec2 launch script hang

Ey-chih, That makes more sense. This is a known issue that will be fixed as part of SPARK-5242 . Charles, Thanks for the info. In your case, when does spark-ec2 hang? Only when the specified path to the identity file doesn't exist? Or also when y

Re: Extending Scala style checks

Reynold Xin wrote: > Thanks. I added one. > > > On Wed, Oct 8, 2014 at 8:49 AM, Nicholas Chammas < > nicholas.cham...@gmail.com> wrote: > >> I've created SPARK-3849: Automate remaining Scala style rules >> > <https://issues.apache.org/jira/browse/SPARK-

[jira] [Created] (SPARK-5434) Preserve spaces in path to spark-ec2

Nicholas Chammas created SPARK-5434: --- Summary: Preserve spaces in path to spark-ec2 Key: SPARK-5434 URL: https://issues.apache.org/jira/browse/SPARK-5434 Project: Spark Issue Type: Bug

Re: spark 1.2 ec2 launch script hang

For those who found that absolute vs. relative path for the pem file mattered, what OS and shell are you using? What version of Spark are you using? ~/ vs. absolute path shouldn’t matter. Your shell will expand the ~/ to the absolute path before sending it to spark-ec2. (i.e. tilde expansion.) Ab

Re: saving rdd to multiple files named by the key

There is also SPARK-3533 , which proposes to add a convenience method for this. ​ On Mon Jan 26 2015 at 10:38:56 PM Aniket Bhatnagar < aniket.bhatna...@gmail.com> wrote: > This might be helpful: > http://stackoverflow.com/questions/23995040/write-

[jira] [Commented] (SPARK-2008) Enhance spark-ec2 to be able to add and remove slaves to an existing cluster

[ https://issues.apache.org/jira/browse/SPARK-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14293805#comment-14293805 ] Nicholas Chammas commented on SPARK-2008: - This isn't implemented

Does spark-ec2 support Windows?

Is spark-ec2 supposed to run normally from Windows (e.g. to launch a cluster)? I ask because I don’t see mention of Windows anywhere in relation to spark-ec2, and there is an open PR that checks file permis

[jira] [Created] (SPARK-5398) Support the eu-central-1 region for spark-ec2

Nicholas Chammas created SPARK-5398: --- Summary: Support the eu-central-1 region for spark-ec2 Key: SPARK-5398 URL: https://issues.apache.org/jira/browse/SPARK-5398 Project: Spark Issue Type

Re: Analyzing data from non-standard data sources (e.g. AWS Redshift)

I believe databricks provides an rdd interface to redshift. Did you check spark-packages.org? On 2015년 1월 24일 (토) at 오전 6:45 Denis Mikhalkin wrote: > Hello, > > we've got some analytics data in AWS Redshift. The data is being > constantly updated. > > I'd like to be able to write a query against

Re: Discourse: A proposed alternative to the Spark User list

https://issues.apache.org/jira/browse/SPARK-5390 On Fri Jan 23 2015 at 12:05:00 PM Gerard Maas wrote: > +1 > > On Fri, Jan 23, 2015 at 5:58 PM, Nicholas Chammas < > nicholas.cham...@gmail.com> wrote: > >> That sounds good to me. Shall I open a JIRA / PR about updating

[jira] [Comment Edited] (SPARK-5390) Encourage users to post on Stack Overflow in Community Docs

[ https://issues.apache.org/jira/browse/SPARK-5390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14290187#comment-14290187 ] Nicholas Chammas edited comment on SPARK-5390 at 1/23/15 11:0

[jira] [Commented] (SPARK-5390) Encourage users to post on Stack Overflow in Community Docs

[ https://issues.apache.org/jira/browse/SPARK-5390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14290187#comment-14290187 ] Nicholas Chammas commented on SPARK-5390: - cc [~pwendell] > Encourage u

[jira] [Commented] (SPARK-5390) Encourage users to post on Stack Overflow in Community Docs

[ https://issues.apache.org/jira/browse/SPARK-5390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14290185#comment-14290185 ] Nicholas Chammas commented on SPARK-5390: - Updated accordingly. > En

[jira] [Updated] (SPARK-5390) Encourage users to post on Stack Overflow in Community Docs

[ https://issues.apache.org/jira/browse/SPARK-5390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-5390: Description: As [discussed extensively on the user list|http://apache-spark-user-list

<    9   10   11   12   13   14   15   16   17   18   >