[jira] [Commented] (SPARK-1391) BlockManager cannot transfer blocks larger than 2G in size

2014-04-02 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13958014#comment-13958014 ] Shivaram Venkataraman commented on SPARK-1391: -- Oh and yes, I'd be happy to

[jira] [Commented] (SPARK-1391) BlockManager cannot transfer blocks larger than 2G in size

2014-04-02 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13958013#comment-13958013 ] Shivaram Venkataraman commented on SPARK-1391: -- I am not using any fastutil

[jira] [Commented] (SPARK-1391) BlockManager cannot transfer blocks larger than 2G in size

2014-04-04 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13960100#comment-13960100 ] Shivaram Venkataraman commented on SPARK-1391: -- Thanks for the patch. I will

[jira] [Created] (SPARK-1614) Move Mesos protobufs out of TaskState

2014-04-24 Thread Shivaram Venkataraman (JIRA)
Shivaram Venkataraman created SPARK-1614: Summary: Move Mesos protobufs out of TaskState Key: SPARK-1614 URL: https://issues.apache.org/jira/browse/SPARK-1614 Project: Spark Issue

[jira] [Commented] (SPARK-2046) Support config properties that are changeable across tasks/stages within a job

2014-06-05 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14019458#comment-14019458 ] Shivaram Venkataraman commented on SPARK-2046: -- FWIW I have an older

[jira] [Commented] (SPARK-2316) StorageStatusListener should avoid O(blocks) operations

2014-07-17 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14065256#comment-14065256 ] Shivaram Venkataraman commented on SPARK-2316: -- I'd just like to add that in

[jira] [Created] (SPARK-2563) Make number of connection retries configurable

2014-07-17 Thread Shivaram Venkataraman (JIRA)
Shivaram Venkataraman created SPARK-2563: Summary: Make number of connection retries configurable Key: SPARK-2563 URL: https://issues.apache.org/jira/browse/SPARK-2563 Project: Spark

[jira] [Commented] (SPARK-2563) Make number of connection retries configurable

2014-07-17 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14065735#comment-14065735 ] Shivaram Venkataraman commented on SPARK-2563: --

[jira] [Commented] (SPARK-2316) StorageStatusListener should avoid O(blocks) operations

2014-07-25 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14074692#comment-14074692 ] Shivaram Venkataraman commented on SPARK-2316: -- On a related note, can we

[jira] [Updated] (SPARK-2563) Re-open sockets to handle connect timeouts

2014-07-28 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivaram Venkataraman updated SPARK-2563: - Description: In a large EC2 cluster, I often see the first shuffle stage in a

[jira] [Comment Edited] (SPARK-2563) Re-open sockets to handle connect timeouts

2014-07-28 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14065735#comment-14065735 ] Shivaram Venkataraman edited comment on SPARK-2563 at 7/28/14 5:43 PM:

[jira] [Created] (SPARK-2723) Block Manager should catch exceptions in putValues

2014-07-28 Thread Shivaram Venkataraman (JIRA)
Shivaram Venkataraman created SPARK-2723: Summary: Block Manager should catch exceptions in putValues Key: SPARK-2723 URL: https://issues.apache.org/jira/browse/SPARK-2723 Project: Spark

[jira] [Created] (SPARK-2774) Set preferred locations for reduce tasks

2014-07-31 Thread Shivaram Venkataraman (JIRA)
Shivaram Venkataraman created SPARK-2774: Summary: Set preferred locations for reduce tasks Key: SPARK-2774 URL: https://issues.apache.org/jira/browse/SPARK-2774 Project: Spark Issue

[jira] [Created] (SPARK-2950) Add GC time and Shuffle Write time to JobLogger output

2014-08-09 Thread Shivaram Venkataraman (JIRA)
Shivaram Venkataraman created SPARK-2950: Summary: Add GC time and Shuffle Write time to JobLogger output Key: SPARK-2950 URL: https://issues.apache.org/jira/browse/SPARK-2950 Project: Spark

[jira] [Updated] (SPARK-2950) Add GC time and Shuffle Write time to JobLogger output

2014-08-10 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivaram Venkataraman updated SPARK-2950: - Fix Version/s: 1.2.0 Add GC time and Shuffle Write time to JobLogger output

[jira] [Resolved] (SPARK-2950) Add GC time and Shuffle Write time to JobLogger output

2014-08-10 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivaram Venkataraman resolved SPARK-2950. -- Resolution: Fixed Add GC time and Shuffle Write time to JobLogger output

[jira] [Commented] (SPARK-3215) Add remote interface for SparkContext

2014-08-27 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14112855#comment-14112855 ] Shivaram Venkataraman commented on SPARK-3215: -- This looks very interesting

[jira] [Created] (SPARK-3659) Set EC2 version to 1.1.0 in master branch

2014-09-23 Thread Shivaram Venkataraman (JIRA)
Shivaram Venkataraman created SPARK-3659: Summary: Set EC2 version to 1.1.0 in master branch Key: SPARK-3659 URL: https://issues.apache.org/jira/browse/SPARK-3659 Project: Spark

[jira] [Created] (SPARK-3674) Add support for launching YARN clusters in spark-ec2

2014-09-23 Thread Shivaram Venkataraman (JIRA)
Shivaram Venkataraman created SPARK-3674: Summary: Add support for launching YARN clusters in spark-ec2 Key: SPARK-3674 URL: https://issues.apache.org/jira/browse/SPARK-3674 Project: Spark

[jira] [Commented] (SPARK-3522) Make spark-ec2 verbosity configurable

2014-09-28 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14151348#comment-14151348 ] Shivaram Venkataraman commented on SPARK-3522: -- It would be good, but I think

[jira] [Commented] (SPARK-2008) Enhance spark-ec2 to be able to add and remove slaves to an existing cluster

2014-09-30 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14153770#comment-14153770 ] Shivaram Venkataraman commented on SPARK-2008: -- This will be a very useful

[jira] [Commented] (SPARK-3434) Distributed block matrix

2014-09-30 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14153774#comment-14153774 ] Shivaram Venkataraman commented on SPARK-3434: -- I'll post a design doc by

[jira] [Commented] (SPARK-3821) Develop an automated way of creating Spark images (AMI, Docker, and others)

2014-10-07 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14162755#comment-14162755 ] Shivaram Venkataraman commented on SPARK-3821: -- 1. Yes - the same stuff is

[jira] [Commented] (SPARK-3434) Distributed block matrix

2014-10-10 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14167478#comment-14167478 ] Shivaram Venkataraman commented on SPARK-3434: -- ~brkyvz -- We are just adding

[jira] [Comment Edited] (SPARK-3434) Distributed block matrix

2014-10-10 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14167478#comment-14167478 ] Shivaram Venkataraman edited comment on SPARK-3434 at 10/10/14 8:45 PM:

[jira] [Assigned] (SPARK-3434) Distributed block matrix

2014-10-14 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivaram Venkataraman reassigned SPARK-3434: Assignee: Shivaram Venkataraman Distributed block matrix

[jira] [Commented] (SPARK-3434) Distributed block matrix

2014-10-14 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171272#comment-14171272 ] Shivaram Venkataraman commented on SPARK-3434: -- Sorry for the delay in

[jira] [Commented] (SPARK-3957) Broadcast variable memory usage not reflected in UI

2014-10-16 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173954#comment-14173954 ] Shivaram Venkataraman commented on SPARK-3957: -- I think it needs to be

[jira] [Created] (SPARK-3973) Print callSite information for broadcast variables

2014-10-16 Thread Shivaram Venkataraman (JIRA)
Shivaram Venkataraman created SPARK-3973: Summary: Print callSite information for broadcast variables Key: SPARK-3973 URL: https://issues.apache.org/jira/browse/SPARK-3973 Project: Spark

[jira] [Created] (SPARK-4030) `destroy` method in Broadcast should be public

2014-10-20 Thread Shivaram Venkataraman (JIRA)
Shivaram Venkataraman created SPARK-4030: Summary: `destroy` method in Broadcast should be public Key: SPARK-4030 URL: https://issues.apache.org/jira/browse/SPARK-4030 Project: Spark

[jira] [Created] (SPARK-4031) Read broadcast variables on use

2014-10-20 Thread Shivaram Venkataraman (JIRA)
Shivaram Venkataraman created SPARK-4031: Summary: Read broadcast variables on use Key: SPARK-4031 URL: https://issues.apache.org/jira/browse/SPARK-4031 Project: Spark Issue Type:

[jira] [Commented] (SPARK-4030) `destroy` method in Broadcast should be public

2014-10-21 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14178532#comment-14178532 ] Shivaram Venkataraman commented on SPARK-4030: -- Yes - there is a bunch of

[jira] [Commented] (SPARK-4030) `destroy` method in Broadcast should be public

2014-10-23 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14182181#comment-14182181 ] Shivaram Venkataraman commented on SPARK-4030: -- Great -- I'll send a PR and

[jira] [Assigned] (SPARK-4030) `destroy` method in Broadcast should be public

2014-10-24 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivaram Venkataraman reassigned SPARK-4030: Assignee: Shivaram Venkataraman `destroy` method in Broadcast should be

[jira] [Resolved] (SPARK-4030) `destroy` method in Broadcast should be public

2014-10-27 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivaram Venkataraman resolved SPARK-4030. -- Resolution: Fixed Fix Version/s: 1.2.0 `destroy` method in Broadcast

[jira] [Commented] (SPARK-4030) `destroy` method in Broadcast should be public

2014-10-27 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14185263#comment-14185263 ] Shivaram Venkataraman commented on SPARK-4030: -- Issue resolved by pull

[jira] [Resolved] (SPARK-4031) Read broadcast variables on use

2014-10-28 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivaram Venkataraman resolved SPARK-4031. -- Resolution: Fixed Fix Version/s: 1.2.0 Read broadcast variables on use

[jira] [Commented] (SPARK-4031) Read broadcast variables on use

2014-10-28 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14187122#comment-14187122 ] Shivaram Venkataraman commented on SPARK-4031: -- Issue resolved by pull

[jira] [Updated] (SPARK-4137) Relative paths don't get handled correctly by spark-ec2

2014-11-05 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivaram Venkataraman updated SPARK-4137: - Assignee: Nicholas Chammas Relative paths don't get handled correctly by

[jira] [Commented] (SPARK-3821) Develop an automated way of creating Spark images (AMI, Docker, and others)

2014-11-08 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14203757#comment-14203757 ] Shivaram Venkataraman commented on SPARK-3821: -- [~nchammas] Thanks for

[jira] [Commented] (SPARK-3821) Develop an automated way of creating Spark images (AMI, Docker, and others)

2014-11-10 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205131#comment-14205131 ] Shivaram Venkataraman commented on SPARK-3821: -- Regarding reducing init time,

[jira] [Commented] (SPARK-3337) Paranoid quoting in shell to allow install dirs with spaces within.

2014-11-17 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14215640#comment-14215640 ] Shivaram Venkataraman commented on SPARK-3337: -- [~andrewor14] can we pull

[jira] [Commented] (SPARK-3963) Support getting task-scoped properties from TaskContext

2014-12-01 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230878#comment-14230878 ] Shivaram Venkataraman commented on SPARK-3963: -- [~pwendell] This looks pretty

[jira] [Commented] (SPARK-3963) Support getting task-scoped properties from TaskContext

2014-12-01 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230900#comment-14230900 ] Shivaram Venkataraman commented on SPARK-3963: -- Thanks. I somehow missed

[jira] [Commented] (SPARK-2075) Anonymous classes are missing from Spark distribution

2014-12-18 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14252053#comment-14252053 ] Shivaram Venkataraman commented on SPARK-2075: -- So just to make sure I

[jira] [Commented] (SPARK-2075) Anonymous classes are missing from Spark distribution

2014-12-18 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14252360#comment-14252360 ] Shivaram Venkataraman commented on SPARK-2075: -- Hmm -- looking at the release

[jira] [Commented] (SPARK-2075) Anonymous classes are missing from Spark distribution

2014-12-21 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14255364#comment-14255364 ] Shivaram Venkataraman commented on SPARK-2075: -- [~sunrui] What I can see from

[jira] [Commented] (SPARK-4977) spark-ec2 start resets all the spark/conf configurations

2014-12-27 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14259480#comment-14259480 ] Shivaram Venkataraman commented on SPARK-4977: -- I've run into this before

[jira] [Commented] (SPARK-3821) Develop an automated way of creating Spark images (AMI, Docker, and others)

2015-01-02 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14263193#comment-14263193 ] Shivaram Venkataraman commented on SPARK-3821: -- Yeah you are right that the

[jira] [Commented] (SPARK-3821) Develop an automated way of creating Spark images (AMI, Docker, and others)

2015-01-02 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14263181#comment-14263181 ] Shivaram Venkataraman commented on SPARK-3821: -- [~nchammas] Thanks for the

[jira] [Commented] (SPARK-5386) Reduce fails with vectors of big length

2015-01-23 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14289679#comment-14289679 ] Shivaram Venkataraman commented on SPARK-5386: -- Couple of things that might

[jira] [Commented] (SPARK-5386) Reduce fails with vectors of big length

2015-01-23 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14289731#comment-14289731 ] Shivaram Venkataraman commented on SPARK-5386: -- Note that having 2 worker

[jira] [Commented] (SPARK-5386) Reduce fails with vectors of big length

2015-01-23 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14289957#comment-14289957 ] Shivaram Venkataraman commented on SPARK-5386: -- Results are merged on the

[jira] [Created] (SPARK-5654) Integrate SparkR into Apache Spark

2015-02-06 Thread Shivaram Venkataraman (JIRA)
Shivaram Venkataraman created SPARK-5654: Summary: Integrate SparkR into Apache Spark Key: SPARK-5654 URL: https://issues.apache.org/jira/browse/SPARK-5654 Project: Spark Issue Type:

[jira] [Commented] (SPARK-5246) spark/spark-ec2.py cannot start Spark master in VPC if local DNS name does not resolve

2015-01-18 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14282117#comment-14282117 ] Shivaram Venkataraman commented on SPARK-5246: -- Yes - this can be resolved.

[jira] [Commented] (SPARK-3821) Develop an automated way of creating Spark images (AMI, Docker, and others)

2015-01-14 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14277334#comment-14277334 ] Shivaram Venkataraman commented on SPARK-3821: -- Regarding the pre-built

[jira] [Commented] (SPARK-1302) httpd doesn't start in spark-ec2 (cc2.8xlarge)

2015-02-11 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14316541#comment-14316541 ] Shivaram Venkataraman commented on SPARK-1302: -- [~soid] Could you let us

[jira] [Commented] (SPARK-5629) Add spark-ec2 action to return info about an existing cluster

2015-02-17 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14325354#comment-14325354 ] Shivaram Venkataraman commented on SPARK-5629: -- This sounds fine to me and I

[jira] [Commented] (SPARK-757) Deserialization Exception partway into long running job with Netty - MLbase

2015-02-17 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14324687#comment-14324687 ] Shivaram Venkataraman commented on SPARK-757: - Since the shuffle implementation

[jira] [Commented] (SPARK-5629) Add spark-ec2 action to return info about an existing cluster

2015-02-17 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14324907#comment-14324907 ] Shivaram Venkataraman commented on SPARK-5629: -- Is there an example output

[jira] [Commented] (SPARK-5008) Persistent HDFS does not recognize EBS Volumes

2015-01-10 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272830#comment-14272830 ] Shivaram Venkataraman commented on SPARK-5008: -- Hmm I think

[jira] [Commented] (SPARK-5122) Remove Shark from spark-ec2

2015-01-06 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14267296#comment-14267296 ] Shivaram Venkataraman commented on SPARK-5122: -- Yes I think removing shark

[jira] [Updated] (SPARK-4948) Use pssh instead of bash-isms and remove unnecessary operations

2015-01-06 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivaram Venkataraman updated SPARK-4948: - Assignee: Nicholas Chammas Use pssh instead of bash-isms and remove unnecessary

[jira] [Commented] (SPARK-3821) Develop an automated way of creating Spark images (AMI, Docker, and others)

2015-01-13 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14276505#comment-14276505 ] Shivaram Venkataraman commented on SPARK-3821: -- [~nchammas] Yes -- That

[jira] [Commented] (SPARK-5676) License missing from spark-ec2 repo

2015-02-09 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14313269#comment-14313269 ] Shivaram Venkataraman commented on SPARK-5676: -- Yes - it is managed as a

[jira] [Commented] (SPARK-5676) License missing from spark-ec2 repo

2015-02-09 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14313210#comment-14313210 ] Shivaram Venkataraman commented on SPARK-5676: -- Just to clarify some things:

[jira] [Commented] (SPARK-6231) Join on two tables (generated from same one) is broken

2015-03-20 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14372447#comment-14372447 ] Shivaram Venkataraman commented on SPARK-6231: -- [~marmbrus] I've sent the

[jira] [Commented] (SPARK-6246) spark-ec2 can't handle clusters with 100 nodes

2015-03-12 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355403#comment-14355403 ] Shivaram Venkataraman commented on SPARK-6246: -- Hmm - This seems like a bad

[jira] [Commented] (SPARK-5134) Bump default Hadoop version to 2+

2015-03-08 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14352317#comment-14352317 ] Shivaram Venkataraman commented on SPARK-5134: -- Yeah so this did change in

[jira] [Commented] (SPARK-6220) Allow extended EC2 options to be passed through spark-ec2

2015-03-08 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14352514#comment-14352514 ] Shivaram Venkataraman commented on SPARK-6220: -- Seems like a good idea and

[jira] [Commented] (SPARK-5134) Bump default Hadoop version to 2+

2015-03-08 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14352374#comment-14352374 ] Shivaram Venkataraman commented on SPARK-5134: -- Yeah if you exclude Spark's

[jira] [Updated] (SPARK-6088) UI is malformed when tasks fetch remote results

2015-02-28 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivaram Venkataraman updated SPARK-6088: - Attachment: Screenshot 2015-02-28 18.24.42.png UI is malformed when tasks fetch

[jira] [Created] (SPARK-6089) Size of task result fetched can't be found in UI

2015-02-28 Thread Shivaram Venkataraman (JIRA)
Shivaram Venkataraman created SPARK-6089: Summary: Size of task result fetched can't be found in UI Key: SPARK-6089 URL: https://issues.apache.org/jira/browse/SPARK-6089 Project: Spark

[jira] [Commented] (SPARK-6088) UI is malformed when tasks fetch remote results

2015-02-28 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341896#comment-14341896 ] Shivaram Venkataraman commented on SPARK-6088: -- Also for some reason the get

[jira] [Updated] (SPARK-6089) Size of task result fetched can't be found in UI

2015-02-28 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivaram Venkataraman updated SPARK-6089: - Description: When you do a large collect the amount of data fetched as task

[jira] [Resolved] (SPARK-6881) Change the checkpoint directory name from checkpoints to checkpoint

2015-04-13 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivaram Venkataraman resolved SPARK-6881. -- Resolution: Fixed Fix Version/s: 1.4.0 Issue resolved by pull request

[jira] [Updated] (SPARK-6818) Support column deletion in SparkR DataFrame API

2015-04-23 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivaram Venkataraman updated SPARK-6818: - Assignee: Sun Rui Support column deletion in SparkR DataFrame API

[jira] [Resolved] (SPARK-6818) Support column deletion in SparkR DataFrame API

2015-04-23 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivaram Venkataraman resolved SPARK-6818. -- Resolution: Fixed Fix Version/s: 1.4.0 Issue resolved by pull request

[jira] [Updated] (SPARK-6797) Add support for YARN cluster mode

2015-04-22 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivaram Venkataraman updated SPARK-6797: - Assignee: Sun Rui Add support for YARN cluster mode

[jira] [Resolved] (SPARK-7033) Use JavaRDD.partitions() instead of JavaRDD.splits()

2015-04-24 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivaram Venkataraman resolved SPARK-7033. -- Resolution: Fixed Fix Version/s: 1.4.0 Issue resolved by pull request

[jira] [Resolved] (SPARK-6852) Accept numeric as numPartitions in SparkR

2015-04-24 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivaram Venkataraman resolved SPARK-6852. -- Resolution: Fixed Fix Version/s: 1.4.0 Issue resolved by pull request

[jira] [Updated] (SPARK-6852) Accept numeric as numPartitions in SparkR

2015-04-24 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivaram Venkataraman updated SPARK-6852: - Assignee: Sun Rui Accept numeric as numPartitions in SparkR

[jira] [Updated] (SPARK-6824) Fill the docs for DataFrame API in SparkR

2015-04-29 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivaram Venkataraman updated SPARK-6824: - Issue Type: Sub-task (was: New Feature) Parent: SPARK-7228 Fill the

[jira] [Updated] (SPARK-6815) Support accumulators in R

2015-04-29 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivaram Venkataraman updated SPARK-6815: - Target Version/s: 1.5.0 (was: 1.4.0) Support accumulators in R

[jira] [Updated] (SPARK-6825) Data sources implementation to support `sequenceFile`

2015-04-29 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivaram Venkataraman updated SPARK-6825: - Target Version/s: 1.5.0 (was: 1.4.0) Data sources implementation to support

[jira] [Updated] (SPARK-6816) Add SparkConf API to configure SparkR

2015-04-29 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivaram Venkataraman updated SPARK-6816: - Target Version/s: 1.5.0 (was: 1.4.0) Add SparkConf API to configure SparkR

[jira] [Updated] (SPARK-6838) Explore using Reference Classes instead of S4 objects

2015-04-29 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivaram Venkataraman updated SPARK-6838: - Target Version/s: 1.5.0 (was: 1.4.0) Explore using Reference Classes instead of

[jira] [Updated] (SPARK-6803) [SparkR] Support SparkR Streaming

2015-04-29 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivaram Venkataraman updated SPARK-6803: - Target Version/s: 1.5.0 (was: 1.4.0) [SparkR] Support SparkR Streaming

[jira] [Updated] (SPARK-6833) Extend `addPackage` so that any given R file can be sourced in the worker before functions are run.

2015-04-29 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivaram Venkataraman updated SPARK-6833: - Target Version/s: 1.5.0 (was: 1.4.0) Extend `addPackage` so that any given R

[jira] [Updated] (SPARK-6813) SparkR style guide

2015-04-29 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivaram Venkataraman updated SPARK-6813: - Target Version/s: 1.5.0 (was: 1.4.0) SparkR style guide --

[jira] [Updated] (SPARK-6809) Make numPartitions optional in pairRDD APIs

2015-04-29 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivaram Venkataraman updated SPARK-6809: - Priority: Major (was: Critical) Make numPartitions optional in pairRDD APIs

[jira] [Updated] (SPARK-6820) Convert NAs to null type in SparkR DataFrames

2015-04-29 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivaram Venkataraman updated SPARK-6820: - Priority: Critical (was: Major) Convert NAs to null type in SparkR DataFrames

[jira] [Updated] (SPARK-6799) Add dataframe examples for SparkR

2015-04-29 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivaram Venkataraman updated SPARK-6799: - Issue Type: Sub-task (was: Improvement) Parent: SPARK-7228 Add

[jira] [Updated] (SPARK-6809) Make numPartitions optional in pairRDD APIs

2015-04-29 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivaram Venkataraman updated SPARK-6809: - Target Version/s: 1.5.0 (was: 1.4.0) Make numPartitions optional in pairRDD

[jira] [Updated] (SPARK-6826) `hashCode` support for arbitrary R objects

2015-04-29 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivaram Venkataraman updated SPARK-6826: - Target Version/s: 1.5.0 (was: 1.4.0) `hashCode` support for arbitrary R objects

[jira] [Created] (SPARK-7230) Make RDD API private in SparkR for Spark 1.4

2015-04-29 Thread Shivaram Venkataraman (JIRA)
Shivaram Venkataraman created SPARK-7230: Summary: Make RDD API private in SparkR for Spark 1.4 Key: SPARK-7230 URL: https://issues.apache.org/jira/browse/SPARK-7230 Project: Spark

[jira] [Updated] (SPARK-6814) Support sorting for any data type in SparkR

2015-04-29 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivaram Venkataraman updated SPARK-6814: - Target Version/s: 1.5.0 (was: 1.4.0) Support sorting for any data type in

[jira] [Created] (SPARK-7228) SparkR public API for 1.4 release

2015-04-29 Thread Shivaram Venkataraman (JIRA)
Shivaram Venkataraman created SPARK-7228: Summary: SparkR public API for 1.4 release Key: SPARK-7228 URL: https://issues.apache.org/jira/browse/SPARK-7228 Project: Spark Issue Type:

[jira] [Updated] (SPARK-6832) Handle partial reads in SparkR JVM to worker communication

2015-04-29 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivaram Venkataraman updated SPARK-6832: - Target Version/s: 1.5.0 (was: 1.4.0) Handle partial reads in SparkR JVM to

[jira] [Updated] (SPARK-7226) Support math functions in R DataFrame

2015-04-29 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivaram Venkataraman updated SPARK-7226: - Priority: Critical (was: Major) Support math functions in R DataFrame

  1   2   3   4   5   6   7   8   9   10   >