[jira] [Commented] (SPARK-2365) Add IndexedRDD, an efficient updatable key-value store

[ https://issues.apache.org/jira/browse/SPARK-2365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14140647#comment-14140647 ] Imran Rashid commented on SPARK-2365: - This looks fantastic. I think it will also see

[jira] [Created] (SPARK-4206) BlockManager warnings in local mode: Block $blockId already exists on this machine; not re-adding it

Imran Rashid created SPARK-4206: --- Summary: BlockManager warnings in local mode: Block $blockId already exists on this machine; not re-adding it Key: SPARK-4206 URL: https://issues.apache.org/jira/browse/SPARK-4206

[jira] [Updated] (SPARK-4206) BlockManager warnings in local mode: Block $blockId already exists on this machine; not re-adding it

[ https://issues.apache.org/jira/browse/SPARK-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-4206: Description: When running in local mode, you often get log warning messages like: WARN

[jira] [Commented] (SPARK-4206) BlockManager warnings in local mode: Block $blockId already exists on this machine; not re-adding it

[ https://issues.apache.org/jira/browse/SPARK-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14194574#comment-14194574 ] Imran Rashid commented on SPARK-4206: - thanks Sean -- sorry I accidentally created the

[jira] [Commented] (SPARK-4206) BlockManager warnings in local mode: Block $blockId already exists on this machine; not re-adding it

[ https://issues.apache.org/jira/browse/SPARK-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14194793#comment-14194793 ] Imran Rashid commented on SPARK-4206: - a little more confirmation: If I change this

[jira] [Commented] (SPARK-4206) BlockManager warnings in local mode: Block $blockId already exists on this machine; not re-adding it

[ https://issues.apache.org/jira/browse/SPARK-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205081#comment-14205081 ] Imran Rashid commented on SPARK-4206: - actually, it looks like this was kind of fixed

[jira] [Commented] (SPARK-664) Accumulator updates should get locally merged before sent to the driver

[ https://issues.apache.org/jira/browse/SPARK-664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14214038#comment-14214038 ] Imran Rashid commented on SPARK-664: Hi [~aash] , thanks for taking another look at

[jira] [Commented] (SPARK-603) add simple Counter API

[ https://issues.apache.org/jira/browse/SPARK-603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14214142#comment-14214142 ] Imran Rashid commented on SPARK-603: Hey, this was originally reported by me too

[jira] [Created] (SPARK-4746) integration tests should be seseparated from faster unit tests

Imran Rashid created SPARK-4746: --- Summary: integration tests should be seseparated from faster unit tests Key: SPARK-4746 URL: https://issues.apache.org/jira/browse/SPARK-4746 Project: Spark

[jira] [Updated] (SPARK-4746) integration tests should be separated from faster unit tests

[ https://issues.apache.org/jira/browse/SPARK-4746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-4746: Summary: integration tests should be separated from faster unit tests (was: integration tests

[jira] [Commented] (SPARK-4746) integration tests should be separated from faster unit tests

[ https://issues.apache.org/jira/browse/SPARK-4746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14241383#comment-14241383 ] Imran Rashid commented on SPARK-4746: - I think this can be done two ways: (1) by

[jira] [Commented] (SPARK-3655) Support sorting of values in addition to keys (i.e. secondary sort)

[ https://issues.apache.org/jira/browse/SPARK-3655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14254107#comment-14254107 ] Imran Rashid commented on SPARK-3655: - I think secondary sort is a great addition,

[jira] [Commented] (SPARK-3655) Support sorting of values in addition to keys (i.e. secondary sort)

[ https://issues.apache.org/jira/browse/SPARK-3655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14254942#comment-14254942 ] Imran Rashid commented on SPARK-3655: - Hey Koert, good questions about the types, I

[jira] [Commented] (SPARK-3644) REST API for Spark application info (jobs / stages / tasks / storage info)

[ https://issues.apache.org/jira/browse/SPARK-3644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14292370#comment-14292370 ] Imran Rashid commented on SPARK-3644: - [~joshrosen] Hi Josh, I've got time to

[jira] [Updated] (SPARK-3454) Expose JSON representation of data shown in WebUI

[ https://issues.apache.org/jira/browse/SPARK-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-3454: Attachment: sparkmonitoringjsondesign.pdf Expose JSON representation of data shown in WebUI

[jira] [Commented] (SPARK-3454) Expose JSON representation of data shown in WebUI

[ https://issues.apache.org/jira/browse/SPARK-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14298747#comment-14298747 ] Imran Rashid commented on SPARK-3454: - design doc attached, would love any feedback

[jira] [Assigned] (SPARK-1061) allow Hadoop RDDs to be read w/ a partitioner

[ https://issues.apache.org/jira/browse/SPARK-1061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-1061: --- Assignee: Imran Rashid allow Hadoop RDDs to be read w/ a partitioner

[jira] [Created] (SPARK-5574) Utils.createDirectory ignores namePrefix

Imran Rashid created SPARK-5574: --- Summary: Utils.createDirectory ignores namePrefix Key: SPARK-5574 URL: https://issues.apache.org/jira/browse/SPARK-5574 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-4746) integration tests should be separated from faster unit tests

[ https://issues.apache.org/jira/browse/SPARK-4746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14277646#comment-14277646 ] Imran Rashid commented on SPARK-4746: - submitted a PR:

[jira] [Commented] (SPARK-4746) integration tests should be separated from faster unit tests

[ https://issues.apache.org/jira/browse/SPARK-4746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14277651#comment-14277651 ] Imran Rashid commented on SPARK-4746: - btw if anybody else wants to futz around with

[jira] [Commented] (SPARK-4746) integration tests should be separated from faster unit tests

[ https://issues.apache.org/jira/browse/SPARK-4746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14279524#comment-14279524 ] Imran Rashid commented on SPARK-4746: - This doesn't work as well as I thought -- all

[jira] [Commented] (SPARK-3622) Provide a custom transformation that can output multiple RDDs

[ https://issues.apache.org/jira/browse/SPARK-3622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14279602#comment-14279602 ] Imran Rashid commented on SPARK-3622: - In some ways this kinda reminds of the problem

[jira] [Updated] (SPARK-733) Add documentation on use of accumulators in lazy transformation

[ https://issues.apache.org/jira/browse/SPARK-733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-733: --- Fix Version/s: 1.2.1 1.3.0 Add documentation on use of accumulators in lazy

[jira] [Closed] (SPARK-733) Add documentation on use of accumulators in lazy transformation

[ https://issues.apache.org/jira/browse/SPARK-733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid closed SPARK-733. -- Resolution: Fixed Fixed by https://github.com/apache/spark/pull/4022 Add documentation on use of

[jira] [Commented] (SPARK-3622) Provide a custom transformation that can output multiple RDDs

[ https://issues.apache.org/jira/browse/SPARK-3622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14287883#comment-14287883 ] Imran Rashid commented on SPARK-3622: - Is there any chance the immediate use case is

[jira] [Created] (SPARK-5786) Documentation of Narrow Dependencies

Imran Rashid created SPARK-5786: --- Summary: Documentation of Narrow Dependencies Key: SPARK-5786 URL: https://issues.apache.org/jira/browse/SPARK-5786 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-5785) Pyspark does not support narrow dependencies

Imran Rashid created SPARK-5785: --- Summary: Pyspark does not support narrow dependencies Key: SPARK-5785 URL: https://issues.apache.org/jira/browse/SPARK-5785 Project: Spark Issue Type:

[jira] [Commented] (SPARK-1476) 2GB limit in spark for blocks

[ https://issues.apache.org/jira/browse/SPARK-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14327394#comment-14327394 ] Imran Rashid commented on SPARK-1476: - Based on discussion on the dev list,

[jira] [Commented] (SPARK-3454) Expose JSON representation of data shown in WebUI

[ https://issues.apache.org/jira/browse/SPARK-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14294311#comment-14294311 ] Imran Rashid commented on SPARK-3454: - There are a few ways we could move forward on

[jira] [Resolved] (SPARK-5467) DStreams should provide windowing based on timestamps from the data (as opposed to wall clock time)

[ https://issues.apache.org/jira/browse/SPARK-5467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-5467. - Resolution: Duplicate DStreams should provide windowing based on timestamps from the data (as

[jira] [Commented] (SPARK-5467) DStreams should provide windowing based on timestamps from the data (as opposed to wall clock time)

[ https://issues.apache.org/jira/browse/SPARK-5467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14296047#comment-14296047 ] Imran Rashid commented on SPARK-5467: - shoot, sorry I missed that jira. (I swear I

[jira] [Created] (SPARK-5467) DStreams should provide windowing based on timestamps from the data (as opposed to wall clock time)

Imran Rashid created SPARK-5467: --- Summary: DStreams should provide windowing based on timestamps from the data (as opposed to wall clock time) Key: SPARK-5467 URL: https://issues.apache.org/jira/browse/SPARK-5467

[jira] [Commented] (SPARK-3298) [SQL] registerAsTable / registerTempTable overwrites old tables

[ https://issues.apache.org/jira/browse/SPARK-3298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14290962#comment-14290962 ] Imran Rashid commented on SPARK-3298: - If {{allowOverwrite}} defaulted to {{true}},

[jira] [Commented] (SPARK-2688) Need a way to run multiple data pipeline concurrently

[ https://issues.apache.org/jira/browse/SPARK-2688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14291214#comment-14291214 ] Imran Rashid commented on SPARK-2688: - [~airhorns] I completely agree with your use

[jira] [Commented] (SPARK-5236) java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.MutableAny cannot be cast to org.apache.spark.sql.catalyst.expressions.MutableInt

[ https://issues.apache.org/jira/browse/SPARK-5236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14295524#comment-14295524 ] Imran Rashid commented on SPARK-5236: - [~alexbaretta] I dont' think there is any doubt

[jira] [Updated] (SPARK-5785) Pyspark does not support narrow dependencies

[ https://issues.apache.org/jira/browse/SPARK-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-5785: Description: joins ( cogroups etc.) are always considered to have wide dependencies in pyspark,

[jira] [Commented] (SPARK-1476) 2GB limit in spark for blocks

[ https://issues.apache.org/jira/browse/SPARK-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329554#comment-14329554 ] Imran Rashid commented on SPARK-1476: - I spent a little time with [~sandyr] on this

[jira] [Commented] (SPARK-5928) Remote Shuffle Blocks cannot be more than 2 GB

[ https://issues.apache.org/jira/browse/SPARK-5928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329568#comment-14329568 ] Imran Rashid commented on SPARK-5928: - (just edited the description -- I mistakenly

[jira] [Commented] (SPARK-5928) Remote Shuffle Blocks cannot be more than 2 GB

[ https://issues.apache.org/jira/browse/SPARK-5928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329720#comment-14329720 ] Imran Rashid commented on SPARK-5928: - Actually, there *is* some weirdness in how

[jira] [Created] (SPARK-5928) Remote Shuffle Blocks cannot be more than 2 GB

Imran Rashid created SPARK-5928: --- Summary: Remote Shuffle Blocks cannot be more than 2 GB Key: SPARK-5928 URL: https://issues.apache.org/jira/browse/SPARK-5928 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-5928) Remote Shuffle Blocks cannot be more than 2 GB

[ https://issues.apache.org/jira/browse/SPARK-5928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-5928: Description: If a shuffle block is over 2GB, the shuffle fails, with an uninformative exception.

[jira] [Commented] (SPARK-1391) BlockManager cannot transfer blocks larger than 2G in size

[ https://issues.apache.org/jira/browse/SPARK-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329605#comment-14329605 ] Imran Rashid commented on SPARK-1391: - [~coderplay], I assume you are no longer

[jira] [Commented] (SPARK-5928) Remote Shuffle Blocks cannot be more than 2 GB

[ https://issues.apache.org/jira/browse/SPARK-5928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329584#comment-14329584 ] Imran Rashid commented on SPARK-5928: - Here are some thoughts on we *might* fix this.

[jira] [Commented] (SPARK-5928) Remote Shuffle Blocks cannot be more than 2 GB

[ https://issues.apache.org/jira/browse/SPARK-5928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329623#comment-14329623 ] Imran Rashid commented on SPARK-5928: - sometimes this also results in exceptions like

[jira] [Commented] (SPARK-1391) BlockManager cannot transfer blocks larger than 2G in size

[ https://issues.apache.org/jira/browse/SPARK-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329613#comment-14329613 ] Imran Rashid commented on SPARK-1391: - Here is a minimal program to demonstrate the

[jira] [Commented] (SPARK-4746) integration tests should be separated from faster unit tests

[ https://issues.apache.org/jira/browse/SPARK-4746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14363351#comment-14363351 ] Imran Rashid commented on SPARK-4746: - gonna wait a few days to see if anyone is

[jira] [Closed] (SPARK-4746) integration tests should be separated from faster unit tests

[ https://issues.apache.org/jira/browse/SPARK-4746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid closed SPARK-4746. --- Resolution: Fixed integration tests should be separated from faster unit tests

[jira] [Created] (SPARK-6365) jetty-security needed for SPARK_PREPEND_CLASSES to work

Imran Rashid created SPARK-6365: --- Summary: jetty-security needed for SPARK_PREPEND_CLASSES to work Key: SPARK-6365 URL: https://issues.apache.org/jira/browse/SPARK-6365 Project: Spark Issue

[jira] [Resolved] (SPARK-6365) jetty-security needed for SPARK_PREPEND_CLASSES to work

[ https://issues.apache.org/jira/browse/SPARK-6365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-6365. - Resolution: Fixed Fix Version/s: 1.3.1 Issue resolved by pull request 5071

[jira] [Commented] (SPARK-6190) create LargeByteBuffer abstraction for eliminating 2GB limit on blocks

[ https://issues.apache.org/jira/browse/SPARK-6190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353687#comment-14353687 ] Imran Rashid commented on SPARK-6190: - Another observation as I've dug into the

[jira] [Commented] (SPARK-6190) create LargeByteBuffer abstraction for eliminating 2GB limit on blocks

[ https://issues.apache.org/jira/browse/SPARK-6190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14357609#comment-14357609 ] Imran Rashid commented on SPARK-6190: - Hi [~rxin], I've been adding scatterered notes

[jira] [Commented] (SPARK-6190) create LargeByteBuffer abstraction for eliminating 2GB limit on blocks

[ https://issues.apache.org/jira/browse/SPARK-6190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14350596#comment-14350596 ] Imran Rashid commented on SPARK-6190: - actually, there isn't any need to change the

[jira] [Updated] (SPARK-1391) BlockManager cannot transfer blocks larger than 2G in size

[ https://issues.apache.org/jira/browse/SPARK-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-1391: Attachment: BlockLimitDesign.pdf design doc BlockManager cannot transfer blocks larger than 2G in

[jira] [Comment Edited] (SPARK-1391) BlockManager cannot transfer blocks larger than 2G in size

[ https://issues.apache.org/jira/browse/SPARK-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338648#comment-14338648 ] Imran Rashid edited comment on SPARK-1391 at 3/2/15 10:10 PM: --

[jira] [Commented] (SPARK-1391) BlockManager cannot transfer blocks larger than 2G in size

[ https://issues.apache.org/jira/browse/SPARK-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14344286#comment-14344286 ] Imran Rashid commented on SPARK-1391: - [~rxin] Sure thing, I can break it into

[jira] [Commented] (SPARK-603) add simple Counter API

[ https://issues.apache.org/jira/browse/SPARK-603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14345204#comment-14345204 ] Imran Rashid commented on SPARK-603: Hi [~srowen] I don't think anyone is actively

[jira] [Closed] (SPARK-603) add simple Counter API

[ https://issues.apache.org/jira/browse/SPARK-603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid closed SPARK-603. -- Resolution: Won't Fix add simple Counter API -- Key: SPARK-603

[jira] [Assigned] (SPARK-5949) Driver program has to register roaring bitmap classes used by spark with Kryo when number of partitions is greater than 2000

[ https://issues.apache.org/jira/browse/SPARK-5949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-5949: --- Assignee: Imran Rashid Driver program has to register roaring bitmap classes used by spark

[jira] [Commented] (SPARK-5949) Driver program has to register roaring bitmap classes used by spark with Kryo when number of partitions is greater than 2000

[ https://issues.apache.org/jira/browse/SPARK-5949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14345696#comment-14345696 ] Imran Rashid commented on SPARK-5949: - Thanks [~ptorok]. I've updated the PR. Can

[jira] [Created] (SPARK-6190) create LargeByteBuffer abstraction for eliminating 2GB limit on blocks

Imran Rashid created SPARK-6190: --- Summary: create LargeByteBuffer abstraction for eliminating 2GB limit on blocks Key: SPARK-6190 URL: https://issues.apache.org/jira/browse/SPARK-6190 Project: Spark

[jira] [Updated] (SPARK-6190) create LargeByteBuffer abstraction for eliminating 2GB limit on blocks

[ https://issues.apache.org/jira/browse/SPARK-6190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-6190: Attachment: LargeByteBuffer.pdf design doc create LargeByteBuffer abstraction for eliminating 2GB

[jira] [Commented] (SPARK-6190) create LargeByteBuffer abstraction for eliminating 2GB limit on blocks

[ https://issues.apache.org/jira/browse/SPARK-6190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14349191#comment-14349191 ] Imran Rashid commented on SPARK-6190: - Hi [~rxin], I've attached a design doc here. I

[jira] [Commented] (SPARK-5949) Driver program has to register roaring bitmap classes used by spark with Kryo when number of partitions is greater than 2000

[ https://issues.apache.org/jira/browse/SPARK-5949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14346825#comment-14346825 ] Imran Rashid commented on SPARK-5949: - Thanks Peter. {{Array\[Long\]}} is there,

[jira] [Commented] (SPARK-5928) Remote Shuffle Blocks cannot be more than 2 GB

[ https://issues.apache.org/jira/browse/SPARK-5928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14332490#comment-14332490 ] Imran Rashid commented on SPARK-5928: - so I'm pretty sure the issue is not just large

[jira] [Commented] (SPARK-5928) Remote Shuffle Blocks cannot be more than 2 GB

[ https://issues.apache.org/jira/browse/SPARK-5928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14332480#comment-14332480 ] Imran Rashid commented on SPARK-5928: - Thanks Kay, that does make sense. I think that

[jira] [Updated] (SPARK-5945) Spark should not retry a stage infinitely on a FetchFailedException

[ https://issues.apache.org/jira/browse/SPARK-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-5945: Description: While investigating SPARK-5928, I noticed some very strange behavior in the way spark

[jira] [Updated] (SPARK-5945) Spark should not retry a stage infinitely on a FetchFailedException

[ https://issues.apache.org/jira/browse/SPARK-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-5945: Description: While investigating SPARK-5928, I noticed some very strange behavior in the way spark

[jira] [Created] (SPARK-5945) Spark should not retry a stage infinitely on a FetchFailedException

Imran Rashid created SPARK-5945: --- Summary: Spark should not retry a stage infinitely on a FetchFailedException Key: SPARK-5945 URL: https://issues.apache.org/jira/browse/SPARK-5945 Project: Spark

[jira] [Commented] (SPARK-1391) BlockManager cannot transfer blocks larger than 2G in size

[ https://issues.apache.org/jira/browse/SPARK-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338648#comment-14338648 ] Imran Rashid commented on SPARK-1391: - The one complication here comes from the

[jira] [Closed] (SPARK-4746) integration tests should be separated from faster unit tests

[ https://issues.apache.org/jira/browse/SPARK-4746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid closed SPARK-4746. --- Resolution: Won't Fix looks like there isn't interest in this, closing to clean up jira integration

[jira] [Commented] (SPARK-5928) Remote Shuffle Blocks cannot be more than 2 GB

[ https://issues.apache.org/jira/browse/SPARK-5928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14376430#comment-14376430 ] Imran Rashid commented on SPARK-5928: - sorry to hear that [~douglaz]. To help

[jira] [Commented] (SPARK-5945) Spark should not retry a stage infinitely on a FetchFailedException

[ https://issues.apache.org/jira/browse/SPARK-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14376454#comment-14376454 ] Imran Rashid commented on SPARK-5945: - Hi [~ilganeli], sorry for taking a while to

[jira] [Updated] (SPARK-5945) Spark should not retry a stage infinitely on a FetchFailedException

[ https://issues.apache.org/jira/browse/SPARK-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-5945: Assignee: Ilya Ganelin Spark should not retry a stage infinitely on a FetchFailedException

[jira] [Comment Edited] (SPARK-6980) Akka timeout exceptions indicate which conf controls them

[ https://issues.apache.org/jira/browse/SPARK-6980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14512148#comment-14512148 ] Imran Rashid edited comment on SPARK-6980 at 4/25/15 1:20 AM: --

[jira] [Commented] (SPARK-6980) Akka timeout exceptions indicate which conf controls them

[ https://issues.apache.org/jira/browse/SPARK-6980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14512148#comment-14512148 ] Imran Rashid commented on SPARK-6980: - Hi [~bryanc] [~harshg], sorry I didn't notice

[jira] [Commented] (SPARK-5945) Spark should not retry a stage infinitely on a FetchFailedException

[ https://issues.apache.org/jira/browse/SPARK-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14508041#comment-14508041 ] Imran Rashid commented on SPARK-5945: - good point about moving the design discussion

[jira] [Commented] (SPARK-5945) Spark should not retry a stage infinitely on a FetchFailedException

[ https://issues.apache.org/jira/browse/SPARK-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14508455#comment-14508455 ] Imran Rashid commented on SPARK-5945: - I think we want to do something more than just

[jira] [Commented] (SPARK-5945) Spark should not retry a stage infinitely on a FetchFailedException

[ https://issues.apache.org/jira/browse/SPARK-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14518807#comment-14518807 ] Imran Rashid commented on SPARK-5945: - [~kayousterhout] can you please clarify -- did

[jira] [Commented] (SPARK-7075) Project Tungsten: Improving Physical Execution and Memory Management

[ https://issues.apache.org/jira/browse/SPARK-7075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14519956#comment-14519956 ] Imran Rashid commented on SPARK-7075: - everything sounds awesome, but can we see

[jira] [Commented] (SPARK-7075) Project Tungsten: Improving Physical Execution and Memory Management

[ https://issues.apache.org/jira/browse/SPARK-7075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14522004#comment-14522004 ] Imran Rashid commented on SPARK-7075: - Thanks Reynold. But to be clear, there was

[jira] [Resolved] (SPARK-6999) infinite recursion with createDataFrame(JavaRDD[Row], java.util.List[String])

[ https://issues.apache.org/jira/browse/SPARK-6999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-6999. - Resolution: Fixed Fix Version/s: 1.4.0 Issue resolved by pull request 5804

[jira] [Commented] (SPARK-7283) Ignore InterruptedException in SparkUncaughtExceptionHandler

[ https://issues.apache.org/jira/browse/SPARK-7283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14522281#comment-14522281 ] Imran Rashid commented on SPARK-7283: - I need a little more information -- what goes

[jira] [Updated] (SPARK-7308) Should there be multiple concurrent attempts for one stage?

[ https://issues.apache.org/jira/browse/SPARK-7308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-7308: Description: Currently, when there is a fetch failure, you can end up with multiple concurrent

[jira] [Commented] (SPARK-7308) Should there be multiple concurrent attempts for one stage?

[ https://issues.apache.org/jira/browse/SPARK-7308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523828#comment-14523828 ] Imran Rashid commented on SPARK-7308: - In our discussion on

[jira] [Created] (SPARK-7308) Should there be multiple concurrent attempts for one stage?

Imran Rashid created SPARK-7308: --- Summary: Should there be multiple concurrent attempts for one stage? Key: SPARK-7308 URL: https://issues.apache.org/jira/browse/SPARK-7308 Project: Spark

[jira] [Commented] (SPARK-7308) Should there be multiple concurrent attempts for one stage?

[ https://issues.apache.org/jira/browse/SPARK-7308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523761#comment-14523761 ] Imran Rashid commented on SPARK-7308: - ping [~kayousterhout] Should there be

[jira] [Commented] (SPARK-7308) Should there be multiple concurrent attempts for one stage?

[ https://issues.apache.org/jira/browse/SPARK-7308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523819#comment-14523819 ] Imran Rashid commented on SPARK-7308: - I've got some code to reproduce the error in

[jira] [Resolved] (SPARK-3454) Expose JSON representation of data shown in WebUI

[ https://issues.apache.org/jira/browse/SPARK-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-3454. - Resolution: Fixed Fix Version/s: 1.4.0 Issue resolved by pull request 4435

[jira] [Commented] (SPARK-7308) Should there be multiple concurrent attempts for one stage?

[ https://issues.apache.org/jira/browse/SPARK-7308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14529618#comment-14529618 ] Imran Rashid commented on SPARK-7308: - I forgot to take {{TaskSetManager.isZombie}}

[jira] [Reopened] (SPARK-3454) Expose JSON representation of data shown in WebUI

[ https://issues.apache.org/jira/browse/SPARK-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reopened SPARK-3454: - need to fix some issues w/ test files in pr ... Expose JSON representation of data shown in WebUI

[jira] [Commented] (SPARK-6980) Akka timeout exceptions indicate which conf controls them

[ https://issues.apache.org/jira/browse/SPARK-6980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14532659#comment-14532659 ] Imran Rashid commented on SPARK-6980: - Hi [~zsxwing][~rxin] -- I know you are working

[jira] [Updated] (SPARK-6980) Akka timeout exceptions indicate which conf controls them

[ https://issues.apache.org/jira/browse/SPARK-6980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-6980: Description: If you hit one of the akka timeouts, you just get an exception like {code}

[jira] [Commented] (SPARK-6889) Streamline contribution process with update to Contribution wiki, JIRA rules

[ https://issues.apache.org/jira/browse/SPARK-6889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14496251#comment-14496251 ] Imran Rashid commented on SPARK-6889: - +1 on the proposed changes. I think that

[jira] [Commented] (SPARK-6980) Akka timeout exceptions indicate which conf controls them

[ https://issues.apache.org/jira/browse/SPARK-6980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14499789#comment-14499789 ] Imran Rashid commented on SPARK-6980: - I don't think I'll get to this for a little

[jira] [Created] (SPARK-6980) Akka timeout exceptions indicate which conf controls them

Imran Rashid created SPARK-6980: --- Summary: Akka timeout exceptions indicate which conf controls them Key: SPARK-6980 URL: https://issues.apache.org/jira/browse/SPARK-6980 Project: Spark Issue

[jira] [Updated] (SPARK-6980) Akka timeout exceptions indicate which conf controls them

[ https://issues.apache.org/jira/browse/SPARK-6980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-6980: Labels: starter (was: ) Akka timeout exceptions indicate which conf controls them

[jira] [Commented] (SPARK-6980) Akka timeout exceptions indicate which conf controls them

[ https://issues.apache.org/jira/browse/SPARK-6980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500572#comment-14500572 ] Imran Rashid commented on SPARK-6980: - awesome, thanks [~bryanc]! Feel free to ping

[jira] [Assigned] (SPARK-6980) Akka timeout exceptions indicate which conf controls them

[ https://issues.apache.org/jira/browse/SPARK-6980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-6980: --- Assignee: Imran Rashid Akka timeout exceptions indicate which conf controls them

[jira] [Updated] (SPARK-6190) create LargeByteBuffer abstraction for eliminating 2GB limit on blocks

[ https://issues.apache.org/jira/browse/SPARK-6190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-6190: Attachment: LargeByteBuffer_v3.pdf create LargeByteBuffer abstraction for eliminating 2GB limit on

[jira] [Updated] (SPARK-6190) create LargeByteBuffer abstraction for eliminating 2GB limit on blocks

[ https://issues.apache.org/jira/browse/SPARK-6190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-6190: Attachment: (was: LargeByteBuffer_v2.pdf) create LargeByteBuffer abstraction for eliminating

  1   2   3   4   5   6   7   8   9   10   >