[jira] [Commented] (SPARK-4952) In some cases, spark on yarn failed to start

[ https://issues.apache.org/jira/browse/SPARK-4952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258089#comment-14258089 ] Apache Spark commented on SPARK-4952: - User 'witgo' has created a pull request for

[jira] [Created] (SPARK-4954) Add spark version information in log for standalone mode

Zhang, Liye created SPARK-4954: -- Summary: Add spark version information in log for standalone mode Key: SPARK-4954 URL: https://issues.apache.org/jira/browse/SPARK-4954 Project: Spark Issue

[jira] [Commented] (SPARK-4954) Add spark version information in log for standalone mode

[ https://issues.apache.org/jira/browse/SPARK-4954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258122#comment-14258122 ] Apache Spark commented on SPARK-4954: - User 'liyezhang556520' has created a pull

[jira] [Created] (SPARK-4955) Executor does not get killed after configured interval.

Chengxiang Li created SPARK-4955: Summary: Executor does not get killed after configured interval. Key: SPARK-4955 URL: https://issues.apache.org/jira/browse/SPARK-4955 Project: Spark Issue

[jira] [Commented] (SPARK-4955) Executor does not get killed after configured interval.

[ https://issues.apache.org/jira/browse/SPARK-4955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258138#comment-14258138 ] Chengxiang Li commented on SPARK-4955: -- I verified this feature with Hive on Spark,

[jira] [Commented] (SPARK-4955) Executor does not get killed after configured interval.

[ https://issues.apache.org/jira/browse/SPARK-4955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258139#comment-14258139 ] Chengxiang Li commented on SPARK-4955: -- cc:[~andrewor14] Executor does not get

[jira] [Created] (SPARK-4956) Vector Initialization error when initialize a Sparse Vector by calling Vectors.sparse(size, indices, values)

liaoyuxi created SPARK-4956: --- Summary: Vector Initialization error when initialize a Sparse Vector by calling Vectors.sparse(size, indices, values) Key: SPARK-4956 URL: https://issues.apache.org/jira/browse/SPARK-4956

[jira] [Commented] (SPARK-4956) Vector Initialization error when initialize a Sparse Vector by calling Vectors.sparse(size, indices, values)

[ https://issues.apache.org/jira/browse/SPARK-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258225#comment-14258225 ] Sean Owen commented on SPARK-4956: -- Yes, I see the same behavior. Breeze doesn't check

[jira] [Created] (SPARK-4957) TaskScheduler: when no resources are available: backoff after # of tries and crash.

Nathan Bijnens created SPARK-4957: - Summary: TaskScheduler: when no resources are available: backoff after # of tries and crash. Key: SPARK-4957 URL: https://issues.apache.org/jira/browse/SPARK-4957

[jira] [Created] (SPARK-4958) Bake common tools like ganglia into Spark AMI

Nicholas Chammas created SPARK-4958: --- Summary: Bake common tools like ganglia into Spark AMI Key: SPARK-4958 URL: https://issues.apache.org/jira/browse/SPARK-4958 Project: Spark Issue

[jira] [Updated] (SPARK-4947) Use EC2 status checks to know when to test SSH availability

[ https://issues.apache.org/jira/browse/SPARK-4947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-4947: -- Assignee: Nicholas Chammas Use EC2 status checks to know when to test SSH availability

[jira] [Updated] (SPARK-4939) Python updateStateByKey example hang in local mode

[ https://issues.apache.org/jira/browse/SPARK-4939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-4939: - Assignee: Davies Liu Python updateStateByKey example hang in local mode

[jira] [Commented] (SPARK-1312) Batch should read based on the batch interval provided in the StreamingContext

[ https://issues.apache.org/jira/browse/SPARK-1312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258536#comment-14258536 ] Tathagata Das commented on SPARK-1312: -- This has probably been solved in Spark 1.2.0

[jira] [Commented] (SPARK-1312) Batch should read based on the batch interval provided in the StreamingContext

[ https://issues.apache.org/jira/browse/SPARK-1312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258537#comment-14258537 ] Tathagata Das commented on SPARK-1312: -- I will try to add a unit test to make sure

[jira] [Resolved] (SPARK-4297) Build warning fixes omnibus

[ https://issues.apache.org/jira/browse/SPARK-4297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-4297. --- Resolution: Fixed Fix Version/s: 1.3.0 Issue resolved by pull request 3157

[jira] [Updated] (SPARK-4297) Build warning fixes omnibus

[ https://issues.apache.org/jira/browse/SPARK-4297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-4297: -- Assignee: Sean Owen Build warning fixes omnibus --- Key:

[jira] [Commented] (SPARK-4133) PARSING_ERROR(2) when upgrading issues from 1.0.2 to 1.1.0

[ https://issues.apache.org/jira/browse/SPARK-4133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258545#comment-14258545 ] Tathagata Das commented on SPARK-4133: -- [~derrickburns] Are you sure you are not

[jira] [Commented] (SPARK-4631) Add real unit test for MQTT

[ https://issues.apache.org/jira/browse/SPARK-4631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258546#comment-14258546 ] Tathagata Das commented on SPARK-4631: -- [~prabeeshk] Any update on the original issue

[jira] [Commented] (SPARK-4835) Streaming saveAs*HadoopFiles() methods may throw FileAlreadyExistsException during checkpoint recovery

[ https://issues.apache.org/jira/browse/SPARK-4835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258547#comment-14258547 ] Tathagata Das commented on SPARK-4835: -- That is a very good point. Lets brainstorm on

[jira] [Updated] (SPARK-1854) Add a version of StreamingContext.fileStream that take hadoop conf object

[ https://issues.apache.org/jira/browse/SPARK-1854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-1854: - Priority: Major (was: Critical) Add a version of StreamingContext.fileStream that take hadoop

[jira] [Commented] (SPARK-1854) Add a version of StreamingContext.fileStream that take hadoop conf object

[ https://issues.apache.org/jira/browse/SPARK-1854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258552#comment-14258552 ] Tathagata Das commented on SPARK-1854: -- What I meant was to use the `hadoopConf`

[jira] [Closed] (SPARK-1854) Add a version of StreamingContext.fileStream that take hadoop conf object

[ https://issues.apache.org/jira/browse/SPARK-1854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das closed SPARK-1854. Resolution: Won't Fix Add a version of StreamingContext.fileStream that take hadoop conf object

[jira] [Commented] (SPARK-2892) Socket Receiver does not stop when streaming context is stopped

[ https://issues.apache.org/jira/browse/SPARK-2892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258553#comment-14258553 ] Tathagata Das commented on SPARK-2892: -- [~ilayaperumalg] Now that SPARK-4802 has been

[jira] [Commented] (SPARK-4940) Document or Support more evenly distributing cores for Mesos mode

[ https://issues.apache.org/jira/browse/SPARK-4940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258570#comment-14258570 ] Timothy Chen commented on SPARK-4940: - Potentially I think there are two ways to help

[jira] [Commented] (SPARK-4940) Document or Support more evenly distributing cores for Mesos mode

[ https://issues.apache.org/jira/browse/SPARK-4940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258571#comment-14258571 ] Timothy Chen commented on SPARK-4940: - [~gmaas] Document or Support more evenly

[jira] [Created] (SPARK-4959) Attributes are case sensitive when using a select query from a projection

Andy Konwinski created SPARK-4959: - Summary: Attributes are case sensitive when using a select query from a projection Key: SPARK-4959 URL: https://issues.apache.org/jira/browse/SPARK-4959 Project:

[jira] [Updated] (SPARK-4959) Attributes are case sensitive when using a select query from a projection

[ https://issues.apache.org/jira/browse/SPARK-4959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andy Konwinski updated SPARK-4959: -- Description: Per [~marmbrus], see this line of code, where we should be using an attribute map

[jira] [Updated] (SPARK-4959) Attributes are case sensitive when using a select query from a projection

[ https://issues.apache.org/jira/browse/SPARK-4959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andy Konwinski updated SPARK-4959: -- Description: Per [~marmbrus], see this line of code, where we should be using an attribute map

[jira] [Commented] (SPARK-4924) Factor out code to launch Spark applications into a separate library

[ https://issues.apache.org/jira/browse/SPARK-4924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258578#comment-14258578 ] Marcelo Vanzin commented on SPARK-4924: --- I've been playing with some code to achieve

[jira] [Commented] (SPARK-3146) Improve the flexibility of Spark Streaming Kafka API to offer user the ability to process message before storing into BM

[ https://issues.apache.org/jira/browse/SPARK-3146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258593#comment-14258593 ] Tathagata Das commented on SPARK-3146: -- I am a little wary of adding any

[jira] [Created] (SPARK-4960) Interceptor pattern in receivers

Tathagata Das created SPARK-4960: Summary: Interceptor pattern in receivers Key: SPARK-4960 URL: https://issues.apache.org/jira/browse/SPARK-4960 Project: Spark Issue Type: New Feature

[jira] [Commented] (SPARK-4960) Interceptor pattern in receivers

[ https://issues.apache.org/jira/browse/SPARK-4960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258595#comment-14258595 ] Tathagata Das commented on SPARK-4960: -- SPARK-3146 wants to add similar functionality

[jira] [Commented] (SPARK-4960) Interceptor pattern in receivers

[ https://issues.apache.org/jira/browse/SPARK-4960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258596#comment-14258596 ] Tathagata Das commented on SPARK-4960: -- If any one is interested is interested in

[jira] [Comment Edited] (SPARK-2388) Streaming from multiple different Kafka topics is problematic

[ https://issues.apache.org/jira/browse/SPARK-2388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258597#comment-14258597 ] Tathagata Das edited comment on SPARK-2388 at 12/25/14 12:26 AM:

[jira] [Commented] (SPARK-2388) Streaming from multiple different Kafka topics is problematic

[ https://issues.apache.org/jira/browse/SPARK-2388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258597#comment-14258597 ] Tathagata Das commented on SPARK-2388: -- You can always create multiple kafka streams

[jira] [Closed] (SPARK-2388) Streaming from multiple different Kafka topics is problematic

[ https://issues.apache.org/jira/browse/SPARK-2388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das closed SPARK-2388. Resolution: Invalid Streaming from multiple different Kafka topics is problematic

[jira] [Commented] (SPARK-3146) Improve the flexibility of Spark Streaming Kafka API to offer user the ability to process message before storing into BM

[ https://issues.apache.org/jira/browse/SPARK-3146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258598#comment-14258598 ] Tathagata Das commented on SPARK-3146: -- [~c...@koeninger.org] [~jerryshao] If either

[jira] [Updated] (SPARK-4893) Clean up uses of System.setProperty in unit tests

[ https://issues.apache.org/jira/browse/SPARK-4893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-4893: -- Summary: Clean up uses of System.setProperty in unit tests (was: Use test fixture to reset system

[jira] [Updated] (SPARK-4893) Clean up uses of System.setProperty in unit tests

[ https://issues.apache.org/jira/browse/SPARK-4893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-4893: -- Description: Several of our tests call {{System.setProperty}} (or test code which implicitly sets

[jira] [Comment Edited] (SPARK-3146) Improve the flexibility of Spark Streaming Kafka API to offer user the ability to process message before storing into BM

[ https://issues.apache.org/jira/browse/SPARK-3146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258598#comment-14258598 ] Tathagata Das edited comment on SPARK-3146 at 12/25/14 12:32 AM:

[jira] [Updated] (SPARK-4873) Use `Future.zip` instead of `Future.flatMap`(for-loop) in WriteAheadLogBasedBlockHandler

[ https://issues.apache.org/jira/browse/SPARK-4873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-4873: Description: Use Future.zip instead of Future.flatMap(for-loop). zip implies these two Futures will

[jira] [Commented] (SPARK-4960) Interceptor pattern in receivers

[ https://issues.apache.org/jira/browse/SPARK-4960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258618#comment-14258618 ] Saisai Shao commented on SPARK-4960: I'd like to take a crack and provide a general

[jira] [Created] (SPARK-4961) Put HadoopRDD.getPartitions forward to reduce DAGScheduler.JobSubmitted processing time

YanTang Zhai created SPARK-4961: --- Summary: Put HadoopRDD.getPartitions forward to reduce DAGScheduler.JobSubmitted processing time Key: SPARK-4961 URL: https://issues.apache.org/jira/browse/SPARK-4961

[jira] [Created] (SPARK-4962) Put TaskScheduler.start back in SparkContext to shorten cluster resources occupation period

YanTang Zhai created SPARK-4962: --- Summary: Put TaskScheduler.start back in SparkContext to shorten cluster resources occupation period Key: SPARK-4962 URL: https://issues.apache.org/jira/browse/SPARK-4962

[jira] [Commented] (SPARK-2387) Remove the stage barrier for better resource utilization

[ https://issues.apache.org/jira/browse/SPARK-2387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258626#comment-14258626 ] Xuefu Zhang commented on SPARK-2387: cc: [~sandyr] I think the purpose of this

[jira] [Commented] (SPARK-4961) Put HadoopRDD.getPartitions forward to reduce DAGScheduler.JobSubmitted processing time

[ https://issues.apache.org/jira/browse/SPARK-4961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258631#comment-14258631 ] Apache Spark commented on SPARK-4961: - User 'YanTangZhai' has created a pull request

[jira] [Resolved] (SPARK-4873) Use `Future.zip` instead of `Future.flatMap`(for-loop) in WriteAheadLogBasedBlockHandler

[ https://issues.apache.org/jira/browse/SPARK-4873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-4873. -- Resolution: Fixed Fix Version/s: 1.2.1 1.3.0 Use `Future.zip`

[jira] [Commented] (SPARK-4960) Interceptor pattern in receivers

[ https://issues.apache.org/jira/browse/SPARK-4960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258639#comment-14258639 ] Hari Shreedharan commented on SPARK-4960: - Awesome! One idea I had is to call it

[jira] [Commented] (SPARK-3847) Enum.hashCode is only consistent within the same JVM

[ https://issues.apache.org/jira/browse/SPARK-3847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258649#comment-14258649 ] Apache Spark commented on SPARK-3847: - User 'JoshRosen' has created a pull request for

[jira] [Commented] (SPARK-4956) Vector Initialization error when initialize a Sparse Vector by calling Vectors.sparse(size, indices, values)

[ https://issues.apache.org/jira/browse/SPARK-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258656#comment-14258656 ] liaoyuxi commented on SPARK-4956: - I've start a pull request by ordering the indices

[jira] [Commented] (SPARK-4959) Attributes are case sensitive when using a select query from a projection

[ https://issues.apache.org/jira/browse/SPARK-4959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258657#comment-14258657 ] Apache Spark commented on SPARK-4959: - User 'chenghao-intel' has created a pull

[jira] [Created] (SPARK-4963) SchemaRDD.sample may return wrong results

Cheng Lian created SPARK-4963: - Summary: SchemaRDD.sample may return wrong results Key: SPARK-4963 URL: https://issues.apache.org/jira/browse/SPARK-4963 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-4923) Maven build should keep publishing spark-repl

[ https://issues.apache.org/jira/browse/SPARK-4923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258666#comment-14258666 ] Saisai Shao commented on SPARK-4923: Hey [~pwendell], for some projects like

[jira] [Created] (SPARK-4964) Exactly-once semantics for Kafka

Cody Koeninger created SPARK-4964: - Summary: Exactly-once semantics for Kafka Key: SPARK-4964 URL: https://issues.apache.org/jira/browse/SPARK-4964 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-4964) Exactly-once semantics for Kafka

[ https://issues.apache.org/jira/browse/SPARK-4964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258668#comment-14258668 ] Apache Spark commented on SPARK-4964: - User 'koeninger' has created a pull request for

[jira] [Commented] (SPARK-4964) Exactly-once semantics for Kafka

[ https://issues.apache.org/jira/browse/SPARK-4964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258670#comment-14258670 ] Cody Koeninger commented on SPARK-4964: --- Usage example of the dstream for the

[jira] [Created] (SPARK-4965) The MemoryOverhead value is not correct

meiyoula created SPARK-4965: --- Summary: The MemoryOverhead value is not correct Key: SPARK-4965 URL: https://issues.apache.org/jira/browse/SPARK-4965 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-4966) The MemoryOverhead value is not correct

meiyoula created SPARK-4966: --- Summary: The MemoryOverhead value is not correct Key: SPARK-4966 URL: https://issues.apache.org/jira/browse/SPARK-4966 Project: Spark Issue Type: Bug