[jira] [Commented] (SPARK-3333) Large number of partitions causes OOM

[ https://issues.apache.org/jira/browse/SPARK-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117062#comment-14117062 ] Josh Rosen commented on SPARK-: --- [~shivaram] and I discussed this; we have a few

[jira] [Updated] (SPARK-3330) Successive test runs with different profiles fail SparkSubmitSuite

[ https://issues.apache.org/jira/browse/SPARK-3330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-3330: --- Assignee: Sean Owen Successive test runs with different profiles fail SparkSubmitSuite

[jira] [Updated] (SPARK-3330) Successive test runs with different profiles fail SparkSubmitSuite

[ https://issues.apache.org/jira/browse/SPARK-3330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-3330: --- Assignee: (was: Sean Owen) Successive test runs with different profiles fail

[jira] [Updated] (SPARK-3328) ./make-distribution.sh --with-tachyon build is broken

[ https://issues.apache.org/jira/browse/SPARK-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-3328: --- Summary: ./make-distribution.sh --with-tachyon build is broken (was: --with-tachyon build is

[jira] [Commented] (SPARK-3322) Log a ConnectionManager error when the application ends

[ https://issues.apache.org/jira/browse/SPARK-3322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117117#comment-14117117 ] Prashant Sharma commented on SPARK-3322: I remember seeing this discussion

[jira] [Commented] (SPARK-3306) Addition of external resource dependency in executors

[ https://issues.apache.org/jira/browse/SPARK-3306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117138#comment-14117138 ] Prashant Sharma commented on SPARK-3306: I am not 100% sure, why you are referring

[jira] [Updated] (SPARK-3306) Addition of external resource dependency in executors

[ https://issues.apache.org/jira/browse/SPARK-3306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-3306: --- Target Version/s: (was: 1.3.0) Addition of external resource dependency in executors

[jira] [Commented] (SPARK-3292) Shuffle Tasks run incessantly even though there's no inputs

[ https://issues.apache.org/jira/browse/SPARK-3292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117156#comment-14117156 ] Prashant Sharma commented on SPARK-3292: This issue reminds me of a pull request

[jira] [Comment Edited] (SPARK-3292) Shuffle Tasks run incessantly even though there's no inputs

[ https://issues.apache.org/jira/browse/SPARK-3292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117156#comment-14117156 ] Prashant Sharma edited comment on SPARK-3292 at 9/1/14 8:12 AM:

[jira] [Created] (SPARK-3335) [Spark SQL] In pyspark, cannot use broadcast variables in UDF

kay feng created SPARK-3335: --- Summary: [Spark SQL] In pyspark, cannot use broadcast variables in UDF Key: SPARK-3335 URL: https://issues.apache.org/jira/browse/SPARK-3335 Project: Spark Issue

[jira] [Updated] (SPARK-3335) [Spark SQL] In pyspark, cannot use broadcast variables in UDF

[ https://issues.apache.org/jira/browse/SPARK-3335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kay feng updated SPARK-3335: Description: Running pyspark on a spark cluster with standalone master, it cannot use broadcast variables

[jira] [Created] (SPARK-3336) [Spark SQL] In pyspark, cannot group by field on UDF

kay feng created SPARK-3336: --- Summary: [Spark SQL] In pyspark, cannot group by field on UDF Key: SPARK-3336 URL: https://issues.apache.org/jira/browse/SPARK-3336 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-3335) [Spark SQL] In pyspark, cannot use broadcast variables in UDF

[ https://issues.apache.org/jira/browse/SPARK-3335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kay feng updated SPARK-3335: Description: Running pyspark on a spark cluster with standalone master, spark sql cannot use broadcast

[jira] [Commented] (SPARK-3282) It should support multiple receivers at one socketInputDStream

[ https://issues.apache.org/jira/browse/SPARK-3282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117208#comment-14117208 ] Prashant Sharma commented on SPARK-3282: You can create as many receivers as you

[jira] [Commented] (SPARK-3275) Socket receiver can not recover when the socket server restarted

[ https://issues.apache.org/jira/browse/SPARK-3275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117216#comment-14117216 ] Prashant Sharma commented on SPARK-3275: AFAIK, from my conversation with TD once.

[jira] [Commented] (SPARK-3071) Increase default driver memory

[ https://issues.apache.org/jira/browse/SPARK-3071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117241#comment-14117241 ] Prashant Sharma commented on SPARK-3071: This makes sense, but looking at it from

[jira] [Commented] (SPARK-3292) Shuffle Tasks run incessantly even though there's no inputs

[ https://issues.apache.org/jira/browse/SPARK-3292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117308#comment-14117308 ] guowei commented on SPARK-3292: --- i've created PR 2192 to fix it . but i don't know how to

[jira] [Commented] (SPARK-3178) setting SPARK_WORKER_MEMORY to a value without a label (m or g) sets the worker memory limit to zero

[ https://issues.apache.org/jira/browse/SPARK-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117350#comment-14117350 ] Apache Spark commented on SPARK-3178: - User 'bbejeck' has created a pull request for

[jira] [Commented] (SPARK-3328) ./make-distribution.sh --with-tachyon build is broken

[ https://issues.apache.org/jira/browse/SPARK-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117361#comment-14117361 ] Apache Spark commented on SPARK-3328: - User 'prudhvije' has created a pull request for

[jira] [Created] (SPARK-3337) Paranoid quoting in shell to allow install dirs with spaces within.

Prashant Sharma created SPARK-3337: -- Summary: Paranoid quoting in shell to allow install dirs with spaces within. Key: SPARK-3337 URL: https://issues.apache.org/jira/browse/SPARK-3337 Project: Spark

[jira] [Updated] (SPARK-3337) Paranoid quoting in shell to allow install dirs with spaces within.

[ https://issues.apache.org/jira/browse/SPARK-3337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-3337: --- Component/s: Spark Core Paranoid quoting in shell to allow install dirs with spaces within.

[jira] [Updated] (SPARK-3337) Paranoid quoting in shell to allow install dirs with spaces within.

[ https://issues.apache.org/jira/browse/SPARK-3337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-3337: --- Component/s: Build Paranoid quoting in shell to allow install dirs with spaces within.

[jira] [Commented] (SPARK-2096) Correctly parse dot notations for accessing an array of structs

[ https://issues.apache.org/jira/browse/SPARK-2096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117381#comment-14117381 ] Apache Spark commented on SPARK-2096: - User 'cloud-fan' has created a pull request for

[jira] [Commented] (SPARK-3191) Add explanation of support building spark with maven in http proxy environment

[ https://issues.apache.org/jira/browse/SPARK-3191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117400#comment-14117400 ] zhengbing li commented on SPARK-3191: - Another way to solve this issuse is to set

[jira] [Commented] (SPARK-1405) parallel Latent Dirichlet Allocation (LDA) atop of spark in MLlib

[ https://issues.apache.org/jira/browse/SPARK-1405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117453#comment-14117453 ] Guoqiang Li commented on SPARK-1405: Hi all [PR

[jira] [Comment Edited] (SPARK-1405) parallel Latent Dirichlet Allocation (LDA) atop of spark in MLlib

[ https://issues.apache.org/jira/browse/SPARK-1405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117453#comment-14117453 ] Guoqiang Li edited comment on SPARK-1405 at 9/1/14 2:25 PM: Hi

[jira] [Commented] (SPARK-2594) Add CACHE TABLE name AS SELECT ...

[ https://issues.apache.org/jira/browse/SPARK-2594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117464#comment-14117464 ] Ravindra Pesala commented on SPARK-2594: Thank you Michael. Following are the

[jira] [Commented] (SPARK-3098) In some cases, operation zipWithIndex get a wrong results

[ https://issues.apache.org/jira/browse/SPARK-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117558#comment-14117558 ] Ye Xianjin commented on SPARK-3098: --- hi, [~srowen] and [~gq], I think what [~matei]

[jira] [Commented] (SPARK-2022) Spark 1.0.0 is failing if mesos.coarse set to true

[ https://issues.apache.org/jira/browse/SPARK-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117581#comment-14117581 ] Timothy Chen commented on SPARK-2022: - This should be resolved now,

[jira] [Commented] (SPARK-3098) In some cases, operation zipWithIndex get a wrong results

[ https://issues.apache.org/jira/browse/SPARK-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117622#comment-14117622 ] Matei Zaharia commented on SPARK-3098: -- It's true that the ordering of values after a

[jira] [Updated] (SPARK-2312) Spark Actors do not handle unknown messages in their receive methods

[ https://issues.apache.org/jira/browse/SPARK-2312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-2312: -- Assignee: Josh Rosen Spark Actors do not handle unknown messages in their receive methods

[jira] [Updated] (SPARK-2312) Spark Actors do not handle unknown messages in their receive methods

[ https://issues.apache.org/jira/browse/SPARK-2312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-2312: -- Assignee: Isaias Barroso (was: Josh Rosen) Spark Actors do not handle unknown messages in their

[jira] [Created] (SPARK-3338) Respect user setting of spark.submit.pyFiles

Andrew Or created SPARK-3338: Summary: Respect user setting of spark.submit.pyFiles Key: SPARK-3338 URL: https://issues.apache.org/jira/browse/SPARK-3338 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-3338) Respect user setting of spark.submit.pyFiles

[ https://issues.apache.org/jira/browse/SPARK-3338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-3338: - Component/s: (was: PySpark) Respect user setting of spark.submit.pyFiles

[jira] [Commented] (SPARK-3333) Large number of partitions causes OOM

[ https://issues.apache.org/jira/browse/SPARK-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117770#comment-14117770 ] Patrick Wendell commented on SPARK-: [~nchammas]. I think the default number

[jira] [Comment Edited] (SPARK-3333) Large number of partitions causes OOM

[ https://issues.apache.org/jira/browse/SPARK-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117770#comment-14117770 ] Patrick Wendell edited comment on SPARK- at 9/1/14 10:51 PM:

[jira] [Created] (SPARK-3339) Support for skipping json lines that fail to parse

Michael Armbrust created SPARK-3339: --- Summary: Support for skipping json lines that fail to parse Key: SPARK-3339 URL: https://issues.apache.org/jira/browse/SPARK-3339 Project: Spark Issue

[jira] [Updated] (SPARK-3339) Support for skipping json lines that fail to parse

[ https://issues.apache.org/jira/browse/SPARK-3339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-3339: Assignee: Yin Huai Support for skipping json lines that fail to parse

[jira] [Updated] (SPARK-3308) Ability to read JSON Arrays as tables

[ https://issues.apache.org/jira/browse/SPARK-3308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-3308: Assignee: Yin Huai Ability to read JSON Arrays as tables

[jira] [Commented] (SPARK-3319) Resolve spark.jars, spark.files, and spark.submit.pyFiles etc.

[ https://issues.apache.org/jira/browse/SPARK-3319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117775#comment-14117775 ] Apache Spark commented on SPARK-3319: - User 'andrewor14' has created a pull request

[jira] [Created] (SPARK-3340) Deprecate ADD_JARS and ADD_FILES

Andrew Or created SPARK-3340: Summary: Deprecate ADD_JARS and ADD_FILES Key: SPARK-3340 URL: https://issues.apache.org/jira/browse/SPARK-3340 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-3338) Respect user setting of spark.submit.pyFiles

[ https://issues.apache.org/jira/browse/SPARK-3338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117801#comment-14117801 ] Andrew Or commented on SPARK-3338: -- https://github.com/apache/spark/pull/2232 Respect

[jira] [Assigned] (SPARK-2638) Improve concurrency of fetching Map outputs

[ https://issues.apache.org/jira/browse/SPARK-2638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen reassigned SPARK-2638: - Assignee: Josh Rosen Improve concurrency of fetching Map outputs

[jira] [Resolved] (SPARK-1174) Adding port configuration for HttpFileServer

[ https://issues.apache.org/jira/browse/SPARK-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-1174. Resolution: Duplicate Adding port configuration for HttpFileServer

[jira] [Updated] (SPARK-2895) Support mapPartitionsWithContext in Spark Java API

[ https://issues.apache.org/jira/browse/SPARK-2895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chengxiang Li updated SPARK-2895: - Description: This is a requirement from Hive on Spark, mapPartitionsWithContext only exists in

[jira] [Resolved] (SPARK-2078) Use ISO8601 date formats in logging

[ https://issues.apache.org/jira/browse/SPARK-2078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-2078. Resolution: Won't Fix Discussion on the JIRA suggested let's either keep the current one or

[jira] [Updated] (SPARK-2078) Use ISO8601 date formats in logging

[ https://issues.apache.org/jira/browse/SPARK-2078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2078: --- Issue Type: Improvement (was: Bug) Use ISO8601 date formats in logging

[jira] [Commented] (SPARK-3179) Add task OutputMetrics

[ https://issues.apache.org/jira/browse/SPARK-3179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117825#comment-14117825 ] Sandy Ryza commented on SPARK-3179: --- Hi Michael, Happy to help review your code or

[jira] [Resolved] (SPARK-2773) Shuffle：use growth rate to predict if need to spill

[ https://issues.apache.org/jira/browse/SPARK-2773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-2773. Resolution: Invalid Shuffle：use growth rate to predict if need to spill

[jira] [Resolved] (SPARK-2773) Shuffle：use growth rate to predict if need to spill

[ https://issues.apache.org/jira/browse/SPARK-2773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-2773. Resolution: Won't Fix I don't think this is needed now that SPARK-2316 is fixed. This queue

[jira] [Reopened] (SPARK-2773) Shuffle：use growth rate to predict if need to spill

[ https://issues.apache.org/jira/browse/SPARK-2773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell reopened SPARK-2773: Shuffle：use growth rate to predict if need to spill

[jira] [Updated] (SPARK-3067) JobProgressPage could not show Fair Scheduler Pools section sometimes

[ https://issues.apache.org/jira/browse/SPARK-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3067: --- Target Version/s: 1.2.0 JobProgressPage could not show Fair Scheduler Pools section

[jira] [Commented] (SPARK-3333) Large number of partitions causes OOM

[ https://issues.apache.org/jira/browse/SPARK-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117853#comment-14117853 ] Nicholas Chammas commented on SPARK-: - It looks like the default number of

[jira] [Comment Edited] (SPARK-3333) Large number of partitions causes OOM

[ https://issues.apache.org/jira/browse/SPARK-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117853#comment-14117853 ] Nicholas Chammas edited comment on SPARK- at 9/2/14 3:13 AM:

[jira] [Commented] (SPARK-3168) The ServletContextHandler of webui lacks a SessionManager

[ https://issues.apache.org/jira/browse/SPARK-3168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117855#comment-14117855 ] Reynold Xin commented on SPARK-3168: [~tgraves] - can you take a look? I think you

[jira] [Resolved] (SPARK-3135) Avoid memory copy in TorrentBroadcast serialization

[ https://issues.apache.org/jira/browse/SPARK-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-3135. Resolution: Fixed Fix Version/s: 1.2.0 Assignee: Reynold Xin Avoid memory copy in

[jira] [Updated] (SPARK-3333) Large number of partitions causes OOM

[ https://issues.apache.org/jira/browse/SPARK-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-: --- Target Version/s: 1.1.0 Large number of partitions causes OOM

[jira] [Updated] (SPARK-3333) Large number of partitions causes OOM

[ https://issues.apache.org/jira/browse/SPARK-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-: --- Priority: Blocker (was: Major) Large number of partitions causes OOM

[jira] [Commented] (SPARK-3333) Large number of partitions causes OOM

[ https://issues.apache.org/jira/browse/SPARK-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117905#comment-14117905 ] Josh Rosen commented on SPARK-: --- Still investigating. I tried this on my laptop by

[jira] [Commented] (SPARK-3005) Spark with Mesos fine-grained mode throws UnsupportedOperationException in MesosSchedulerBackend.killTask()

[ https://issues.apache.org/jira/browse/SPARK-3005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117909#comment-14117909 ] Patrick Wendell commented on SPARK-3005: I think it might be safe to just define

[jira] [Updated] (SPARK-3005) Spark with Mesos fine-grained mode throws UnsupportedOperationException in MesosSchedulerBackend.killTask()

[ https://issues.apache.org/jira/browse/SPARK-3005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3005: --- Target Version/s: 1.1.1, 1.2.0 Spark with Mesos fine-grained mode throws

[jira] [Updated] (SPARK-1239) Don't fetch all map output statuses at each reducer during shuffles

[ https://issues.apache.org/jira/browse/SPARK-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated SPARK-1239: -- Summary: Don't fetch all map output statuses at each reducer during shuffles (was: Don't fetch all map

[jira] [Commented] (SPARK-1239) Don't fetch all map output statuses at each reducer during shuffles

[ https://issues.apache.org/jira/browse/SPARK-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117914#comment-14117914 ] Patrick Wendell commented on SPARK-1239: I think [~andrewor] has this in his

[jira] [Commented] (SPARK-1701) Inconsistent naming: slice or partition

[ https://issues.apache.org/jira/browse/SPARK-1701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117917#comment-14117917 ] Nicholas Chammas commented on SPARK-1701: - OK, so it sounds like action is: *