[jira] [Created] (SPARK-4471) blockManagerIdFromJson function throws exception while BlockManagerId be null in MetadataFetchFailedException

SuYan created SPARK-4471: Summary: blockManagerIdFromJson function throws exception while BlockManagerId be null in MetadataFetchFailedException Key: SPARK-4471 URL: https://issues.apache.org/jira/browse/SPARK-4471

[jira] [Commented] (SPARK-4471) blockManagerIdFromJson function throws exception while BlockManagerId be null in MetadataFetchFailedException

[ https://issues.apache.org/jira/browse/SPARK-4471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14215972#comment-14215972 ] Apache Spark commented on SPARK-4471: - User 'suyanNone' has created a pull request for

[jira] [Closed] (SPARK-4418) Remove restriction of binary arithmetic operations for fixed-precision decimal.

[ https://issues.apache.org/jira/browse/SPARK-4418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin closed SPARK-4418. Resolution: Not a Problem Remove restriction of binary arithmetic operations for fixed-precision

[jira] [Created] (SPARK-4472) Print Spark context available as sc. only when SparkContext is created successfully

Shixiong Zhu created SPARK-4472: --- Summary: Print Spark context available as sc. only when SparkContext is created successfully Key: SPARK-4472 URL: https://issues.apache.org/jira/browse/SPARK-4472

[jira] [Commented] (SPARK-4472) Print Spark context available as sc. only when SparkContext is created successfully

[ https://issues.apache.org/jira/browse/SPARK-4472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14216017#comment-14216017 ] Apache Spark commented on SPARK-4472: - User 'zsxwing' has created a pull request for

[jira] [Commented] (SPARK-4429) Build for Scala 2.11 using sbt fails.

[ https://issues.apache.org/jira/browse/SPARK-4429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14216039#comment-14216039 ] Apache Spark commented on SPARK-4429: - User 'ueshin' has created a pull request for

[jira] [Comment Edited] (SPARK-4426) The symbol of BitwiseOr is wrong, should not be ''

[ https://issues.apache.org/jira/browse/SPARK-4426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14213567#comment-14213567 ] ch.prasad edited comment on SPARK-4426 at 11/18/14 11:28 AM: -

[jira] [Comment Edited] (SPARK-4426) The symbol of BitwiseOr is wrong, should not be ''

[ https://issues.apache.org/jira/browse/SPARK-4426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14213567#comment-14213567 ] ch.prasad edited comment on SPARK-4426 at 11/18/14 11:28 AM: -

[jira] [Commented] (SPARK-4454) Race condition in DAGScheduler

[ https://issues.apache.org/jira/browse/SPARK-4454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14216080#comment-14216080 ] Apache Spark commented on SPARK-4454: - User 'mag-' has created a pull request for this

[jira] [Commented] (SPARK-2418) Custom checkpointing with an external function as parameter

[ https://issues.apache.org/jira/browse/SPARK-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14216251#comment-14216251 ] Apache Spark commented on SPARK-2418: - User 'Forevian' has created a pull request for

[jira] [Created] (SPARK-4473) [Core] StageInfo should have ActiveJob's group ID as a field

Aniket Bhatnagar created SPARK-4473: --- Summary: [Core] StageInfo should have ActiveJob's group ID as a field Key: SPARK-4473 URL: https://issues.apache.org/jira/browse/SPARK-4473 Project: Spark

[jira] [Commented] (SPARK-4258) NPE with new Parquet Filters

[ https://issues.apache.org/jira/browse/SPARK-4258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14216329#comment-14216329 ] Cheng Lian commented on SPARK-4258: --- Reproduced this issue with the following test case

[jira] [Comment Edited] (SPARK-4258) NPE with new Parquet Filters

[ https://issues.apache.org/jira/browse/SPARK-4258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14216329#comment-14216329 ] Cheng Lian edited comment on SPARK-4258 at 11/18/14 3:51 PM: -

[jira] [Created] (SPARK-4474) Improve handling of jars that cannot be included in the uber jar

Jim Lim created SPARK-4474: -- Summary: Improve handling of jars that cannot be included in the uber jar Key: SPARK-4474 URL: https://issues.apache.org/jira/browse/SPARK-4474 Project: Spark Issue

[jira] [Updated] (SPARK-4474) Improve handling of jars that cannot be included in the uber jar

[ https://issues.apache.org/jira/browse/SPARK-4474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jim Lim updated SPARK-4474: --- Description: Please refer to this [pull request|https://github.com/apache/spark/pull/3238] for more details.

[jira] [Updated] (SPARK-4474) Improve handling of jars that cannot be included in the uber jar

[ https://issues.apache.org/jira/browse/SPARK-4474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jim Lim updated SPARK-4474: --- Description: Please refer to this [pull request|https://github.com/apache/spark/pull/3238] for more details.

[jira] [Comment Edited] (SPARK-3630) Identify cause of Kryo+Snappy PARSING_ERROR

[ https://issues.apache.org/jira/browse/SPARK-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14216523#comment-14216523 ] Arun Ahuja edited comment on SPARK-3630 at 11/18/14 6:08 PM: -

[jira] [Resolved] (SPARK-4435) Add setThreshold in Python LogisticRegressionModel and SVMModel

[ https://issues.apache.org/jira/browse/SPARK-4435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-4435. -- Issue resolved by pull request 3305 [https://github.com/apache/spark/pull/3305] Add setThreshold

[jira] [Resolved] (SPARK-4396) Support lookup by index in Rating

[ https://issues.apache.org/jira/browse/SPARK-4396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-4396. -- Resolution: Fixed Fix Version/s: 1.2.0 Issue resolved by pull request 3261

[jira] [Commented] (SPARK-4467) Number of elements read is never reset in ExternalSorter

[ https://issues.apache.org/jira/browse/SPARK-4467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14216564#comment-14216564 ] Apache Spark commented on SPARK-4467: - User 'tsdeng' has created a pull request for

[jira] [Comment Edited] (SPARK-3630) Identify cause of Kryo+Snappy PARSING_ERROR

[ https://issues.apache.org/jira/browse/SPARK-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14216523#comment-14216523 ] Arun Ahuja edited comment on SPARK-3630 at 11/18/14 6:59 PM: -

[jira] [Created] (SPARK-4475) PySpark failed to initialize if localhost can not be resolved

Davies Liu created SPARK-4475: - Summary: PySpark failed to initialize if localhost can not be resolved Key: SPARK-4475 URL: https://issues.apache.org/jira/browse/SPARK-4475 Project: Spark Issue

[jira] [Commented] (SPARK-4452) Shuffle data structures can starve others on the same thread for memory

[ https://issues.apache.org/jira/browse/SPARK-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14216691#comment-14216691 ] Matei Zaharia commented on SPARK-4452: -- BTW I've thought about this more and here's

[jira] [Updated] (SPARK-4393) Memory leak in connection manager timeout thread

[ https://issues.apache.org/jira/browse/SPARK-4393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-4393: -- Fix Version/s: 1.1.1 Memory leak in connection manager timeout thread

[jira] [Commented] (SPARK-668) JavaRDDLike.flatMap(PairFlatMapFunction) may fail with typechecking errors

[ https://issues.apache.org/jira/browse/SPARK-668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14216715#comment-14216715 ] Josh Rosen commented on SPARK-668: -- For reference, I think that this was caused by

[jira] [Closed] (SPARK-4434) spark-submit cluster deploy mode JAR URLs are broken in 1.1.1

[ https://issues.apache.org/jira/browse/SPARK-4434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or closed SPARK-4434. Resolution: Fixed Fix Version/s: 1.2.0 1.1.1 Reverted in 1.1.1, and fixed in

[jira] [Created] (SPARK-4476) Use MapType for dict in json which has unique keys in each row.

Davies Liu created SPARK-4476: - Summary: Use MapType for dict in json which has unique keys in each row. Key: SPARK-4476 URL: https://issues.apache.org/jira/browse/SPARK-4476 Project: Spark

[jira] [Updated] (SPARK-4476) Use MapType for dict in json which has unique keys in each row.

[ https://issues.apache.org/jira/browse/SPARK-4476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-4476: -- Description: For the jsonRDD like this: {code} {a: 1} {b: 2} {c: 3} {d: 4} {e: 5} {code} It

[jira] [Updated] (SPARK-4404) SparkSubmitDriverBootstrapper should stop after its SparkSubmit sub-process ends

[ https://issues.apache.org/jira/browse/SPARK-4404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-4404: -- Priority: Blocker (was: Major) SparkSubmitDriverBootstrapper should stop after its SparkSubmit

[jira] [Commented] (SPARK-2554) CountDistinct and SumDistinct should do partial aggregation

[ https://issues.apache.org/jira/browse/SPARK-2554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14216790#comment-14216790 ] Apache Spark commented on SPARK-2554: - User 'ravipesala' has created a pull request

[jira] [Updated] (SPARK-4404) SparkSubmitDriverBootstrapper should stop after its SparkSubmit sub-process ends

[ https://issues.apache.org/jira/browse/SPARK-4404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-4404: - Affects Version/s: (was: 1.2.0) 1.1.0 SparkSubmitDriverBootstrapper should

[jira] [Updated] (SPARK-4404) SparkSubmitDriverBootstrapper should stop after its SparkSubmit sub-process ends

[ https://issues.apache.org/jira/browse/SPARK-4404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-4404: - Affects Version/s: (was: 1.1.0) 1.2.0 SparkSubmitDriverBootstrapper should

[jira] [Commented] (SPARK-4476) Use MapType for dict in json which has unique keys in each row.

[ https://issues.apache.org/jira/browse/SPARK-4476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14216826#comment-14216826 ] Yin Huai commented on SPARK-4476: - Seems a possible approach is to convert the StructType

[jira] [Resolved] (SPARK-4017) Progress bar in console

[ https://issues.apache.org/jira/browse/SPARK-4017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-4017. Resolution: Fixed Fix Version/s: 1.2.0 Progress bar in console

[jira] [Commented] (SPARK-4476) Use MapType for dict in json which has unique keys in each row.

[ https://issues.apache.org/jira/browse/SPARK-4476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14216837#comment-14216837 ] Davies Liu commented on SPARK-4476: --- Not started yet, it's better if you could take it,

[jira] [Commented] (SPARK-4452) Shuffle data structures can starve others on the same thread for memory

[ https://issues.apache.org/jira/browse/SPARK-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14216933#comment-14216933 ] Sandy Ryza commented on SPARK-4452: --- One issue with a limits-by-object approach is that

[jira] [Closed] (SPARK-4463) Add (de)select all button for additional metrics in webUI

[ https://issues.apache.org/jira/browse/SPARK-4463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or closed SPARK-4463. Resolution: Fixed Fix Version/s: 1.2.0 Add (de)select all button for additional metrics in webUI

[jira] [Updated] (SPARK-4463) Add (de)select all button for additional metrics in webUI

[ https://issues.apache.org/jira/browse/SPARK-4463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-4463: - Affects Version/s: 1.2.0 Add (de)select all button for additional metrics in webUI

[jira] [Commented] (SPARK-3060) spark-shell.cmd doesn't accept application options in Windows OS

[ https://issues.apache.org/jira/browse/SPARK-3060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14217053#comment-14217053 ] Apache Spark commented on SPARK-3060: - User 'tsudukim' has created a pull request for

[jira] [Resolved] (SPARK-4306) LogisticRegressionWithLBFGS support for PySpark MLlib

[ https://issues.apache.org/jira/browse/SPARK-4306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-4306. -- Resolution: Fixed Fix Version/s: 1.2.0 Issue resolved by pull request 3307

[jira] [Resolved] (SPARK-3721) Broadcast Variables above 2GB break in PySpark

[ https://issues.apache.org/jira/browse/SPARK-3721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-3721. --- Resolution: Fixed Fix Version/s: 1.2.0 Issue resolved by pull request 2659

[jira] [Updated] (SPARK-4377) ZooKeeperPersistenceEngine: java.lang.IllegalStateException: Trying to deserialize a serialized ActorRef without an ActorSystem in scope.

[ https://issues.apache.org/jira/browse/SPARK-4377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-4377: -- Assignee: Prashant Sharma [~prashant_], could you take a look at this? I think that you modified the

[jira] [Resolved] (SPARK-4433) Racing condition in zipWithIndex

[ https://issues.apache.org/jira/browse/SPARK-4433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-4433. -- Resolution: Fixed Fix Version/s: 1.0.3 1.1.1 1.2.0

[jira] [Updated] (SPARK-4433) Racing condition in zipWithIndex

[ https://issues.apache.org/jira/browse/SPARK-4433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-4433: - Target Version/s: 1.1.1, 1.2.0, 1.0.3 (was: 1.2.0, 1.0.3, 1.1.2) Racing condition in

[jira] [Commented] (SPARK-799) Windows versions of the deploy scripts

[ https://issues.apache.org/jira/browse/SPARK-799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14217136#comment-14217136 ] Masayoshi TSUZUKI commented on SPARK-799: - I think Powershell is better. Linux

[jira] [Updated] (SPARK-4327) Python API for RDD.randomSplit()

[ https://issues.apache.org/jira/browse/SPARK-4327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-4327: - Assignee: Davies Liu Python API for RDD.randomSplit()

[jira] [Resolved] (SPARK-4327) Python API for RDD.randomSplit()

[ https://issues.apache.org/jira/browse/SPARK-4327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-4327. -- Resolution: Fixed Fix Version/s: 1.2.0 Issue resolved by pull request 3193

[jira] [Updated] (SPARK-4377) ZooKeeperPersistenceEngine: java.lang.IllegalStateException: Trying to deserialize a serialized ActorRef without an ActorSystem in scope.

[ https://issues.apache.org/jira/browse/SPARK-4377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4377: --- Target Version/s: 1.2.0 ZooKeeperPersistenceEngine: java.lang.IllegalStateException: Trying

[jira] [Resolved] (SPARK-4281) Yarn shuffle service jars need to include dependencies

[ https://issues.apache.org/jira/browse/SPARK-4281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-4281. Resolution: Fixed Fix Version/s: 1.2.0 Yarn shuffle service jars need to include

[jira] [Updated] (SPARK-1856) Standardize MLlib interfaces

[ https://issues.apache.org/jira/browse/SPARK-1856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-1856: - Target Version/s: 1.3.0 (was: 1.2.0) Standardize MLlib interfaces

[jira] [Updated] (SPARK-3702) Standardize MLlib classes for learners, models

[ https://issues.apache.org/jira/browse/SPARK-3702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-3702: - Target Version/s: 1.3.0 (was: 1.2.0) Standardize MLlib classes for learners, models

[jira] [Updated] (SPARK-4452) Shuffle data structures can starve others on the same thread for memory

[ https://issues.apache.org/jira/browse/SPARK-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4452: --- Component/s: Spark Core Shuffle data structures can starve others on the same thread for

[jira] [Created] (SPARK-4477) remove numpy from RDDSampler of PySpark

Davies Liu created SPARK-4477: - Summary: remove numpy from RDDSampler of PySpark Key: SPARK-4477 URL: https://issues.apache.org/jira/browse/SPARK-4477 Project: Spark Issue Type: New Feature

[jira] [Updated] (SPARK-4477) remove numpy from RDDSampler of PySpark

[ https://issues.apache.org/jira/browse/SPARK-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-4477: -- Affects Version/s: 1.2.0 0.8.1 0.9.1

[jira] [Created] (SPARK-4478) totalRegisteredExecutors not updated properly

Akshat Aranya created SPARK-4478: Summary: totalRegisteredExecutors not updated properly Key: SPARK-4478 URL: https://issues.apache.org/jira/browse/SPARK-4478 Project: Spark Issue Type: Bug

[jira] [Reopened] (SPARK-4404) SparkSubmitDriverBootstrapper should stop after its SparkSubmit sub-process ends

[ https://issues.apache.org/jira/browse/SPARK-4404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell reopened SPARK-4404: SparkSubmitDriverBootstrapper should stop after its SparkSubmit sub-process ends

[jira] [Closed] (SPARK-4404) SparkSubmitDriverBootstrapper should stop after its SparkSubmit sub-process ends

[ https://issues.apache.org/jira/browse/SPARK-4404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell closed SPARK-4404. -- Resolution: Fixed SparkSubmitDriverBootstrapper should stop after its SparkSubmit sub-process

[jira] [Resolved] (SPARK-4404) SparkSubmitDriverBootstrapper should stop after its SparkSubmit sub-process ends

[ https://issues.apache.org/jira/browse/SPARK-4404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-4404. Resolution: Fixed SparkSubmitDriverBootstrapper should stop after its SparkSubmit

[jira] [Closed] (SPARK-4404) SparkSubmitDriverBootstrapper should stop after its SparkSubmit sub-process ends

[ https://issues.apache.org/jira/browse/SPARK-4404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell closed SPARK-4404. -- SparkSubmitDriverBootstrapper should stop after its SparkSubmit sub-process ends

[jira] [Resolved] (SPARK-4404) SparkSubmitDriverBootstrapper should stop after its SparkSubmit sub-process ends

[ https://issues.apache.org/jira/browse/SPARK-4404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-4404. Resolution: Fixed SparkSubmitDriverBootstrapper should stop after its SparkSubmit

[jira] [Commented] (SPARK-4477) remove numpy from RDDSampler of PySpark

[ https://issues.apache.org/jira/browse/SPARK-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14217183#comment-14217183 ] Apache Spark commented on SPARK-4477: - User 'davies' has created a pull request for

[jira] [Closed] (SPARK-4339) Make fixedPoint Configurable in Analyzer

[ https://issues.apache.org/jira/browse/SPARK-4339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DoingDone9 closed SPARK-4339. - Resolution: Not a Problem Make fixedPoint Configurable in Analyzer

[jira] [Created] (SPARK-4479) Avoid unnecessary defensive copies when Sort based shuffle is on

Michael Armbrust created SPARK-4479: --- Summary: Avoid unnecessary defensive copies when Sort based shuffle is on Key: SPARK-4479 URL: https://issues.apache.org/jira/browse/SPARK-4479 Project: Spark

[jira] [Updated] (SPARK-4452) Shuffle data structures can starve others on the same thread for memory

[ https://issues.apache.org/jira/browse/SPARK-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-4452: - Priority: Critical (was: Blocker) Shuffle data structures can starve others on the same thread for

[jira] [Created] (SPARK-4480) Avoid many small spills in external data structures

Andrew Or created SPARK-4480: Summary: Avoid many small spills in external data structures Key: SPARK-4480 URL: https://issues.apache.org/jira/browse/SPARK-4480 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-4452) Shuffle data structures can starve others on the same thread for memory

[ https://issues.apache.org/jira/browse/SPARK-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-4452: - Target Version/s: 1.1.1, 1.2.0, 1.3.0 (was: 1.1.1, 1.2.0) Shuffle data structures can starve others on

[jira] [Commented] (SPARK-4452) Shuffle data structures can starve others on the same thread for memory

[ https://issues.apache.org/jira/browse/SPARK-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14217253#comment-14217253 ] Andrew Or commented on SPARK-4452: -- I have opened a JIRA that targets on fixing this on a

[jira] [Commented] (SPARK-4480) Avoid many small spills in external data structures

[ https://issues.apache.org/jira/browse/SPARK-4480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14217272#comment-14217272 ] Apache Spark commented on SPARK-4480: - User 'andrewor14' has created a pull request

[jira] [Commented] (SPARK-4452) Shuffle data structures can starve others on the same thread for memory

[ https://issues.apache.org/jira/browse/SPARK-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14217274#comment-14217274 ] Andrew Or commented on SPARK-4452: -- [~matei] I have implemented your first suggestion

[jira] [Commented] (SPARK-4480) Avoid many small spills in external data structures

[ https://issues.apache.org/jira/browse/SPARK-4480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14217296#comment-14217296 ] Andrew Or commented on SPARK-4480: -- https://github.com/apache/spark/pull/3353 Avoid

[jira] [Issue Comment Deleted] (SPARK-4480) Avoid many small spills in external data structures

[ https://issues.apache.org/jira/browse/SPARK-4480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-4480: - Comment: was deleted (was: https://github.com/apache/spark/pull/3353) Avoid many small spills in

[jira] [Commented] (SPARK-3633) Fetches failure observed after SPARK-2711

[ https://issues.apache.org/jira/browse/SPARK-3633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14217300#comment-14217300 ] Andrew Or commented on SPARK-3633: -- I have filed SPARK-4480 as an immediate fix. The

[jira] [Updated] (SPARK-4380) Executor full of log spilling in-memory map of 0 MB to disk

[ https://issues.apache.org/jira/browse/SPARK-4380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-4380: - Target Version/s: 1.1.1, 1.2.0 Executor full of log spilling in-memory map of 0 MB to disk

[jira] [Commented] (SPARK-4380) Executor full of log spilling in-memory map of 0 MB to disk

[ https://issues.apache.org/jira/browse/SPARK-4380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14217304#comment-14217304 ] Andrew Or commented on SPARK-4380: -- Reopened this to pull it into 1.1.1 Executor full

[jira] [Reopened] (SPARK-4380) Executor full of log spilling in-memory map of 0 MB to disk

[ https://issues.apache.org/jira/browse/SPARK-4380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or reopened SPARK-4380: -- Assignee: Andrew Or (was: Hong Shen) Executor full of log spilling in-memory map of 0 MB to disk

[jira] [Commented] (SPARK-4480) Avoid many small spills in external data structures

[ https://issues.apache.org/jira/browse/SPARK-4480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14217305#comment-14217305 ] Apache Spark commented on SPARK-4480: - User 'andrewor14' has created a pull request

[jira] [Created] (SPARK-4481) Some comments for `updateStateByKey` are wrong

Shixiong Zhu created SPARK-4481: --- Summary: Some comments for `updateStateByKey` are wrong Key: SPARK-4481 URL: https://issues.apache.org/jira/browse/SPARK-4481 Project: Spark Issue Type:

[jira] [Commented] (SPARK-4380) Executor full of log spilling in-memory map of 0 MB to disk

[ https://issues.apache.org/jira/browse/SPARK-4380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14217313#comment-14217313 ] Apache Spark commented on SPARK-4380: - User 'andrewor14' has created a pull request

[jira] [Commented] (SPARK-4481) Some comments for `updateStateByKey` are wrong

[ https://issues.apache.org/jira/browse/SPARK-4481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14217314#comment-14217314 ] Apache Spark commented on SPARK-4481: - User 'zsxwing' has created a pull request for

[jira] [Updated] (SPARK-2918) EXPLAIN doens't support the CTAS

[ https://issues.apache.org/jira/browse/SPARK-2918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Hao updated SPARK-2918: - Summary: EXPLAIN doens't support the CTAS (was: EXPLAIN doens't support the native command) EXPLAIN

[jira] [Commented] (SPARK-2918) EXPLAIN doens't support the native command

[ https://issues.apache.org/jira/browse/SPARK-2918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14217323#comment-14217323 ] Apache Spark commented on SPARK-2918: - User 'chenghao-intel' has created a pull

[jira] [Commented] (SPARK-4452) Shuffle data structures can starve others on the same thread for memory

[ https://issues.apache.org/jira/browse/SPARK-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14217331#comment-14217331 ] Matei Zaharia commented on SPARK-4452: -- Forced spilling is orthogonal to how you set

[jira] [Created] (SPARK-4482) ReceivedBlockTracker's write ahead log is enabled by default

Tathagata Das created SPARK-4482: Summary: ReceivedBlockTracker's write ahead log is enabled by default Key: SPARK-4482 URL: https://issues.apache.org/jira/browse/SPARK-4482 Project: Spark

[jira] [Commented] (SPARK-4482) ReceivedBlockTracker's write ahead log is enabled by default

[ https://issues.apache.org/jira/browse/SPARK-4482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14217336#comment-14217336 ] Apache Spark commented on SPARK-4482: - User 'tdas' has created a pull request for this

[jira] [Commented] (SPARK-4452) Shuffle data structures can starve others on the same thread for memory

[ https://issues.apache.org/jira/browse/SPARK-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14217340#comment-14217340 ] Sandy Ryza commented on SPARK-4452: --- [~matei] my point is not that forced spilling

[jira] [Closed] (SPARK-4380) Executor full of log spilling in-memory map of 0 MB to disk

[ https://issues.apache.org/jira/browse/SPARK-4380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or closed SPARK-4380. Resolution: Fixed Fix Version/s: 1.1.1 Executor full of log spilling in-memory map of 0 MB to disk

[jira] [Created] (SPARK-4483) Optimization about reduce memory costs during the HashOuterJoin

Yi Tian created SPARK-4483: -- Summary: Optimization about reduce memory costs during the HashOuterJoin Key: SPARK-4483 URL: https://issues.apache.org/jira/browse/SPARK-4483 Project: Spark Issue

[jira] [Updated] (SPARK-4483) Optimization about reduce memory costs during the HashOuterJoin

[ https://issues.apache.org/jira/browse/SPARK-4483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Tian updated SPARK-4483: --- Description: In {{HashOuterJoin.scala}}, spark read data from both side of join operation before zip them

[jira] [Commented] (SPARK-2321) Design a proper progress reporting event listener API

[ https://issues.apache.org/jira/browse/SPARK-2321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14217485#comment-14217485 ] Rui Li commented on SPARK-2321: --- Hi [~joshrosen], Shall we make {{SparkJobInfo}} and

[jira] [Resolved] (SPARK-4441) Close Tachyon client when TachyonBlockManager is shut down

[ https://issues.apache.org/jira/browse/SPARK-4441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-4441. Resolution: Fixed Fix Version/s: 1.2.0 Close Tachyon client when

[jira] [Resolved] (SPARK-4432) Resource(InStream) is not closed in TachyonStore

[ https://issues.apache.org/jira/browse/SPARK-4432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-4432. Resolution: Fixed Fix Version/s: 1.2.0 Assignee: shimingfei

[jira] [Commented] (SPARK-2397) Get rid of LocalHiveContext

[ https://issues.apache.org/jira/browse/SPARK-2397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14217494#comment-14217494 ] Nathan M commented on SPARK-2397: - Just curious why this has been Deprecated. It was

[jira] [Updated] (SPARK-4441) Close Tachyon client when TachyonBlockManager is shut down

[ https://issues.apache.org/jira/browse/SPARK-4441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4441: --- Assignee: shimingfei Close Tachyon client when TachyonBlockManager is shut down

[jira] [Updated] (SPARK-2808) update kafka to version 0.8.2

[ https://issues.apache.org/jira/browse/SPARK-2808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-2808: --- Summary: update kafka to version 0.8.2 (was: update kafka to version 0.8.1) update kafka to

[jira] [Commented] (SPARK-2808) update kafka to version 0.8.2

[ https://issues.apache.org/jira/browse/SPARK-2808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14217528#comment-14217528 ] Prashant Sharma commented on SPARK-2808: Kafka 0.8.2-beta for scala 2.11 is