[jira] [Commented] (SPARK-17498) StringIndexer.setHandleInvalid sohuld have another option 'new'

2016-09-12 Thread Vincent (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15483514#comment-15483514 ] Vincent commented on SPARK-17498: - Here is what we cc [~qhuang] see about this issue and correct me if

[jira] [Commented] (SPARK-17498) StringIndexer.setHandleInvalid sohuld have another option 'new'

2016-09-12 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15483552#comment-15483552 ] Sean Owen commented on SPARK-17498: --- This is more band-aid than anything. Really, the assumption is

[jira] [Comment Edited] (SPARK-17498) StringIndexer.setHandleInvalid sohuld have another option 'new'

2016-09-12 Thread Vincent (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15483514#comment-15483514 ] Vincent edited comment on SPARK-17498 at 9/12/16 8:55 AM: -- Here is how we cc

[jira] [Updated] (SPARK-17503) Memory leak in Memory store when unable to cache the whole RDD

2016-09-12 Thread Sean Zhong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Zhong updated SPARK-17503: --- Summary: Memory leak in Memory store when unable to cache the whole RDD (was: Memory leak in Memory

[jira] [Updated] (SPARK-17503) Memory leak in Memory store when unable to cache the whole RDD in memory

2016-09-12 Thread Sean Zhong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Zhong updated SPARK-17503: --- Summary: Memory leak in Memory store when unable to cache the whole RDD in memory (was: Memory leak

[jira] [Updated] (SPARK-17503) Memory leak in Memory store when unable to cache the whole RDD in memory

2016-09-12 Thread Sean Zhong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Zhong updated SPARK-17503: --- Description: h2.Problem description: The following query triggers out of memory error. {code}

[jira] [Comment Edited] (SPARK-17498) StringIndexer.setHandleInvalid sohuld have another option 'new'

2016-09-12 Thread Vincent (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15483514#comment-15483514 ] Vincent edited comment on SPARK-17498 at 9/12/16 8:43 AM: -- Here is what we cc

[jira] [Created] (SPARK-17504) Spark App Handle from SparkLauncher always returns UNKNOWN app state when used with Mesos in Client Mode

2016-09-12 Thread Adam Jakubowski (JIRA)
Adam Jakubowski created SPARK-17504: --- Summary: Spark App Handle from SparkLauncher always returns UNKNOWN app state when used with Mesos in Client Mode Key: SPARK-17504 URL:

[jira] [Closed] (SPARK-17500) The DiskBytesSpilled metric in ExternalMerger && ExternalGroupBy is not right

2016-09-12 Thread DjvuLee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DjvuLee closed SPARK-17500. --- Resolution: Not A Bug > The DiskBytesSpilled metric in ExternalMerger && ExternalGroupBy is not right >

[jira] [Updated] (SPARK-17503) Memory leak in Memory store when unable to cache the whole RDD in memory

2016-09-12 Thread Sean Zhong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Zhong updated SPARK-17503: --- Attachment: Screen Shot 2016-09-12 at 4.34.19 PM.png Screen Shot 2016-09-12 at

[jira] [Updated] (SPARK-17504) Spark App Handle from SparkLauncher always returns UNKNOWN app state when used with Mesos in Client Mode

2016-09-12 Thread Adam Jakubowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Jakubowski updated SPARK-17504: Description: Spark App Handle returned from Spark Launcher when used with Mesos in Client

[jira] [Assigned] (SPARK-17502) Multiple Bugs in DDL Statements on Temporary Views

2016-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17502: Assignee: (was: Apache Spark) > Multiple Bugs in DDL Statements on Temporary Views >

[jira] [Assigned] (SPARK-17502) Multiple Bugs in DDL Statements on Temporary Views

2016-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17502: Assignee: Apache Spark > Multiple Bugs in DDL Statements on Temporary Views >

[jira] [Created] (SPARK-17502) Multiple Bugs in DDL Statements on Temporary Views

2016-09-12 Thread Xiao Li (JIRA)
Xiao Li created SPARK-17502: --- Summary: Multiple Bugs in DDL Statements on Temporary Views Key: SPARK-17502 URL: https://issues.apache.org/jira/browse/SPARK-17502 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-17502) Multiple Bugs in DDL Statements on Temporary Views

2016-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15483219#comment-15483219 ] Apache Spark commented on SPARK-17502: -- User 'gatorsmile' has created a pull request for this issue:

[jira] [Updated] (SPARK-17503) Memory leak in Memory store when unable to cache the whole RDD

2016-09-12 Thread Sean Zhong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Zhong updated SPARK-17503: --- Description: h2.Problem description: The following query triggers out of memory error. {code}

[jira] [Updated] (SPARK-17503) Memory leak in Memory store when unable to cache the whole RDD

2016-09-12 Thread Sean Zhong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Zhong updated SPARK-17503: --- Description: h2.Problem description: The following query triggers out of memory error. {code}

[jira] [Assigned] (SPARK-17462) Check for places within MLlib which should use VersionUtils to parse Spark version strings

2016-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17462: Assignee: (was: Apache Spark) > Check for places within MLlib which should use

[jira] [Commented] (SPARK-17462) Check for places within MLlib which should use VersionUtils to parse Spark version strings

2016-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15483285#comment-15483285 ] Apache Spark commented on SPARK-17462: -- User 'VinceShieh' has created a pull request for this issue:

[jira] [Assigned] (SPARK-17462) Check for places within MLlib which should use VersionUtils to parse Spark version strings

2016-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17462: Assignee: Apache Spark > Check for places within MLlib which should use VersionUtils to

[jira] [Commented] (SPARK-17503) Memory leak in Memory store when unable to cache the whole RDD

2016-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15483381#comment-15483381 ] Apache Spark commented on SPARK-17503: -- User 'clockfly' has created a pull request for this issue:

[jira] [Commented] (SPARK-17498) StringIndexer.setHandleInvalid sohuld have another option 'new'

2016-09-12 Thread Miao Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15483383#comment-15483383 ] Miao Wang commented on SPARK-17498: --- Can you give a concrete example? > StringIndexer.setHandleInvalid

[jira] [Updated] (SPARK-17503) Memory leak in Memory store when unable to cache the whole RDD

2016-09-12 Thread Sean Zhong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Zhong updated SPARK-17503: --- Description: h2.Problem description: The following query triggers out of memory error. {code}

[jira] [Created] (SPARK-17503) Memory leak in Memory store which unable to cache whole RDD

2016-09-12 Thread Sean Zhong (JIRA)
Sean Zhong created SPARK-17503: -- Summary: Memory leak in Memory store which unable to cache whole RDD Key: SPARK-17503 URL: https://issues.apache.org/jira/browse/SPARK-17503 Project: Spark

[jira] [Assigned] (SPARK-17503) Memory leak in Memory store when unable to cache the whole RDD

2016-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17503: Assignee: (was: Apache Spark) > Memory leak in Memory store when unable to cache the

[jira] [Assigned] (SPARK-17503) Memory leak in Memory store when unable to cache the whole RDD

2016-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17503: Assignee: Apache Spark > Memory leak in Memory store when unable to cache the whole RDD >

[jira] [Updated] (SPARK-17503) Memory leak in Memory store when unable to cache the whole RDD in memory

2016-09-12 Thread Sean Zhong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Zhong updated SPARK-17503: --- Description: h2.Problem description: The following query triggers out of memory error. {code}

[jira] [Commented] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs

2016-09-12 Thread Alexander Kasper (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15483413#comment-15483413 ] Alexander Kasper commented on SPARK-17321: -- I guess then we encountered the 1% where the NM

[jira] [Updated] (SPARK-17503) Memory leak in Memory store when unable to cache the whole RDD in memory

2016-09-12 Thread Sean Zhong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Zhong updated SPARK-17503: --- Affects Version/s: (was: 1.6.2) (was: 2.0.0) Target Version/s:

[jira] [Commented] (SPARK-6567) Large linear model parallelism via a join and reduceByKey

2016-09-12 Thread WangJianfei (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15483287#comment-15483287 ] WangJianfei commented on SPARK-6567: @Reza Zadeh Any progress about this problems? Thanks. codlife >

[jira] [Comment Edited] (SPARK-6567) Large linear model parallelism via a join and reduceByKey

2016-09-12 Thread WangJianfei (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15483287#comment-15483287 ] WangJianfei edited comment on SPARK-6567 at 9/12/16 6:57 AM: - Reza Zadeh Any

[jira] [Commented] (SPARK-17503) Memory leak in Memory store when unable to cache the whole RDD

2016-09-12 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15483326#comment-15483326 ] Sean Owen commented on SPARK-17503: --- cache() means "cache in memory" only. There is a persist() call

[jira] [Commented] (SPARK-17462) Check for places within MLlib which should use VersionUtils to parse Spark version strings

2016-09-12 Thread Peng Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15483336#comment-15483336 ] Peng Meng commented on SPARK-17462: --- hi [~josephkb], I am busy this days, I am glad VinceShieh can help

[jira] [Commented] (SPARK-17503) Memory leak in Memory store when unable to cache the whole RDD

2016-09-12 Thread Sean Zhong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15483389#comment-15483389 ] Sean Zhong commented on SPARK-17503: [~sowen] I have modified the title to mean "cache in memory" >

[jira] [Assigned] (SPARK-17505) Add setBins for BinaryClassificationMetrics in mlllb/evaluation

2016-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17505: Assignee: Apache Spark > Add setBins for BinaryClassificationMetrics in mlllb/evaluation

[jira] [Assigned] (SPARK-17505) Add setBins for BinaryClassificationMetrics in mlllb/evaluation

2016-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17505: Assignee: (was: Apache Spark) > Add setBins for BinaryClassificationMetrics in

[jira] [Created] (SPARK-17505) Add setBins for BinaryClassificationMetrics in mlllb/evaluation

2016-09-12 Thread Peng Meng (JIRA)
Peng Meng created SPARK-17505: - Summary: Add setBins for BinaryClassificationMetrics in mlllb/evaluation Key: SPARK-17505 URL: https://issues.apache.org/jira/browse/SPARK-17505 Project: Spark

[jira] [Commented] (SPARK-17505) Add setBins for BinaryClassificationMetrics in mlllb/evaluation

2016-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15483806#comment-15483806 ] Apache Spark commented on SPARK-17505: -- User 'mpjlu' has created a pull request for this issue:

[jira] [Updated] (SPARK-17453) Broadcast block already exists in MemoryStore

2016-09-12 Thread Chris Bannister (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Bannister updated SPARK-17453: Component/s: Spark Core > Broadcast block already exists in MemoryStore >

[jira] [Updated] (SPARK-17171) DAG will list all partitions in the graph

2016-09-12 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-17171: -- Assignee: cen yuhai > DAG will list all partitions in the graph >

[jira] [Resolved] (SPARK-17171) DAG will list all partitions in the graph

2016-09-12 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-17171. --- Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 14737

[jira] [Resolved] (SPARK-17447) performance improvement in Partitioner.DefaultPartitioner

2016-09-12 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-17447. --- Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 15039

[jira] [Commented] (SPARK-17397) Show example of what to do when awaitTermination() throws an Exception

2016-09-12 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15483832#comment-15483832 ] Sean Owen commented on SPARK-17397: --- [~spirom] would you like to follow up on this? > Show example of

[jira] [Resolved] (SPARK-17466) Error message is not very clear

2016-09-12 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-17466. --- Resolution: Not A Problem > Error message is not very clear > --- > >

[jira] [Commented] (SPARK-17498) StringIndexer.setHandleInvalid sohuld have another option 'new'

2016-09-12 Thread miroslav Balaz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15484129#comment-15484129 ] miroslav Balaz commented on SPARK-17498: No I meant, that it should return 3 and 3 for "d" and

[jira] [Comment Edited] (SPARK-17498) StringIndexer.setHandleInvalid sohuld have another option 'new'

2016-09-12 Thread miroslav Balaz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15484129#comment-15484129 ] miroslav Balaz edited comment on SPARK-17498 at 9/12/16 1:46 PM: - No I

[jira] [Comment Edited] (SPARK-17501) Re-register BlockManager again and again

2016-09-12 Thread cen yuhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15484185#comment-15484185 ] cen yuhai edited comment on SPARK-17501 at 9/12/16 1:54 PM: I can't hardly

[jira] [Commented] (SPARK-17501) Re-register BlockManager again and again

2016-09-12 Thread cen yuhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15484190#comment-15484190 ] cen yuhai commented on SPARK-17501: --- I will try to create a unit test for it. > Re-register

[jira] [Commented] (SPARK-17501) Re-register BlockManager again and again

2016-09-12 Thread cen yuhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15484185#comment-15484185 ] cen yuhai commented on SPARK-17501: --- I can't hardly reproduce this error. But maybe I found the root

[jira] [Created] (SPARK-17506) Improve the check double values equality rule

2016-09-12 Thread Jiang Xingbo (JIRA)
Jiang Xingbo created SPARK-17506: Summary: Improve the check double values equality rule Key: SPARK-17506 URL: https://issues.apache.org/jira/browse/SPARK-17506 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-17506) Improve the check double values equality rule

2016-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17506: Assignee: Apache Spark > Improve the check double values equality rule >

[jira] [Commented] (SPARK-17506) Improve the check double values equality rule

2016-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15484443#comment-15484443 ] Apache Spark commented on SPARK-17506: -- User 'jiangxb1987' has created a pull request for this

[jira] [Assigned] (SPARK-17506) Improve the check double values equality rule

2016-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17506: Assignee: (was: Apache Spark) > Improve the check double values equality rule >

[jira] [Assigned] (SPARK-17424) Dataset job fails from unsound substitution in ScalaReflect

2016-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17424: Assignee: (was: Apache Spark) > Dataset job fails from unsound substitution in

[jira] [Assigned] (SPARK-17424) Dataset job fails from unsound substitution in ScalaReflect

2016-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17424: Assignee: Apache Spark > Dataset job fails from unsound substitution in ScalaReflect >

[jira] [Commented] (SPARK-17424) Dataset job fails from unsound substitution in ScalaReflect

2016-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15484715#comment-15484715 ] Apache Spark commented on SPARK-17424: -- User 'rdblue' has created a pull request for this issue:

[jira] [Commented] (SPARK-17424) Dataset job fails from unsound substitution in ScalaReflect

2016-09-12 Thread Ryan Blue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15484718#comment-15484718 ] Ryan Blue commented on SPARK-17424: --- I'm adding the above fix in a PR. This fix works for us (the job

[jira] [Commented] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs

2016-09-12 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15484748#comment-15484748 ] Thomas Graves commented on SPARK-17321: --- Not sure I follow this comment. So you are using NM

[jira] [Assigned] (SPARK-17463) Serialization of accumulators in heartbeats is not thread-safe

2016-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17463: Assignee: Shixiong Zhu (was: Apache Spark) > Serialization of accumulators in heartbeats

[jira] [Updated] (SPARK-17409) Query in CTAS is Optimized Twice

2016-09-12 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-17409: --- Labels: correctness (was: ) > Query in CTAS is Optimized Twice > >

[jira] [Assigned] (SPARK-17507) check weight vector size in ANN

2016-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17507: Assignee: Apache Spark > check weight vector size in ANN >

[jira] [Assigned] (SPARK-17507) check weight vector size in ANN

2016-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17507: Assignee: (was: Apache Spark) > check weight vector size in ANN >

[jira] [Commented] (SPARK-17507) check weight vector size in ANN

2016-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15484586#comment-15484586 ] Apache Spark commented on SPARK-17507: -- User 'WeichenXu123' has created a pull request for this

[jira] [Assigned] (SPARK-14818) Move sketch and mllibLocal out from mima exclusion

2016-09-12 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen reassigned SPARK-14818: -- Assignee: Josh Rosen > Move sketch and mllibLocal out from mima exclusion >

[jira] [Assigned] (SPARK-14818) Move sketch and mllibLocal out from mima exclusion

2016-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-14818: Assignee: Apache Spark (was: Josh Rosen) > Move sketch and mllibLocal out from mima

[jira] [Commented] (SPARK-17445) Reference an ASF page as the main place to find third-party packages

2016-09-12 Thread Matei Zaharia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15484732#comment-15484732 ] Matei Zaharia commented on SPARK-17445: --- Sounds good to me. > Reference an ASF page as the main

[jira] [Commented] (SPARK-17508) Setting weightCol to None in ML library causes an error

2016-09-12 Thread Evan Zamir (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15484861#comment-15484861 ] Evan Zamir commented on SPARK-17508: Just ran the same snippet of code setting weightCol="" and that

[jira] [Updated] (SPARK-17503) Memory leak in Memory store when unable to cache the whole RDD in memory

2016-09-12 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-17503: --- Assignee: Sean Zhong > Memory leak in Memory store when unable to cache the whole RDD in memory >

[jira] [Commented] (SPARK-17494) Floor function rounds up during join

2016-09-12 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15484952#comment-15484952 ] Josh Rosen commented on SPARK-17494: This also seems to affect Spark 2.0, except there it always

[jira] [Resolved] (SPARK-17483) Minor refactoring and cleanup in BlockManager block status reporting and block removal

2016-09-12 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-17483. Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 15036

[jira] [Updated] (SPARK-17494) Floor function rounds up during join

2016-09-12 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-17494: --- Labels: correctness (was: ) > Floor function rounds up during join >

[jira] [Updated] (SPARK-17494) Floor function rounds up during join

2016-09-12 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-17494: --- Affects Version/s: 2.0.0 > Floor function rounds up during join >

[jira] [Updated] (SPARK-17509) When wrapping catalyst datatype to Hive data type avoid pattern matching

2016-09-12 Thread Sital Kedia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sital Kedia updated SPARK-17509: Affects Version/s: 2.0.0 Description: Profiling a job, we saw that patten matching in

[jira] [Created] (SPARK-17510) Set Streaming MaxRate Independently For Multiple Streams

2016-09-12 Thread Jeff Nadler (JIRA)
Jeff Nadler created SPARK-17510: --- Summary: Set Streaming MaxRate Independently For Multiple Streams Key: SPARK-17510 URL: https://issues.apache.org/jira/browse/SPARK-17510 Project: Spark Issue

[jira] [Commented] (SPARK-2352) [MLLIB] Add Artificial Neural Network (ANN) to Spark

2016-09-12 Thread Alessio (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15485130#comment-15485130 ] Alessio commented on SPARK-2352: Pretty strange that this post with such hype is still "In progress" after

[jira] [Updated] (SPARK-17503) Memory leak in Memory store when unable to cache the whole RDD in memory

2016-09-12 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-17503: --- Target Version/s: 2.0.1, 2.1.0 (was: 2.1.0) > Memory leak in Memory store when unable to cache the

[jira] [Commented] (SPARK-15406) Structured streaming support for consuming from Kafka

2016-09-12 Thread Chris Parmer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15484602#comment-15484602 ] Chris Parmer commented on SPARK-15406: -- For my team, we are just primarily interested in the SQL /

[jira] [Commented] (SPARK-17508) Setting weightCol to None in ML library causes an error

2016-09-12 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15484704#comment-15484704 ] Sean Owen commented on SPARK-17508: --- This looks a lot like the problem solved in SPARK-14931 /

[jira] [Updated] (SPARK-16742) Kerberos support for Spark on Mesos

2016-09-12 Thread Michael Gummelt (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Gummelt updated SPARK-16742: Description: We at Mesosphere have written Kerberos support for Spark on Mesos. We'll be

[jira] [Commented] (SPARK-17471) Add compressed method for Matrix class

2016-09-12 Thread Seth Hendrickson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15484775#comment-15484775 ] Seth Hendrickson commented on SPARK-17471: -- [~yanboliang] Do you have any updates on this? We

[jira] [Commented] (SPARK-17477) SparkSQL cannot handle schema evolution from Int -> Long when parquet files have Int as its type while hive metastore has Long as its type

2016-09-12 Thread Gang Wu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15484629#comment-15484629 ] Gang Wu commented on SPARK-17477: - [~hyukjin.kwon] I agree with you. But both issues are targeting at

[jira] [Created] (SPARK-17508) Setting weightCol to None in ML library causes an error

2016-09-12 Thread Evan Zamir (JIRA)
Evan Zamir created SPARK-17508: -- Summary: Setting weightCol to None in ML library causes an error Key: SPARK-17508 URL: https://issues.apache.org/jira/browse/SPARK-17508 Project: Spark Issue

[jira] [Commented] (SPARK-17463) Serialization of accumulators in heartbeats is not thread-safe

2016-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15484769#comment-15484769 ] Apache Spark commented on SPARK-17463: -- User 'zsxwing' has created a pull request for this issue:

[jira] [Assigned] (SPARK-17463) Serialization of accumulators in heartbeats is not thread-safe

2016-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17463: Assignee: Apache Spark (was: Shixiong Zhu) > Serialization of accumulators in heartbeats

[jira] [Updated] (SPARK-16742) Kerberos support for Spark on Mesos

2016-09-12 Thread Michael Gummelt (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Gummelt updated SPARK-16742: Description: We at Mesosphere have written Kerberos support for Spark on Mesos. We'll be

[jira] [Commented] (SPARK-17463) Serialization of accumulators in heartbeats is not thread-safe

2016-09-12 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15484720#comment-15484720 ] Shixiong Zhu commented on SPARK-17463: -- [~joshrosen] I think we can just leave LongAccum as it is.

[jira] [Updated] (SPARK-16742) Kerberos support for Spark on Mesos

2016-09-12 Thread Michael Gummelt (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Gummelt updated SPARK-16742: Description: We at Mesosphere have written Kerberos support for Spark on Mesos. We'll be

[jira] [Assigned] (SPARK-17463) Serialization of accumulators in heartbeats is not thread-safe

2016-09-12 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-17463: Assignee: Shixiong Zhu > Serialization of accumulators in heartbeats is not thread-safe >

[jira] [Assigned] (SPARK-17509) When wrapping catalyst datatype to Hive data type avoid pattern matching

2016-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17509: Assignee: (was: Apache Spark) > When wrapping catalyst datatype to Hive data type

[jira] [Assigned] (SPARK-17509) When wrapping catalyst datatype to Hive data type avoid pattern matching

2016-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17509: Assignee: Apache Spark > When wrapping catalyst datatype to Hive data type avoid pattern

[jira] [Commented] (SPARK-17509) When wrapping catalyst datatype to Hive data type avoid pattern matching

2016-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15485327#comment-15485327 ] Apache Spark commented on SPARK-17509: -- User 'sitalkedia' has created a pull request for this issue:

[jira] [Assigned] (SPARK-15621) BatchEvalPythonExec fails with OOM

2016-09-12 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu reassigned SPARK-15621: -- Assignee: Davies Liu > BatchEvalPythonExec fails with OOM >

[jira] [Updated] (SPARK-5575) Artificial neural networks for MLlib deep learning

2016-09-12 Thread Alexander Ulanov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Ulanov updated SPARK-5575: Description: *Goal:* Implement various types of artificial neural networks *Motivation:*

[jira] [Commented] (SPARK-17463) Serialization of accumulators in heartbeats is not thread-safe

2016-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15485420#comment-15485420 ] Apache Spark commented on SPARK-17463: -- User 'zsxwing' has created a pull request for this issue:

[jira] [Commented] (SPARK-15406) Structured streaming support for consuming from Kafka

2016-09-12 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15486131#comment-15486131 ] Cody Koeninger commented on SPARK-15406: I've got a minimal working Source and SourceProvider, at

[jira] [Created] (SPARK-17507) check weight vector size in ANN

2016-09-12 Thread Weichen Xu (JIRA)
Weichen Xu created SPARK-17507: -- Summary: check weight vector size in ANN Key: SPARK-17507 URL: https://issues.apache.org/jira/browse/SPARK-17507 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-17477) SparkSQL cannot handle schema evolution from Int -> Long when parquet files have Int as its type while hive metastore has Long as its type

2016-09-12 Thread Gang Wu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gang Wu updated SPARK-17477: Target Version/s: (was: 2.1.0) > SparkSQL cannot handle schema evolution from Int -> Long when parquet

[jira] [Assigned] (SPARK-14818) Move sketch and mllibLocal out from mima exclusion

2016-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-14818: Assignee: Josh Rosen (was: Apache Spark) > Move sketch and mllibLocal out from mima

[jira] [Commented] (SPARK-14818) Move sketch and mllibLocal out from mima exclusion

2016-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15484691#comment-15484691 ] Apache Spark commented on SPARK-14818: -- User 'JoshRosen' has created a pull request for this issue:

[jira] [Commented] (SPARK-17508) Setting weightCol to None in ML library causes an error

2016-09-12 Thread Evan Zamir (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15484850#comment-15484850 ] Evan Zamir commented on SPARK-17508: Yep, I'm running 2.0.0. You can see in the error messages above

  1   2   >