[jira] [Created] (SPARK-13757) support quoted column names in schema string at types.py#_parse_datatype_string

2016-03-08 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-13757: --- Summary: support quoted column names in schema string at types.py#_parse_datatype_string Key: SPARK-13757 URL: https://issues.apache.org/jira/browse/SPARK-13757

[jira] [Commented] (SPARK-13758) Error message is misleading when RDD refer to null spark context

2016-03-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186337#comment-15186337 ] Apache Spark commented on SPARK-13758: -- User 'mwws' has created a pull request for this issue:

[jira] [Assigned] (SPARK-13758) Error message is misleading when RDD refer to null spark context

2016-03-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-13758: Assignee: (was: Apache Spark) > Error message is misleading when RDD refer to null

[jira] [Assigned] (SPARK-13758) Error message is misleading when RDD refer to null spark context

2016-03-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-13758: Assignee: Apache Spark > Error message is misleading when RDD refer to null spark context

[jira] [Commented] (SPARK-12719) SQL generation support for generators (including UDTF)

2016-03-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186349#comment-15186349 ] Apache Spark commented on SPARK-12719: -- User 'dilipbiswal' has created a pull request for this

[jira] [Commented] (SPARK-10620) Look into whether accumulator mechanism can replace TaskMetrics

2016-03-08 Thread Liwei Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186379#comment-15186379 ] Liwei Lin commented on SPARK-10620: --- hi [~andrewor14], in the "\[3\] A Simpler Accumulator API" section

[jira] [Updated] (SPARK-13761) Deprecate validateParams

2016-03-08 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-13761: -- Description: Deprecate validateParams() method here:

[jira] [Comment Edited] (SPARK-13523) Reuse the exchanges in a query

2016-03-08 Thread Mark Hamstra (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15185967#comment-15185967 ] Mark Hamstra edited comment on SPARK-13523 at 3/8/16 10:36 PM: --- Yes that is

[jira] [Updated] (SPARK-13702) Use diamond operator for generic instance creation in Java code

2016-03-08 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-13702: -- Description: In order to make docs/example (and other related code) more simple and readable,

[jira] [Updated] (SPARK-13744) Dataframe RDD caching increases the input size for subsequent stages

2016-03-08 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stavros Kontopoulos updated SPARK-13744: Attachment: stages.png > Dataframe RDD caching increases the input size for

[jira] [Commented] (SPARK-13750) Fix sizeInBytes for HadoopFSRelation

2016-03-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186055#comment-15186055 ] Apache Spark commented on SPARK-13750: -- User 'davies' has created a pull request for this issue:

[jira] [Updated] (SPARK-13593) improve the `createDataFrame` method to accept data type string and verify the data

2016-03-08 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated SPARK-13593: - Assignee: Wenchen Fan > improve the `createDataFrame` method to accept data type string and verify >

[jira] [Resolved] (SPARK-13625) PySpark-ML method to get list of params for an obj should not check property attr

2016-03-08 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley resolved SPARK-13625. --- Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 11476

[jira] [Resolved] (SPARK-13754) Keep old data source name for backwards compatibility

2016-03-08 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-13754. -- Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 11589

[jira] [Updated] (SPARK-13754) Keep old data source name for backwards compatibility

2016-03-08 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-13754: - Assignee: Hossein Falaki > Keep old data source name for backwards compatibility >

[jira] [Created] (SPARK-13759) Add IsNotNull constraints for expressions with an inequality

2016-03-08 Thread Sameer Agarwal (JIRA)
Sameer Agarwal created SPARK-13759: -- Summary: Add IsNotNull constraints for expressions with an inequality Key: SPARK-13759 URL: https://issues.apache.org/jira/browse/SPARK-13759 Project: Spark

[jira] [Updated] (SPARK-7146) Should ML sharedParams be a public API?

2016-03-08 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-7146: - Target Version/s: 2.0.0 (was: ) > Should ML sharedParams be a public API? >

[jira] [Comment Edited] (SPARK-13719) Bad JSON record raises java.lang.ClassCastException

2016-03-08 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186276#comment-15186276 ] Hyukjin Kwon edited comment on SPARK-13719 at 3/9/16 1:34 AM: -- [~rxin]

[jira] [Commented] (SPARK-13719) Bad JSON record raises java.lang.ClassCastException

2016-03-08 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186276#comment-15186276 ] Hyukjin Kwon commented on SPARK-13719: -- [~rxin] Actually, shouldn't we maybe need modes such as

[jira] [Resolved] (SPARK-7286) Precedence of operator not behaving properly

2016-03-08 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-7286. Resolution: Fixed Assignee: Jakob Odersky Fix Version/s: 2.0.0 > Precedence of

[jira] [Created] (SPARK-13761) Deprecate validateParams

2016-03-08 Thread Joseph K. Bradley (JIRA)
Joseph K. Bradley created SPARK-13761: - Summary: Deprecate validateParams Key: SPARK-13761 URL: https://issues.apache.org/jira/browse/SPARK-13761 Project: Spark Issue Type: Improvement

[jira] [Assigned] (SPARK-13760) Fix BigDecimal constructor for FloatType

2016-03-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-13760: Assignee: Apache Spark > Fix BigDecimal constructor for FloatType >

[jira] [Assigned] (SPARK-13760) Fix BigDecimal constructor for FloatType

2016-03-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-13760: Assignee: (was: Apache Spark) > Fix BigDecimal constructor for FloatType >

[jira] [Commented] (SPARK-13760) Fix BigDecimal constructor for FloatType

2016-03-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186375#comment-15186375 ] Apache Spark commented on SPARK-13760: -- User 'sameeragarwal' has created a pull request for this

[jira] [Commented] (SPARK-13761) Deprecate validateParams

2016-03-08 Thread yuhao yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186405#comment-15186405 ] yuhao yang commented on SPARK-13761: Hi [~josephkb], do you mind if I work on this? > Deprecate

[jira] [Commented] (SPARK-13660) CommitFailureTestRelationSuite floods the logs with garbage

2016-03-08 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186245#comment-15186245 ] Shixiong Zhu commented on SPARK-13660: -- Just go ahead! > CommitFailureTestRelationSuite floods the

[jira] [Created] (SPARK-13758) Error message is misleading when RDD refer to null spark context

2016-03-08 Thread Mao, Wei (JIRA)
Mao, Wei created SPARK-13758: Summary: Error message is misleading when RDD refer to null spark context Key: SPARK-13758 URL: https://issues.apache.org/jira/browse/SPARK-13758 Project: Spark

[jira] [Resolved] (SPARK-13750) Fix sizeInBytes for HadoopFSRelation

2016-03-08 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-13750. -- Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 11590

[jira] [Assigned] (SPARK-13759) Add IsNotNull constraints for expressions with an inequality

2016-03-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-13759: Assignee: Apache Spark > Add IsNotNull constraints for expressions with an inequality >

[jira] [Commented] (SPARK-13759) Add IsNotNull constraints for expressions with an inequality

2016-03-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186307#comment-15186307 ] Apache Spark commented on SPARK-13759: -- User 'sameeragarwal' has created a pull request for this

[jira] [Assigned] (SPARK-13759) Add IsNotNull constraints for expressions with an inequality

2016-03-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-13759: Assignee: (was: Apache Spark) > Add IsNotNull constraints for expressions with an

[jira] [Created] (SPARK-13760) Fix BigDecimal constructor for FloatType

2016-03-08 Thread Sameer Agarwal (JIRA)
Sameer Agarwal created SPARK-13760: -- Summary: Fix BigDecimal constructor for FloatType Key: SPARK-13760 URL: https://issues.apache.org/jira/browse/SPARK-13760 Project: Spark Issue Type:

[jira] [Created] (SPARK-13762) support column names only in schema string at createDataFrame

2016-03-08 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-13762: --- Summary: support column names only in schema string at createDataFrame Key: SPARK-13762 URL: https://issues.apache.org/jira/browse/SPARK-13762 Project: Spark

[jira] [Updated] (SPARK-13762) support only column names in schema string at createDataFrame

2016-03-08 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-13762: Summary: support only column names in schema string at createDataFrame (was: support column names

[jira] [Comment Edited] (SPARK-13719) Bad JSON record raises java.lang.ClassCastException

2016-03-08 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186276#comment-15186276 ] Hyukjin Kwon edited comment on SPARK-13719 at 3/9/16 1:33 AM: -- [~rxin]

[jira] [Commented] (SPARK-12697) Allow adding new streams without stopping Spark streaming context

2016-03-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186432#comment-15186432 ] Apache Spark commented on SPARK-12697: -- User 'zuowang' has created a pull request for this issue:

[jira] [Assigned] (SPARK-12697) Allow adding new streams without stopping Spark streaming context

2016-03-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12697: Assignee: Apache Spark > Allow adding new streams without stopping Spark streaming

[jira] [Assigned] (SPARK-12697) Allow adding new streams without stopping Spark streaming context

2016-03-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12697: Assignee: (was: Apache Spark) > Allow adding new streams without stopping Spark

[jira] [Commented] (SPARK-13743) Adding configurable support for Spark Streaming gracefull timeout

2016-03-08 Thread Liwei Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186431#comment-15186431 ] Liwei Lin commented on SPARK-13743: --- [~nyuval] thanks for reporting this! Here 1 hour is just the

[jira] [Created] (SPARK-13764) Parse modes in JSON data source

2016-03-08 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-13764: Summary: Parse modes in JSON data source Key: SPARK-13764 URL: https://issues.apache.org/jira/browse/SPARK-13764 Project: Spark Issue Type: New Feature

[jira] [Commented] (SPARK-13719) Bad JSON record raises java.lang.ClassCastException

2016-03-08 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186554#comment-15186554 ] Hyukjin Kwon commented on SPARK-13719: -- I opened a JIRA here, SPARK-13764. Could we maybe make

[jira] [Commented] (SPARK-13764) Parse modes in JSON data source

2016-03-08 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186557#comment-15186557 ] Hyukjin Kwon commented on SPARK-13764: -- I will try to work on this (after looking a bit deeper). >

[jira] [Created] (SPARK-13766) Inconsistent file extensions and omitting file extensions written by CSV, TEXT and JSON data sources

2016-03-08 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-13766: Summary: Inconsistent file extensions and omitting file extensions written by CSV, TEXT and JSON data sources Key: SPARK-13766 URL:

[jira] [Updated] (SPARK-13765) method specialStateTransition(int, IntStream) is exceeding the 65535 bytes limit

2016-03-08 Thread Xin Ren (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xin Ren updated SPARK-13765: Description: Eclipse-Scala IDE is complaining on Java Problem (*please see attached screenshot*), but

[jira] [Commented] (SPARK-13766) Inconsistent file extensions and omitting file extensions written by CSV, TEXT and JSON data sources

2016-03-08 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186576#comment-15186576 ] Hyukjin Kwon commented on SPARK-13766: -- I will work on this. > Inconsistent file extensions and

[jira] [Updated] (SPARK-13766) Inconsistent file extensions and omitted file extensions written by CSV, TEXT and JSON data sources

2016-03-08 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-13766: - Summary: Inconsistent file extensions and omitted file extensions written by CSV, TEXT and JSON

[jira] [Commented] (SPARK-13393) Column mismatch issue in left_outer join using Spark DataFrame

2016-03-08 Thread Adrian Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186652#comment-15186652 ] Adrian Wang commented on SPARK-13393: - In your example, df1("name") and df2("name") is exactly the

[jira] [Comment Edited] (SPARK-13393) Column mismatch issue in left_outer join using Spark DataFrame

2016-03-08 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186679#comment-15186679 ] Xiao Li edited comment on SPARK-13393 at 3/9/16 7:38 AM: - Yeah, I agree. CC

[jira] [Commented] (SPARK-13392) KafkaSink for Metrics

2016-03-08 Thread Liwei Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186445#comment-15186445 ] Liwei Lin commented on SPARK-13392: --- Thanks for working on this! You don't have to be assigned to

[jira] [Commented] (SPARK-13393) Column mismatch issue in left_outer join using Spark DataFrame

2016-03-08 Thread Adrian Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186642#comment-15186642 ] Adrian Wang commented on SPARK-13393: - This is another issue; here we are talking about `varadha` and

[jira] [Created] (SPARK-13768) Set hive conf failed use --hiveconf when beeline connect to thriftserver

2016-03-08 Thread Weizhong (JIRA)
Weizhong created SPARK-13768: Summary: Set hive conf failed use --hiveconf when beeline connect to thriftserver Key: SPARK-13768 URL: https://issues.apache.org/jira/browse/SPARK-13768 Project: Spark

[jira] [Comment Edited] (SPARK-13393) Column mismatch issue in left_outer join using Spark DataFrame

2016-03-08 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186655#comment-15186655 ] Xiao Li edited comment on SPARK-13393 at 3/9/16 7:25 AM: - This is not right. We

[jira] [Commented] (SPARK-13393) Column mismatch issue in left_outer join using Spark DataFrame

2016-03-08 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186655#comment-15186655 ] Xiao Li commented on SPARK-13393: - This is not right. We can resolve it. For example, {code} val

[jira] [Assigned] (SPARK-13763) Remove Project when its projectList is Empty

2016-03-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-13763: Assignee: Apache Spark > Remove Project when its projectList is Empty >

[jira] [Commented] (SPARK-13763) Remove Project when its projectList is Empty

2016-03-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186479#comment-15186479 ] Apache Spark commented on SPARK-13763: -- User 'gatorsmile' has created a pull request for this issue:

[jira] [Updated] (SPARK-13765) method specialStateTransition(int, IntStream) is exceeding the 65535 bytes limit

2016-03-08 Thread Xin Ren (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xin Ren updated SPARK-13765: Attachment: Screen Shot 2016-03-08 at 9.52.48 PM.png > method specialStateTransition(int, IntStream) is

[jira] [Created] (SPARK-13765) method specialStateTransition(int, IntStream) is exceeding the 65535 bytes limit

2016-03-08 Thread Xin Ren (JIRA)
Xin Ren created SPARK-13765: --- Summary: method specialStateTransition(int, IntStream) is exceeding the 65535 bytes limit Key: SPARK-13765 URL: https://issues.apache.org/jira/browse/SPARK-13765 Project:

[jira] [Updated] (SPARK-13765) method specialStateTransition(int, IntStream) is exceeding the 65535 bytes limit

2016-03-08 Thread Xin Ren (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xin Ren updated SPARK-13765: Description: IDE is complaining on Java Problem (attached screenshot) {code} The code of method

[jira] [Commented] (SPARK-13393) Column mismatch issue in left_outer join using Spark DataFrame

2016-03-08 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186601#comment-15186601 ] Xiao Li commented on SPARK-13393: - Hi, [~adrian-wang] You know, this is a well-known issue. The

[jira] [Created] (SPARK-13767) py4j.protocol.Py4JNetworkError: An error occurred while trying to connect to the Java server

2016-03-08 Thread Poonam Agrawal (JIRA)
Poonam Agrawal created SPARK-13767: -- Summary: py4j.protocol.Py4JNetworkError: An error occurred while trying to connect to the Java server Key: SPARK-13767 URL: https://issues.apache.org/jira/browse/SPARK-13767

[jira] [Updated] (SPARK-13767) py4j.protocol.Py4JNetworkError: An error occurred while trying to connect to the Java server

2016-03-08 Thread Poonam Agrawal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Poonam Agrawal updated SPARK-13767: --- Description: I am trying to create spark context object with the following commands on

[jira] [Commented] (SPARK-13600) Use approxQuantile from DataFrame stats in QuantileDiscretizer

2016-03-08 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186648#comment-15186648 ] Nick Pentreath commented on SPARK-13600: Thanks, that's fine > Use approxQuantile from DataFrame

[jira] [Commented] (SPARK-13393) Column mismatch issue in left_outer join using Spark DataFrame

2016-03-08 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186665#comment-15186665 ] Xiao Li commented on SPARK-13393: - Try it. You will see it works > Column mismatch issue in left_outer

[jira] [Comment Edited] (SPARK-13393) Column mismatch issue in left_outer join using Spark DataFrame

2016-03-08 Thread Adrian Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186660#comment-15186660 ] Adrian Wang edited comment on SPARK-13393 at 3/9/16 7:31 AM: - How do you

[jira] [Commented] (SPARK-13393) Column mismatch issue in left_outer join using Spark DataFrame

2016-03-08 Thread Adrian Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186673#comment-15186673 ] Adrian Wang commented on SPARK-13393: - See my updated comment. That's not reasonable. > Column

[jira] [Created] (SPARK-13763) Remove Project when its projectList is Empty

2016-03-08 Thread Xiao Li (JIRA)
Xiao Li created SPARK-13763: --- Summary: Remove Project when its projectList is Empty Key: SPARK-13763 URL: https://issues.apache.org/jira/browse/SPARK-13763 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-13769) Java Doc needs update in SparkSubmit.scala

2016-03-08 Thread Ahmed Kamal (JIRA)
Ahmed Kamal created SPARK-13769: --- Summary: Java Doc needs update in SparkSubmit.scala Key: SPARK-13769 URL: https://issues.apache.org/jira/browse/SPARK-13769 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-13393) Column mismatch issue in left_outer join using Spark DataFrame

2016-03-08 Thread Adrian Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186660#comment-15186660 ] Adrian Wang commented on SPARK-13393: - How do you resolve it? Both sides are `df`, so we can resolve

[jira] [Issue Comment Deleted] (SPARK-13393) Column mismatch issue in left_outer join using Spark DataFrame

2016-03-08 Thread Adrian Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adrian Wang updated SPARK-13393: Comment: was deleted (was: How do you resolve it? Both sides are `df`, so we can resolve df("key")

[jira] [Commented] (SPARK-13393) Column mismatch issue in left_outer join using Spark DataFrame

2016-03-08 Thread Adrian Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186661#comment-15186661 ] Adrian Wang commented on SPARK-13393: - How do you resolve it? Both sides are `df`, so we can resolve

[jira] [Commented] (SPARK-13393) Column mismatch issue in left_outer join using Spark DataFrame

2016-03-08 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186679#comment-15186679 ] Xiao Li commented on SPARK-13393: - Yeah, I agree. CC [~marmbrus][~rxin]. They need to make a decision

[jira] [Created] (SPARK-13770) Document the ML feature Interaction

2016-03-08 Thread Abbass Marouni (JIRA)
Abbass Marouni created SPARK-13770: -- Summary: Document the ML feature Interaction Key: SPARK-13770 URL: https://issues.apache.org/jira/browse/SPARK-13770 Project: Spark Issue Type:

[jira] [Commented] (SPARK-5581) When writing sorted map output file, avoid open / close between each partition

2016-03-08 Thread Sital Kedia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186488#comment-15186488 ] Sital Kedia commented on SPARK-5581: [~joshrosen] - The issue is not only open/close the file output

[jira] [Updated] (SPARK-13766) Inconsistent file extensions and omitting file extensions written by CSV, TEXT and JSON data sources

2016-03-08 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-13766: - Description: Currently, the output (part-files) from CSV, TEXT and JSON data sources do not

[jira] [Updated] (SPARK-13766) Inconsistent file extensions and omitting file extensions written by CSV, TEXT and JSON data sources

2016-03-08 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-13766: - Description: Currently, the output (part-files) from CSV, TEXT and JSON data sources do not

[jira] [Commented] (SPARK-13393) Column mismatch issue in left_outer join using Spark DataFrame

2016-03-08 Thread Adrian Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186605#comment-15186605 ] Adrian Wang commented on SPARK-13393: - So that's the reason I have to introduce the layer of

[jira] [Commented] (SPARK-13393) Column mismatch issue in left_outer join using Spark DataFrame

2016-03-08 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186629#comment-15186629 ] Xiao Li commented on SPARK-13393: - : ) I do not understand how it works. Could you explains it a little

[jira] [Commented] (SPARK-13719) Bad JSON record raises java.lang.ClassCastException

2016-03-08 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186521#comment-15186521 ] Reynold Xin commented on SPARK-13719: - Yes - that would be great if possible. > Bad JSON record

[jira] [Commented] (SPARK-13393) Column mismatch issue in left_outer join using Spark DataFrame

2016-03-08 Thread Adrian Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186580#comment-15186580 ] Adrian Wang commented on SPARK-13393: - Hi [~srinathsmn] In this `errorDF`, both `df('id')` and

[jira] [Commented] (SPARK-13393) Column mismatch issue in left_outer join using Spark DataFrame

2016-03-08 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186644#comment-15186644 ] Xiao Li commented on SPARK-13393: - Fundamentally, they are the same issue. We need a solution for

[jira] [Commented] (SPARK-13393) Column mismatch issue in left_outer join using Spark DataFrame

2016-03-08 Thread Adrian Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186674#comment-15186674 ] Adrian Wang commented on SPARK-13393: - That's the case we should throw exceptions. > Column mismatch

[jira] [Resolved] (SPARK-13593) improve the `createDataFrame` method to accept data type string and verify the data

2016-03-08 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-13593. Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 11444

[jira] [Commented] (SPARK-13756) Reuse Query Fragments

2016-03-08 Thread Mark Hamstra (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15185991#comment-15185991 ] Mark Hamstra commented on SPARK-13756: -- Fragment reuse across queries > Reuse Query Fragments >

[jira] [Updated] (SPARK-13756) Reuse Query Fragments

2016-03-08 Thread Mark Hamstra (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Hamstra updated SPARK-13756: - Description: Query fragments that have been materialized in RDDs can and should be reused either

[jira] [Updated] (SPARK-13485) (Dataset-oriented) API evolution in Spark 2.0

2016-03-08 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-13485: Description: As part of Spark 2.0, we want to create a stable API foundation for Dataset to

[jira] [Commented] (SPARK-12664) Expose raw prediction scores in MultilayerPerceptronClassificationModel

2016-03-08 Thread Gayathri Murali (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186074#comment-15186074 ] Gayathri Murali commented on SPARK-12664: - [~yanboliang] Are you working on this? If not, Can I

[jira] [Commented] (SPARK-13660) CommitFailureTestRelationSuite floods the logs with garbage

2016-03-08 Thread Xin Ren (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186103#comment-15186103 ] Xin Ren commented on SPARK-13660: - Hi Shixiong I'd like to have a try on this one :) >

[jira] [Resolved] (SPARK-13738) Clean up ResolveDataSource

2016-03-08 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-13738. - Resolution: Fixed Fix Version/s: 2.0.0 > Clean up ResolveDataSource >

[jira] [Comment Edited] (SPARK-13744) Dataframe RDD caching increases the input size for subsequent stages

2016-03-08 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186058#comment-15186058 ] Stavros Kontopoulos edited comment on SPARK-13744 at 3/8/16 11:27 PM:

[jira] [Assigned] (SPARK-13578) Make launcher lib and user scripts handle jar directories instead of single assembly file

2016-03-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-13578: Assignee: (was: Apache Spark) > Make launcher lib and user scripts handle jar

[jira] [Commented] (SPARK-13578) Make launcher lib and user scripts handle jar directories instead of single assembly file

2016-03-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186057#comment-15186057 ] Apache Spark commented on SPARK-13578: -- User 'vanzin' has created a pull request for this issue:

[jira] [Commented] (SPARK-13523) Reuse the exchanges in a query

2016-03-08 Thread Mark Hamstra (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15185967#comment-15185967 ] Mark Hamstra commented on SPARK-13523: -- Yes that is a good point. But they are closely enough

[jira] [Commented] (SPARK-7286) Precedence of operator not behaving properly

2016-03-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15185924#comment-15185924 ] Apache Spark commented on SPARK-7286: - User 'jodersky' has created a pull request for this issue:

[jira] [Comment Edited] (SPARK-13523) Reuse the exchanges in a query

2016-03-08 Thread Mark Hamstra (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15185967#comment-15185967 ] Mark Hamstra edited comment on SPARK-13523 at 3/8/16 10:36 PM: --- Yes that is

[jira] [Commented] (SPARK-13744) Dataframe RDD caching increases the input size for subsequent stages

2016-03-08 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186058#comment-15186058 ] Stavros Kontopoulos commented on SPARK-13744: - I understand that, my question is the

[jira] [Assigned] (SPARK-13578) Make launcher lib and user scripts handle jar directories instead of single assembly file

2016-03-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-13578: Assignee: Apache Spark > Make launcher lib and user scripts handle jar directories

[jira] [Assigned] (SPARK-13750) Fix sizeInBytes for HadoopFSRelation

2016-03-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-13750: Assignee: Davies Liu (was: Apache Spark) > Fix sizeInBytes for HadoopFSRelation >

[jira] [Assigned] (SPARK-13750) Fix sizeInBytes for HadoopFSRelation

2016-03-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-13750: Assignee: Apache Spark (was: Davies Liu) > Fix sizeInBytes for HadoopFSRelation >

[jira] [Resolved] (SPARK-13755) Escape quotes in SQL plan visualization node labels

2016-03-08 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-13755. Resolution: Fixed Fix Version/s: 1.6.2 2.0.0 Issue resolved by pull

[jira] [Resolved] (SPARK-13740) add null check for _verify_type in types.py

2016-03-08 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai resolved SPARK-13740. -- Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 11574

  1   2   3   >