[jira] [Created] (SPARK-25908) Remove old deprecated items in Spark 3

2018-11-01 Thread Sean Owen (JIRA)
Sean Owen created SPARK-25908: - Summary: Remove old deprecated items in Spark 3 Key: SPARK-25908 URL: https://issues.apache.org/jira/browse/SPARK-25908 Project: Spark Issue Type: Task

[jira] [Assigned] (SPARK-25908) Remove old deprecated items in Spark 3

2018-11-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25908: Assignee: Sean Owen (was: Apache Spark) > Remove old deprecated items in Spark 3 >

[jira] [Commented] (SPARK-25908) Remove old deprecated items in Spark 3

2018-11-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671666#comment-16671666 ] Apache Spark commented on SPARK-25908: -- User 'srowen' has created a pull request for this issue:

[jira] [Commented] (SPARK-25908) Remove old deprecated items in Spark 3

2018-11-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671668#comment-16671668 ] Apache Spark commented on SPARK-25908: -- User 'srowen' has created a pull request for this issue:

[jira] [Assigned] (SPARK-25908) Remove old deprecated items in Spark 3

2018-11-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25908: Assignee: Apache Spark (was: Sean Owen) > Remove old deprecated items in Spark 3 >

[jira] [Created] (SPARK-25909) Error in documentation: number of cluster managers

2018-11-01 Thread James Lamb (JIRA)
James Lamb created SPARK-25909: -- Summary: Error in documentation: number of cluster managers Key: SPARK-25909 URL: https://issues.apache.org/jira/browse/SPARK-25909 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-25909) Error in documentation: number of cluster managers

2018-11-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25909: Assignee: Apache Spark > Error in documentation: number of cluster managers >

[jira] [Assigned] (SPARK-25909) Error in documentation: number of cluster managers

2018-11-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25909: Assignee: (was: Apache Spark) > Error in documentation: number of cluster managers >

[jira] [Commented] (SPARK-25909) Error in documentation: number of cluster managers

2018-11-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671679#comment-16671679 ] Apache Spark commented on SPARK-25909: -- User 'jameslamb' has created a pull request for this issue:

[jira] [Assigned] (SPARK-25892) AttributeReference.withMetadata method should have return type AttributeReference

2018-11-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25892: Assignee: Apache Spark > AttributeReference.withMetadata method should have return type

[jira] [Assigned] (SPARK-25892) AttributeReference.withMetadata method should have return type AttributeReference

2018-11-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25892: Assignee: (was: Apache Spark) > AttributeReference.withMetadata method should have

[jira] [Commented] (SPARK-25892) AttributeReference.withMetadata method should have return type AttributeReference

2018-11-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671713#comment-16671713 ] Apache Spark commented on SPARK-25892: -- User 'kevinyu98' has created a pull request for this issue:

[jira] [Commented] (SPARK-25892) AttributeReference.withMetadata method should have return type AttributeReference

2018-11-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671714#comment-16671714 ] Apache Spark commented on SPARK-25892: -- User 'kevinyu98' has created a pull request for this issue:

[jira] [Created] (SPARK-25910) accumulator updates from previous stage attempt should not fail

2018-11-01 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-25910: --- Summary: accumulator updates from previous stage attempt should not fail Key: SPARK-25910 URL: https://issues.apache.org/jira/browse/SPARK-25910 Project: Spark

[jira] [Updated] (SPARK-25906) spark-shell cannot handle `-i` option correctly

2018-11-01 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25906: -- Priority: Major (was: Critical) > spark-shell cannot handle `-i` option correctly >

[jira] [Updated] (SPARK-25906) spark-shell cannot handle `-i` option correctly

2018-11-01 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25906: -- Description: This is a regression on Spark 2.4.0. *Spark 2.3.2* {code:java} $ cat test.scala

[jira] [Created] (SPARK-25907) SIGBUS (0xa) when using DataFrameWriter.insertInto

2018-11-01 Thread Alexander Zautke (JIRA)
Alexander Zautke created SPARK-25907: Summary: SIGBUS (0xa) when using DataFrameWriter.insertInto Key: SPARK-25907 URL: https://issues.apache.org/jira/browse/SPARK-25907 Project: Spark

[jira] [Commented] (SPARK-25906) spark-shell cannot handle `-i` option correctly

2018-11-01 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671241#comment-16671241 ] Hyukjin Kwon commented on SPARK-25906: -- Another workaround is to use {{-I}} instead of {{-i}}. >

[jira] [Updated] (SPARK-25907) SIGBUS (0xa) when using DataFrameWriter.insertInto

2018-11-01 Thread Alexander Zautke (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Zautke updated SPARK-25907: - Attachment: hs_err_pid18703.log > SIGBUS (0xa) when using DataFrameWriter.insertInto >

[jira] [Updated] (SPARK-25906) spark-shell cannot handle `-i` option correctly

2018-11-01 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25906: -- Description: This is a regression on Spark 2.4.0. *Spark 2.3.2* {code:java} $ cat test.scala

[jira] [Comment Edited] (SPARK-25906) spark-shell cannot handle `-i` option correctly

2018-11-01 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671264#comment-16671264 ] Hyukjin Kwon edited comment on SPARK-25906 at 11/1/18 8:42 AM: --- The root

[jira] [Commented] (SPARK-25906) spark-shell cannot handle `-i` option correctly

2018-11-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671296#comment-16671296 ] Apache Spark commented on SPARK-25906: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Assigned] (SPARK-25906) spark-shell cannot handle `-i` option correctly

2018-11-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25906: Assignee: (was: Apache Spark) > spark-shell cannot handle `-i` option correctly >

[jira] [Assigned] (SPARK-25906) spark-shell cannot handle `-i` option correctly

2018-11-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25906: Assignee: Apache Spark > spark-shell cannot handle `-i` option correctly >

[jira] [Updated] (SPARK-25906) spark-shell cannot handle `-i` option correctly

2018-11-01 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25906: -- Priority: Critical (was: Major) > spark-shell cannot handle `-i` option correctly >

[jira] [Comment Edited] (SPARK-25906) spark-shell cannot handle `-i` option correctly

2018-11-01 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671161#comment-16671161 ] Dongjoon Hyun edited comment on SPARK-25906 at 11/1/18 6:31 AM: Hi, 

[jira] [Commented] (SPARK-25906) spark-shell cannot handle `-i` option correctly

2018-11-01 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671165#comment-16671165 ] Hyukjin Kwon commented on SPARK-25906: -- Yea, looks this is pretty much related with REPL fix we did

[jira] [Comment Edited] (SPARK-25906) spark-shell cannot handle `-i` option correctly

2018-11-01 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671186#comment-16671186 ] Wenchen Fan edited comment on SPARK-25906 at 11/1/18 7:20 AM: -- thanks for

[jira] [Commented] (SPARK-25906) spark-shell cannot handle `-i` option correctly

2018-11-01 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671186#comment-16671186 ] Wenchen Fan commented on SPARK-25906: - thanks for reporting it! I'll list it as known issus in 2.4.0

[jira] [Commented] (SPARK-25906) spark-shell cannot handle `-i` option correctly

2018-11-01 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671193#comment-16671193 ] Dongjoon Hyun commented on SPARK-25906: --- Thanks, [~cloud_fan]! > spark-shell cannot handle `-i`

[jira] [Created] (SPARK-25906) spark-shell cannot handle `-i` option correctly

2018-11-01 Thread Dongjoon Hyun (JIRA)
Dongjoon Hyun created SPARK-25906: - Summary: spark-shell cannot handle `-i` option correctly Key: SPARK-25906 URL: https://issues.apache.org/jira/browse/SPARK-25906 Project: Spark Issue

[jira] [Commented] (SPARK-25906) spark-shell cannot handle `-i` option correctly

2018-11-01 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671159#comment-16671159 ] Dongjoon Hyun commented on SPARK-25906: --- cc [~cloud_fan] and [~smilegator] . Although this is a

[jira] [Commented] (SPARK-25906) spark-shell cannot handle `-i` option correctly

2018-11-01 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671169#comment-16671169 ] Dongjoon Hyun commented on SPARK-25906: --- Thank you for confirmation. The workaround is to add the

[jira] [Comment Edited] (SPARK-25906) spark-shell cannot handle `-i` option correctly

2018-11-01 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671264#comment-16671264 ] Hyukjin Kwon edited comment on SPARK-25906 at 11/1/18 8:49 AM: --- The root

[jira] [Commented] (SPARK-25906) spark-shell cannot handle `-i` option correctly

2018-11-01 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671161#comment-16671161 ] Dongjoon Hyun commented on SPARK-25906: --- Hi, [~dbtsai] . Could you take a look at this? This

[jira] [Commented] (SPARK-20608) Standby namenodes should be allowed to included in yarn.spark.access.namenodes to support HDFS HA

2018-11-01 Thread Chen Xi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671238#comment-16671238 ] Chen Xi commented on SPARK-20608: - Hi, everyone! I read through your discussion, and tried to use HDFS

[jira] [Comment Edited] (SPARK-25906) spark-shell cannot handle `-i` option correctly

2018-11-01 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671264#comment-16671264 ] Hyukjin Kwon edited comment on SPARK-25906 at 11/1/18 8:37 AM: --- The root

[jira] [Commented] (SPARK-25906) spark-shell cannot handle `-i` option correctly

2018-11-01 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671264#comment-16671264 ] Hyukjin Kwon commented on SPARK-25906: -- The root cause seems to be

[jira] [Updated] (SPARK-25906) spark-shell cannot handle `-i` option correctly

2018-11-01 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-25906: - Component/s: Spark Shell > spark-shell cannot handle `-i` option correctly >

[jira] [Commented] (SPARK-24959) Do not invoke the CSV/JSON parser for empty schema

2018-11-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671553#comment-16671553 ] Apache Spark commented on SPARK-24959: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Updated] (SPARK-24703) Unable to multiply calendar interval

2018-11-01 Thread Priyanka Garg (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyanka Garg updated SPARK-24703: -- Priority: Critical (was: Major) > Unable to multiply calendar interval >

[jira] [Commented] (SPARK-24703) Unable to multiply calendar interval

2018-11-01 Thread Priyanka Garg (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671783#comment-16671783 ] Priyanka Garg commented on SPARK-24703: --- The code changes for this ticket are done. PR is raised.

[jira] [Resolved] (SPARK-25809) Support additional K8S cluster types for integration tests

2018-11-01 Thread Matt Cheah (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Cheah resolved SPARK-25809. Resolution: Fixed > Support additional K8S cluster types for integration tests >

[jira] [Commented] (SPARK-20608) Standby namenodes should be allowed to included in yarn.spark.access.namenodes to support HDFS HA

2018-11-01 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671828#comment-16671828 ] Marcelo Vanzin commented on SPARK-20608: That error doesn't provide a lot of information. Maybe

[jira] [Assigned] (SPARK-25837) Web UI does not respect spark.ui.retainedJobs in some instances

2018-11-01 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25837: -- Assignee: Patrick Brown (was: Apache Spark) > Web UI does not respect

[jira] [Commented] (SPARK-25910) accumulator updates from previous stage attempt should not fail

2018-11-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671812#comment-16671812 ] Apache Spark commented on SPARK-25910: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Assigned] (SPARK-25910) accumulator updates from previous stage attempt should not fail

2018-11-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25910: Assignee: Wenchen Fan (was: Apache Spark) > accumulator updates from previous stage

[jira] [Assigned] (SPARK-25910) accumulator updates from previous stage attempt should not fail

2018-11-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25910: Assignee: Apache Spark (was: Wenchen Fan) > accumulator updates from previous stage

[jira] [Resolved] (SPARK-25837) Web UI does not respect spark.ui.retainedJobs in some instances

2018-11-01 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25837. Resolution: Fixed Fix Version/s: 2.4.1 3.0.0

[jira] [Commented] (SPARK-25906) spark-shell cannot handle `-i` option correctly

2018-11-01 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671665#comment-16671665 ] Dongjoon Hyun commented on SPARK-25906: --- Thank you! [~hyukjin.kwon]. > spark-shell cannot handle

[jira] [Updated] (SPARK-24437) Memory leak in UnsafeHashedRelation

2018-11-01 Thread David Vogelbacher (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Vogelbacher updated SPARK-24437: -- Attachment: Screen Shot 2018-11-01 at 10.38.30 AM.png > Memory leak in

[jira] [Commented] (SPARK-24437) Memory leak in UnsafeHashedRelation

2018-11-01 Thread David Vogelbacher (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671686#comment-16671686 ] David Vogelbacher commented on SPARK-24437: --- Hey [~mgaido], I am seeing something similar in

[jira] [Comment Edited] (SPARK-24437) Memory leak in UnsafeHashedRelation

2018-11-01 Thread David Vogelbacher (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671686#comment-16671686 ] David Vogelbacher edited comment on SPARK-24437 at 11/1/18 2:43 PM:

[jira] [Resolved] (SPARK-25912) Starting history server throwing exception, when enable the local store directory

2018-11-01 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25912. Resolution: Not A Problem > Starting history server throwing exception, when enable the

[jira] [Created] (SPARK-25913) Unary SparkPlan nodes should extend UnaryExecNode

2018-11-01 Thread Maxim Gekk (JIRA)
Maxim Gekk created SPARK-25913: -- Summary: Unary SparkPlan nodes should extend UnaryExecNode Key: SPARK-25913 URL: https://issues.apache.org/jira/browse/SPARK-25913 Project: Spark Issue Type:

[jira] [Commented] (SPARK-25913) Unary SparkPlan nodes should extend UnaryExecNode

2018-11-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16672057#comment-16672057 ] Apache Spark commented on SPARK-25913: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Assigned] (SPARK-25913) Unary SparkPlan nodes should extend UnaryExecNode

2018-11-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25913: Assignee: Apache Spark > Unary SparkPlan nodes should extend UnaryExecNode >

[jira] [Assigned] (SPARK-25913) Unary SparkPlan nodes should extend UnaryExecNode

2018-11-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25913: Assignee: (was: Apache Spark) > Unary SparkPlan nodes should extend UnaryExecNode >

[jira] [Commented] (SPARK-25918) LOAD DATA LOCAL INPATH should handle a relative path

2018-11-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16672159#comment-16672159 ] Apache Spark commented on SPARK-25918: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Assigned] (SPARK-25918) LOAD DATA LOCAL INPATH should handle a relative path

2018-11-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25918: Assignee: (was: Apache Spark) > LOAD DATA LOCAL INPATH should handle a relative path

[jira] [Updated] (SPARK-25914) Separate projection from grouping and aggregate in logical Aggregate

2018-11-01 Thread Maryann Xue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maryann Xue updated SPARK-25914: Description: Currently the Spark SQL logical Aggregate has two expression fields:

[jira] [Assigned] (SPARK-25918) LOAD DATA LOCAL INPATH should handle a relative path

2018-11-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25918: Assignee: Apache Spark > LOAD DATA LOCAL INPATH should handle a relative path >

[jira] [Commented] (SPARK-25918) LOAD DATA LOCAL INPATH should handle a relative path

2018-11-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16672161#comment-16672161 ] Apache Spark commented on SPARK-25918: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Updated] (SPARK-25919) Date value corrupts when tables are "ParquetHiveSerDe" formatted and target table is Partitioned

2018-11-01 Thread Pawan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pawan updated SPARK-25919: -- Description: Hi I found a really strange issue. Below are the steps to reproduce it. This issue occurs only

[jira] [Commented] (SPARK-25912) Starting history server throwing exception, when enable the local store directory

2018-11-01 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16672043#comment-16672043 ] Marcelo Vanzin commented on SPARK-25912: No. As I said, that is how directories work. If you

[jira] [Updated] (SPARK-25914) Separate projection from grouping and aggregate in logical Aggregate

2018-11-01 Thread Maryann Xue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maryann Xue updated SPARK-25914: Description: Currently the Spark SQL logical Aggregate has two expression fields:

[jira] [Created] (SPARK-25917) Spark UI's executors page loads forever when memoryMetrics in None. Fix is to JSON ignore memorymetrics when it is None.

2018-11-01 Thread Rong Tang (JIRA)
Rong Tang created SPARK-25917: - Summary: Spark UI's executors page loads forever when memoryMetrics in None. Fix is to JSON ignore memorymetrics when it is None. Key: SPARK-25917 URL:

[jira] [Created] (SPARK-25921) Python worker reuse causes Barrier tasks to run without BarrierTaskContext

2018-11-01 Thread Bago Amirbekian (JIRA)
Bago Amirbekian created SPARK-25921: --- Summary: Python worker reuse causes Barrier tasks to run without BarrierTaskContext Key: SPARK-25921 URL: https://issues.apache.org/jira/browse/SPARK-25921

[jira] [Created] (SPARK-25916) Add `resultExpressions` in logical `Aggregate`

2018-11-01 Thread Maryann Xue (JIRA)
Maryann Xue created SPARK-25916: --- Summary: Add `resultExpressions` in logical `Aggregate` Key: SPARK-25916 URL: https://issues.apache.org/jira/browse/SPARK-25916 Project: Spark Issue Type:

[jira] [Commented] (SPARK-25921) Python worker reuse causes Barrier tasks to run without BarrierTaskContext

2018-11-01 Thread Bago Amirbekian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16672293#comment-16672293 ] Bago Amirbekian commented on SPARK-25921: - [~mengxr] [~jiangxb1987] Could you have a look. >

[jira] [Commented] (SPARK-25912) Starting history server throwing exception, when enable the local store directory

2018-11-01 Thread shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16672029#comment-16672029 ] shahid commented on SPARK-25912: [~vanzin] okay I agree it is not a bug. Do we need to add in the

[jira] [Created] (SPARK-25914) Separate projection from grouping and aggregate in logical Aggregate

2018-11-01 Thread Maryann Xue (JIRA)
Maryann Xue created SPARK-25914: --- Summary: Separate projection from grouping and aggregate in logical Aggregate Key: SPARK-25914 URL: https://issues.apache.org/jira/browse/SPARK-25914 Project: Spark

[jira] [Assigned] (SPARK-25914) Separate projection from grouping and aggregate in logical Aggregate

2018-11-01 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-25914: --- Assignee: Dilip Biswal > Separate projection from grouping and aggregate in logical Aggregate >

[jira] [Created] (SPARK-25918) LOAD DATA LOCAL INPATH should handle a relative path

2018-11-01 Thread Dongjoon Hyun (JIRA)
Dongjoon Hyun created SPARK-25918: - Summary: LOAD DATA LOCAL INPATH should handle a relative path Key: SPARK-25918 URL: https://issues.apache.org/jira/browse/SPARK-25918 Project: Spark Issue

[jira] [Updated] (SPARK-25921) Python worker reuse causes Barrier tasks to run without BarrierTaskContext

2018-11-01 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-25921: -- Target Version/s: 2.4.1 > Python worker reuse causes Barrier tasks to run without

[jira] [Updated] (SPARK-25921) Python worker reuse causes Barrier tasks to run without BarrierTaskContext

2018-11-01 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-25921: -- Priority: Critical (was: Major) > Python worker reuse causes Barrier tasks to run without

[jira] [Commented] (SPARK-25921) Python worker reuse causes Barrier tasks to run without BarrierTaskContext

2018-11-01 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16672350#comment-16672350 ] Xiangrui Meng commented on SPARK-25921: --- Tested offline. This was caused by PySpark worker reuse.

[jira] [Assigned] (SPARK-25916) Add `resultExpressions` in logical `Aggregate`

2018-11-01 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-25916: --- Assignee: Dilip Biswal (was: Xiao Li) > Add `resultExpressions` in logical `Aggregate` >

[jira] [Issue Comment Deleted] (SPARK-25918) LOAD DATA LOCAL INPATH should handle a relative path

2018-11-01 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25918: -- Comment: was deleted (was: User 'dongjoon-hyun' has created a pull request for this issue:

[jira] [Created] (SPARK-25915) Replace grouping expressions with references in `aggregateExpressions` of logical `Aggregate`

2018-11-01 Thread Maryann Xue (JIRA)
Maryann Xue created SPARK-25915: --- Summary: Replace grouping expressions with references in `aggregateExpressions` of logical `Aggregate` Key: SPARK-25915 URL: https://issues.apache.org/jira/browse/SPARK-25915

[jira] [Assigned] (SPARK-25917) Spark UI's executors page loads forever when memoryMetrics in None. Fix is to JSON ignore memorymetrics when it is None.

2018-11-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25917: Assignee: Apache Spark > Spark UI's executors page loads forever when memoryMetrics in

[jira] [Assigned] (SPARK-25917) Spark UI's executors page loads forever when memoryMetrics in None. Fix is to JSON ignore memorymetrics when it is None.

2018-11-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25917: Assignee: (was: Apache Spark) > Spark UI's executors page loads forever when

[jira] [Commented] (SPARK-25917) Spark UI's executors page loads forever when memoryMetrics in None. Fix is to JSON ignore memorymetrics when it is None.

2018-11-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16672101#comment-16672101 ] Apache Spark commented on SPARK-25917: -- User 'jianjianjiao' has created a pull request for this

[jira] [Created] (SPARK-25919) Date value corrupts when tables are "ParquetHiveSerDe" formatted and target table is Partitioned

2018-11-01 Thread Pawan (JIRA)
Pawan created SPARK-25919: - Summary: Date value corrupts when tables are "ParquetHiveSerDe" formatted and target table is Partitioned Key: SPARK-25919 URL: https://issues.apache.org/jira/browse/SPARK-25919

[jira] [Assigned] (SPARK-25916) Add `resultExpressions` in logical `Aggregate`

2018-11-01 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-25916: --- Assignee: Xiao Li > Add `resultExpressions` in logical `Aggregate` >

[jira] [Assigned] (SPARK-25915) Replace grouping expressions with references in `aggregateExpressions` of logical `Aggregate`

2018-11-01 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-25915: --- Assignee: Dilip Biswal > Replace grouping expressions with references in `aggregateExpressions` of

[jira] [Updated] (SPARK-25918) LOAD DATA LOCAL INPATH should handle a relative path

2018-11-01 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25918: -- Description: Unfortunately, it seems that we missed this in 2.4.0. In Spark 2.4, if the

[jira] [Commented] (SPARK-25901) Barrier mode spawns a bunch of threads that get collected on gc

2018-11-01 Thread yogesh garg (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16672202#comment-16672202 ] yogesh garg commented on SPARK-25901: - [~jiangxb1987] m thanks for approving the PR, can we assign

[jira] [Created] (SPARK-25920) Avoid custom processing of CLI options for cluster submission

2018-11-01 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-25920: -- Summary: Avoid custom processing of CLI options for cluster submission Key: SPARK-25920 URL: https://issues.apache.org/jira/browse/SPARK-25920 Project: Spark

[jira] [Updated] (SPARK-25921) Python worker reuse causes Barrier tasks to run without BarrierTaskContext

2018-11-01 Thread Bago Amirbekian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bago Amirbekian updated SPARK-25921: Description: Running a barrier job after a normal spark job causes the barrier job to run

[jira] [Commented] (SPARK-25925) Spark 2.3.1 retrieves all partitions from Hive Metastore by default

2018-11-01 Thread Alex Ivanov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16672577#comment-16672577 ] Alex Ivanov commented on SPARK-25925: - [~budde], [~michael], can you please share your thoughts? >

[jira] [Updated] (SPARK-25769) UnresolvedAttribute.sql() incorrectly escapes nested columns

2018-11-01 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25769: Labels: (was: sql) > UnresolvedAttribute.sql() incorrectly escapes nested columns >

[jira] [Updated] (SPARK-25921) Python worker reuse causes Barrier tasks to run without BarrierTaskContext

2018-11-01 Thread Bago Amirbekian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bago Amirbekian updated SPARK-25921: Description: Running a barrier job after a normal spark job causes the barrier job to run

[jira] [Updated] (SPARK-25923) SparkR UT Failure (checking CRAN incoming feasibility)

2018-11-01 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25923: -- Description: Currently, the following SparkR error blocks PR builders. {code:java} *

[jira] [Comment Edited] (SPARK-10816) EventTime based sessionization

2018-11-01 Thread Jungtaek Lim (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16672478#comment-16672478 ] Jungtaek Lim edited comment on SPARK-10816 at 11/2/18 2:49 AM: --- UPDATE: I

[jira] [Created] (SPARK-25925) Spark 2.3.1 retrieves all partitions from Hive Metastore by default

2018-11-01 Thread Alex Ivanov (JIRA)
Alex Ivanov created SPARK-25925: --- Summary: Spark 2.3.1 retrieves all partitions from Hive Metastore by default Key: SPARK-25925 URL: https://issues.apache.org/jira/browse/SPARK-25925 Project: Spark

[jira] [Updated] (SPARK-25922) [K8] Spark Driver/Executor "spark-app-selector" label mismatch

2018-11-01 Thread Anmol Khurana (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anmol Khurana updated SPARK-25922: -- Description: Hi, I have been testing Spark 2.4.0 RC4 on Kubernetes  to run Python Spark

[jira] [Updated] (SPARK-25922) [K8] Spark Driver/Executor "spark-app-selector" label mismatch

2018-11-01 Thread Anmol Khurana (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anmol Khurana updated SPARK-25922: -- Description: Hi, I have been testing Spark 2.4.0 RC4 on Kubernetes  to run Python Spark

[jira] [Created] (SPARK-25924) k8s integration tests on jenkins do not save log output

2018-11-01 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-25924: -- Summary: k8s integration tests on jenkins do not save log output Key: SPARK-25924 URL: https://issues.apache.org/jira/browse/SPARK-25924 Project: Spark

[jira] [Commented] (SPARK-10816) EventTime based sessionization

2018-11-01 Thread Jungtaek Lim (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16672478#comment-16672478 ] Jungtaek Lim commented on SPARK-10816: -- UPDATE: I just discovered the performance critical issue on

  1   2   >