[jira] [Commented] (SPARK-31982) Spark sequence doesn't handle date increments that cross DST

2020-06-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17144656#comment-17144656 ] Apache Spark commented on SPARK-31982: -- User 'TJX2014' has created a pull request for this issue:

[jira] [Commented] (SPARK-31982) Spark sequence doesn't handle date increments that cross DST

2020-06-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17144655#comment-17144655 ] Apache Spark commented on SPARK-31982: -- User 'TJX2014' has created a pull request for this issue:

[jira] [Created] (SPARK-32097) Allow reading history log files from multiple directories

2020-06-24 Thread Gaurangi Saxena (Jira)
Gaurangi Saxena created SPARK-32097: --- Summary: Allow reading history log files from multiple directories Key: SPARK-32097 URL: https://issues.apache.org/jira/browse/SPARK-32097 Project: Spark

[jira] [Commented] (SPARK-24615) SPIP: Accelerator-aware task scheduling for Spark

2020-06-24 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17144581#comment-17144581 ] Hyukjin Kwon commented on SPARK-24615: -- [~tgraves], I think we can do. Yes, +1 for separating the

[jira] [Updated] (SPARK-32096) Support top-N sort for Spark SQL rank window function

2020-06-24 Thread Zikun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zikun updated SPARK-32096: -- Description: In Spark SQL, there are two types of sort execution, *_SortExec_* and

[jira] [Updated] (SPARK-32096) Support top-N sort for Spark SQL rank window function

2020-06-24 Thread Zikun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zikun updated SPARK-32096: -- Summary: Support top-N sort for Spark SQL rank window function (was: Support top-N sort for Spark SQL window

[jira] [Created] (SPARK-32096) Support top-N sort for Spark SQL window function

2020-06-24 Thread Zikun (Jira)
Zikun created SPARK-32096: - Summary: Support top-N sort for Spark SQL window function Key: SPARK-32096 URL: https://issues.apache.org/jira/browse/SPARK-32096 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-32095) [DataSource V2] Documentation on SupportsReportStatistics Outdated?

2020-06-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17144399#comment-17144399 ] Apache Spark commented on SPARK-32095: -- User 'emkornfield' has created a pull request for this

[jira] [Assigned] (SPARK-32095) [DataSource V2] Documentation on SupportsReportStatistics Outdated?

2020-06-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32095: Assignee: Apache Spark > [DataSource V2] Documentation on SupportsReportStatistics

[jira] [Assigned] (SPARK-32095) [DataSource V2] Documentation on SupportsReportStatistics Outdated?

2020-06-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32095: Assignee: (was: Apache Spark) > [DataSource V2] Documentation on

[jira] [Updated] (SPARK-32095) [DataSource V2] Documentation on SupportsReportStatistics Outdated?

2020-06-24 Thread Micah Kornfield (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Micah Kornfield updated SPARK-32095: Priority: Minor (was: Major) > [DataSource V2] Documentation on SupportsReportStatistics

[jira] [Created] (SPARK-32095) [DataSource V2] Documentation on SupportsReportStatistics Outdated?

2020-06-24 Thread Micah Kornfield (Jira)
Micah Kornfield created SPARK-32095: --- Summary: [DataSource V2] Documentation on SupportsReportStatistics Outdated? Key: SPARK-32095 URL: https://issues.apache.org/jira/browse/SPARK-32095 Project:

[jira] [Updated] (SPARK-32094) Patch cloudpickle.py with typing module side-effect fix

2020-06-24 Thread Suzen Fylke (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suzen Fylke updated SPARK-32094: Description: Pyspark's cloudpickle.py and versions of cloudpickle below 1.3.0 interfere with

[jira] [Created] (SPARK-32094) Patch cloudpickle.py with typing module side-effect fix

2020-06-24 Thread Suzen Fylke (Jira)
Suzen Fylke created SPARK-32094: --- Summary: Patch cloudpickle.py with typing module side-effect fix Key: SPARK-32094 URL: https://issues.apache.org/jira/browse/SPARK-32094 Project: Spark Issue

[jira] [Commented] (SPARK-24615) SPIP: Accelerator-aware task scheduling for Spark

2020-06-24 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17144194#comment-17144194 ] Thomas Graves commented on SPARK-24615: --- [~mengxr]  [~jiangxb1987]  it would be nice to mark this

[jira] [Resolved] (SPARK-32078) Add a redirect to sql-ref from sql-reference

2020-06-24 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-32078. --- Fix Version/s: 3.1.0 3.0.1 Resolution: Fixed Issue resolved by

[jira] [Updated] (SPARK-31998) Change package references for ArrowBuf

2020-06-24 Thread Bryan Cutler (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler updated SPARK-31998: - Component/s: (was: Spark Core) SQL > Change package references for

[jira] [Updated] (SPARK-31998) Change package references for ArrowBuf

2020-06-24 Thread Bryan Cutler (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler updated SPARK-31998: - Issue Type: Improvement (was: Bug) > Change package references for ArrowBuf >

[jira] [Commented] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

2020-06-24 Thread Min Shen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17144060#comment-17144060 ] Min Shen commented on SPARK-30602: -- Also want to share the production results we have so far. We have

[jira] [Updated] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

2020-06-24 Thread Min Shen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Min Shen updated SPARK-30602: - Attachment: Screen Shot 2020-06-23 at 11.31.22 AM.jpg > SPIP: Support push-based shuffle to improve

[jira] [Updated] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

2020-06-24 Thread Min Shen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Min Shen updated SPARK-30602: - Attachment: (was: Screen Shot 2020-06-17 at 7.01.32 PM.jpg) > SPIP: Support push-based shuffle to

[jira] [Updated] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

2020-06-24 Thread Min Shen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Min Shen updated SPARK-30602: - Attachment: Screen Shot 2020-06-17 at 7.01.32 PM.jpg > SPIP: Support push-based shuffle to improve

[jira] [Updated] (SPARK-32093) Add hadoop-ozone-filesystem jar to ozone profile

2020-06-24 Thread Bharat Viswanadham (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated SPARK-32093: --- Component/s: (was: Spark Core) Build > Add

[jira] [Created] (SPARK-32093) Add hadoop-ozone-filesystem jar to ozone profile

2020-06-24 Thread Bharat Viswanadham (Jira)
Bharat Viswanadham created SPARK-32093: -- Summary: Add hadoop-ozone-filesystem jar to ozone profile Key: SPARK-32093 URL: https://issues.apache.org/jira/browse/SPARK-32093 Project: Spark

[jira] [Commented] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

2020-06-24 Thread Min Shen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17144010#comment-17144010 ] Min Shen commented on SPARK-30602: -- Our paper summarizing the work on this new push-based shuffle was

[jira] [Updated] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

2020-06-24 Thread Min Shen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Min Shen updated SPARK-30602: - Attachment: vldb_2020_magnet_shuffle.pdf > SPIP: Support push-based shuffle to improve shuffle

[jira] [Updated] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

2020-06-24 Thread Min Shen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Min Shen updated SPARK-30602: - Attachment: (was: magnet_shuffle.pdf) > SPIP: Support push-based shuffle to improve shuffle

[jira] [Updated] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

2020-06-24 Thread Min Shen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Min Shen updated SPARK-30602: - Attachment: magnet_shuffle.pdf > SPIP: Support push-based shuffle to improve shuffle efficiency >

[jira] [Updated] (SPARK-32092) CrossvalidatorModel does not save all submodels (it saves only 3)

2020-06-24 Thread An De Rijdt (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] An De Rijdt updated SPARK-32092: Priority: Major (was: Minor) > CrossvalidatorModel does not save all submodels (it saves only 3)

[jira] [Created] (SPARK-32092) CrossvalidatorModel does not save all submodels (it saves only 3)

2020-06-24 Thread An De Rijdt (Jira)
An De Rijdt created SPARK-32092: --- Summary: CrossvalidatorModel does not save all submodels (it saves only 3) Key: SPARK-32092 URL: https://issues.apache.org/jira/browse/SPARK-32092 Project: Spark

[jira] [Assigned] (SPARK-32091) Ignore timeout error when remove blocks on the lost executor

2020-06-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32091: Assignee: Apache Spark > Ignore timeout error when remove blocks on the lost executor >

[jira] [Assigned] (SPARK-32091) Ignore timeout error when remove blocks on the lost executor

2020-06-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32091: Assignee: (was: Apache Spark) > Ignore timeout error when remove blocks on the lost

[jira] [Commented] (SPARK-32091) Ignore timeout error when remove blocks on the lost executor

2020-06-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17143924#comment-17143924 ] Apache Spark commented on SPARK-32091: -- User 'Ngone51' has created a pull request for this issue:

[jira] [Updated] (SPARK-32091) Ignore timeout error when remove blocks on the lost executor

2020-06-24 Thread wuyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyi updated SPARK-32091: - Description: When removing blocks(e.g. RDD, broadcast, shuffle), BlockManagerMaserEndpoint will make RPC calls

[jira] [Created] (SPARK-32091) Ignore timeout error when remove blocks on the lost executor

2020-06-24 Thread wuyi (Jira)
wuyi created SPARK-32091: Summary: Ignore timeout error when remove blocks on the lost executor Key: SPARK-32091 URL: https://issues.apache.org/jira/browse/SPARK-32091 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-32089) Upgrade R version to 4.0.2 in the release DockerFiile

2020-06-24 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-32089. - Fix Version/s: 3.1.0 3.0.1 Resolution: Fixed Issue resolved by pull

[jira] [Assigned] (SPARK-32089) Upgrade R version to 4.0.2 in the release DockerFiile

2020-06-24 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-32089: --- Assignee: Hyukjin Kwon > Upgrade R version to 4.0.2 in the release DockerFiile >

[jira] [Assigned] (SPARK-32087) Allow UserDefinedType to use encoder to deserialize rows in ScalaUDF as well

2020-06-24 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-32087: --- Assignee: wuyi > Allow UserDefinedType to use encoder to deserialize rows in ScalaUDF as

[jira] [Resolved] (SPARK-32087) Allow UserDefinedType to use encoder to deserialize rows in ScalaUDF as well

2020-06-24 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-32087. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 28920

[jira] [Comment Edited] (SPARK-32051) Dataset.foreachPartition returns object

2020-06-24 Thread Frank Oosterhuis (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17143875#comment-17143875 ] Frank Oosterhuis edited comment on SPARK-32051 at 6/24/20, 2:37 PM:

[jira] [Commented] (SPARK-32051) Dataset.foreachPartition returns object

2020-06-24 Thread Frank Oosterhuis (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17143875#comment-17143875 ] Frank Oosterhuis commented on SPARK-32051: -- This is fine:  {code:java} spark.range(100)

[jira] [Assigned] (SPARK-32080) Simplify ArrowColumnVector ListArray accessor

2020-06-24 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-32080: Assignee: Bryan Cutler > Simplify ArrowColumnVector ListArray accessor >

[jira] [Resolved] (SPARK-31998) Change package references for ArrowBuf

2020-06-24 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-31998. -- Fix Version/s: 3.1.0 3.0.1 Resolution: Fixed Issue resolved by pull

[jira] [Assigned] (SPARK-31998) Change package references for ArrowBuf

2020-06-24 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-31998: Assignee: Bryan Cutler > Change package references for ArrowBuf >

[jira] [Resolved] (SPARK-32080) Simplify ArrowColumnVector ListArray accessor

2020-06-24 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32080. -- Fix Version/s: 3.1.0 3.0.1 Resolution: Fixed Issue resolved by pull

[jira] [Assigned] (SPARK-32090) UserDefinedType.equal() does not have symmetry

2020-06-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32090: Assignee: (was: Apache Spark) > UserDefinedType.equal() does not have symmetry >

[jira] [Assigned] (SPARK-32090) UserDefinedType.equal() does not have symmetry

2020-06-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32090: Assignee: Apache Spark > UserDefinedType.equal() does not have symmetry >

[jira] [Commented] (SPARK-32090) UserDefinedType.equal() does not have symmetry

2020-06-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17143832#comment-17143832 ] Apache Spark commented on SPARK-32090: -- User 'Ngone51' has created a pull request for this issue:

[jira] [Created] (SPARK-32090) UserDefinedType.equal() does not have symmetry

2020-06-24 Thread wuyi (Jira)
wuyi created SPARK-32090: Summary: UserDefinedType.equal() does not have symmetry Key: SPARK-32090 URL: https://issues.apache.org/jira/browse/SPARK-32090 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-24615) SPIP: Accelerator-aware task scheduling for Spark

2020-06-24 Thread Chris (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17143799#comment-17143799 ] Chris commented on SPARK-24615: --- This is a great improvement for Spark! Spark can benefit from the power

[jira] [Commented] (SPARK-32089) Upgrade R version to 4.0.2 in the release DockerFiile

2020-06-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17143752#comment-17143752 ] Apache Spark commented on SPARK-32089: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Commented] (SPARK-32089) Upgrade R version to 4.0.2 in the release DockerFiile

2020-06-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17143751#comment-17143751 ] Apache Spark commented on SPARK-32089: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Assigned] (SPARK-32089) Upgrade R version to 4.0.2 in the release DockerFiile

2020-06-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32089: Assignee: Apache Spark > Upgrade R version to 4.0.2 in the release DockerFiile >

[jira] [Assigned] (SPARK-32089) Upgrade R version to 4.0.2 in the release DockerFiile

2020-06-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32089: Assignee: (was: Apache Spark) > Upgrade R version to 4.0.2 in the release

[jira] [Issue Comment Deleted] (SPARK-31918) SparkR CRAN check gives a warning with R 4.0.0 on OSX

2020-06-24 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-31918: - Comment: was deleted (was: User 'HyukjinKwon' has created a pull request for this issue:

[jira] [Commented] (SPARK-31918) SparkR CRAN check gives a warning with R 4.0.0 on OSX

2020-06-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17143749#comment-17143749 ] Apache Spark commented on SPARK-31918: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Created] (SPARK-32089) Upgrade R version to 4.0.2 in the release DockerFiile

2020-06-24 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-32089: Summary: Upgrade R version to 4.0.2 in the release DockerFiile Key: SPARK-32089 URL: https://issues.apache.org/jira/browse/SPARK-32089 Project: Spark Issue

[jira] [Created] (SPARK-32088) test of pyspark.sql.functions.timestamp_seconds failed if non-american timezone setting

2020-06-24 Thread huangtianhua (Jira)
huangtianhua created SPARK-32088: Summary: test of pyspark.sql.functions.timestamp_seconds failed if non-american timezone setting Key: SPARK-32088 URL: https://issues.apache.org/jira/browse/SPARK-32088

[jira] [Commented] (SPARK-32086) RemoveBroadcast RPC failed after executor is shutdown

2020-06-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17143693#comment-17143693 ] Apache Spark commented on SPARK-32086: -- User 'wankunde' has created a pull request for this issue:

[jira] [Assigned] (SPARK-32086) RemoveBroadcast RPC failed after executor is shutdown

2020-06-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32086: Assignee: Apache Spark > RemoveBroadcast RPC failed after executor is shutdown >

[jira] [Assigned] (SPARK-32086) RemoveBroadcast RPC failed after executor is shutdown

2020-06-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32086: Assignee: (was: Apache Spark) > RemoveBroadcast RPC failed after executor is

[jira] [Commented] (SPARK-32086) RemoveBroadcast RPC failed after executor is shutdown

2020-06-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17143692#comment-17143692 ] Apache Spark commented on SPARK-32086: -- User 'wankunde' has created a pull request for this issue:

[jira] [Commented] (SPARK-32087) Allow UserDefinedType to use encoder to deserialize rows in ScalaUDF as well

2020-06-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17143684#comment-17143684 ] Apache Spark commented on SPARK-32087: -- User 'Ngone51' has created a pull request for this issue:

[jira] [Commented] (SPARK-32087) Allow UserDefinedType to use encoder to deserialize rows in ScalaUDF as well

2020-06-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17143682#comment-17143682 ] Apache Spark commented on SPARK-32087: -- User 'Ngone51' has created a pull request for this issue:

[jira] [Assigned] (SPARK-32087) Allow UserDefinedType to use encoder to deserialize rows in ScalaUDF as well

2020-06-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32087: Assignee: Apache Spark > Allow UserDefinedType to use encoder to deserialize rows in

[jira] [Assigned] (SPARK-32087) Allow UserDefinedType to use encoder to deserialize rows in ScalaUDF as well

2020-06-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32087: Assignee: (was: Apache Spark) > Allow UserDefinedType to use encoder to deserialize

[jira] [Updated] (SPARK-32087) Allow UserDefinedType to use encoder to deserialize rows in ScalaUDF as well

2020-06-24 Thread wuyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyi updated SPARK-32087: - Summary: Allow UserDefinedType to use encoder to deserialize rows in ScalaUDF as well (was: Allow

[jira] [Created] (SPARK-32087) Allow UserDenfinedType to use encoder to deserialize rows in ScalaUDF as well

2020-06-24 Thread wuyi (Jira)
wuyi created SPARK-32087: Summary: Allow UserDenfinedType to use encoder to deserialize rows in ScalaUDF as well Key: SPARK-32087 URL: https://issues.apache.org/jira/browse/SPARK-32087 Project: Spark

[jira] [Created] (SPARK-32086) RemoveBroadcast RPC failed after executor is shutdown

2020-06-24 Thread Wan Kun (Jira)
Wan Kun created SPARK-32086: --- Summary: RemoveBroadcast RPC failed after executor is shutdown Key: SPARK-32086 URL: https://issues.apache.org/jira/browse/SPARK-32086 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-32085) Migrate to NumPy documentation style

2020-06-24 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32085: - Description: https://github.com/numpy/numpydoc For example, Before:

[jira] [Created] (SPARK-32085) Migrate to NumPy documentation style

2020-06-24 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-32085: Summary: Migrate to NumPy documentation style Key: SPARK-32085 URL: https://issues.apache.org/jira/browse/SPARK-32085 Project: Spark Issue Type: Umbrella

[jira] [Created] (SPARK-32084) Replace dictionary-based function definitions to proper functions in functions.py

2020-06-24 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-32084: Summary: Replace dictionary-based function definitions to proper functions in functions.py Key: SPARK-32084 URL: https://issues.apache.org/jira/browse/SPARK-32084

[jira] [Commented] (SPARK-32038) Regression in handling NaN values in COUNT(DISTINCT)

2020-06-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17143646#comment-17143646 ] Apache Spark commented on SPARK-32038: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Commented] (SPARK-32038) Regression in handling NaN values in COUNT(DISTINCT)

2020-06-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17143644#comment-17143644 ] Apache Spark commented on SPARK-32038: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Commented] (SPARK-32082) Project Zen: Improving Python usability

2020-06-24 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17143642#comment-17143642 ] Hyukjin Kwon commented on SPARK-32082: -- The JIRA itself is WIP. Some updates and more subtasks are

[jira] [Updated] (SPARK-32082) Project Zen: Improving Python usability

2020-06-24 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32082: - Description: The importance of Python and PySpark has grown radically in the last few years.

[jira] [Resolved] (SPARK-31341) Spark documentation incorrectly claims 3.8 compatibility

2020-06-24 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-31341. -- Resolution: Cannot Reproduce It's fixed in 3.0. > Spark documentation incorrectly claims 3.8

[jira] [Updated] (SPARK-32082) Project Zen: Improving Python usability

2020-06-24 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32082: - Description: The importance of Python and PySpark has grown radically in the last few years.

[jira] [Updated] (SPARK-32082) Project Zen: Improving Python usability

2020-06-24 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32082: - Description: The importance of Python and PySpark has grown radically in the last few years.

[jira] [Updated] (SPARK-32082) Project Zen: Improving Python usability

2020-06-24 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32082: - Description: The importance of Python and PySpark has grown radically in the last few years.

[jira] [Updated] (SPARK-32082) Project Zen: Improving Python usability

2020-06-24 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32082: - Description: The importance of Python and PySpark has grown radically in the last few years.

[jira] [Updated] (SPARK-32082) Project Zen: Improving Python usability

2020-06-24 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32082: - Description: The importance of Python and PySpark has grown radically in the last few years.

[jira] [Commented] (SPARK-32068) Spark 3 UI task launch time show in error time zone

2020-06-24 Thread JinxinTang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17143619#comment-17143619 ] JinxinTang commented on SPARK-32068: Thank you for the issue, hope my PR can help. > Spark 3 UI

[jira] [Assigned] (SPARK-32068) Spark 3 UI task launch time show in error time zone

2020-06-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32068: Assignee: (was: Apache Spark) > Spark 3 UI task launch time show in error time zone

[jira] [Commented] (SPARK-32068) Spark 3 UI task launch time show in error time zone

2020-06-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17143616#comment-17143616 ] Apache Spark commented on SPARK-32068: -- User 'TJX2014' has created a pull request for this issue:

[jira] [Assigned] (SPARK-32068) Spark 3 UI task launch time show in error time zone

2020-06-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32068: Assignee: Apache Spark > Spark 3 UI task launch time show in error time zone >

[jira] [Assigned] (SPARK-31847) DAGSchedulerSuite: Rewrite the test framework to cover most of the existing major features of the Spark Scheduler, mock the necessary part wisely, and make the test fra

2020-06-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31847: Assignee: (was: Apache Spark) > DAGSchedulerSuite: Rewrite the test framework to

[jira] [Commented] (SPARK-31847) DAGSchedulerSuite: Rewrite the test framework to cover most of the existing major features of the Spark Scheduler, mock the necessary part wisely, and make the test fr

2020-06-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17143610#comment-17143610 ] Apache Spark commented on SPARK-31847: -- User 'beliefer' has created a pull request for this issue:

[jira] [Assigned] (SPARK-31847) DAGSchedulerSuite: Rewrite the test framework to cover most of the existing major features of the Spark Scheduler, mock the necessary part wisely, and make the test fra

2020-06-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31847: Assignee: Apache Spark > DAGSchedulerSuite: Rewrite the test framework to cover most of

[jira] [Assigned] (SPARK-32083) Unnecessary tasks are launched when input is empty with AQE

2020-06-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32083: Assignee: Apache Spark > Unnecessary tasks are launched when input is empty with AQE >

[jira] [Assigned] (SPARK-32083) Unnecessary tasks are launched when input is empty with AQE

2020-06-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32083: Assignee: (was: Apache Spark) > Unnecessary tasks are launched when input is empty

[jira] [Commented] (SPARK-32083) Unnecessary tasks are launched when input is empty with AQE

2020-06-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17143608#comment-17143608 ] Apache Spark commented on SPARK-32083: -- User 'manuzhang' has created a pull request for this issue:

[jira] [Commented] (SPARK-30466) remove dependency on jackson-mapper-asl-1.9.13 and jackson-core-asl-1.9.13

2020-06-24 Thread Prashant Sharma (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17143607#comment-17143607 ] Prashant Sharma commented on SPARK-30466: - https://issues.apache.org/jira/browse/HADOOP-15984

[jira] [Created] (SPARK-32083) Unnecessary tasks are launched when input is empty with AQE

2020-06-24 Thread Manu Zhang (Jira)
Manu Zhang created SPARK-32083: -- Summary: Unnecessary tasks are launched when input is empty with AQE Key: SPARK-32083 URL: https://issues.apache.org/jira/browse/SPARK-32083 Project: Spark

[jira] [Assigned] (SPARK-31843) DAGSchedulerSuite: For the pattern of complete + assert, extract the general method

2020-06-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31843: Assignee: (was: Apache Spark) > DAGSchedulerSuite: For the pattern of complete +

[jira] [Commented] (SPARK-31843) DAGSchedulerSuite: For the pattern of complete + assert, extract the general method

2020-06-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17143598#comment-17143598 ] Apache Spark commented on SPARK-31843: -- User 'beliefer' has created a pull request for this issue:

[jira] [Assigned] (SPARK-31843) DAGSchedulerSuite: For the pattern of complete + assert, extract the general method

2020-06-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31843: Assignee: Apache Spark > DAGSchedulerSuite: For the pattern of complete + assert,

[jira] [Commented] (SPARK-31843) DAGSchedulerSuite: For the pattern of complete + assert, extract the general method

2020-06-24 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17143597#comment-17143597 ] Apache Spark commented on SPARK-31843: -- User 'beliefer' has created a pull request for this issue:

[jira] [Updated] (SPARK-32082) Project Zen: Improving Python usability

2020-06-24 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32082: - Epic Name: Project Zen (was: Project Zen: Improving Python usability) > Project Zen: Improving

[jira] [Created] (SPARK-32082) The importance of Python and PySpark has grown radically recently. This ticket targets to improve the usability in PySpark.

2020-06-24 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-32082: Summary: The importance of Python and PySpark has grown radically recently. This ticket targets to improve the usability in PySpark. Key: SPARK-32082 URL:

  1   2   >