[jira] [Updated] (SPARK-35267) nullable field is set to false for integer type when using reflection to get StructType for a case class

2021-04-28 Thread Ganesh Chand (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ganesh Chand updated SPARK-35267: - Description: {code:java} // code placeholder object Util { def toStructType[T](implicit

[jira] [Created] (SPARK-35267) nullable field is set to false for integer type when using reflection to get StructType for a case class

2021-04-28 Thread Ganesh Chand (Jira)
Ganesh Chand created SPARK-35267: Summary: nullable field is set to false for integer type when using reflection to get StructType for a case class Key: SPARK-35267 URL:

[jira] [Resolved] (SPARK-35105) Support multiple paths for ADD FILE/JAR/ARCHIVE commands

2021-04-28 Thread Kousuke Saruta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta resolved SPARK-35105. Fix Version/s: 3.2.0 Resolution: Fixed This issue was resolved in

[jira] [Resolved] (SPARK-35226) JDBC datasources should accept refreshKrb5Config parameter

2021-04-28 Thread Kousuke Saruta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta resolved SPARK-35226. Fix Version/s: 3.2.0 3.1.2 Resolution: Fixed This issue was

[jira] [Commented] (SPARK-35264) Support AQE side broadcastJoin threshold

2021-04-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17335144#comment-17335144 ] Apache Spark commented on SPARK-35264: -- User 'ulysses-you' has created a pull request for this

[jira] [Assigned] (SPARK-35264) Support AQE side broadcastJoin threshold

2021-04-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35264: Assignee: (was: Apache Spark) > Support AQE side broadcastJoin threshold >

[jira] [Assigned] (SPARK-35264) Support AQE side broadcastJoin threshold

2021-04-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35264: Assignee: Apache Spark > Support AQE side broadcastJoin threshold >

[jira] [Commented] (SPARK-35266) Fix an error in BenchmarkBase.scala that occurs when creating a benchmark file in a non-existent directory

2021-04-28 Thread Byungsoo Oh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17335143#comment-17335143 ] Byungsoo Oh commented on SPARK-35266: - I fixed this issue and checked it's working fine. If it's

[jira] [Updated] (SPARK-35264) Support AQE side broadcastJoin threshold

2021-04-28 Thread ulysses you (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ulysses you updated SPARK-35264: Description: The main idea here is that make join config isolation between normal planner and

[jira] [Created] (SPARK-35266) Fix an error in BenchmarkBase.scala that occurs when creating a benchmark file in a non-existent directory

2021-04-28 Thread Byungsoo Oh (Jira)
Byungsoo Oh created SPARK-35266: --- Summary: Fix an error in BenchmarkBase.scala that occurs when creating a benchmark file in a non-existent directory Key: SPARK-35266 URL:

[jira] [Resolved] (SPARK-35135) Duplicate code implementation of `WritablePartitionedIterator`

2021-04-28 Thread wuyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyi resolved SPARK-35135. -- Fix Version/s: 3.2.0 Target Version/s: 3.2.0 Resolution: Fixed Issue resolved by 

[jira] [Assigned] (SPARK-35135) Duplicate code implementation of `WritablePartitionedIterator`

2021-04-28 Thread wuyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyi reassigned SPARK-35135: Assignee: Yang Jie (was: Apache Spark) > Duplicate code implementation of `WritablePartitionedIterator`

[jira] [Updated] (SPARK-35264) Support AQE side broadcastJoin threshold

2021-04-28 Thread ulysses you (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ulysses you updated SPARK-35264: Description: The main idea here is that make join config isolation between normal planner and

[jira] [Commented] (SPARK-34786) read parquet uint64 as decimal

2021-04-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17335110#comment-17335110 ] Apache Spark commented on SPARK-34786: -- User 'yaooqinn' has created a pull request for this issue:

[jira] [Updated] (SPARK-35265) abs return negative

2021-04-28 Thread liuzhenjie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuzhenjie updated SPARK-35265: --- Component/s: (was: Spark Core) PySpark Affects Version/s: 3.1.1

[jira] [Commented] (SPARK-34786) read parquet uint64 as decimal

2021-04-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17335108#comment-17335108 ] Apache Spark commented on SPARK-34786: -- User 'yaooqinn' has created a pull request for this issue:

[jira] [Created] (SPARK-35265) abs return negative

2021-04-28 Thread liuzhenjie (Jira)
liuzhenjie created SPARK-35265: -- Summary: abs return negative Key: SPARK-35265 URL: https://issues.apache.org/jira/browse/SPARK-35265 Project: Spark Issue Type: Bug Components: Spark

[jira] [Updated] (SPARK-35265) abs return negative

2021-04-28 Thread liuzhenjie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuzhenjie updated SPARK-35265: --- Description: from pyspark.sql.functions import lit, abs, concat, hash,col df =

[jira] [Updated] (SPARK-35264) Support AQE side broadcastJoin threshold

2021-04-28 Thread ulysses you (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ulysses you updated SPARK-35264: Description: The main idea here is that make join config isolation between normal planner and

[jira] [Created] (SPARK-35264) Support AQE side broadcastJoin threshold

2021-04-28 Thread ulysses you (Jira)
ulysses you created SPARK-35264: --- Summary: Support AQE side broadcastJoin threshold Key: SPARK-35264 URL: https://issues.apache.org/jira/browse/SPARK-35264 Project: Spark Issue Type:

[jira] [Commented] (SPARK-35227) Replace Bintray with the new repository service for the spark-packages resolver in SparkSubmit

2021-04-28 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17335094#comment-17335094 ] L. C. Hsieh commented on SPARK-35227: - This issue was resolved by

[jira] [Updated] (SPARK-35227) Replace Bintray with the new repository service for the spark-packages resolver in SparkSubmit

2021-04-28 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh updated SPARK-35227: Issue Type: Improvement (was: Task) > Replace Bintray with the new repository service for the

[jira] [Assigned] (SPARK-35227) Replace Bintray with the new repository service for the spark-packages resolver in SparkSubmit

2021-04-28 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh reassigned SPARK-35227: --- Assignee: Bo Zhang > Replace Bintray with the new repository service for the

[jira] [Resolved] (SPARK-35227) Replace Bintray with the new repository service for the spark-packages resolver in SparkSubmit

2021-04-28 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh resolved SPARK-35227. - Resolution: Fixed > Replace Bintray with the new repository service for the spark-packages >

[jira] [Updated] (SPARK-35252) PartitionReaderFactory's Implemention Class of DataSourceV2: sqlConf parameter is null

2021-04-28 Thread lynn (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lynn updated SPARK-35252: - Summary: PartitionReaderFactory's Implemention Class of DataSourceV2: sqlConf parameter is null (was:

[jira] [Commented] (SPARK-34705) Add code-gen for all join types of sort merge join

2021-04-28 Thread Cheng Su (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17335022#comment-17335022 ] Cheng Su commented on SPARK-34705: -- [~advancedxy] - We saw ~10% CPU performance improvement for

[jira] [Commented] (SPARK-35263) Refactor ShuffleBlockFetcherIteratorSuite to reduce duplicated code

2021-04-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17334987#comment-17334987 ] Apache Spark commented on SPARK-35263: -- User 'xkrogen' has created a pull request for this issue:

[jira] [Assigned] (SPARK-35263) Refactor ShuffleBlockFetcherIteratorSuite to reduce duplicated code

2021-04-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35263: Assignee: (was: Apache Spark) > Refactor ShuffleBlockFetcherIteratorSuite to reduce

[jira] [Commented] (SPARK-35263) Refactor ShuffleBlockFetcherIteratorSuite to reduce duplicated code

2021-04-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17334986#comment-17334986 ] Apache Spark commented on SPARK-35263: -- User 'xkrogen' has created a pull request for this issue:

[jira] [Assigned] (SPARK-35263) Refactor ShuffleBlockFetcherIteratorSuite to reduce duplicated code

2021-04-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35263: Assignee: Apache Spark > Refactor ShuffleBlockFetcherIteratorSuite to reduce duplicated

[jira] [Updated] (SPARK-35262) Memory leak when dataset is being persisted

2021-04-28 Thread Igor Amelin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Amelin updated SPARK-35262: Priority: Critical (was: Major) > Memory leak when dataset is being persisted >

[jira] [Created] (SPARK-35263) Refactor ShuffleBlockFetcherIteratorSuite to reduce duplicated code

2021-04-28 Thread Erik Krogen (Jira)
Erik Krogen created SPARK-35263: --- Summary: Refactor ShuffleBlockFetcherIteratorSuite to reduce duplicated code Key: SPARK-35263 URL: https://issues.apache.org/jira/browse/SPARK-35263 Project: Spark

[jira] [Created] (SPARK-35262) Memory leak when dataset is being persisted

2021-04-28 Thread Igor Amelin (Jira)
Igor Amelin created SPARK-35262: --- Summary: Memory leak when dataset is being persisted Key: SPARK-35262 URL: https://issues.apache.org/jira/browse/SPARK-35262 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-35259) ExternalBlockHandler metrics have misleading unit in the name

2021-04-28 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated SPARK-35259: Description: Today {{ExternalBlockHandler}} exposes a few {{Timer}} metrics: {code} // Time

[jira] [Updated] (SPARK-35259) ExternalBlockHandler metrics have misleading unit in the name

2021-04-28 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated SPARK-35259: Description: Today {{ExternalBlockHandler}} exposes a few {{Timer}} metrics: {code} // Time

[jira] [Commented] (SPARK-35259) ExternalBlockHandler metrics have misleading unit in the name

2021-04-28 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17334932#comment-17334932 ] Erik Krogen commented on SPARK-35259: - I have a PR for this but it is based on the PR for

[jira] [Created] (SPARK-35261) Support static invoke for stateless UDF

2021-04-28 Thread Chao Sun (Jira)
Chao Sun created SPARK-35261: Summary: Support static invoke for stateless UDF Key: SPARK-35261 URL: https://issues.apache.org/jira/browse/SPARK-35261 Project: Spark Issue Type: Sub-task

[jira] [Assigned] (SPARK-35258) Enhance ESS ExternalBlockHandler with additional block rate-based metrics and histograms

2021-04-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35258: Assignee: Apache Spark > Enhance ESS ExternalBlockHandler with additional block

[jira] [Assigned] (SPARK-35258) Enhance ESS ExternalBlockHandler with additional block rate-based metrics and histograms

2021-04-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35258: Assignee: (was: Apache Spark) > Enhance ESS ExternalBlockHandler with additional

[jira] [Commented] (SPARK-35258) Enhance ESS ExternalBlockHandler with additional block rate-based metrics and histograms

2021-04-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17334929#comment-17334929 ] Apache Spark commented on SPARK-35258: -- User 'xkrogen' has created a pull request for this issue:

[jira] [Updated] (SPARK-34981) Implement V2 function resolution and evaluation

2021-04-28 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-34981: - Parent: SPARK-35260 Issue Type: Sub-task (was: Improvement) > Implement V2 function resolution

[jira] [Created] (SPARK-35260) DataSourceV2 Function Catalog implementation

2021-04-28 Thread Chao Sun (Jira)
Chao Sun created SPARK-35260: Summary: DataSourceV2 Function Catalog implementation Key: SPARK-35260 URL: https://issues.apache.org/jira/browse/SPARK-35260 Project: Spark Issue Type: Umbrella

[jira] [Commented] (SPARK-35244) invoke should throw the original exception

2021-04-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17334925#comment-17334925 ] Apache Spark commented on SPARK-35244: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Created] (SPARK-35259) ExternalBlockHandler metrics have incorrect unit in the name

2021-04-28 Thread Erik Krogen (Jira)
Erik Krogen created SPARK-35259: --- Summary: ExternalBlockHandler metrics have incorrect unit in the name Key: SPARK-35259 URL: https://issues.apache.org/jira/browse/SPARK-35259 Project: Spark

[jira] [Updated] (SPARK-35259) ExternalBlockHandler metrics have misleading unit in the name

2021-04-28 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated SPARK-35259: Summary: ExternalBlockHandler metrics have misleading unit in the name (was:

[jira] [Created] (SPARK-35258) Enhance ESS ExternalBlockHandler with additional block rate-based metrics and histograms

2021-04-28 Thread Erik Krogen (Jira)
Erik Krogen created SPARK-35258: --- Summary: Enhance ESS ExternalBlockHandler with additional block rate-based metrics and histograms Key: SPARK-35258 URL: https://issues.apache.org/jira/browse/SPARK-35258

[jira] [Commented] (SPARK-34887) Port/integrate Koalas dependencies into PySpark

2021-04-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17334918#comment-17334918 ] Apache Spark commented on SPARK-34887: -- User 'xinrong-databricks' has created a pull request for

[jira] [Assigned] (SPARK-34887) Port/integrate Koalas dependencies into PySpark

2021-04-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34887: Assignee: (was: Apache Spark) > Port/integrate Koalas dependencies into PySpark >

[jira] [Assigned] (SPARK-34887) Port/integrate Koalas dependencies into PySpark

2021-04-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34887: Assignee: Apache Spark > Port/integrate Koalas dependencies into PySpark >

[jira] [Assigned] (SPARK-34887) Port/integrate Koalas dependencies into PySpark

2021-04-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34887: Assignee: (was: Apache Spark) > Port/integrate Koalas dependencies into PySpark >

[jira] [Commented] (SPARK-34887) Port/integrate Koalas dependencies into PySpark

2021-04-28 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17334915#comment-17334915 ] Xinrong Meng commented on SPARK-34887: -- May I work on this ticket? > Port/integrate Koalas

[jira] [Assigned] (SPARK-34981) Implement V2 function resolution and evaluation

2021-04-28 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-34981: --- Assignee: Chao Sun > Implement V2 function resolution and evaluation >

[jira] [Resolved] (SPARK-34981) Implement V2 function resolution and evaluation

2021-04-28 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-34981. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 32082

[jira] [Commented] (SPARK-34705) Add code-gen for all join types of sort merge join

2021-04-28 Thread Xianjin YE (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17334830#comment-17334830 ] Xianjin YE commented on SPARK-34705: [~chengsu] could you share some number of the CPU performance

[jira] [Commented] (SPARK-18188) Add checksum for block of broadcast

2021-04-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-18188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17334826#comment-17334826 ] Apache Spark commented on SPARK-18188: -- User 'Ngone51' has created a pull request for this issue:

[jira] [Commented] (SPARK-35257) Let `HadoopVersionInfoSuite` can use SPARK_VERSIONS_SUITE_IVY_PATH to speed up

2021-04-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17334798#comment-17334798 ] Apache Spark commented on SPARK-35257: -- User 'LuciferYang' has created a pull request for this

[jira] [Assigned] (SPARK-35257) Let `HadoopVersionInfoSuite` can use SPARK_VERSIONS_SUITE_IVY_PATH to speed up

2021-04-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35257: Assignee: Apache Spark > Let `HadoopVersionInfoSuite` can use

[jira] [Commented] (SPARK-35257) Let `HadoopVersionInfoSuite` can use SPARK_VERSIONS_SUITE_IVY_PATH to speed up

2021-04-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17334797#comment-17334797 ] Apache Spark commented on SPARK-35257: -- User 'LuciferYang' has created a pull request for this

[jira] [Assigned] (SPARK-35257) Let `HadoopVersionInfoSuite` can use SPARK_VERSIONS_SUITE_IVY_PATH to speed up

2021-04-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35257: Assignee: (was: Apache Spark) > Let `HadoopVersionInfoSuite` can use

[jira] [Created] (SPARK-35257) Let `HadoopVersionInfoSuite` can use SPARK_VERSIONS_SUITE_IVY_PATH to speed up

2021-04-28 Thread Yang Jie (Jira)
Yang Jie created SPARK-35257: Summary: Let `HadoopVersionInfoSuite` can use SPARK_VERSIONS_SUITE_IVY_PATH to speed up Key: SPARK-35257 URL: https://issues.apache.org/jira/browse/SPARK-35257 Project:

[jira] [Updated] (SPARK-35256) str_to_map + split performance regression

2021-04-28 Thread Ondrej Kokes (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ondrej Kokes updated SPARK-35256: - Description: I'm seeing almost double the runtime between 3.0.1 and 3.1.1 in my pipeline that

[jira] [Created] (SPARK-35256) str_to_map + split performance regression

2021-04-28 Thread Ondrej Kokes (Jira)
Ondrej Kokes created SPARK-35256: Summary: str_to_map + split performance regression Key: SPARK-35256 URL: https://issues.apache.org/jira/browse/SPARK-35256 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-35255) Automated formatting for Scala Code for Blank Lines.

2021-04-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17334738#comment-17334738 ] Apache Spark commented on SPARK-35255: -- User 'lipzhu' has created a pull request for this issue:

[jira] [Assigned] (SPARK-35255) Automated formatting for Scala Code for Blank Lines.

2021-04-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35255: Assignee: (was: Apache Spark) > Automated formatting for Scala Code for Blank Lines.

[jira] [Assigned] (SPARK-35255) Automated formatting for Scala Code for Blank Lines.

2021-04-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35255: Assignee: Apache Spark > Automated formatting for Scala Code for Blank Lines. >

[jira] [Commented] (SPARK-35255) Automated formatting for Scala Code for Blank Lines.

2021-04-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17334736#comment-17334736 ] Apache Spark commented on SPARK-35255: -- User 'lipzhu' has created a pull request for this issue:

[jira] [Created] (SPARK-35255) Automated formatting for Scala Code for Blank Lines.

2021-04-28 Thread Zhu, Lipeng (Jira)
Zhu, Lipeng created SPARK-35255: --- Summary: Automated formatting for Scala Code for Blank Lines. Key: SPARK-35255 URL: https://issues.apache.org/jira/browse/SPARK-35255 Project: Spark Issue

[jira] [Assigned] (SPARK-35254) Upgrade SBT to 1.5.1

2021-04-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35254: Assignee: (was: Apache Spark) > Upgrade SBT to 1.5.1 > > >

[jira] [Commented] (SPARK-35254) Upgrade SBT to 1.5.1

2021-04-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17334687#comment-17334687 ] Apache Spark commented on SPARK-35254: -- User 'lipzhu' has created a pull request for this issue:

[jira] [Assigned] (SPARK-35254) Upgrade SBT to 1.5.1

2021-04-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35254: Assignee: Apache Spark > Upgrade SBT to 1.5.1 > > >

[jira] [Created] (SPARK-35254) Upgrade SBT to 1.5.1

2021-04-28 Thread Zhu, Lipeng (Jira)
Zhu, Lipeng created SPARK-35254: --- Summary: Upgrade SBT to 1.5.1 Key: SPARK-35254 URL: https://issues.apache.org/jira/browse/SPARK-35254 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-35159) extract doc of hive format

2021-04-28 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-35159: - Fix Version/s: 3.1.2 3.0.3 > extract doc of hive format >

[jira] [Commented] (SPARK-35229) Spark Job web page is extremely slow while there are more than 1500 events in timeline

2021-04-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17334633#comment-17334633 ] Apache Spark commented on SPARK-35229: -- User 'sarutak' has created a pull request for this issue:

[jira] [Commented] (SPARK-35229) Spark Job web page is extremely slow while there are more than 1500 events in timeline

2021-04-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17334632#comment-17334632 ] Apache Spark commented on SPARK-35229: -- User 'sarutak' has created a pull request for this issue:

[jira] [Assigned] (SPARK-35229) Spark Job web page is extremely slow while there are more than 1500 events in timeline

2021-04-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35229: Assignee: Apache Spark > Spark Job web page is extremely slow while there are more than

[jira] [Assigned] (SPARK-35229) Spark Job web page is extremely slow while there are more than 1500 events in timeline

2021-04-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35229: Assignee: (was: Apache Spark) > Spark Job web page is extremely slow while there are

[jira] [Commented] (SPARK-34781) Eliminate LEFT SEMI/ANTI join to its left child side with AQE

2021-04-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17334631#comment-17334631 ] Apache Spark commented on SPARK-34781: -- User 'ulysses-you' has created a pull request for this

[jira] [Commented] (SPARK-11844) can not read class org.apache.parquet.format.PageHeader: don't know what type: 13

2021-04-28 Thread Nick Hryhoriev (Jira)
[ https://issues.apache.org/jira/browse/SPARK-11844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17334577#comment-17334577 ] Nick Hryhoriev commented on SPARK-11844: [~Xu_Guang_Lv] My issue is also always reproducible

[jira] [Commented] (SPARK-35159) extract doc of hive format

2021-04-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17334569#comment-17334569 ] Apache Spark commented on SPARK-35159: -- User 'AngersZh' has created a pull request for this

[jira] [Commented] (SPARK-35159) extract doc of hive format

2021-04-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17334567#comment-17334567 ] Apache Spark commented on SPARK-35159: -- User 'AngersZh' has created a pull request for this

[jira] [Commented] (SPARK-35159) extract doc of hive format

2021-04-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17334566#comment-17334566 ] Apache Spark commented on SPARK-35159: -- User 'AngersZh' has created a pull request for this

[jira] [Assigned] (SPARK-35021) Group exception messages in connector/catalog

2021-04-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35021: Assignee: Apache Spark > Group exception messages in connector/catalog >

[jira] [Assigned] (SPARK-35021) Group exception messages in connector/catalog

2021-04-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35021: Assignee: (was: Apache Spark) > Group exception messages in connector/catalog >

[jira] [Commented] (SPARK-35021) Group exception messages in connector/catalog

2021-04-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17334551#comment-17334551 ] Apache Spark commented on SPARK-35021: -- User 'beliefer' has created a pull request for this issue:

[jira] [Issue Comment Deleted] (SPARK-35021) Group exception messages in connector/catalog

2021-04-28 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-35021: --- Comment: was deleted (was: I'm working on.) > Group exception messages in connector/catalog >

[jira] [Resolved] (SPARK-35214) OptimizeSkewedJoin support ShuffledHashJoinExec

2021-04-28 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-35214. -- Fix Version/s: 3.2.0 Assignee: ulysses you Resolution: Fixed Resolved

[jira] [Updated] (SPARK-33976) Add a dedicated SQL document page for the TRANSFORM-related functionality,

2021-04-28 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-33976: - Fix Version/s: 3.1.2 3.0.3 > Add a dedicated SQL document page for

[jira] [Commented] (SPARK-35021) Group exception messages in connector/catalog

2021-04-28 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17334524#comment-17334524 ] jiaan.geng commented on SPARK-35021: I'm working on. > Group exception messages in

[jira] [Commented] (SPARK-35229) Spark Job web page is extremely slow while there are more than 1500 events in timeline

2021-04-28 Thread Kousuke Saruta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17334511#comment-17334511 ] Kousuke Saruta commented on SPARK-35229: [~tiehexue]Thanks for the report. I'll try to mitigate

[jira] [Commented] (SPARK-33976) Add a dedicated SQL document page for the TRANSFORM-related functionality,

2021-04-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17334491#comment-17334491 ] Apache Spark commented on SPARK-33976: -- User 'AngersZh' has created a pull request for this

[jira] [Commented] (SPARK-35253) Upgrade Janino from 3.0.x to 3.1.x

2021-04-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17334488#comment-17334488 ] Apache Spark commented on SPARK-35253: -- User 'LuciferYang' has created a pull request for this

[jira] [Assigned] (SPARK-35253) Upgrade Janino from 3.0.x to 3.1.x

2021-04-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35253: Assignee: (was: Apache Spark) > Upgrade Janino from 3.0.x to 3.1.x >

[jira] [Commented] (SPARK-35253) Upgrade Janino from 3.0.x to 3.1.x

2021-04-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17334486#comment-17334486 ] Apache Spark commented on SPARK-35253: -- User 'LuciferYang' has created a pull request for this

[jira] [Assigned] (SPARK-35253) Upgrade Janino from 3.0.x to 3.1.x

2021-04-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35253: Assignee: Apache Spark > Upgrade Janino from 3.0.x to 3.1.x >

[jira] [Commented] (SPARK-33976) Add a dedicated SQL document page for the TRANSFORM-related functionality,

2021-04-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17334485#comment-17334485 ] Apache Spark commented on SPARK-33976: -- User 'AngersZh' has created a pull request for this

[jira] [Created] (SPARK-35253) Upgrade Janino from 3.0.x to 3.1.x

2021-04-28 Thread Yang Jie (Jira)
Yang Jie created SPARK-35253: Summary: Upgrade Janino from 3.0.x to 3.1.x Key: SPARK-35253 URL: https://issues.apache.org/jira/browse/SPARK-35253 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-35244) invoke should throw the original exception

2021-04-28 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-35244: Fix Version/s: 3.1.2 3.0.3 > invoke should throw the original exception >

[jira] [Resolved] (SPARK-35085) Get columns operation should handle ANSI interval column properly

2021-04-28 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-35085. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 32345

[jira] [Assigned] (SPARK-35085) Get columns operation should handle ANSI interval column properly

2021-04-28 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-35085: Assignee: jiaan.geng > Get columns operation should handle ANSI interval column properly >