[jira] [Commented] (SPARK-32693) Compare two dataframes with same schema except nullable property

2020-08-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17186859#comment-17186859 ] Apache Spark commented on SPARK-32693: -- User 'viirya' has created a pull request fo

[jira] [Commented] (SPARK-32693) Compare two dataframes with same schema except nullable property

2020-08-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17186857#comment-17186857 ] Apache Spark commented on SPARK-32693: -- User 'viirya' has created a pull request fo

[jira] [Commented] (SPARK-32385) Publish a "bill of materials" (BOM) descriptor for Spark with correct versions of various dependencies

2020-08-28 Thread Vladimir Matveev (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17186855#comment-17186855 ] Vladimir Matveev commented on SPARK-32385: -- > I am still not quite sure what th

[jira] [Commented] (SPARK-32693) Compare two dataframes with same schema except nullable property

2020-08-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17186854#comment-17186854 ] Apache Spark commented on SPARK-32693: -- User 'viirya' has created a pull request fo

[jira] [Commented] (SPARK-32693) Compare two dataframes with same schema except nullable property

2020-08-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17186853#comment-17186853 ] Apache Spark commented on SPARK-32693: -- User 'viirya' has created a pull request fo

[jira] [Updated] (SPARK-19256) Hive bucketing write support

2020-08-28 Thread Cheng Su (Jira)
[ https://issues.apache.org/jira/browse/SPARK-19256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Su updated SPARK-19256: - Affects Version/s: 3.1.0 > Hive bucketing write support > > >

[jira] [Commented] (SPARK-32385) Publish a "bill of materials" (BOM) descriptor for Spark with correct versions of various dependencies

2020-08-28 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17186850#comment-17186850 ] Sean R. Owen commented on SPARK-32385: -- OK, so it's just a different theory of orga

[jira] [Comment Edited] (SPARK-32385) Publish a "bill of materials" (BOM) descriptor for Spark with correct versions of various dependencies

2020-08-28 Thread Vladimir Matveev (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17186846#comment-17186846 ] Vladimir Matveev edited comment on SPARK-32385 at 8/28/20, 11:37 PM: -

[jira] [Commented] (SPARK-32385) Publish a "bill of materials" (BOM) descriptor for Spark with correct versions of various dependencies

2020-08-28 Thread Vladimir Matveev (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17186846#comment-17186846 ] Vladimir Matveev commented on SPARK-32385: -- Sorry for the delayed response! >

[jira] [Created] (SPARK-32732) Convert schema only once in OrcSerializer

2020-08-28 Thread Muhammad Samir Khan (Jira)
Muhammad Samir Khan created SPARK-32732: --- Summary: Convert schema only once in OrcSerializer Key: SPARK-32732 URL: https://issues.apache.org/jira/browse/SPARK-32732 Project: Spark Issue

[jira] [Commented] (SPARK-32731) Add tests for arrays/maps of nested structs to ReadSchemaSuite to test structs reuse

2020-08-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17186826#comment-17186826 ] Apache Spark commented on SPARK-32731: -- User 'msamirkhan' has created a pull reques

[jira] [Commented] (SPARK-32731) Add tests for arrays/maps of nested structs to ReadSchemaSuite to test structs reuse

2020-08-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17186825#comment-17186825 ] Apache Spark commented on SPARK-32731: -- User 'msamirkhan' has created a pull reques

[jira] [Assigned] (SPARK-32731) Add tests for arrays/maps of nested structs to ReadSchemaSuite to test structs reuse

2020-08-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32731: Assignee: (was: Apache Spark) > Add tests for arrays/maps of nested structs to ReadSc

[jira] [Assigned] (SPARK-32731) Add tests for arrays/maps of nested structs to ReadSchemaSuite to test structs reuse

2020-08-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32731: Assignee: Apache Spark > Add tests for arrays/maps of nested structs to ReadSchemaSuite t

[jira] [Updated] (SPARK-32731) Add tests for arrays/maps of nested structs to ReadSchemaSuite to test structs reuse

2020-08-28 Thread Muhammad Samir Khan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Muhammad Samir Khan updated SPARK-32731: Summary: Add tests for arrays/maps of nested structs to ReadSchemaSuite to test st

[jira] [Created] (SPARK-32731) Added tests for arrays/maps of nested structs to ReadSchemaSuite to test structs reuse

2020-08-28 Thread Muhammad Samir Khan (Jira)
Muhammad Samir Khan created SPARK-32731: --- Summary: Added tests for arrays/maps of nested structs to ReadSchemaSuite to test structs reuse Key: SPARK-32731 URL: https://issues.apache.org/jira/browse/SPARK-327

[jira] [Assigned] (SPARK-32730) Improve LeftSemi SortMergeJoin right side buffering

2020-08-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32730: Assignee: Apache Spark > Improve LeftSemi SortMergeJoin right side buffering > --

[jira] [Commented] (SPARK-32730) Improve LeftSemi SortMergeJoin right side buffering

2020-08-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17186791#comment-17186791 ] Apache Spark commented on SPARK-32730: -- User 'peter-toth' has created a pull reques

[jira] [Commented] (SPARK-32730) Improve LeftSemi SortMergeJoin right side buffering

2020-08-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17186792#comment-17186792 ] Apache Spark commented on SPARK-32730: -- User 'peter-toth' has created a pull reques

[jira] [Assigned] (SPARK-32730) Improve LeftSemi SortMergeJoin right side buffering

2020-08-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32730: Assignee: (was: Apache Spark) > Improve LeftSemi SortMergeJoin right side buffering >

[jira] [Created] (SPARK-32730) Improve LeftSemi SortMergeJoin right side buffering

2020-08-28 Thread Peter Toth (Jira)
Peter Toth created SPARK-32730: -- Summary: Improve LeftSemi SortMergeJoin right side buffering Key: SPARK-32730 URL: https://issues.apache.org/jira/browse/SPARK-32730 Project: Spark Issue Type: I

[jira] [Assigned] (SPARK-32639) Support GroupType parquet mapkey field

2020-08-28 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-32639: --- Assignee: Chen Zhang > Support GroupType parquet mapkey field > ---

[jira] [Resolved] (SPARK-32639) Support GroupType parquet mapkey field

2020-08-28 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-32639. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29451 [https://gith

[jira] [Resolved] (SPARK-32704) Logging plan changes for execution

2020-08-28 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-32704. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29544 [https://gith

[jira] [Assigned] (SPARK-32704) Logging plan changes for execution

2020-08-28 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-32704: --- Assignee: Takeshi Yamamuro > Logging plan changes for execution > -

[jira] [Updated] (SPARK-32721) Simplify if clauses with null and boolean

2020-08-28 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-32721: - Description: The following if clause: {code:sql} if(p, null, false) {code} can be simplified to: {code:

[jira] [Updated] (SPARK-32721) Simplify if clauses with null and boolean

2020-08-28 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-32721: - Description: The following if clause: {code:sql} if(p, null, false) {code} can be simplified to: {code:

[jira] [Resolved] (SPARK-32729) Fill up missing since version for math expressions

2020-08-28 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32729. -- Fix Version/s: 3.1.0 Assignee: Kent Yao Resolution: Fixed Fixed in https://git

[jira] [Updated] (SPARK-32729) Fill up missing since version for math expressions

2020-08-28 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-32729: - Description: the mark of since version is absent for more than 40 math expressions, which were added i

[jira] [Commented] (SPARK-32729) Fill up missing since version for math expressions

2020-08-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17186589#comment-17186589 ] Apache Spark commented on SPARK-32729: -- User 'yaooqinn' has created a pull request

[jira] [Assigned] (SPARK-32729) Fill up missing since version for math expressions

2020-08-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32729: Assignee: Apache Spark > Fill up missing since version for math expressions > ---

[jira] [Assigned] (SPARK-32729) Fill up missing since version for math expressions

2020-08-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32729: Assignee: (was: Apache Spark) > Fill up missing since version for math expressions >

[jira] [Created] (SPARK-32729) Fill up missing since version for math expressions

2020-08-28 Thread Kent Yao (Jira)
Kent Yao created SPARK-32729: Summary: Fill up missing since version for math expressions Key: SPARK-32729 URL: https://issues.apache.org/jira/browse/SPARK-32729 Project: Spark Issue Type: Improv

[jira] [Updated] (SPARK-32728) Using groupby with rand creates different values when joining table with itself

2020-08-28 Thread Joachim Bargsten (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joachim Bargsten updated SPARK-32728: - Description: When running following query in a python3 notebook on a cluster with *multi

[jira] [Updated] (SPARK-32728) Using groupby with rand creates different values when joining table with itself

2020-08-28 Thread Joachim Bargsten (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joachim Bargsten updated SPARK-32728: - Environment: I tested it with Azure Databricks 7.2 (& 6.6) (includes Apache Spark 3.0.0

[jira] [Updated] (SPARK-32728) Using groupby with rand creates different values when joining table with itself

2020-08-28 Thread Joachim Bargsten (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joachim Bargsten updated SPARK-32728: - Environment: I tested it with  environment,Azure Databricks 7.2 (& 6.6) (includes Apache

[jira] [Created] (SPARK-32728) Using groupby with rand creates different values when joining table with itself

2020-08-28 Thread Joachim Bargsten (Jira)
Joachim Bargsten created SPARK-32728: Summary: Using groupby with rand creates different values when joining table with itself Key: SPARK-32728 URL: https://issues.apache.org/jira/browse/SPARK-32728

[jira] [Updated] (SPARK-32728) Using groupby with rand creates different values when joining table with itself

2020-08-28 Thread Joachim Bargsten (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joachim Bargsten updated SPARK-32728: - Description: When running following query on a cluster with *multiple workers (>1)*, the

[jira] [Commented] (SPARK-32693) Compare two dataframes with same schema except nullable property

2020-08-28 Thread david bernuau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17186517#comment-17186517 ] david bernuau commented on SPARK-32693: --- thanks > Compare two dataframes with sam

[jira] [Assigned] (SPARK-32717) Add a AQEOptimizer for AdaptiveSparkPlanExec

2020-08-28 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-32717: Assignee: wuyi > Add a AQEOptimizer for AdaptiveSparkPlanExec > -

[jira] [Resolved] (SPARK-32717) Add a AQEOptimizer for AdaptiveSparkPlanExec

2020-08-28 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32717. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29559 [https://gi

[jira] [Assigned] (SPARK-32727) replace CaseWhen with If when there is only one case

2020-08-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32727: Assignee: Apache Spark > replace CaseWhen with If when there is only one case > -

[jira] [Updated] (SPARK-32726) Filter by column alias in where clause

2020-08-28 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-32726: Description: {{group by}} and {{order by}} clause support it. but {{where}} does not support it:

[jira] [Assigned] (SPARK-32727) replace CaseWhen with If when there is only one case

2020-08-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32727: Assignee: (was: Apache Spark) > replace CaseWhen with If when there is only one case

[jira] [Commented] (SPARK-32727) replace CaseWhen with If when there is only one case

2020-08-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17186446#comment-17186446 ] Apache Spark commented on SPARK-32727: -- User 'tanelk' has created a pull request fo

[jira] [Created] (SPARK-32727) replace CaseWhen with If when there is only one case

2020-08-28 Thread Tanel Kiis (Jira)
Tanel Kiis created SPARK-32727: -- Summary: replace CaseWhen with If when there is only one case Key: SPARK-32727 URL: https://issues.apache.org/jira/browse/SPARK-32727 Project: Spark Issue Type:

[jira] [Updated] (SPARK-32723) Security Vulnerability due to JQuery version in Spark Master/Worker UI

2020-08-28 Thread Rohit Mishra (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohit Mishra updated SPARK-32723: - Target Version/s: (was: 3.1.0) > Security Vulnerability due to JQuery version in Spark Master/

[jira] [Created] (SPARK-32726) Filter by column alias in where clause

2020-08-28 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-32726: --- Summary: Filter by column alias in where clause Key: SPARK-32726 URL: https://issues.apache.org/jira/browse/SPARK-32726 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-32723) Security Vulnerability due to JQuery version in Spark Master/Worker UI

2020-08-28 Thread Rohit Mishra (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17186403#comment-17186403 ] Rohit Mishra commented on SPARK-32723: -- [~ashish23aks], Please refrain from marking

[jira] [Created] (SPARK-32725) Getting jar vulnerability for kryo-shaded for 4.0 version

2020-08-28 Thread Wasi (Jira)
Wasi created SPARK-32725: Summary: Getting jar vulnerability for kryo-shaded for 4.0 version Key: SPARK-32725 URL: https://issues.apache.org/jira/browse/SPARK-32725 Project: Spark Issue Type: Depende

[jira] [Created] (SPARK-32724) java.io.IOException: Stream is corrupted when tried to inner join 4 huge tables. Currently using pyspark version 2.4.0-cdh6.3.1

2020-08-28 Thread Kannan (Jira)
Kannan created SPARK-32724: -- Summary: java.io.IOException: Stream is corrupted when tried to inner join 4 huge tables. Currently using pyspark version 2.4.0-cdh6.3.1 Key: SPARK-32724 URL: https://issues.apache.org/jira