[jira] [Commented] (SPARK-31783) Performance test on dense and sparse datasets

2020-05-21 Thread zhengruifeng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112823#comment-17112823 ] zhengruifeng commented on SPARK-31783: -- ping [~srowen] [~mengxr] [~weichenxu123] [~huaxingao] I am

[jira] [Updated] (SPARK-31784) Fix test BarrierTaskContextSuite."share messages with allGather() call"

2020-05-21 Thread wuyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyi updated SPARK-31784: - Environment:   {code:java}  {code}   was:   {code:java} test("share messages with allGather() call") {

[jira] [Created] (SPARK-31784) Fix test BarrierTaskContextSuite."share messages with allGather() call"

2020-05-21 Thread wuyi (Jira)
wuyi created SPARK-31784: Summary: Fix test BarrierTaskContextSuite."share messages with allGather() call" Key: SPARK-31784 URL: https://issues.apache.org/jira/browse/SPARK-31784 Project: Spark

[jira] [Updated] (SPARK-31784) Fix test BarrierTaskContextSuite."share messages with allGather() call"

2020-05-21 Thread wuyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyi updated SPARK-31784: - Description: {code:java} test("share messages with allGather() call") { val conf = new SparkConf()

[jira] [Commented] (SPARK-31783) Performance test on dense and sparse datasets

2020-05-21 Thread zhengruifeng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112850#comment-17112850 ] zhengruifeng commented on SPARK-31783: -- if blockSize==1, original code path dealing with each

[jira] [Updated] (SPARK-31784) Fix test BarrierTaskContextSuite."share messages with allGather() call"

2020-05-21 Thread wuyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyi updated SPARK-31784: - Description: {code:java} test("share messages with allGather() call") { val conf = new SparkConf()

[jira] [Updated] (SPARK-31784) Fix test BarrierTaskContextSuite."share messages with allGather() call"

2020-05-21 Thread wuyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyi updated SPARK-31784: - Description: {code:java} test("share messages with allGather() call") { val conf = new SparkConf()

[jira] [Updated] (SPARK-31784) Fix test BarrierTaskContextSuite."share messages with allGather() call"

2020-05-21 Thread wuyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyi updated SPARK-31784: - Description: {code:java} test("share messages with allGather() call") { val conf = new SparkConf()

[jira] [Assigned] (SPARK-31784) Fix test BarrierTaskContextSuite."share messages with allGather() call"

2020-05-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31784: Assignee: Apache Spark > Fix test BarrierTaskContextSuite."share messages with

[jira] [Assigned] (SPARK-31784) Fix test BarrierTaskContextSuite."share messages with allGather() call"

2020-05-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31784: Assignee: (was: Apache Spark) > Fix test BarrierTaskContextSuite."share messages

[jira] [Commented] (SPARK-31761) Sql Div operator can result in incorrect output for int_min

2020-05-21 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112924#comment-17112924 ] Hyukjin Kwon commented on SPARK-31761: -- [~sandeep.katta2007] feel free to open a followup PR

[jira] [Commented] (SPARK-31761) Sql Div operator can result in incorrect output for int_min

2020-05-21 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112918#comment-17112918 ] Hyukjin Kwon commented on SPARK-31761: -- If it overflows, looks like it just should be guided by

[jira] [Commented] (SPARK-31784) Fix test BarrierTaskContextSuite."share messages with allGather() call"

2020-05-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112913#comment-17112913 ] Apache Spark commented on SPARK-31784: -- User 'Ngone51' has created a pull request for this issue:

[jira] [Commented] (SPARK-31448) Difference in Storage Levels used in cache() and persist() for pyspark dataframes

2020-05-21 Thread Abhishek Dixit (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113035#comment-17113035 ] Abhishek Dixit commented on SPARK-31448: [~tianshi] [~hyukjin.kwon] Any update on this? >

[jira] [Commented] (SPARK-31785) Add a helper function to test all parquet readers

2020-05-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113113#comment-17113113 ] Apache Spark commented on SPARK-31785: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Assigned] (SPARK-31785) Add a helper function to test all parquet readers

2020-05-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31785: Assignee: Apache Spark > Add a helper function to test all parquet readers >

[jira] [Commented] (SPARK-31785) Add a helper function to test all parquet readers

2020-05-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113110#comment-17113110 ] Apache Spark commented on SPARK-31785: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Assigned] (SPARK-31785) Add a helper function to test all parquet readers

2020-05-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31785: Assignee: (was: Apache Spark) > Add a helper function to test all parquet readers >

[jira] [Created] (SPARK-31785) Add a helper function to test all parquet readers

2020-05-21 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-31785: -- Summary: Add a helper function to test all parquet readers Key: SPARK-31785 URL: https://issues.apache.org/jira/browse/SPARK-31785 Project: Spark Issue Type:

[jira] [Updated] (SPARK-31788) Cannot convert org.apache.spark.api.java.JavaPairRDD to org.apache.spark.api.java.JavaRDD

2020-05-21 Thread Sanket Reddy (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sanket Reddy updated SPARK-31788: - Description: Union RDD of Pair RDD's seems to have issues SparkSession available as 'spark'.

[jira] [Updated] (SPARK-31788) Error when creating UnionRDD of PairRDDs

2020-05-21 Thread Sanket Reddy (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sanket Reddy updated SPARK-31788: - Summary: Error when creating UnionRDD of PairRDDs (was: Cannot convert

[jira] [Commented] (SPARK-31765) Upgrade HtmlUnit >= 2.37.0

2020-05-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113610#comment-17113610 ] Apache Spark commented on SPARK-31765: -- User 'gengliangwang' has created a pull request for this

[jira] [Commented] (SPARK-31765) Upgrade HtmlUnit >= 2.37.0

2020-05-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113609#comment-17113609 ] Apache Spark commented on SPARK-31765: -- User 'gengliangwang' has created a pull request for this

[jira] [Reopened] (SPARK-31765) Upgrade HtmlUnit >= 2.37.0

2020-05-21 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang reopened SPARK-31765: > Upgrade HtmlUnit >= 2.37.0 > -- > > Key:

[jira] [Updated] (SPARK-31786) Exception on submitting Spark-Pi to Kubernetes 1.17.3

2020-05-21 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31786: -- Target Version/s: 2.4.6, 3.0.0 > Exception on submitting Spark-Pi to Kubernetes 1.17.3 >

[jira] [Created] (SPARK-31790) cast scenarios may generate different results between Hive and Spark

2020-05-21 Thread philipse (Jira)
philipse created SPARK-31790: Summary: cast scenarios may generate different results between Hive and Spark Key: SPARK-31790 URL: https://issues.apache.org/jira/browse/SPARK-31790 Project: Spark

[jira] [Assigned] (SPARK-31788) Error when creating UnionRDD of PairRDDs

2020-05-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31788: Assignee: (was: Apache Spark) > Error when creating UnionRDD of PairRDDs >

[jira] [Assigned] (SPARK-31788) Error when creating UnionRDD of PairRDDs

2020-05-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31788: Assignee: Apache Spark > Error when creating UnionRDD of PairRDDs >

[jira] [Commented] (SPARK-31788) Error when creating UnionRDD of PairRDDs

2020-05-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113625#comment-17113625 ] Apache Spark commented on SPARK-31788: -- User 'redsanket' has created a pull request for this issue:

[jira] [Commented] (SPARK-31788) Error when creating UnionRDD of PairRDDs

2020-05-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113626#comment-17113626 ] Apache Spark commented on SPARK-31788: -- User 'redsanket' has created a pull request for this issue:

[jira] [Resolved] (SPARK-31785) Add a helper function to test all parquet readers

2020-05-21 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-31785. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 28598

[jira] [Assigned] (SPARK-31785) Add a helper function to test all parquet readers

2020-05-21 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-31785: Assignee: Maxim Gekk > Add a helper function to test all parquet readers >

[jira] [Assigned] (SPARK-31765) Upgrade HtmlUnit >= 2.37.0

2020-05-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31765: Assignee: Apache Spark (was: Kousuke Saruta) > Upgrade HtmlUnit >= 2.37.0 >

[jira] [Assigned] (SPARK-31765) Upgrade HtmlUnit >= 2.37.0

2020-05-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31765: Assignee: Kousuke Saruta (was: Apache Spark) > Upgrade HtmlUnit >= 2.37.0 >

[jira] [Assigned] (SPARK-31765) Upgrade HtmlUnit >= 2.37.0

2020-05-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31765: Assignee: Kousuke Saruta (was: Apache Spark) > Upgrade HtmlUnit >= 2.37.0 >

[jira] [Commented] (SPARK-31788) Error when creating UnionRDD of PairRDDs

2020-05-21 Thread Sanket Reddy (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113623#comment-17113623 ] Sanket Reddy commented on SPARK-31788: -- Took a naive dig at it 

[jira] [Commented] (SPARK-29854) lpad and rpad built in function not throw Exception for invalid len value

2020-05-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113644#comment-17113644 ] Apache Spark commented on SPARK-29854: -- User 'maropu' has created a pull request for this issue:

[jira] [Resolved] (SPARK-31049) Support nested adjacent generators, e.g., explode(explode(v))

2020-05-21 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-31049. -- Resolution: Won't Fix > Support nested adjacent generators, e.g., explode(explode(v))

[jira] [Assigned] (SPARK-31790) cast scenarios may generate different results between Hive and Spark

2020-05-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31790: Assignee: Apache Spark > cast scenarios may generate different results between Hive and

[jira] [Commented] (SPARK-31790) cast scenarios may generate different results between Hive and Spark

2020-05-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113715#comment-17113715 ] Apache Spark commented on SPARK-31790: -- User 'GuoPhilipse' has created a pull request for this

[jira] [Assigned] (SPARK-31790) cast scenarios may generate different results between Hive and Spark

2020-05-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31790: Assignee: (was: Apache Spark) > cast scenarios may generate different results

[jira] [Comment Edited] (SPARK-31786) Exception on submitting Spark-Pi to Kubernetes 1.17.3

2020-05-21 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113611#comment-17113611 ] Dongjoon Hyun edited comment on SPARK-31786 at 5/21/20, 11:13 PM: -- I

[jira] [Commented] (SPARK-31786) Exception on submitting Spark-Pi to Kubernetes 1.17.3

2020-05-21 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113613#comment-17113613 ] Dongjoon Hyun commented on SPARK-31786: --- [~holden] and [~dbtsai]. I raised this issue as a blocker

[jira] [Commented] (SPARK-31761) Sql Div operator can result in incorrect output for int_min

2020-05-21 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113706#comment-17113706 ] Jungtaek Lim commented on SPARK-31761: -- Let's make sure priority is marked properly - sounds like

[jira] [Assigned] (SPARK-31761) Sql Div operator can result in incorrect output for int_min

2020-05-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31761: Assignee: (was: Apache Spark) > Sql Div operator can result in incorrect output for

[jira] [Commented] (SPARK-31761) Sql Div operator can result in incorrect output for int_min

2020-05-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113523#comment-17113523 ] Apache Spark commented on SPARK-31761: -- User 'sandeep-katta' has created a pull request for this

[jira] [Commented] (SPARK-31788) Error when creating UnionRDD of PairRDDs

2020-05-21 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113574#comment-17113574 ] Thomas Graves commented on SPARK-31788: --- [~sanket991]the link you provided is not to public Apache

[jira] [Created] (SPARK-31789) SparkSubmitOperator could not get Exit Code after log stream interrupted by k8s old resource version execption

2020-05-21 Thread Dylan Yao (Jira)
Dylan Yao created SPARK-31789: - Summary: SparkSubmitOperator could not get Exit Code after log stream interrupted by k8s old resource version execption Key: SPARK-31789 URL:

[jira] [Comment Edited] (SPARK-31761) Sql Div operator can result in incorrect output for int_min

2020-05-21 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113706#comment-17113706 ] Jungtaek Lim edited comment on SPARK-31761 at 5/22/20, 2:58 AM: Let's

[jira] [Commented] (SPARK-31786) Exception on submitting Spark-Pi to Kubernetes 1.17.3

2020-05-21 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113575#comment-17113575 ] Dongjoon Hyun commented on SPARK-31786: --- Thank you for reporting, [~maver1ck]. > Exception on

[jira] [Assigned] (SPARK-31786) Exception on submitting Spark-Pi to Kubernetes 1.17.3

2020-05-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31786: Assignee: Apache Spark > Exception on submitting Spark-Pi to Kubernetes 1.17.3 >

[jira] [Assigned] (SPARK-31786) Exception on submitting Spark-Pi to Kubernetes 1.17.3

2020-05-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31786: Assignee: (was: Apache Spark) > Exception on submitting Spark-Pi to Kubernetes

[jira] [Commented] (SPARK-31786) Exception on submitting Spark-Pi to Kubernetes 1.17.3

2020-05-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113587#comment-17113587 ] Apache Spark commented on SPARK-31786: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Commented] (SPARK-31786) Exception on submitting Spark-Pi to Kubernetes 1.17.3

2020-05-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113588#comment-17113588 ] Apache Spark commented on SPARK-31786: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Updated] (SPARK-31786) Exception on submitting Spark-Pi to Kubernetes 1.17.3

2020-05-21 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31786: -- Priority: Blocker (was: Major) > Exception on submitting Spark-Pi to Kubernetes 1.17.3 >

[jira] [Updated] (SPARK-31786) Exception on submitting Spark-Pi to Kubernetes 1.17.3

2020-05-21 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31786: -- Affects Version/s: 3.0.0 > Exception on submitting Spark-Pi to Kubernetes 1.17.3 >

[jira] [Commented] (SPARK-31786) Exception on submitting Spark-Pi to Kubernetes 1.17.3

2020-05-21 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113611#comment-17113611 ] Dongjoon Hyun commented on SPARK-31786: --- I also verified that Apache Spark 3.0.0-RC2 fails, too.

[jira] [Updated] (SPARK-31789) SparkSubmitOperator could not get Exit Code after log stream interrupted by k8s old resource version execption

2020-05-21 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-31789: - Priority: Major (was: Blocker) > SparkSubmitOperator could not get Exit Code after log stream

[jira] [Commented] (SPARK-31789) SparkSubmitOperator could not get Exit Code after log stream interrupted by k8s old resource version execption

2020-05-21 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113708#comment-17113708 ] Jungtaek Lim commented on SPARK-31789: -- critical / blocker are tend to be reserved for committers.

[jira] [Commented] (SPARK-31788) Error when creating UnionRDD of PairRDDs

2020-05-21 Thread Sanket Reddy (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113566#comment-17113566 ] Sanket Reddy commented on SPARK-31788: --

[jira] [Commented] (SPARK-31787) Fix Minikube.getIfNewMinikubeStatus to understand 1.5+

2020-05-21 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113537#comment-17113537 ] Dongjoon Hyun commented on SPARK-31787: --- This is merged to `branch-2.4` with the original

[jira] [Resolved] (SPARK-31787) Fix Minikube.getIfNewMinikubeStatus to understand 1.5+

2020-05-21 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-31787. --- Fix Version/s: 2.4.6 Assignee: Marcelo Masiero Vanzin Resolution: Fixed >

[jira] [Commented] (SPARK-31786) Exception on submitting Spark-Pi to Kubernetes 1.17.3

2020-05-21 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113595#comment-17113595 ] Dongjoon Hyun commented on SPARK-31786: --- BTW, is there any chance for you to test Apache Spark 3.0

[jira] [Comment Edited] (SPARK-31788) Error when creating UnionRDD of PairRDDs

2020-05-21 Thread Sanket Reddy (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113566#comment-17113566 ] Sanket Reddy edited comment on SPARK-31788 at 5/21/20, 11:37 PM: -

[jira] [Updated] (SPARK-31788) Error when creating UnionRDD of PairRDDs

2020-05-21 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-31788: -- Priority: Blocker (was: Major) > Error when creating UnionRDD of PairRDDs >

[jira] [Updated] (SPARK-31387) HiveThriftServer2Listener update methods fail with unknown operation/session id

2020-05-21 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31387: -- Fix Version/s: (was: 3.0.0) 3.1.0 > HiveThriftServer2Listener update

[jira] [Commented] (SPARK-31696) Support spark.kubernetes.driver.service.annotation

2020-05-21 Thread Canbin Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113729#comment-17113729 ] Canbin Zheng commented on SPARK-31696: -- Hi [~dongjoon]! Are there some scenarios that the users

[jira] [Commented] (SPARK-31555) Improve cache block migration

2020-05-21 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113736#comment-17113736 ] Holden Karau commented on SPARK-31555: -- Rocking. We're running into an issue in master with the

[jira] [Created] (SPARK-31791) Improve cache block migration test reliability

2020-05-21 Thread Holden Karau (Jira)
Holden Karau created SPARK-31791: Summary: Improve cache block migration test reliability Key: SPARK-31791 URL: https://issues.apache.org/jira/browse/SPARK-31791 Project: Spark Issue Type:

[jira] [Updated] (SPARK-31790) cast scenarios may generate different results between Hive and Spark

2020-05-21 Thread philipse (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] philipse updated SPARK-31790: - Description: `CAST(n,TIMESTAMPTYPE)` If n is Byte/Short/Int/Long data type, Hive treat n as

[jira] [Updated] (SPARK-31761) Sql Div operator can result in incorrect output for int_min

2020-05-21 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-31761: Priority: Blocker (was: Major) > Sql Div operator can result in incorrect output for int_min >

[jira] [Commented] (SPARK-31761) Sql Div operator can result in incorrect output for int_min

2020-05-21 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113728#comment-17113728 ] Wenchen Fan commented on SPARK-31761: - I've set it as blocker. > Sql Div operator can result in

[jira] [Commented] (SPARK-31786) Exception on submitting Spark-Pi to Kubernetes 1.17.3

2020-05-21 Thread Canbin Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113724#comment-17113724 ] Canbin Zheng commented on SPARK-31786: -- It seems the same issue as

[jira] [Resolved] (SPARK-29303) UI updates for stage level scheduling

2020-05-21 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-29303. --- Fix Version/s: 3.1.0 Assignee: Thomas Graves Resolution: Fixed > UI updates

[jira] [Updated] (SPARK-31788) Cannot convert org.apache.spark.api.java.JavaPairRDD to org.apache.spark.api.java.JavaRDD

2020-05-21 Thread Sanket Reddy (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sanket Reddy updated SPARK-31788: - Description: Pair RDD conversion seems to have issues SparkSession available as 'spark'. >>>

[jira] [Commented] (SPARK-16323) Avoid unnecessary cast when doing integral divide

2020-05-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-16323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113522#comment-17113522 ] Apache Spark commented on SPARK-16323: -- User 'sandeep-katta' has created a pull request for this

[jira] [Commented] (SPARK-31761) Sql Div operator can result in incorrect output for int_min

2020-05-21 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113309#comment-17113309 ] Wenchen Fan commented on SPARK-31761: - [~sandeep.katta2007] are you interested at fixing it? > Sql

[jira] [Commented] (SPARK-31761) Sql Div operator can result in incorrect output for int_min

2020-05-21 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113308#comment-17113308 ] Wenchen Fan commented on SPARK-31761: - I think this is a breaking change and we should fix it. The

[jira] [Updated] (SPARK-31763) DataFrame.inputFiles() not Available

2020-05-21 Thread Felix Kizhakkel Jose (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Kizhakkel Jose updated SPARK-31763: - Issue Type: Bug (was: New Feature) > DataFrame.inputFiles() not Available >

[jira] [Resolved] (SPARK-31765) Upgrade HtmlUnit >= 2.37.0

2020-05-21 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-31765. Resolution: Fixed > Upgrade HtmlUnit >= 2.37.0 > -- > >

[jira] [Commented] (SPARK-31765) Upgrade HtmlUnit >= 2.37.0

2020-05-21 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113460#comment-17113460 ] Gengliang Wang commented on SPARK-31765: This issue is resolved in

[jira] [Updated] (SPARK-31787) Fix Minikube.getIfNewMinikubeStatus to understand 1.5+

2020-05-21 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31787: -- Summary: Fix Minikube.getIfNewMinikubeStatus to understand 1.5+ (was: Support Minikube 1.5.x

[jira] [Updated] (SPARK-31688) Refactor pagination framework for spark web UI pages

2020-05-21 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-31688: - Issue Type: Improvement (was: Bug) > Refactor pagination framework for spark web UI pages >

[jira] [Resolved] (SPARK-31688) Refactor pagination framework for spark web UI pages

2020-05-21 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-31688. -- Fix Version/s: 3.1.0 Assignee: Rakesh Raushan Resolution: Fixed Resolved by

[jira] [Commented] (SPARK-31761) Sql Div operator can result in incorrect output for int_min

2020-05-21 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113451#comment-17113451 ] Dongjoon Hyun commented on SPARK-31761: --- If possible, could you try to find a way not to revert

[jira] [Commented] (SPARK-31786) Exception on submitting Spark-Pi to Kubernetes 1.17.3

2020-05-21 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-31786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113429#comment-17113429 ] Maciej Bryński commented on SPARK-31786: I think this is related to:

[jira] [Resolved] (SPARK-31354) SparkSession Lifecycle methods to fix memory leak

2020-05-21 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-31354. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 28128

[jira] [Commented] (SPARK-31761) Sql Div operator can result in incorrect output for int_min

2020-05-21 Thread Sandeep Katta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113337#comment-17113337 ] Sandeep Katta commented on SPARK-31761: --- I can fix this, I will raise PR > Sql Div operator can

[jira] [Commented] (SPARK-31761) Sql Div operator can result in incorrect output for int_min

2020-05-21 Thread Sandeep Katta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113344#comment-17113344 ] Sandeep Katta commented on SPARK-31761: --- but to fix this do we need to revert 

[jira] [Assigned] (SPARK-31354) SparkSession Lifecycle methods to fix memory leak

2020-05-21 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-31354: --- Assignee: Vinoo Ganesh > SparkSession Lifecycle methods to fix memory leak >

[jira] [Commented] (SPARK-31787) Support Minikube 1.5.x in K8s IT

2020-05-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113503#comment-17113503 ] Apache Spark commented on SPARK-31787: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Assigned] (SPARK-31787) Support Minikube 1.5.x in K8s IT

2020-05-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31787: Assignee: Apache Spark > Support Minikube 1.5.x in K8s IT >

[jira] [Assigned] (SPARK-31787) Support Minikube 1.5.x in K8s IT

2020-05-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31787: Assignee: (was: Apache Spark) > Support Minikube 1.5.x in K8s IT >

[jira] [Commented] (SPARK-31774) getting the Caused by: org.apache.spark.sql.catalyst.errors.package$TreeNodeException: Binding attribute, at org.apache.spark.sql.catalyst.errors.package$.attachTree(p

2020-05-21 Thread Pankaj Tiwari (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113297#comment-17113297 ] Pankaj Tiwari commented on SPARK-31774: --- Hi [~jobitmathew] , I have a column name like "name ->

[jira] [Created] (SPARK-31788) Cannot convert org.apache.spark.api.java.JavaPairRDD to org.apache.spark.api.java.JavaRDD

2020-05-21 Thread Sanket Reddy (Jira)
Sanket Reddy created SPARK-31788: Summary: Cannot convert org.apache.spark.api.java.JavaPairRDD to org.apache.spark.api.java.JavaRDD Key: SPARK-31788 URL: https://issues.apache.org/jira/browse/SPARK-31788

[jira] [Commented] (SPARK-31761) Sql Div operator can result in incorrect output for int_min

2020-05-21 Thread Sandeep Katta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113453#comment-17113453 ] Sandeep Katta commented on SPARK-31761: --- okay I will try to fix it  > Sql Div operator can result

[jira] [Updated] (SPARK-31787) Support Minikube 1.5.x in K8s IT

2020-05-21 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31787: -- Priority: Minor (was: Major) > Support Minikube 1.5.x in K8s IT >

[jira] [Created] (SPARK-31787) Support Minikube 1.5.x in K8s IT

2020-05-21 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-31787: - Summary: Support Minikube 1.5.x in K8s IT Key: SPARK-31787 URL: https://issues.apache.org/jira/browse/SPARK-31787 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-31761) Sql Div operator can result in incorrect output for int_min

2020-05-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113524#comment-17113524 ] Apache Spark commented on SPARK-31761: -- User 'sandeep-katta' has created a pull request for this

[jira] [Assigned] (SPARK-31761) Sql Div operator can result in incorrect output for int_min

2020-05-21 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31761: Assignee: Apache Spark > Sql Div operator can result in incorrect output for int_min >

  1   2   >