[jira] [Resolved] (SPARK-28723) Upgrade to Hive 2.3.6 for HiveMetastore Client and Hadoop-3.2 profile

[ https://issues.apache.org/jira/browse/SPARK-28723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-28723. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25443

[jira] [Commented] (SPARK-28833) Document ALTER VIEW Statement in SQL Reference.

[ https://issues.apache.org/jira/browse/SPARK-28833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16914795#comment-16914795 ] kevin yu commented on SPARK-28833: -- @aman omer : I am about the half way there.. and you can help

[jira] [Commented] (SPARK-28778) Shuffle jobs fail due to incorrect advertised address when running in virtual network

[ https://issues.apache.org/jira/browse/SPARK-28778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16914762#comment-16914762 ] Dongjoon Hyun commented on SPARK-28778: --- You are added to the Apache Spark contributor group.

[jira] [Assigned] (SPARK-28778) Shuffle jobs fail due to incorrect advertised address when running in virtual network

[ https://issues.apache.org/jira/browse/SPARK-28778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-28778: - Assignee: Anton Kirillov > Shuffle jobs fail due to incorrect advertised address when

[jira] [Resolved] (SPARK-28778) Shuffle jobs fail due to incorrect advertised address when running in virtual network

[ https://issues.apache.org/jira/browse/SPARK-28778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-28778. --- Fix Version/s: 3.0.0 Resolution: Fixed This is resolved via 

[jira] [Resolved] (SPARK-28858) add tree-based transformation in the py side

[ https://issues.apache.org/jira/browse/SPARK-28858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler resolved SPARK-28858. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25566

[jira] [Assigned] (SPARK-28858) add tree-based transformation in the py side

[ https://issues.apache.org/jira/browse/SPARK-28858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler reassigned SPARK-28858: Assignee: zhengruifeng > add tree-based transformation in the py side >

[jira] [Comment Edited] (SPARK-23519) Create View Commands Fails with The view output (col1,col1) contains duplicate column name

[ https://issues.apache.org/jira/browse/SPARK-23519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16914660#comment-16914660 ] hemanth meka edited comment on SPARK-23519 at 8/23/19 9:50 PM: --- PR raised

[jira] [Comment Edited] (SPARK-23519) Create View Commands Fails with The view output (col1,col1) contains duplicate column name

[ https://issues.apache.org/jira/browse/SPARK-23519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16914660#comment-16914660 ] hemanth meka edited comment on SPARK-23519 at 8/23/19 9:50 PM: --- PR raised

[jira] [Commented] (SPARK-23519) Create View Commands Fails with The view output (col1,col1) contains duplicate column name

[ https://issues.apache.org/jira/browse/SPARK-23519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16914660#comment-16914660 ] hemanth meka commented on SPARK-23519: -- PR raised

[jira] [Created] (SPARK-28863) Add an AlreadyPlanned logical node that skips query planning

Burak Yavuz created SPARK-28863: --- Summary: Add an AlreadyPlanned logical node that skips query planning Key: SPARK-28863 URL: https://issues.apache.org/jira/browse/SPARK-28863 Project: Spark

[jira] [Assigned] (SPARK-28839) ExecutorMonitor$Tracker NullPointerException

[ https://issues.apache.org/jira/browse/SPARK-28839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-28839: -- Assignee: Hyukjin Kwon > ExecutorMonitor$Tracker NullPointerException >

[jira] [Resolved] (SPARK-28839) ExecutorMonitor$Tracker NullPointerException

[ https://issues.apache.org/jira/browse/SPARK-28839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-28839. Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25551

[jira] [Resolved] (SPARK-28482) Data incomplete when using pandas udf in Python 3

[ https://issues.apache.org/jira/browse/SPARK-28482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler resolved SPARK-28482. -- Resolution: Not A Problem No problem [~jiangyu1211] ! I will resolve this then. In general, I

[jira] [Comment Edited] (SPARK-28833) Document ALTER VIEW Statement in SQL Reference.

[ https://issues.apache.org/jira/browse/SPARK-28833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16914462#comment-16914462 ] Aman Omer edited comment on SPARK-28833 at 8/23/19 4:48 PM: [~kevinyu98] 

[jira] [Comment Edited] (SPARK-28833) Document ALTER VIEW Statement in SQL Reference.

[ https://issues.apache.org/jira/browse/SPARK-28833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16914462#comment-16914462 ] Aman Omer edited comment on SPARK-28833 at 8/23/19 4:47 PM: [~kevinyu98] 

[jira] [Commented] (SPARK-28833) Document ALTER VIEW Statement in SQL Reference.

[ https://issues.apache.org/jira/browse/SPARK-28833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16914462#comment-16914462 ] Aman Omer commented on SPARK-28833: --- [~kevinyu98] Since I have worked on CREATE VIEW document

[jira] [Created] (SPARK-28862) Read Impala view with UNION ALL operator

francesco created SPARK-28862: - Summary: Read Impala view with UNION ALL operator Key: SPARK-28862 URL: https://issues.apache.org/jira/browse/SPARK-28862 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-27330) ForeachWriter is not being closed once a batch is aborted

[ https://issues.apache.org/jira/browse/SPARK-27330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16914321#comment-16914321 ] Dongjoon Hyun commented on SPARK-27330: --- Thank you, [~eyalzit]. You are added to the Apache Spark

[jira] [Assigned] (SPARK-27330) ForeachWriter is not being closed once a batch is aborted

[ https://issues.apache.org/jira/browse/SPARK-27330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-27330: - Assignee: Eyal Zituny > ForeachWriter is not being closed once a batch is aborted >

[jira] [Resolved] (SPARK-27330) ForeachWriter is not being closed once a batch is aborted

[ https://issues.apache.org/jira/browse/SPARK-27330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-27330. --- Fix Version/s: 3.0.0 2.4.4 Resolution: Fixed > ForeachWriter is

[jira] [Updated] (SPARK-28025) HDFSBackedStateStoreProvider should not leak .crc files

[ https://issues.apache.org/jira/browse/SPARK-28025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-28025: -- Fix Version/s: 2.4.4 > HDFSBackedStateStoreProvider should not leak .crc files >

[jira] [Updated] (SPARK-25474) Support `spark.sql.statistics.fallBackToHdfs` in data source tables

[ https://issues.apache.org/jira/browse/SPARK-25474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25474: -- Fix Version/s: (was: 3.0.0) > Support `spark.sql.statistics.fallBackToHdfs` in data

[jira] [Reopened] (SPARK-25474) Support `spark.sql.statistics.fallBackToHdfs` in data source tables

[ https://issues.apache.org/jira/browse/SPARK-25474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reopened SPARK-25474: --- Assignee: (was: shahid) > Support `spark.sql.statistics.fallBackToHdfs` in data

[jira] [Created] (SPARK-28861) Jetty property handling

Ketan created SPARK-28861: - Summary: Jetty property handling Key: SPARK-28861 URL: https://issues.apache.org/jira/browse/SPARK-28861 Project: Spark Issue Type: Wish Components: Spark

[jira] [Resolved] (SPARK-28741) New optional mode: Throw exceptions when casting to integers causes overflow

[ https://issues.apache.org/jira/browse/SPARK-28741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-28741. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25461

[jira] [Assigned] (SPARK-28741) New optional mode: Throw exceptions when casting to integers causes overflow

[ https://issues.apache.org/jira/browse/SPARK-28741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-28741: --- Assignee: Gengliang Wang > New optional mode: Throw exceptions when casting to integers

[jira] [Resolved] (SPARK-28716) Add id to Exchange and Subquery's stringArgs method for easier identifying their reuses in query plans

[ https://issues.apache.org/jira/browse/SPARK-28716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell resolved SPARK-28716. --- Fix Version/s: 3.0.0 Assignee: Ali Afroozeh Resolution: Fixed > Add

[jira] [Resolved] (SPARK-28836) Remove the canonicalize(attributes) method from PlanExpression

[ https://issues.apache.org/jira/browse/SPARK-28836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell resolved SPARK-28836. --- Fix Version/s: 3.0.0 Assignee: Ali Afroozeh Resolution: Fixed >

[jira] [Commented] (SPARK-18112) Spark2.x does not support read data from Hive 2.x metastore

[ https://issues.apache.org/jira/browse/SPARK-18112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16914192#comment-16914192 ] Rahul Kulkarni commented on SPARK-18112: Getting this issue on Spark 2.4.3 with Hive 2.3.3. Also

[jira] [Updated] (SPARK-28836) Remove the canonicalize(attributes) method from PlanExpression

[ https://issues.apache.org/jira/browse/SPARK-28836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ali Afroozeh updated SPARK-28836: - Summary: Remove the canonicalize(attributes) method from PlanExpression (was: Remove the

[jira] [Updated] (SPARK-28836) Remove the canonicalize() method

[ https://issues.apache.org/jira/browse/SPARK-28836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ali Afroozeh updated SPARK-28836: - Summary: Remove the canonicalize() method (was: Improve canonicalize API) > Remove the

[jira] [Updated] (SPARK-28836) Improve canonicalize API

[ https://issues.apache.org/jira/browse/SPARK-28836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ali Afroozeh updated SPARK-28836: - Description: The canonicalize(attrs: AttributeSeq) method in PlanExpression is somewhat

[jira] [Updated] (SPARK-28836) Improve canonicalize API

[ https://issues.apache.org/jira/browse/SPARK-28836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ali Afroozeh updated SPARK-28836: - Description: The canonicalize(attrs: AttributeSeq) method in PlanExpression is somewhat

[jira] [Commented] (SPARK-28851) Connect HBase using Spark SQL in Spark 2.x

[ https://issues.apache.org/jira/browse/SPARK-28851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16914094#comment-16914094 ] Hyukjin Kwon commented on SPARK-28851: -- That will likely be an issue in HBase connector. For

[jira] [Resolved] (SPARK-28851) Connect HBase using Spark SQL in Spark 2.x

[ https://issues.apache.org/jira/browse/SPARK-28851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-28851. -- Resolution: Invalid > Connect HBase using Spark SQL in Spark 2.x >

[jira] [Resolved] (SPARK-28854) Zipping iterators in mapPartitions will fail

[ https://issues.apache.org/jira/browse/SPARK-28854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-28854. -- Resolution: Invalid > Zipping iterators in mapPartitions will fail >

[jira] [Commented] (SPARK-28854) Zipping iterators in mapPartitions will fail

[ https://issues.apache.org/jira/browse/SPARK-28854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16914091#comment-16914091 ] Hyukjin Kwon commented on SPARK-28854: -- Seems {{xs}} is an iterator and it's all consumed at the

[jira] [Updated] (SPARK-28854) Zipping iterators in mapPartitions will fail

[ https://issues.apache.org/jira/browse/SPARK-28854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-28854: - Description: {code} scala> sc.parallelize(Seq(1, 2, 3)).mapPartitions(xs =>

[jira] [Issue Comment Deleted] (SPARK-28854) Zipping iterators in mapPartitions will fail

[ https://issues.apache.org/jira/browse/SPARK-28854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-28854: - Comment: was deleted (was: Your {{xs.map(2*)}} produces: {code} scala> sc.parallelize(Seq(1,

[jira] [Commented] (SPARK-28854) Zipping iterators in mapPartitions will fail

[ https://issues.apache.org/jira/browse/SPARK-28854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16914085#comment-16914085 ] Hyukjin Kwon commented on SPARK-28854: -- Your {{xs.map(2*)}} produces: {code} scala>

[jira] [Assigned] (SPARK-28857) Clean up the comments of PR template during merging

[ https://issues.apache.org/jira/browse/SPARK-28857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-28857: Assignee: Dongjoon Hyun > Clean up the comments of PR template during merging >

[jira] [Resolved] (SPARK-28857) Clean up the comments of PR template during merging

[ https://issues.apache.org/jira/browse/SPARK-28857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-28857. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25564

[jira] [Commented] (SPARK-28836) Improve canonicalize API

[ https://issues.apache.org/jira/browse/SPARK-28836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16914059#comment-16914059 ] Hyukjin Kwon commented on SPARK-28836: -- Sure, that's fine but [~afroozeh] please clarify what issue

[jira] [Updated] (SPARK-28860) Using ColumnStats of join key to get TableAccessCardinality when finding star joins in ReorderJoinRule

[ https://issues.apache.org/jira/browse/SPARK-28860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lai Zhou updated SPARK-28860: - Description: Now the star-schema detection uses TableAccessCardinality to reorder DimTables  when

[jira] [Created] (SPARK-28860) Using ColumnStats of join key to get TableAccessCardinality when finding star joins in ReorderJoinRule

Lai Zhou created SPARK-28860: Summary: Using ColumnStats of join key to get TableAccessCardinality when finding star joins in ReorderJoinRule Key: SPARK-28860 URL: https://issues.apache.org/jira/browse/SPARK-28860

[jira] [Updated] (SPARK-28859) Remove value check of MEMORY_OFFHEAP_SIZE in declaration section

[ https://issues.apache.org/jira/browse/SPARK-28859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-28859: - Affects Version/s: (was: 2.4.3) 3.0.0 > Remove value check of

[jira] [Commented] (SPARK-28859) Remove value check of MEMORY_OFFHEAP_SIZE in declaration section

[ https://issues.apache.org/jira/browse/SPARK-28859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16914016#comment-16914016 ] Yang Jie commented on SPARK-28859: -- cc [~tgraves]  > Remove value check of MEMORY_OFFHEAP_SIZE in

[jira] [Updated] (SPARK-28642) Hide credentials in show create table

[ https://issues.apache.org/jira/browse/SPARK-28642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-28642: -- Fix Version/s: 2.4.4 > Hide credentials in show create table >

[jira] [Created] (SPARK-28859) Remove value check of MEMORY_OFFHEAP_SIZE in declaration section

Yang Jie created SPARK-28859: Summary: Remove value check of MEMORY_OFFHEAP_SIZE in declaration section Key: SPARK-28859 URL: https://issues.apache.org/jira/browse/SPARK-28859 Project: Spark

[jira] [Created] (SPARK-28858) add tree-based transformation in the py side

zhengruifeng created SPARK-28858: Summary: add tree-based transformation in the py side Key: SPARK-28858 URL: https://issues.apache.org/jira/browse/SPARK-28858 Project: Spark Issue Type:

[jira] [Commented] (SPARK-28836) Improve canonicalize API

[ https://issues.apache.org/jira/browse/SPARK-28836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16913973#comment-16913973 ] Dongjoon Hyun commented on SPARK-28836: --- This issue content is switched with SPARK-28835. >

[jira] [Resolved] (SPARK-28835) Introduce TPCDSSchema

[ https://issues.apache.org/jira/browse/SPARK-28835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-28835. --- Fix Version/s: 3.0.0 Assignee: Ali Afroozeh Resolution: Fixed This is

[jira] [Updated] (SPARK-28836) Improve canonicalize API

[ https://issues.apache.org/jira/browse/SPARK-28836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-28836: -- Description: This PR improves the `canonicalize` API by removing the method `def

[jira] [Updated] (SPARK-28835) Introduce TPCDSSchema

[ https://issues.apache.org/jira/browse/SPARK-28835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-28835: -- Description: This PR extracts the schema information of TPCDS tables into a separate class

[jira] [Updated] (SPARK-28836) Improve canonicalize API

[ https://issues.apache.org/jira/browse/SPARK-28836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-28836: -- Summary: Improve canonicalize API (was: Introduce TPCDSSchema) > Improve canonicalize API >

[jira] [Updated] (SPARK-28835) Introduce TPCDSSchema

[ https://issues.apache.org/jira/browse/SPARK-28835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-28835: -- Summary: Introduce TPCDSSchema (was: Improve canonicalize API) > Introduce TPCDSSchema >

[jira] [Reopened] (SPARK-28836) Introduce TPCDSSchema

[ https://issues.apache.org/jira/browse/SPARK-28836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reopened SPARK-28836: --- > Introduce TPCDSSchema > - > > Key: SPARK-28836 >

[jira] [Commented] (SPARK-28836) Introduce TPCDSSchema

[ https://issues.apache.org/jira/browse/SPARK-28836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16913971#comment-16913971 ] Dongjoon Hyun commented on SPARK-28836: --- Oops. Sorry, [~hyukjin.kwon]. I merged this because the

[jira] [Resolved] (SPARK-28319) DataSourceV2: Support SHOW TABLES

[ https://issues.apache.org/jira/browse/SPARK-28319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-28319. - Fix Version/s: 3.0.0 Assignee: Terry Kim Resolution: Fixed > DataSourceV2:

[jira] [Assigned] (SPARK-28025) HDFSBackedStateStoreProvider should not leak .crc files

[ https://issues.apache.org/jira/browse/SPARK-28025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-28025: Assignee: Jungtaek Lim > HDFSBackedStateStoreProvider should not leak .crc files >

[jira] [Resolved] (SPARK-28025) HDFSBackedStateStoreProvider should not leak .crc files

[ https://issues.apache.org/jira/browse/SPARK-28025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-28025. -- Fix Version/s: 3.0.0 Resolution: Fixed > HDFSBackedStateStoreProvider should not leak

[jira] [Commented] (SPARK-23519) Create View Commands Fails with The view output (col1,col1) contains duplicate column name

[ https://issues.apache.org/jira/browse/SPARK-23519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16913965#comment-16913965 ] hemanth meka commented on SPARK-23519: -- I have a fix for this. checkColumnNameDuplication is