[jira] [Resolved] (SPARK-32722) Update document type conversion for Pandas UDFs (pyarrow 1.0.1, pandas 1.1.1, Python 3.7)

2020-08-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32722. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29569 [https://gi

[jira] [Assigned] (SPARK-32722) Update document type conversion for Pandas UDFs (pyarrow 1.0.1, pandas 1.1.1, Python 3.7)

2020-08-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-32722: Assignee: Hyukjin Kwon > Update document type conversion for Pandas UDFs (pyarrow 1.0.1,

[jira] [Updated] (SPARK-32723) Security Vulnerability due to JQuery version in Spark Master/Worker UI

2020-08-27 Thread Ashish Kumar Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Kumar Singh updated SPARK-32723: --- Summary: Security Vulnerability due to JQuery version in Spark Master/Worker UI (wa

[jira] [Updated] (SPARK-32723) Security Vulnerability due to Jquery version in Spark Master/Worker UI

2020-08-27 Thread Ashish Kumar Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Kumar Singh updated SPARK-32723: --- Labels: Security (was: ) > Security Vulnerability due to Jquery version in Spark Ma

[jira] [Created] (SPARK-32723) Security Vulnerability due to Jquery version in Spark Master/Worker UI

2020-08-27 Thread Ashish Kumar Singh (Jira)
Ashish Kumar Singh created SPARK-32723: -- Summary: Security Vulnerability due to Jquery version in Spark Master/Worker UI Key: SPARK-32723 URL: https://issues.apache.org/jira/browse/SPARK-32723 Pr

[jira] [Assigned] (SPARK-32183) User Guide - PySpark Usage Guide for Pandas with Apache Arrow

2020-08-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-32183: Assignee: Hyukjin Kwon > User Guide - PySpark Usage Guide for Pandas with Apache Arrow >

[jira] [Resolved] (SPARK-32183) User Guide - PySpark Usage Guide for Pandas with Apache Arrow

2020-08-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32183. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29548 [https://gi

[jira] [Resolved] (SPARK-28216) Add calculate local directory size to SQLTestUtils

2020-08-27 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-28216. - Resolution: Fixed > Add calculate local directory size to SQLTestUtils > ---

[jira] [Assigned] (SPARK-28216) Add calculate local directory size to SQLTestUtils

2020-08-27 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-28216: --- Assignee: Yuming Wang > Add calculate local directory size to SQLTestUtils > --

[jira] [Updated] (SPARK-28216) Add calculate local directory size to SQLTestUtils

2020-08-27 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-28216: Affects Version/s: (was: 3.1.0) 3.0.0 > Add calculate local directory s

[jira] [Updated] (SPARK-28216) Add calculate local directory size to SQLTestUtils

2020-08-27 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-28216: Fix Version/s: 3.0.0 > Add calculate local directory size to SQLTestUtils > --

[jira] [Resolved] (SPARK-29868) Add a benchmark for Adaptive Execution

2020-08-27 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-29868. - Resolution: Won't Fix > Add a benchmark for Adaptive Execution > ---

[jira] [Resolved] (SPARK-31288) Remove `getRawTable` in `alterPartitions`

2020-08-27 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-31288. - Resolution: Won't Fix > Remove `getRawTable` in `alterPartitions` >

[jira] [Updated] (SPARK-14922) Alter Table Drop Partition Using Predicate-based Partition Spec

2020-08-27 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-14922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-14922: Target Version/s: 3.1.0 (was: 3.0.0) > Alter Table Drop Partition Using Predicate-based Partition

[jira] [Assigned] (SPARK-32722) Update document type conversion for Pandas UDFs (pyarrow 1.0.1, pandas 1.1.1, Python 3.7)

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32722: Assignee: Apache Spark > Update document type conversion for Pandas UDFs (pyarrow 1.0.1,

[jira] [Assigned] (SPARK-32722) Update document type conversion for Pandas UDFs (pyarrow 1.0.1, pandas 1.1.1, Python 3.7)

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32722: Assignee: (was: Apache Spark) > Update document type conversion for Pandas UDFs (pyar

[jira] [Commented] (SPARK-32722) Update document type conversion for Pandas UDFs (pyarrow 1.0.1, pandas 1.1.1, Python 3.7)

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17186259#comment-17186259 ] Apache Spark commented on SPARK-32722: -- User 'HyukjinKwon' has created a pull reque

[jira] [Updated] (SPARK-32722) Update document type conversion for Pandas UDFs (pyarrow 1.0.1, pandas 1.1.1, Python 3.7)

2020-08-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32722: - Description: Update the chart generated by SPARK-25798. > Update document type conversion for Pa

[jira] [Created] (SPARK-32722) Update document type conversion for Pandas UDFs (pyarrow 1.0.1, pandas 1.1.1, Python 3.7)

2020-08-27 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-32722: Summary: Update document type conversion for Pandas UDFs (pyarrow 1.0.1, pandas 1.1.1, Python 3.7) Key: SPARK-32722 URL: https://issues.apache.org/jira/browse/SPARK-32722

[jira] [Commented] (SPARK-28612) DataSourceV2: Add new DataFrameWriter API for v2

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17186229#comment-17186229 ] Apache Spark commented on SPARK-28612: -- User 'HeartSaVioR' has created a pull reque

[jira] [Commented] (SPARK-28612) DataSourceV2: Add new DataFrameWriter API for v2

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17186228#comment-17186228 ] Apache Spark commented on SPARK-28612: -- User 'HeartSaVioR' has created a pull reque

[jira] [Updated] (SPARK-32708) Query optimization fails to reuse exchange with DataSourceV2

2020-08-27 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-32708: - Affects Version/s: 2.4.7 > Query optimization fails to reuse exchange with DataSourceV2

[jira] [Updated] (SPARK-32708) Query optimization fails to reuse exchange with DataSourceV2

2020-08-27 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-32708: - Affects Version/s: (was: 2.4.7) (was: 2.4.5)

[jira] [Commented] (SPARK-32721) Simplify if clauses with null and boolean

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17186209#comment-17186209 ] Apache Spark commented on SPARK-32721: -- User 'sunchao' has created a pull request f

[jira] [Assigned] (SPARK-32721) Simplify if clauses with null and boolean

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32721: Assignee: Apache Spark > Simplify if clauses with null and boolean >

[jira] [Commented] (SPARK-32721) Simplify if clauses with null and boolean

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17186206#comment-17186206 ] Apache Spark commented on SPARK-32721: -- User 'sunchao' has created a pull request f

[jira] [Assigned] (SPARK-32721) Simplify if clauses with null and boolean

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32721: Assignee: (was: Apache Spark) > Simplify if clauses with null and boolean > -

[jira] [Comment Edited] (SPARK-32716) Add function mapPartitionsWithInputSplitAndIndex to NewHadoopRDD

2020-08-27 Thread Patrick Grandjean (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17186202#comment-17186202 ] Patrick Grandjean edited comment on SPARK-32716 at 8/28/20, 2:02 AM: -

[jira] [Commented] (SPARK-32716) Add function mapPartitionsWithInputSplitAndIndex to NewHadoopRDD

2020-08-27 Thread Patrick Grandjean (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17186202#comment-17186202 ] Patrick Grandjean commented on SPARK-32716: --- Yes, but not something that provi

[jira] [Commented] (SPARK-31561) Add QUALIFY Clause

2020-08-27 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17186199#comment-17186199 ] Yuming Wang commented on SPARK-31561: - Note that we should forbid the following case

[jira] [Updated] (SPARK-32721) Simplify if clauses with null and boolean

2020-08-27 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-32721: - Description: The following if clause: {code:sql} if(p, null, false) {code} can be simplified to: {code:

[jira] [Created] (SPARK-32721) Simplify if clauses with null and boolean

2020-08-27 Thread Chao Sun (Jira)
Chao Sun created SPARK-32721: Summary: Simplify if clauses with null and boolean Key: SPARK-32721 URL: https://issues.apache.org/jira/browse/SPARK-32721 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-32721) Simplify if clauses with null and boolean

2020-08-27 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-32721: - Description: The following if clause: {code:sql} if(p, null, false) {code} can be simplified to: {code:

[jira] [Commented] (SPARK-32720) Spark 3 Fails to Cast DateType to StringType when comparing result of Max

2020-08-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17186196#comment-17186196 ] Hyukjin Kwon commented on SPARK-32720: -- I can't reproduce it by doing below: {code

[jira] [Resolved] (SPARK-32693) Compare two dataframes with same schema except nullable property

2020-08-27 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-32693. -- Fix Version/s: 3.1.0 Assignee: L. C. Hsieh Resolution: Fixed Resolved

[jira] [Resolved] (SPARK-32716) Add function mapPartitionsWithInputSplitAndIndex to NewHadoopRDD

2020-08-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32716. -- Resolution: Won't Fix > Add function mapPartitionsWithInputSplitAndIndex to NewHadoopRDD > ---

[jira] [Commented] (SPARK-32716) Add function mapPartitionsWithInputSplitAndIndex to NewHadoopRDD

2020-08-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17186192#comment-17186192 ] Hyukjin Kwon commented on SPARK-32716: -- RDD APIs are currently rather in a maintena

[jira] [Commented] (SPARK-32720) Spark 3 Fails to Cast DateType to StringType when comparing result of Max

2020-08-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17186191#comment-17186191 ] Hyukjin Kwon commented on SPARK-32720: -- [~markus.notti] can you make the reproducer

[jira] [Commented] (SPARK-32629) Record metrics of extra BitSet/HashSet in full outer shuffled hash join

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17186190#comment-17186190 ] Apache Spark commented on SPARK-32629: -- User 'c21' has created a pull request for t

[jira] [Assigned] (SPARK-32629) Record metrics of extra BitSet/HashSet in full outer shuffled hash join

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32629: Assignee: (was: Apache Spark) > Record metrics of extra BitSet/HashSet in full outer

[jira] [Assigned] (SPARK-32629) Record metrics of extra BitSet/HashSet in full outer shuffled hash join

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32629: Assignee: Apache Spark > Record metrics of extra BitSet/HashSet in full outer shuffled ha

[jira] [Commented] (SPARK-24994) When the data type of the field is converted to other types, it can also support pushdown to parquet

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17186173#comment-17186173 ] Apache Spark commented on SPARK-24994: -- User 'sunchao' has created a pull request f

[jira] [Assigned] (SPARK-32708) Query optimization fails to reuse exchange with DataSourceV2

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32708: Assignee: (was: Apache Spark) > Query optimization fails to reuse exchange with DataS

[jira] [Commented] (SPARK-32708) Query optimization fails to reuse exchange with DataSourceV2

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17186158#comment-17186158 ] Apache Spark commented on SPARK-32708: -- User 'mingjialiu' has created a pull reques

[jira] [Assigned] (SPARK-32708) Query optimization fails to reuse exchange with DataSourceV2

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32708: Assignee: Apache Spark > Query optimization fails to reuse exchange with DataSourceV2 > -

[jira] [Updated] (SPARK-32720) Spark 3 Fails to Cast DateType to StringType when comparing result of Max

2020-08-27 Thread Markus (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Markus updated SPARK-32720: --- Description: When reading from parquet, using basepath option, the some_date is inferred and read as a Date

[jira] [Updated] (SPARK-32720) Spark 3 Fails to Cast DateType to StringType when comparing result of Max

2020-08-27 Thread Markus (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Markus updated SPARK-32720: --- Description: When reading from parquet, using basepath option, the some_date is inferred and read as a Date

[jira] [Updated] (SPARK-32720) Spark 3 Fails to Cast DateType to StringType when comparing result of Max

2020-08-27 Thread Markus (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Markus updated SPARK-32720: --- Description: When reading from parquet, using basepath option, the some_date is inferred and read as a Date

[jira] [Created] (SPARK-32720) Spark 3 Fails to Cast DateType to StringType when comparing result of Max

2020-08-27 Thread Markus (Jira)
Markus created SPARK-32720: -- Summary: Spark 3 Fails to Cast DateType to StringType when comparing result of Max Key: SPARK-32720 URL: https://issues.apache.org/jira/browse/SPARK-32720 Project: Spark

[jira] [Commented] (SPARK-32719) Add Flake8 check for missing imports

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17186115#comment-17186115 ] Apache Spark commented on SPARK-32719: -- User 'Fokko' has created a pull request for

[jira] [Assigned] (SPARK-32719) Add Flake8 check for missing imports

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32719: Assignee: Apache Spark > Add Flake8 check for missing imports > -

[jira] [Commented] (SPARK-32719) Add Flake8 check for missing imports

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17186114#comment-17186114 ] Apache Spark commented on SPARK-32719: -- User 'Fokko' has created a pull request for

[jira] [Assigned] (SPARK-32719) Add Flake8 check for missing imports

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32719: Assignee: (was: Apache Spark) > Add Flake8 check for missing imports > --

[jira] [Created] (SPARK-32719) Add Flake8 check for missing imports

2020-08-27 Thread Fokko Driesprong (Jira)
Fokko Driesprong created SPARK-32719: Summary: Add Flake8 check for missing imports Key: SPARK-32719 URL: https://issues.apache.org/jira/browse/SPARK-32719 Project: Spark Issue Type: Impr

[jira] [Updated] (SPARK-19256) Hive bucketing write support

2020-08-27 Thread Cheng Su (Jira)
[ https://issues.apache.org/jira/browse/SPARK-19256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Su updated SPARK-19256: - Description: Update (2020 by Cheng Su): We use this JIRA to track progress for Hive bucketing write sup

[jira] [Commented] (SPARK-19256) Hive bucketing write support

2020-08-27 Thread Cheng Su (Jira)
[ https://issues.apache.org/jira/browse/SPARK-19256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17186023#comment-17186023 ] Cheng Su commented on SPARK-19256: -- [~viirya] - > Do you mean Hive ORC/Parquet buckete

[jira] [Assigned] (SPARK-32713) Support execId placeholder in executor PVC conf

2020-08-27 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-32713: - Assignee: Dongjoon Hyun > Support execId placeholder in executor PVC conf > ---

[jira] [Resolved] (SPARK-32713) Support execId placeholder in executor PVC conf

2020-08-27 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-32713. --- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29557 [https://

[jira] [Commented] (SPARK-32718) remove unnecessary keywords for interval units

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17185936#comment-17185936 ] Apache Spark commented on SPARK-32718: -- User 'cloud-fan' has created a pull request

[jira] [Assigned] (SPARK-32718) remove unnecessary keywords for interval units

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32718: Assignee: Apache Spark (was: Wenchen Fan) > remove unnecessary keywords for interval uni

[jira] [Assigned] (SPARK-32718) remove unnecessary keywords for interval units

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32718: Assignee: Wenchen Fan (was: Apache Spark) > remove unnecessary keywords for interval uni

[jira] [Commented] (SPARK-32718) remove unnecessary keywords for interval units

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17185934#comment-17185934 ] Apache Spark commented on SPARK-32718: -- User 'cloud-fan' has created a pull request

[jira] [Created] (SPARK-32718) remove unnecessary keywords for interval units

2020-08-27 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-32718: --- Summary: remove unnecessary keywords for interval units Key: SPARK-32718 URL: https://issues.apache.org/jira/browse/SPARK-32718 Project: Spark Issue Type: Impr

[jira] [Commented] (SPARK-31511) Make BytesToBytesMap iterator() thread-safe

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17185919#comment-17185919 ] Apache Spark commented on SPARK-31511: -- User 'hvanhovell' has created a pull reques

[jira] [Commented] (SPARK-32636) AsyncEventQueue: Exception scala.Some cannot be cast to java.lang.String

2020-08-27 Thread Muhammad Abubakar (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17185866#comment-17185866 ] Muhammad Abubakar commented on SPARK-32636: --- Hi [~hyukjin.kwon], Thanks for l

[jira] [Assigned] (SPARK-32701) mapreduce.fileoutputcommitter.algorithm.version default depends on runtime environment

2020-08-27 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-32701: Assignee: Waleed Fateem > mapreduce.fileoutputcommitter.algorithm.version default depends

[jira] [Resolved] (SPARK-32701) mapreduce.fileoutputcommitter.algorithm.version default depends on runtime environment

2020-08-27 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-32701. -- Fix Version/s: 3.1.0 3.0.1 Resolution: Fixed Issue resolved by pull

[jira] [Updated] (SPARK-30654) Update Docs Bootstrap to 4.4.1

2020-08-27 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-30654: - Fix Version/s: 3.1.0 > Update Docs Bootstrap to 4.4.1 > -- > >

[jira] [Commented] (SPARK-32717) Add a AQEOptimizer for AdaptiveSparkPlanExec

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17185838#comment-17185838 ] Apache Spark commented on SPARK-32717: -- User 'Ngone51' has created a pull request f

[jira] [Commented] (SPARK-32717) Add a AQEOptimizer for AdaptiveSparkPlanExec

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17185837#comment-17185837 ] Apache Spark commented on SPARK-32717: -- User 'Ngone51' has created a pull request f

[jira] [Assigned] (SPARK-32717) Add a AQEOptimizer for AdaptiveSparkPlanExec

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32717: Assignee: Apache Spark > Add a AQEOptimizer for AdaptiveSparkPlanExec > -

[jira] [Assigned] (SPARK-32717) Add a AQEOptimizer for AdaptiveSparkPlanExec

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32717: Assignee: (was: Apache Spark) > Add a AQEOptimizer for AdaptiveSparkPlanExec > --

[jira] [Created] (SPARK-32717) Add a AQEOptimizer for AdaptiveSparkPlanExec

2020-08-27 Thread wuyi (Jira)
wuyi created SPARK-32717: Summary: Add a AQEOptimizer for AdaptiveSparkPlanExec Key: SPARK-32717 URL: https://issues.apache.org/jira/browse/SPARK-32717 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-32715) Broadcast block pieces may memory leak

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17185825#comment-17185825 ] Apache Spark commented on SPARK-32715: -- User 'LantaoJin' has created a pull request

[jira] [Assigned] (SPARK-32715) Broadcast block pieces may memory leak

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32715: Assignee: Apache Spark > Broadcast block pieces may memory leak > ---

[jira] [Commented] (SPARK-32715) Broadcast block pieces may memory leak

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17185824#comment-17185824 ] Apache Spark commented on SPARK-32715: -- User 'LantaoJin' has created a pull request

[jira] [Assigned] (SPARK-32715) Broadcast block pieces may memory leak

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32715: Assignee: (was: Apache Spark) > Broadcast block pieces may memory leak >

[jira] [Created] (SPARK-32716) Add function mapPartitionsWithInputSplitAndIndex to NewHadoopRDD

2020-08-27 Thread Patrick Grandjean (Jira)
Patrick Grandjean created SPARK-32716: - Summary: Add function mapPartitionsWithInputSplitAndIndex to NewHadoopRDD Key: SPARK-32716 URL: https://issues.apache.org/jira/browse/SPARK-32716 Project: S

[jira] [Updated] (SPARK-32715) Broadcast block pieces may memory leak

2020-08-27 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32715: --- Description: We use Spark thrift-server as a long-running service. A bad query submitted a heavy Br

[jira] [Created] (SPARK-32715) Broadcast block pieces may memory leak

2020-08-27 Thread Lantao Jin (Jira)
Lantao Jin created SPARK-32715: -- Summary: Broadcast block pieces may memory leak Key: SPARK-32715 URL: https://issues.apache.org/jira/browse/SPARK-32715 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-32681) PySpark type hints support

2020-08-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32681: - Description: https://github.com/zero323/pyspark-stubs demonstrates a lot of benefits to improv

[jira] [Updated] (SPARK-32714) Port pyspark-stubs

2020-08-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32714: - Description: Port https://github.com/zero323/pyspark-stubs into PySpark. This was being discusse

[jira] [Updated] (SPARK-32681) PySpark type hints support

2020-08-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32681: - Description: We're discussing about porting https://github.com/zero323/pyspark-stubs in the dev

[jira] [Updated] (SPARK-32714) Port pyspark-stubs

2020-08-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32714: - Description: Port https://github.com/zero323/pyspark-stubs > Port pyspark-stubs > --

[jira] [Assigned] (SPARK-32681) PySpark type hints support

2020-08-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-32681: Assignee: Maciej Szymkiewicz > PySpark type hints support > -- >

[jira] [Created] (SPARK-32714) Port pyspark-stubs

2020-08-27 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-32714: Summary: Port pyspark-stubs Key: SPARK-32714 URL: https://issues.apache.org/jira/browse/SPARK-32714 Project: Spark Issue Type: Sub-task Components:

[jira] [Commented] (SPARK-32713) Support execId placeholder in executor PVC conf

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17185780#comment-17185780 ] Apache Spark commented on SPARK-32713: -- User 'dongjoon-hyun' has created a pull req

[jira] [Assigned] (SPARK-32713) Support execId placeholder in executor PVC conf

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32713: Assignee: (was: Apache Spark) > Support execId placeholder in executor PVC conf > ---

[jira] [Assigned] (SPARK-32713) Support execId placeholder in executor PVC conf

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32713: Assignee: Apache Spark > Support execId placeholder in executor PVC conf > --

[jira] [Commented] (SPARK-32713) Support execId placeholder in executor PVC conf

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17185778#comment-17185778 ] Apache Spark commented on SPARK-32713: -- User 'dongjoon-hyun' has created a pull req

[jira] [Created] (SPARK-32713) Support execId placeholder in executor PVC conf

2020-08-27 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-32713: - Summary: Support execId placeholder in executor PVC conf Key: SPARK-32713 URL: https://issues.apache.org/jira/browse/SPARK-32713 Project: Spark Issue Type:

[jira] [Commented] (SPARK-27951) ANSI SQL: NTH_VALUE function

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17185691#comment-17185691 ] Apache Spark commented on SPARK-27951: -- User 'beliefer' has created a pull request

[jira] [Commented] (SPARK-27951) ANSI SQL: NTH_VALUE function

2020-08-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17185690#comment-17185690 ] Apache Spark commented on SPARK-27951: -- User 'beliefer' has created a pull request