[jira] [Created] (SPARK-46208) Use specific Pandas version for API specifictations

2023-12-01 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-46208: --- Summary: Use specific Pandas version for API specifictations Key: SPARK-46208 URL: https://issues.apache.org/jira/browse/SPARK-46208 Project: Spark Issue Type:

[jira] [Created] (SPARK-46206) Use a narrower scope exception for SQL processor

2023-12-01 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-46206: --- Summary: Use a narrower scope exception for SQL processor Key: SPARK-46206 URL: https://issues.apache.org/jira/browse/SPARK-46206 Project: Spark Issue Type: Bu

[jira] [Created] (SPARK-46169) Assign appropriate JIRA numbers to unlabeled TODO items for DataFrame API.

2023-11-28 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-46169: --- Summary: Assign appropriate JIRA numbers to unlabeled TODO items for DataFrame API. Key: SPARK-46169 URL: https://issues.apache.org/jira/browse/SPARK-46169 Project: Spa

[jira] [Created] (SPARK-46168) Add axis parameter to DataFrame.idxmin & idxmax

2023-11-28 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-46168: --- Summary: Add axis parameter to DataFrame.idxmin & idxmax Key: SPARK-46168 URL: https://issues.apache.org/jira/browse/SPARK-46168 Project: Spark Issue Type: Sub

[jira] [Created] (SPARK-46167) Add axis, pct and na_option parameter to DataFrame.rank

2023-11-28 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-46167: --- Summary: Add axis, pct and na_option parameter to DataFrame.rank Key: SPARK-46167 URL: https://issues.apache.org/jira/browse/SPARK-46167 Project: Spark Issue T

[jira] [Created] (SPARK-46165) Improve axis parameter for DataFrame.all to support columns.

2023-11-28 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-46165: --- Summary: Improve axis parameter for DataFrame.all to support columns. Key: SPARK-46165 URL: https://issues.apache.org/jira/browse/SPARK-46165 Project: Spark I

[jira] [Created] (SPARK-46166) Add axis and skipna parameters to DataFrame.any

2023-11-28 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-46166: --- Summary: Add axis and skipna parameters to DataFrame.any Key: SPARK-46166 URL: https://issues.apache.org/jira/browse/SPARK-46166 Project: Spark Issue Type: Sub

[jira] [Created] (SPARK-46164) Add include and exclude parameters for DataFrame.describe

2023-11-28 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-46164: --- Summary: Add include and exclude parameters for DataFrame.describe Key: SPARK-46164 URL: https://issues.apache.org/jira/browse/SPARK-46164 Project: Spark Issue

[jira] [Created] (SPARK-46163) Add filter_func and errors parameter for DataFrame.update

2023-11-28 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-46163: --- Summary: Add filter_func and errors parameter for DataFrame.update Key: SPARK-46163 URL: https://issues.apache.org/jira/browse/SPARK-46163 Project: Spark Issue

[jira] [Created] (SPARK-46160) Add freq and axis parameters to DataFrame.shift

2023-11-28 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-46160: --- Summary: Add freq and axis parameters to DataFrame.shift Key: SPARK-46160 URL: https://issues.apache.org/jira/browse/SPARK-46160 Project: Spark Issue Type: Sub

[jira] [Created] (SPARK-46162) Improve axis parameter for DataFrame.nunique to support columns.

2023-11-28 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-46162: --- Summary: Improve axis parameter for DataFrame.nunique to support columns. Key: SPARK-46162 URL: https://issues.apache.org/jira/browse/SPARK-46162 Project: Spark

[jira] [Created] (SPARK-46161) Improve axis parameter for DataFrame.diff to support columns.

2023-11-28 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-46161: --- Summary: Improve axis parameter for DataFrame.diff to support columns. Key: SPARK-46161 URL: https://issues.apache.org/jira/browse/SPARK-46161 Project: Spark

[jira] [Created] (SPARK-46159) Improve axis parameter for DataFrame.at_time to support columns.

2023-11-28 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-46159: --- Summary: Improve axis parameter for DataFrame.at_time to support columns. Key: SPARK-46159 URL: https://issues.apache.org/jira/browse/SPARK-46159 Project: Spark

[jira] [Created] (SPARK-46158) Improve axis parameter for DataFrame.xs to support columns.

2023-11-28 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-46158: --- Summary: Improve axis parameter for DataFrame.xs to support columns. Key: SPARK-46158 URL: https://issues.apache.org/jira/browse/SPARK-46158 Project: Spark Is

[jira] [Created] (SPARK-46157) Add `axis` parameter for DataFrame.aggregate.

2023-11-28 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-46157: --- Summary: Add `axis` parameter for DataFrame.aggregate. Key: SPARK-46157 URL: https://issues.apache.org/jira/browse/SPARK-46157 Project: Spark Issue Type: Sub-t

[jira] [Created] (SPARK-46129) Add GitHub link icon to PySpark documentation header

2023-11-27 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-46129: --- Summary: Add GitHub link icon to PySpark documentation header Key: SPARK-46129 URL: https://issues.apache.org/jira/browse/SPARK-46129 Project: Spark Issue Type

[jira] [Created] (SPARK-46123) Using brighter color for document title for better visibility

2023-11-27 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-46123: --- Summary: Using brighter color for document title for better visibility Key: SPARK-46123 URL: https://issues.apache.org/jira/browse/SPARK-46123 Project: Spark

[jira] [Created] (SPARK-46117) Enhancing readability of PySpark API reference by hiding verbose typehints.

2023-11-26 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-46117: --- Summary: Enhancing readability of PySpark API reference by hiding verbose typehints. Key: SPARK-46117 URL: https://issues.apache.org/jira/browse/SPARK-46117 Project: Sp

[jira] [Updated] (SPARK-46116) Adding "Q&A Support" and "Mailing Lists" link into PySpark doc homepage.

2023-11-26 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-46116: Description: It is aimed at improving user engagement and providing quick access to community sup

[jira] [Updated] (SPARK-46116) Adding "Q&A Support" and "Mailing Lists" link into PySpark doc homepage.

2023-11-26 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-46116: Summary: Adding "Q&A Support" and "Mailing Lists" link into PySpark doc homepage. (was: Enriching

[jira] [Updated] (SPARK-46116) Enriching PySpark doc with "Useful links" including Q&A Support and Mailing Lists

2023-11-26 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-46116: Summary: Enriching PySpark doc with "Useful links" including Q&A Support and Mailing Lists (was:

[jira] [Updated] (SPARK-46116) Enriching "Useful links" on PySpark docs including "Q&A Support" and "Mailing Lists"

2023-11-26 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-46116: Summary: Enriching "Useful links" on PySpark docs including "Q&A Support" and "Mailing Lists" (wa

[jira] [Updated] (SPARK-46116) Enriching PySpark doc with "Useful links" Including Q&A Support and Mailing Lists

2023-11-26 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-46116: Summary: Enriching PySpark doc with "Useful links" Including Q&A Support and Mailing Lists (was:

[jira] [Updated] (SPARK-46116) Enriching PySpark Documentation with "Useful Links" Including Q&A Support and Mailing Lists

2023-11-26 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-46116: Summary: Enriching PySpark Documentation with "Useful Links" Including Q&A Support and Mailing Lis

[jira] [Created] (SPARK-46116) Add "Q&A Support" Link to PySpark Documentation Homepage

2023-11-26 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-46116: --- Summary: Add "Q&A Support" Link to PySpark Documentation Homepage Key: SPARK-46116 URL: https://issues.apache.org/jira/browse/SPARK-46116 Project: Spark Issue

[jira] [Created] (SPARK-46112) Enforce usage of PySpark-specific Exceptions over built-in Python Exceptions

2023-11-26 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-46112: --- Summary: Enforce usage of PySpark-specific Exceptions over built-in Python Exceptions Key: SPARK-46112 URL: https://issues.apache.org/jira/browse/SPARK-46112 Project: S

[jira] [Created] (SPARK-46111) Add copyright to the PySpark official documentation.

2023-11-26 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-46111: --- Summary: Add copyright to the PySpark official documentation. Key: SPARK-46111 URL: https://issues.apache.org/jira/browse/SPARK-46111 Project: Spark Issue Type

[jira] [Created] (SPARK-46087) Sync PySpark dependencies in docs and dev requirements

2023-11-23 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-46087: --- Summary: Sync PySpark dependencies in docs and dev requirements Key: SPARK-46087 URL: https://issues.apache.org/jira/browse/SPARK-46087 Project: Spark Issue Ty

[jira] [Updated] (SPARK-46016) Fix pandas API support list properly

2023-11-23 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-46016: Summary: Fix pandas API support list properly (was: Correct Supported pandas API list) > Fix pan

[jira] [Updated] (SPARK-46016) Correct Supported pandas API list

2023-11-23 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-46016: Summary: Correct Supported pandas API list (was: Fix the script for Supported pandas API to work

[jira] [Updated] (SPARK-46084) Refactor data type casting operation for Categorical type.

2023-11-23 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-46084: Description: Using official API for better performance and readability. (was: Using official API

[jira] [Created] (SPARK-46084) Refactor data type casting operation for Categorical type.

2023-11-23 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-46084: --- Summary: Refactor data type casting operation for Categorical type. Key: SPARK-46084 URL: https://issues.apache.org/jira/browse/SPARK-46084 Project: Spark Issu

[jira] [Created] (SPARK-46065) Refactor `(DataFrame|Series).factorize()` to use `create_map`.

2023-11-22 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-46065: --- Summary: Refactor `(DataFrame|Series).factorize()` to use `create_map`. Key: SPARK-46065 URL: https://issues.apache.org/jira/browse/SPARK-46065 Project: Spark

[jira] [Created] (SPARK-46045) Add individual categories for `Options and settings` to API reference

2023-11-21 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-46045: --- Summary: Add individual categories for `Options and settings` to API reference Key: SPARK-46045 URL: https://issues.apache.org/jira/browse/SPARK-46045 Project: Spark

[jira] [Created] (SPARK-46022) Remove deprecated functions APIs from documents

2023-11-20 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-46022: --- Summary: Remove deprecated functions APIs from documents Key: SPARK-46022 URL: https://issues.apache.org/jira/browse/SPARK-46022 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-46017) PySpark doc build doesn't work properly on Mac

2023-11-20 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-46017: Summary: PySpark doc build doesn't work properly on Mac (was: PySpark doc build doesn't work prop

[jira] [Updated] (SPARK-46017) PySpark doc build doesn't work properly on Mac

2023-11-20 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-46017: Description: PySpark doc build is working properly on GitHub CI, but doesn't work properly on loca

[jira] [Updated] (SPARK-46017) PySpark doc build doesn't work properly on local environment

2023-11-20 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-46017: Summary: PySpark doc build doesn't work properly on local environment (was: PySpark doc build doe

[jira] [Created] (SPARK-46017) PySpark doc build doesn't work properly on local env

2023-11-20 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-46017: --- Summary: PySpark doc build doesn't work properly on local env Key: SPARK-46017 URL: https://issues.apache.org/jira/browse/SPARK-46017 Project: Spark Issue Type

[jira] [Created] (SPARK-46016) Fix the script for Supported pandas API to work properly

2023-11-20 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-46016: --- Summary: Fix the script for Supported pandas API to work properly Key: SPARK-46016 URL: https://issues.apache.org/jira/browse/SPARK-46016 Project: Spark Issue

[jira] [Created] (SPARK-46015) Fix broken link for Koalas issues

2023-11-20 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-46015: --- Summary: Fix broken link for Koalas issues Key: SPARK-46015 URL: https://issues.apache.org/jira/browse/SPARK-46015 Project: Spark Issue Type: Bug Com

[jira] [Updated] (SPARK-45997) Remove deprecated APIs from legacy Koalas

2023-11-19 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-45997: Summary: Remove deprecated APIs from legacy Koalas (was: Remove deprecated Koalas signatures) >

[jira] [Updated] (SPARK-45997) Remove deprecated Koalas signatures

2023-11-19 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-45997: Description: We should remove the deprecated Koalas signatures to cleanup the API surface. (was:

[jira] [Updated] (SPARK-45997) Remove deprecated Koalas signatures

2023-11-19 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-45997: Summary: Remove deprecated Koalas signatures (was: Update Koalas Migration Guide) > Remove depre

[jira] [Created] (SPARK-45997) Update Koalas Migration Guide

2023-11-19 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-45997: --- Summary: Update Koalas Migration Guide Key: SPARK-45997 URL: https://issues.apache.org/jira/browse/SPARK-45997 Project: Spark Issue Type: Bug Compone

[jira] [Created] (SPARK-45966) Add missing methods for API reference.

2023-11-16 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-45966: --- Summary: Add missing methods for API reference. Key: SPARK-45966 URL: https://issues.apache.org/jira/browse/SPARK-45966 Project: Spark Issue Type: Bug

[jira] [Reopened] (SPARK-45551) Enhance PySpark testing utils

2023-11-13 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee reopened SPARK-45551: - > Enhance PySpark testing utils > - > > Key: SPARK-45551

[jira] [Resolved] (SPARK-45551) Enhance PySpark testing utils

2023-11-13 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee resolved SPARK-45551. - Resolution: Fixed > Enhance PySpark testing utils > - > >

[jira] [Updated] (SPARK-45913) Make the internal attributes private from PySpark errors.

2023-11-13 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-45913: Summary: Make the internal attributes private from PySpark errors. (was: Make the internal attrib

[jira] [Updated] (SPARK-45913) Make the internal attributes private from `PySparkException`.

2023-11-13 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-45913: Summary: Make the internal attributes private from `PySparkException`. (was: Hide the internal at

[jira] [Updated] (SPARK-45913) Hide the internal attributes from PySparkException.

2023-11-13 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-45913: Description: There are some APIs from PySparkException are exposed to user space which should have

[jira] [Updated] (SPARK-45913) Hide the internal attributes from PySparkException.

2023-11-13 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-45913: Summary: Hide the internal attributes from PySparkException. (was: Hide the internal APIs from Py

[jira] [Updated] (SPARK-45913) Hide the internal APIs from PySparkException.

2023-11-13 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-45913: Description: There are some APIs are exposed to user space which should have not been as below: (

[jira] [Updated] (SPARK-45913) Hide the internal APIs from PySparkException.

2023-11-13 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-45913: Description: There are some APIs from PySparkException are exposed to user space which should have

[jira] [Created] (SPARK-45913) Hide the internal APIs from PySparkException.

2023-11-13 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-45913: --- Summary: Hide the internal APIs from PySparkException. Key: SPARK-45913 URL: https://issues.apache.org/jira/browse/SPARK-45913 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-45812) Upgrade Pandas to 2.1.2

2023-11-06 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-45812: --- Summary: Upgrade Pandas to 2.1.2 Key: SPARK-45812 URL: https://issues.apache.org/jira/browse/SPARK-45812 Project: Spark Issue Type: Sub-task Componen

[jira] [Created] (SPARK-45718) Remove remaining deprecated Pandas APIs from Spark 3.4.0

2023-10-30 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-45718: --- Summary: Remove remaining deprecated Pandas APIs from Spark 3.4.0 Key: SPARK-45718 URL: https://issues.apache.org/jira/browse/SPARK-45718 Project: Spark Issue

[jira] [Reopened] (SPARK-45673) Enhancing clarity and usability of PySpark error messages

2023-10-29 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee reopened SPARK-45673: - > Enhancing clarity and usability of PySpark error messages > --

[jira] [Resolved] (SPARK-45673) Enhancing clarity and usability of PySpark error messages

2023-10-29 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee resolved SPARK-45673. - Resolution: Fixed > Enhancing clarity and usability of PySpark error messages >

[jira] [Updated] (SPARK-45674) Improve error message for JVM-dependent attributes on Spark Connect.

2023-10-25 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-45674: Summary: Improve error message for JVM-dependent attributes on Spark Connect. (was: Improve `NOT_

[jira] [Created] (SPARK-45674) Improve `NOT_IMPLEMENTED` error for `sparkContext` on Spark Connect.

2023-10-25 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-45674: --- Summary: Improve `NOT_IMPLEMENTED` error for `sparkContext` on Spark Connect. Key: SPARK-45674 URL: https://issues.apache.org/jira/browse/SPARK-45674 Project: Spark

[jira] [Created] (SPARK-45673) Enhancing clarity and usability of PySpark error messages

2023-10-25 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-45673: --- Summary: Enhancing clarity and usability of PySpark error messages Key: SPARK-45673 URL: https://issues.apache.org/jira/browse/SPARK-45673 Project: Spark Issue

[jira] [Updated] (SPARK-45551) Enhance PySpark testing utils

2023-10-23 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-45551: Summary: Enhance PySpark testing utils (was: Enchance PySpark testing utils) > Enhance PySpark t

[jira] [Created] (SPARK-45635) Cleanup unused import for PySpark testing

2023-10-23 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-45635: --- Summary: Cleanup unused import for PySpark testing Key: SPARK-45635 URL: https://issues.apache.org/jira/browse/SPARK-45635 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-45634) Remove `get_dtype_counts` from Pandas API on Spark

2023-10-23 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-45634: --- Summary: Remove `get_dtype_counts` from Pandas API on Spark Key: SPARK-45634 URL: https://issues.apache.org/jira/browse/SPARK-45634 Project: Spark Issue Type:

[jira] [Created] (SPARK-45566) Introduce Pandas-like testing utils for Pandas API on Spark

2023-10-16 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-45566: --- Summary: Introduce Pandas-like testing utils for Pandas API on Spark Key: SPARK-45566 URL: https://issues.apache.org/jira/browse/SPARK-45566 Project: Spark Is

[jira] [Updated] (SPARK-45553) Deprecate assertPandasOnSparkEqual

2023-10-16 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-45553: Description: We will add new APIs for DataFrame, Series and Index separately, and we should deprec

[jira] [Updated] (SPARK-45553) Deprecate assertPandasOnSparkEqual

2023-10-16 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-45553: Summary: Deprecate assertPandasOnSparkEqual (was: Introduce flexible parameters to assertPandasOn

[jira] [Created] (SPARK-45555) Returning a debuggable object for failed assertion

2023-10-16 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-4: --- Summary: Returning a debuggable object for failed assertion Key: SPARK-4 URL: https://issues.apache.org/jira/browse/SPARK-4 Project: Spark Issue Type:

[jira] [Created] (SPARK-45554) Introduce flexible parameter to assertSchemaEqual

2023-10-16 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-45554: --- Summary: Introduce flexible parameter to assertSchemaEqual Key: SPARK-45554 URL: https://issues.apache.org/jira/browse/SPARK-45554 Project: Spark Issue Type: S

[jira] [Created] (SPARK-45553) Introduce flexible parameters to assertPandasOnSparkEqual

2023-10-16 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-45553: --- Summary: Introduce flexible parameters to assertPandasOnSparkEqual Key: SPARK-45553 URL: https://issues.apache.org/jira/browse/SPARK-45553 Project: Spark Issue

[jira] [Created] (SPARK-45552) Introduce flexible parameters to assertDataFrameEqual

2023-10-16 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-45552: --- Summary: Introduce flexible parameters to assertDataFrameEqual Key: SPARK-45552 URL: https://issues.apache.org/jira/browse/SPARK-45552 Project: Spark Issue Typ

[jira] [Created] (SPARK-45551) Enchance PySpark testing utils

2023-10-16 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-45551: --- Summary: Enchance PySpark testing utils Key: SPARK-45551 URL: https://issues.apache.org/jira/browse/SPARK-45551 Project: Spark Issue Type: Umbrella C

[jira] [Updated] (SPARK-45550) Remove deprecated APIs from Pandas API on Spark

2023-10-16 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-45550: Summary: Remove deprecated APIs from Pandas API on Spark (was: Remove to_spark_io) > Remove depr

[jira] [Updated] (SPARK-45550) Remove to_spark_io

2023-10-16 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-45550: Summary: Remove to_spark_io (was: Remove deprecated PySpark APIs) > Remove to_spark_io >

[jira] [Created] (SPARK-45550) Remove deprecated PySpark APIs

2023-10-16 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-45550: --- Summary: Remove deprecated PySpark APIs Key: SPARK-45550 URL: https://issues.apache.org/jira/browse/SPARK-45550 Project: Spark Issue Type: Bug Compon

[jira] [Created] (SPARK-45476) Raise exception directly instead of calling `resolveColumnsByPosition`

2023-10-09 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-45476: --- Summary: Raise exception directly instead of calling `resolveColumnsByPosition` Key: SPARK-45476 URL: https://issues.apache.org/jira/browse/SPARK-45476 Project: Spark

[jira] [Updated] (SPARK-43656) Enable NumPy compat tests

2023-10-04 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-43656: Summary: Enable NumPy compat tests (was: Fix pyspark.sql.column._to_java_column to accept Connect

[jira] [Resolved] (SPARK-44101) Support pandas 2

2023-09-27 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee resolved SPARK-44101. - Resolution: Fixed > Support pandas 2 > > > Key: SPARK-44101 >

[jira] [Updated] (SPARK-45308) Enable `GroupbySplitApplyTests.test_split_apply_combine_on_series` for pandas 2.0.0.

2023-09-24 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-45308: Summary: Enable `GroupbySplitApplyTests.test_split_apply_combine_on_series` for pandas 2.0.0. (wa

[jira] [Created] (SPARK-45308) Enable`GroupbySplitApplyTests.test_split_apply_combine_on_series` for pandas 2.0.0.

2023-09-24 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-45308: --- Summary: Enable`GroupbySplitApplyTests.test_split_apply_combine_on_series` for pandas 2.0.0. Key: SPARK-45308 URL: https://issues.apache.org/jira/browse/SPARK-45308 Pr

[jira] [Updated] (SPARK-43877) Fix behavior difference for compare binary functions.

2023-09-21 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-43877: Epic Link: SPARK-39375 > Fix behavior difference for compare binary functions. > -

[jira] [Updated] (SPARK-43877) Fix behavior difference for compare binary functions.

2023-09-21 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-43877: Parent: (was: SPARK-42497) Issue Type: Improvement (was: Sub-task) > Fix behavior dif

[jira] [Resolved] (SPARK-43623) Enable DefaultIndexParityTests.test_index_distributed_sequence_cleanup.

2023-09-21 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee resolved SPARK-43623. - Resolution: Duplicate > Enable DefaultIndexParityTests.test_index_distributed_sequence_cleanup.

[jira] [Resolved] (SPARK-43159) Refine `column_op` to use lambda function instead of Column API.

2023-09-21 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee resolved SPARK-43159. - Resolution: Won't Fix > Refine `column_op` to use lambda function instead of Column API. > -

[jira] [Updated] (SPARK-42965) metadata mismatch for StructField when running some tests.

2023-09-21 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-42965: Parent: (was: SPARK-42497) Issue Type: Improvement (was: Sub-task) > metadata mismatc

[jira] [Updated] (SPARK-42965) metadata mismatch for StructField when running some tests.

2023-09-21 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-42965: Epic Link: SPARK-39375 > metadata mismatch for StructField when running some tests. >

[jira] [Updated] (SPARK-45267) Change the default value for `numeric_only`.

2023-09-21 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-45267: Summary: Change the default value for `numeric_only`. (was: Changed the default value for `numeri

[jira] [Created] (SPARK-45267) Changed the default value for `numeric_only`.

2023-09-21 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-45267: --- Summary: Changed the default value for `numeric_only`. Key: SPARK-45267 URL: https://issues.apache.org/jira/browse/SPARK-45267 Project: Spark Issue Type: Sub-t

[jira] [Comment Edited] (SPARK-45264) Configurable error when generating Python docs

2023-09-21 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17767764#comment-17767764 ] Haejoon Lee edited comment on SPARK-45264 at 9/22/23 12:28 AM: ---

[jira] [Commented] (SPARK-45264) Configurable error when generating Python docs

2023-09-21 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17767764#comment-17767764 ] Haejoon Lee commented on SPARK-45264: - Currently the PySpark documentation build req

[jira] [Created] (SPARK-45247) Upgrade Pandas to 2.1.1

2023-09-20 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-45247: --- Summary: Upgrade Pandas to 2.1.1 Key: SPARK-45247 URL: https://issues.apache.org/jira/browse/SPARK-45247 Project: Spark Issue Type: Sub-task Componen

[jira] [Created] (SPARK-45246) Encourage using latest jinja2 other than documentation build

2023-09-20 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-45246: --- Summary: Encourage using latest jinja2 other than documentation build Key: SPARK-45246 URL: https://issues.apache.org/jira/browse/SPARK-45246 Project: Spark I

[jira] [Updated] (SPARK-45228) Update `test_axis_on_dataframe` when Pandas regression is fixed

2023-09-19 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-45228: Summary: Update `test_axis_on_dataframe` when Pandas regression is fixed (was: Restore `test_axis

[jira] [Created] (SPARK-45228) Restore `test_axis_on_dataframe` in normal state when Pandas regression is fixed

2023-09-19 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-45228: --- Summary: Restore `test_axis_on_dataframe` in normal state when Pandas regression is fixed Key: SPARK-45228 URL: https://issues.apache.org/jira/browse/SPARK-45228 Projec

[jira] [Updated] (SPARK-43432) Fix `min_periods` for Rolling to work same as pandas

2023-09-19 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-43432: Parent: (was: SPARK-44101) Issue Type: Improvement (was: Sub-task) > Fix `min_periods

[jira] [Reopened] (SPARK-43433) Match `GroupBy.nth` behavior with new pandas behavior

2023-09-19 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee reopened SPARK-43433: - > Match `GroupBy.nth` behavior with new pandas behavior > --

[jira] [Commented] (SPARK-44033) Support list-like for binary ops

2023-09-17 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17766170#comment-17766170 ] Haejoon Lee commented on SPARK-44033: - No worries! Let me take a look soon. Thanks f

[jira] [Resolved] (SPARK-45185) Ignore type check for preventing unexpected linter failure

2023-09-16 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee resolved SPARK-45185. - Resolution: Won't Fix > Ignore type check for preventing unexpected linter failure > ---

[jira] [Resolved] (SPARK-43630) Implement `localCheckpoint` for Spark Connect DataFrame

2023-09-16 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee resolved SPARK-43630. - Resolution: Won't Fix We can't support RDD-dependent feature in Spark Connect. > Implement `loc

<    1   2   3   4   5   6   7   8   9   10   >