[jira] [Commented] (SPARK-44734) Add documentation for type casting rules in Python UDFs/UDTFs

2023-10-18 Thread BingKun Pan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1043#comment-1043 ] BingKun Pan commented on SPARK-44734: - Okay, let me take a look at the two JIRAs above first. > Add

[jira] [Resolved] (SPARK-45220) Refine docstring of `DataFrame.join`

2023-10-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-45220. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43039

[jira] [Assigned] (SPARK-45220) Refine docstring of `DataFrame.join`

2023-10-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-45220: Assignee: Allison Wang > Refine docstring of `DataFrame.join` >

[jira] [Resolved] (SPARK-45589) Supplementary exception class

2023-10-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-45589. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43427

[jira] [Assigned] (SPARK-45589) Supplementary exception class

2023-10-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-45589: Assignee: BingKun Pan > Supplementary exception class > - >

[jira] [Resolved] (SPARK-45588) Minor scaladoc improvement in StreamingForeachBatchHelper

2023-10-18 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-45588. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43424

[jira] [Assigned] (SPARK-45588) Minor scaladoc improvement in StreamingForeachBatchHelper

2023-10-18 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-45588: Assignee: Raghu Angadi > Minor scaladoc improvement in StreamingForeachBatchHelper >

[jira] [Updated] (SPARK-45603) merge_spark_pr shall notice us about GITHUB_OAUTH_KEY expiry

2023-10-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45603: --- Labels: pull-request-available (was: ) > merge_spark_pr shall notice us about

[jira] [Assigned] (SPARK-45553) Deprecate assertPandasOnSparkEqual

2023-10-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-45553: Assignee: Haejoon Lee > Deprecate assertPandasOnSparkEqual >

[jira] [Resolved] (SPARK-45553) Deprecate assertPandasOnSparkEqual

2023-10-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-45553. -- Fix Version/s: 3.5.1 4.0.0 Resolution: Fixed Issue resolved by pull

[jira] [Created] (SPARK-45603) merge_spark_pr shall notice us about GITHUB_OAUTH_KEY expiry

2023-10-18 Thread Kent Yao (Jira)
Kent Yao created SPARK-45603: Summary: merge_spark_pr shall notice us about GITHUB_OAUTH_KEY expiry Key: SPARK-45603 URL: https://issues.apache.org/jira/browse/SPARK-45603 Project: Spark Issue

[jira] [Updated] (SPARK-45546) Make publish-snapshot support package first then deploy

2023-10-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-45546: - Fix Version/s: (was: 4.0.0) > Make publish-snapshot support package first then deploy >

[jira] [Resolved] (SPARK-45546) Make publish-snapshot support package first then deploy

2023-10-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-45546. -- Resolution: Invalid > Make publish-snapshot support package first then deploy >

[jira] [Assigned] (SPARK-45546) Make publish-snapshot support package first then deploy

2023-10-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-45546: Assignee: (was: Yang Jie) > Make publish-snapshot support package first then deploy

[jira] [Reopened] (SPARK-45546) Make publish-snapshot support package first then deploy

2023-10-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reopened SPARK-45546: -- Reverted at https://github.com/apache/spark/commit/706872d4de2374d1faf84d8706611a092c0b6e76 and

[jira] [Updated] (SPARK-45602) Replace `s.c.MapOps.filterKeys` with `s.c.MapOps.view.filterKeys`

2023-10-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45602: --- Labels: pull-request-available (was: ) > Replace `s.c.MapOps.filterKeys` with

[jira] [Created] (SPARK-45602) Replace `s.c.MapOps.filterKeys` with `s.c.MapOps.view.filterKeys`

2023-10-18 Thread Yang Jie (Jira)
Yang Jie created SPARK-45602: Summary: Replace `s.c.MapOps.filterKeys` with `s.c.MapOps.view.filterKeys` Key: SPARK-45602 URL: https://issues.apache.org/jira/browse/SPARK-45602 Project: Spark

[jira] [Updated] (SPARK-45601) stackoverflow when executing rule ExtractWindowExpressions

2023-10-18 Thread JacobZheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JacobZheng updated SPARK-45601: --- Description: I am encountering stackoverflow errors while executing the following test case. I

[jira] [Created] (SPARK-45601) stackoverflow when executing rule ExtractWindowExpressions

2023-10-18 Thread JacobZheng (Jira)
JacobZheng created SPARK-45601: -- Summary: stackoverflow when executing rule ExtractWindowExpressions Key: SPARK-45601 URL: https://issues.apache.org/jira/browse/SPARK-45601 Project: Spark Issue

[jira] [Updated] (SPARK-45596) Use java.lang.ref.Cleaner instead of org.apache.spark.sql.connect.client.util.Cleaner

2023-10-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45596: --- Labels: pull-request-available (was: ) > Use java.lang.ref.Cleaner instead of >

[jira] [Updated] (SPARK-45555) Returning a debuggable object for failed assertion

2023-10-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-4?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-4: --- Labels: pull-request-available (was: ) > Returning a debuggable object for failed

[jira] [Assigned] (SPARK-45507) Correctness bug in correlated scalar subqueries with COUNT aggregates

2023-10-18 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-45507: --- Assignee: Andy Lam > Correctness bug in correlated scalar subqueries with COUNT aggregates

[jira] [Resolved] (SPARK-45507) Correctness bug in correlated scalar subqueries with COUNT aggregates

2023-10-18 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-45507. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43341

[jira] [Commented] (SPARK-44817) SPIP: Incremental Stats Collection

2023-10-18 Thread Abhinav Kumar (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17776976#comment-17776976 ] Abhinav Kumar commented on SPARK-44817: --- [~rakson] [~gurwls223] [~cloud_fan] - We find this issue

[jira] [Assigned] (SPARK-45586) Reduce compilation time for large expression trees

2023-10-18 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-45586: --- Assignee: Kelvin Jiang > Reduce compilation time for large expression trees >

[jira] [Resolved] (SPARK-45586) Reduce compilation time for large expression trees

2023-10-18 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-45586. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43420

[jira] [Updated] (SPARK-45543) InferWindowGroupLimit causes bug if the other window functions haven't the same window frame as the rank-like functions

2023-10-18 Thread Jiaan Geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiaan Geng updated SPARK-45543: --- Summary: InferWindowGroupLimit causes bug if the other window functions haven't the same window

[jira] [Updated] (SPARK-45543) InferWindowGroupLimit causes bug if the window frame is different between rank-like functions and others

2023-10-18 Thread Jiaan Geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiaan Geng updated SPARK-45543: --- Summary: InferWindowGroupLimit causes bug if the window frame is different between rank-like

[jira] [Updated] (SPARK-45543) InferWindowGroupLimit causes bug if the other window functions haven't the same window frame as the rank-like functions

2023-10-18 Thread Jiaan Geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiaan Geng updated SPARK-45543: --- Summary: InferWindowGroupLimit causes bug if the other window functions haven't the same window

[jira] [Resolved] (SPARK-45585) Fix time format and redirection issues in SparkSubmit tests

2023-10-18 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-45585. -- Fix Version/s: 3.5.1 4.0.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-45585) Fix time format and redirection issues in SparkSubmit tests

2023-10-18 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao reassigned SPARK-45585: Assignee: Kent Yao > Fix time format and redirection issues in SparkSubmit tests >

[jira] [Commented] (SPARK-45023) SPIP: Python Stored Procedures

2023-10-18 Thread Abhinav Kumar (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17776963#comment-17776963 ] Abhinav Kumar commented on SPARK-45023: --- Not sure where we are with this. Looks like we are not

[jira] [Created] (SPARK-45600) Separate the Python data source logic from DataFrameReader

2023-10-18 Thread Allison Wang (Jira)
Allison Wang created SPARK-45600: Summary: Separate the Python data source logic from DataFrameReader Key: SPARK-45600 URL: https://issues.apache.org/jira/browse/SPARK-45600 Project: Spark

[jira] [Updated] (SPARK-44347) Upgrade janino to 3.1.10

2023-10-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-44347: --- Labels: pull-request-available (was: ) > Upgrade janino to 3.1.10 >

[jira] [Assigned] (SPARK-45582) Streaming aggregation in complete mode should not refer to store instance after commit

2023-10-18 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-45582: Assignee: Anish Shrigondekar > Streaming aggregation in complete mode should not refer

[jira] [Resolved] (SPARK-45582) Streaming aggregation in complete mode should not refer to store instance after commit

2023-10-18 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-45582. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43413

[jira] [Comment Edited] (SPARK-44734) Add documentation for type casting rules in Python UDFs/UDTFs

2023-10-18 Thread Philip Dakin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17776921#comment-17776921 ] Philip Dakin edited comment on SPARK-44734 at 10/18/23 9:47 PM:

[jira] [Commented] (SPARK-44734) Add documentation for type casting rules in Python UDFs/UDTFs

2023-10-18 Thread Philip Dakin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17776921#comment-17776921 ] Philip Dakin commented on SPARK-44734: -- [~panbingkun] please make sure changes operate well with

[jira] [Updated] (SPARK-45599) Percentile can produce a wrong answer if -0.0 and 0.0 are mixed in the dataset

2023-10-18 Thread Robert Joseph Evans (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated SPARK-45599: Priority: Blocker (was: Major) > Percentile can produce a wrong answer if -0.0

[jira] [Updated] (SPARK-45599) Percentile can produce a wrong answer if -0.0 and 0.0 are mixed in the dataset

2023-10-18 Thread Robert Joseph Evans (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated SPARK-45599: Labels: data-corruption (was: ) > Percentile can produce a wrong answer if -0.0

[jira] [Created] (SPARK-45599) Percentile can produce a wrong answer if -0.0 and 0.0 are mixed in the dataset

2023-10-18 Thread Robert Joseph Evans (Jira)
Robert Joseph Evans created SPARK-45599: --- Summary: Percentile can produce a wrong answer if -0.0 and 0.0 are mixed in the dataset Key: SPARK-45599 URL: https://issues.apache.org/jira/browse/SPARK-45599

[jira] [Updated] (SPARK-45230) Adjust sorter for Aggregate after SMJ

2023-10-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45230: --- Labels: pull-request-available (was: ) > Adjust sorter for Aggregate after SMJ >

[jira] [Updated] (SPARK-45598) Delta table 3.0-rc2 not working with Spark Connect 3.5.0

2023-10-18 Thread Faiz Halde (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Faiz Halde updated SPARK-45598: --- Description: Spark version 3.5.0 Spark Connect version 3.5.0 Delta table 3.0-rc2 Spark connect

[jira] [Updated] (SPARK-44526) Porting k8s PVC reuse logic to spark standalone

2023-10-18 Thread Faiz Halde (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Faiz Halde updated SPARK-44526: --- Affects Version/s: 3.4.1 (was: 3.5.0) > Porting k8s PVC reuse logic to

[jira] [Updated] (SPARK-44526) Porting k8s PVC reuse logic to spark standalone

2023-10-18 Thread Faiz Halde (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Faiz Halde updated SPARK-44526: --- Affects Version/s: 3.5.0 (was: 3.4.1) > Porting k8s PVC reuse logic to

[jira] [Created] (SPARK-45598) Delta table 3.0-rc2 not working with Spark Connect 3.5.0

2023-10-18 Thread Faiz Halde (Jira)
Faiz Halde created SPARK-45598: -- Summary: Delta table 3.0-rc2 not working with Spark Connect 3.5.0 Key: SPARK-45598 URL: https://issues.apache.org/jira/browse/SPARK-45598 Project: Spark Issue

[jira] [Updated] (SPARK-45559) Support spark.read.schema(...) for Python data source API

2023-10-18 Thread Allison Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allison Wang updated SPARK-45559: - Description: Support `spark.read.schema(...)` for Python data source read. Add test cases

[jira] [Created] (SPARK-45597) Support creating table using a Python data source in SQL

2023-10-18 Thread Allison Wang (Jira)
Allison Wang created SPARK-45597: Summary: Support creating table using a Python data source in SQL Key: SPARK-45597 URL: https://issues.apache.org/jira/browse/SPARK-45597 Project: Spark

[jira] [Updated] (SPARK-45596) Use java.lang.ref.Cleaner instead of org.apache.spark.sql.connect.client.util.Cleaner

2023-10-18 Thread Min Zhao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Min Zhao updated SPARK-45596: - Description: Now, we have updated JDK to 17,  so should replace this class by

[jira] [Updated] (SPARK-45596) Use java.lang.ref.Cleaner instead of org.apache.spark.sql.connect.client.util.Cleaner

2023-10-18 Thread Min Zhao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Min Zhao updated SPARK-45596: - Attachment: image-2023-10-19-02-25-57-966.png > Use java.lang.ref.Cleaner instead of >

[jira] [Created] (SPARK-45596) Use java.lang.ref.Cleaner instead of org.apache.spark.sql.connect.client.util.Cleaner

2023-10-18 Thread Min Zhao (Jira)
Min Zhao created SPARK-45596: Summary: Use java.lang.ref.Cleaner instead of org.apache.spark.sql.connect.client.util.Cleaner Key: SPARK-45596 URL: https://issues.apache.org/jira/browse/SPARK-45596

[jira] [Resolved] (SPARK-45581) Make SQLSTATEs mandatory

2023-10-18 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-45581. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43412

[jira] [Assigned] (SPARK-45581) Make SQLSTATEs mandatory

2023-10-18 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang reassigned SPARK-45581: -- Assignee: Serge Rielau > Make SQLSTATEs mandatory > > >

[jira] [Updated] (SPARK-45595) Expose SQLSTATE in error message

2023-10-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45595: --- Labels: pull-request-available (was: ) > Expose SQLSTATE in error message >

[jira] [Commented] (SPARK-45583) Spark SQL returning incorrect values for full outer join on keys with the same name.

2023-10-18 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17776783#comment-17776783 ] Bruce Robbins commented on SPARK-45583: --- Strangely, I cannot reproduce. Is some setting required?

[jira] [Resolved] (SPARK-45549) Remove unused `numExistingExecutors` in `CoarseGrainedSchedulerBackend`

2023-10-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-45549. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43383

[jira] [Assigned] (SPARK-45549) Remove unused `numExistingExecutors` in `CoarseGrainedSchedulerBackend`

2023-10-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-45549: - Assignee: xiaoping.huang > Remove unused `numExistingExecutors` in

[jira] [Updated] (SPARK-45595) Expose SQLSTATE in error message

2023-10-18 Thread Serge Rielau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Serge Rielau updated SPARK-45595: - Summary: Expose SQLSTATE in error message (was: Expose SQLSTATRE in errormessage) > Expose

[jira] [Created] (SPARK-45595) Expose SQLSTATRE in errormessage

2023-10-18 Thread Serge Rielau (Jira)
Serge Rielau created SPARK-45595: Summary: Expose SQLSTATRE in errormessage Key: SPARK-45595 URL: https://issues.apache.org/jira/browse/SPARK-45595 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-45594) Auto repartition before writing data into partitioned or bucket table

2023-10-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45594: --- Labels: pull-request-available (was: ) > Auto repartition before writing data into

[jira] [Updated] (SPARK-45594) Auto repartition before writing data into partitioned or bucket table

2023-10-18 Thread Wan Kun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wan Kun updated SPARK-45594: Description: Now, when writing data into partitioned table, there will be at least *dynamicPartitions *

[jira] [Updated] (SPARK-45593) Building a runnable distribution from master code running spark-sql raise error "java.lang.ClassNotFoundException: org.sparkproject.guava.util.concurrent.internal.Intern

2023-10-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45593: --- Labels: pull-request-available (was: ) > Building a runnable distribution from master code

[jira] [Created] (SPARK-45594) Auto repartition before writing data into partitioned or bucket table

2023-10-18 Thread Wan Kun (Jira)
Wan Kun created SPARK-45594: --- Summary: Auto repartition before writing data into partitioned or bucket table Key: SPARK-45594 URL: https://issues.apache.org/jira/browse/SPARK-45594 Project: Spark

[jira] [Updated] (SPARK-45593) Building a runnable distribution from master code running spark-sql raise error "java.lang.ClassNotFoundException: org.sparkproject.guava.util.concurrent.internal.Intern

2023-10-18 Thread yikaifei (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yikaifei updated SPARK-45593: - Description: Building a runnable distribution from master code running spark-sql raise error

[jira] [Updated] (SPARK-45593) Building a runnable distribution from master code running spark-sql raise error "java.lang.ClassNotFoundException: org.sparkproject.guava.util.concurrent.internal.Intern

2023-10-18 Thread yikaifei (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yikaifei updated SPARK-45593: - Description: Building a runnable distribution from master code running spark-sql raise error

[jira] [Updated] (SPARK-45593) Building a runnable distribution from master code running spark-sql raise error "java.lang.ClassNotFoundException: org.sparkproject.guava.util.concurrent.internal.Intern

2023-10-18 Thread yikaifei (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yikaifei updated SPARK-45593: - Description:   Reproducing steps: first, clone spark master code, then: # Build runnable

[jira] [Created] (SPARK-45593) Building a runnable distribution from master code running spark-sql raise error "java.lang.ClassNotFoundException: org.sparkproject.guava.util.concurrent.internal.Intern

2023-10-18 Thread yikaifei (Jira)
yikaifei created SPARK-45593: Summary: Building a runnable distribution from master code running spark-sql raise error "java.lang.ClassNotFoundException: org.sparkproject.guava.util.concurrent.internal.InternalFutureFailureAccess"

[jira] [Updated] (SPARK-45592) AQE and InMemoryTableScanExec correctness bug

2023-10-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45592: --- Labels: pull-request-available (was: ) > AQE and InMemoryTableScanExec correctness bug >

[jira] [Updated] (SPARK-45588) Minor scaladoc improvement in StreamingForeachBatchHelper

2023-10-18 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-45588: - Issue Type: Improvement (was: Bug) Priority: Trivial (was: Major) > Minor scaladoc

[jira] [Updated] (SPARK-45549) Remove unused `numExistingExecutors` in `CoarseGrainedSchedulerBackend`

2023-10-18 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-45549: - Priority: Trivial (was: Minor) > Remove unused `numExistingExecutors` in

[jira] [Created] (SPARK-45592) AQE and InMemoryTableScanExec correctness bug

2023-10-18 Thread Emil Ejbyfeldt (Jira)
Emil Ejbyfeldt created SPARK-45592: -- Summary: AQE and InMemoryTableScanExec correctness bug Key: SPARK-45592 URL: https://issues.apache.org/jira/browse/SPARK-45592 Project: Spark Issue

[jira] [Updated] (SPARK-45552) Introduce flexible parameters to assertDataFrameEqual

2023-10-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45552: --- Labels: pull-request-available (was: ) > Introduce flexible parameters to

[jira] [Commented] (SPARK-45570) Spark job hangs due to task launch thread failed to create

2023-10-18 Thread lifulong (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17776595#comment-17776595 ] lifulong commented on SPARK-45570: -- !image-2023-10-18-18-18-36-132.png! catch thread create exception

[jira] [Updated] (SPARK-45570) Spark job hangs due to task launch thread failed to create

2023-10-18 Thread lifulong (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lifulong updated SPARK-45570: - Attachment: image-2023-10-18-18-18-36-132.png > Spark job hangs due to task launch thread failed to

[jira] [Assigned] (SPARK-45586) Reduce compilation time for large expression trees

2023-10-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-45586: -- Assignee: (was: Apache Spark) > Reduce compilation time for large expression

[jira] [Assigned] (SPARK-45586) Reduce compilation time for large expression trees

2023-10-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-45586: -- Assignee: Apache Spark > Reduce compilation time for large expression trees >

[jira] [Updated] (SPARK-45591) Upgrade ASM to 9.6

2023-10-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45591: --- Labels: pull-request-available (was: ) > Upgrade ASM to 9.6 > -- > >

[jira] [Created] (SPARK-45591) Upgrade ASM to 9.6

2023-10-18 Thread Yang Jie (Jira)
Yang Jie created SPARK-45591: Summary: Upgrade ASM to 9.6 Key: SPARK-45591 URL: https://issues.apache.org/jira/browse/SPARK-45591 Project: Spark Issue Type: Improvement Components:

[jira] [Resolved] (SPARK-45370) Fix python test when ansi mode enabled

2023-10-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-45370. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43168

[jira] [Assigned] (SPARK-45370) Fix python test when ansi mode enabled

2023-10-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-45370: Assignee: BingKun Pan > Fix python test when ansi mode enabled >

[jira] [Updated] (SPARK-45574) Add :: syntax as a shorthand for casting

2023-10-18 Thread Ivan Mitic (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Mitic updated SPARK-45574: --- Description: Adds the `::` syntax as syntactic sugar for casting columns. This is a pretty common

[jira] [Updated] (SPARK-45574) Add :: syntax as a shorthand for casting

2023-10-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45574: --- Labels: pull-request-available release-notes (was: release-notes) > Add :: syntax as a

[jira] [Commented] (SPARK-44734) Add documentation for type casting rules in Python UDFs/UDTFs

2023-10-18 Thread BingKun Pan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17776533#comment-17776533 ] BingKun Pan commented on SPARK-44734: - I work on it. > Add documentation for type casting rules in

[jira] [Resolved] (SPARK-45542) Replace `setSafeMode(HdfsConstants.SafeModeAction, boolean)` with `setSafeMode(SafeModeAction, boolean)`

2023-10-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-45542. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43377

[jira] [Assigned] (SPARK-45542) Replace `setSafeMode(HdfsConstants.SafeModeAction, boolean)` with `setSafeMode(SafeModeAction, boolean)`

2023-10-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-45542: - Assignee: Yang Jie > Replace `setSafeMode(HdfsConstants.SafeModeAction, boolean)` with

[jira] [Assigned] (SPARK-44649) Runtime Filter supports passing equivalent creation side expressions

2023-10-18 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-44649: --- Assignee: Jiaan Geng > Runtime Filter supports passing equivalent creation side

[jira] [Resolved] (SPARK-44649) Runtime Filter supports passing equivalent creation side expressions

2023-10-18 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-44649. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42317

[jira] [Assigned] (SPARK-45558) Introduce a metadata file for streaming stateful operator

2023-10-18 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-45558: Assignee: Chaoqin Li > Introduce a metadata file for streaming stateful operator >

[jira] [Created] (SPARK-45590) okio-1.15.0 CVE-2023-3635

2023-10-18 Thread Colm O hEigeartaigh (Jira)
Colm O hEigeartaigh created SPARK-45590: --- Summary: okio-1.15.0 CVE-2023-3635 Key: SPARK-45590 URL: https://issues.apache.org/jira/browse/SPARK-45590 Project: Spark Issue Type: Task

[jira] [Resolved] (SPARK-45558) Introduce a metadata file for streaming stateful operator

2023-10-18 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-45558. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43393

[jira] [Updated] (SPARK-45589) Supplementary exception class

2023-10-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45589: --- Labels: pull-request-available (was: ) > Supplementary exception class >

[jira] [Created] (SPARK-45589) Supplementary exception class

2023-10-18 Thread BingKun Pan (Jira)
BingKun Pan created SPARK-45589: --- Summary: Supplementary exception class Key: SPARK-45589 URL: https://issues.apache.org/jira/browse/SPARK-45589 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-45553) Deprecate assertPandasOnSparkEqual

2023-10-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45553: --- Labels: pull-request-available (was: ) > Deprecate assertPandasOnSparkEqual >

[jira] [Updated] (SPARK-45570) Spark job hangs due to task launch thread failed to create

2023-10-18 Thread lifulong (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lifulong updated SPARK-45570: - Affects Version/s: 3.5.0 > Spark job hangs due to task launch thread failed to create >

[jira] [Updated] (SPARK-45570) Spark job hangs due to task launch thread failed to create

2023-10-18 Thread lifulong (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lifulong updated SPARK-45570: - Environment: spark.speculation is use default value false spark version 3.1.2   was:

[jira] [Assigned] (SPARK-45587) Skip UNIDOC and MIMA in build GitHub Action job

2023-10-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-45587: - Assignee: Dongjoon Hyun > Skip UNIDOC and MIMA in build GitHub Action job >

[jira] [Resolved] (SPARK-45587) Skip UNIDOC and MIMA in build GitHub Action job

2023-10-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-45587. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43422

[jira] [Resolved] (SPARK-45534) Use `java.lang.ref.Cleaner` instead of `finalize` for `RemoteBlockPushResolver`

2023-10-18 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-45534. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-45534) Use `java.lang.ref.Cleaner` instead of `finalize` for `RemoteBlockPushResolver`

2023-10-18 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-45534: --- Assignee: Min Zhao > Use `java.lang.ref.Cleaner` instead of `finalize` for

[jira] [Assigned] (SPARK-45035) Support ignoreCorruptFiles for multiline CSV

2023-10-18 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-45035: Assignee: Jia Fan > Support ignoreCorruptFiles for multiline CSV >

  1   2   >