[jira] [Resolved] (SPARK-44373) Wrap withActive for Dataset API w/ parse logic

2023-07-11 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-44373. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41938

[jira] [Assigned] (SPARK-44373) Wrap withActive for Dataset API w/ parse logic

2023-07-11 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao reassigned SPARK-44373: Assignee: Kent Yao > Wrap withActive for Dataset API w/ parse logic >

[jira] [Resolved] (SPARK-44334) Status of execution w/ error and w/o jobs shall be FAILED not COMPLETED

2023-07-11 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-44334. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41891

[jira] [Assigned] (SPARK-44334) Status of execution w/ error and w/o jobs shall be FAILED not COMPLETED

2023-07-11 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao reassigned SPARK-44334: Assignee: Kent Yao > Status of execution w/ error and w/o jobs shall be FAILED not COMPLETED >

[jira] [Resolved] (SPARK-44370) Migrate Buf remote generation alpha to remote plugins

2023-07-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-44370. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41933

[jira] [Commented] (SPARK-43755) Spark Connect - decouple query execution from RPC handler

2023-07-11 Thread Snoot.io (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17742252#comment-17742252 ] Snoot.io commented on SPARK-43755: -- User 'juliuszsompolski' has created a pull request for this issue:

[jira] [Created] (SPARK-44381) How to specify parameters in spark-sumbit to make HiveDelegationTokenProvider refresh token regularly

2023-07-11 Thread qingbo jiao (Jira)
qingbo jiao created SPARK-44381: --- Summary: How to specify parameters in spark-sumbit to make HiveDelegationTokenProvider refresh token regularly Key: SPARK-44381 URL:

[jira] [Resolved] (SPARK-44340) Define the computing logic through PartitionEvaluator API and use it in WindowGroupLimitExec

2023-07-11 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-44340. - Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41899

[jira] [Assigned] (SPARK-44340) Define the computing logic through PartitionEvaluator API and use it in WindowGroupLimitExec

2023-07-11 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-44340: --- Assignee: jiaan.geng > Define the computing logic through PartitionEvaluator API and use

[jira] [Commented] (SPARK-44340) Define the computing logic through PartitionEvaluator API and use it in WindowGroupLimitExec

2023-07-11 Thread Snoot.io (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17742246#comment-17742246 ] Snoot.io commented on SPARK-44340: -- User 'beliefer' has created a pull request for this issue:

[jira] [Commented] (SPARK-44340) Define the computing logic through PartitionEvaluator API and use it in WindowGroupLimitExec

2023-07-11 Thread Snoot.io (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17742247#comment-17742247 ] Snoot.io commented on SPARK-44340: -- User 'beliefer' has created a pull request for this issue:

[jira] [Resolved] (SPARK-43665) Enable PandasSQLStringFormatter.vformat to work with Spark Connect

2023-07-11 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-43665. --- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41931

[jira] [Assigned] (SPARK-43665) Enable PandasSQLStringFormatter.vformat to work with Spark Connect

2023-07-11 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-43665: - Assignee: Haejoon Lee > Enable PandasSQLStringFormatter.vformat to work with Spark

[jira] [Resolved] (SPARK-44325) Define the computing logic through PartitionEvaluator API and use it in SortMergeJoinExec

2023-07-11 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-44325. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41884

[jira] [Assigned] (SPARK-44325) Define the computing logic through PartitionEvaluator API and use it in SortMergeJoinExec

2023-07-11 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao reassigned SPARK-44325: Assignee: Vinod KC > Define the computing logic through PartitionEvaluator API and use it in >

[jira] [Resolved] (SPARK-44377) exclude junit5 deps from jersey-test-framework-provider-simple

2023-07-11 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie resolved SPARK-44377. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41944

[jira] [Commented] (SPARK-44377) exclude junit5 deps from jersey-test-framework-provider-simple

2023-07-11 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17742234#comment-17742234 ] Yang Jie commented on SPARK-44377: -- Fixed > exclude junit5 deps from

[jira] [Assigned] (SPARK-44377) exclude junit5 deps from jersey-test-framework-provider-simple

2023-07-11 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie reassigned SPARK-44377: Assignee: Yang Jie > exclude junit5 deps from jersey-test-framework-provider-simple >

[jira] [Updated] (SPARK-44374) Add example code

2023-07-11 Thread Weichen Xu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu updated SPARK-44374: --- Fix Version/s: 3.5.0 > Add example code > > > Key: SPARK-44374 >

[jira] [Resolved] (SPARK-44374) Add example code

2023-07-11 Thread Weichen Xu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu resolved SPARK-44374. Resolution: Done > Add example code > > > Key: SPARK-44374 >

[jira] [Commented] (SPARK-44362) Use PartitionEvaluator API in AggregateInPandasExec,EvalPythonExec,AttachDistributedSequenceExec

2023-07-11 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1774#comment-1774 ] jiaan.geng commented on SPARK-44362: Thank you. > Use PartitionEvaluator API in >

[jira] [Created] (SPARK-44380) Support for UDTF to analyze in Python

2023-07-11 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-44380: - Summary: Support for UDTF to analyze in Python Key: SPARK-44380 URL: https://issues.apache.org/jira/browse/SPARK-44380 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-44217) Allow custom precision for fp approx equality

2023-07-11 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-44217: --- Summary: Allow custom precision for fp approx equality (was: Add assert_approx_df_equality util

[jira] [Resolved] (SPARK-44264) DeepSpeed Distrobutor

2023-07-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-44264. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41770

[jira] [Commented] (SPARK-43513) withColumnRenamed duplicates columns if new column already exists

2023-07-11 Thread Frederik Paradis (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17742175#comment-17742175 ] Frederik Paradis commented on SPARK-43513: -- Hi [~wenxin]. Thank you for your comment. 1.

[jira] [Comment Edited] (SPARK-43513) withColumnRenamed duplicates columns if new column already exists

2023-07-11 Thread Frederik Paradis (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17742175#comment-17742175 ] Frederik Paradis edited comment on SPARK-43513 at 7/11/23 9:02 PM: --- Hi

[jira] [Commented] (SPARK-44279) Upgrade word-wrap

2023-07-11 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-44279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17742146#comment-17742146 ] Bjørn Jørgensen commented on SPARK-44279: - have a look at

[jira] [Commented] (SPARK-44279) Upgrade word-wrap

2023-07-11 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17742139#comment-17742139 ] Sean R. Owen commented on SPARK-44279: -- This is a dumb question, but what is that file? packages

[jira] [Commented] (SPARK-44279) Upgrade word-wrap

2023-07-11 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-44279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17742137#comment-17742137 ] Bjørn Jørgensen commented on SPARK-44279: - [~srowen]

[jira] [Updated] (SPARK-44262) JdbcUtils hardcodes some SQL statements

2023-07-11 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-44262: - Issue Type: Improvement (was: Bug) Priority: Minor (was: Major) > JdbcUtils hardcodes

[jira] [Updated] (SPARK-43439) Drop does not work when passed a string with an alias

2023-07-11 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-43439: - Priority: Minor (was: Major) > Drop does not work when passed a string with an alias >

[jira] [Resolved] (SPARK-44058) Remove deprecated API usage in HiveShim.scala

2023-07-11 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-44058. -- Resolution: Not A Problem > Remove deprecated API usage in HiveShim.scala >

[jira] [Updated] (SPARK-44379) Broadcast Joins taking up too much memory

2023-07-11 Thread Shardul Mahadik (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shardul Mahadik updated SPARK-44379: Description: Context: After migrating to Spark 3 with AQE, we saw a significant increase

[jira] [Commented] (SPARK-44379) Broadcast Joins taking up too much memory

2023-07-11 Thread Shardul Mahadik (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17742126#comment-17742126 ] Shardul Mahadik commented on SPARK-44379: - cc: [~cloud_fan] [~joshrosen] [~mridul] Would be

[jira] [Updated] (SPARK-44379) Broadcast Joins taking up too much memory

2023-07-11 Thread Shardul Mahadik (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shardul Mahadik updated SPARK-44379: Description: Context: After migrating to Spark 3 with AQE, we saw a significant increase

[jira] [Updated] (SPARK-44379) Broadcast Joins taking up too much memory

2023-07-11 Thread Shardul Mahadik (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shardul Mahadik updated SPARK-44379: Attachment: screenshot-1.png > Broadcast Joins taking up too much memory >

[jira] [Updated] (SPARK-44379) Broadcast Joins taking up too much memory

2023-07-11 Thread Shardul Mahadik (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shardul Mahadik updated SPARK-44379: Attachment: screenshot-2.png > Broadcast Joins taking up too much memory >

[jira] [Created] (SPARK-44379) Broadcast Joins taking up too much memory

2023-07-11 Thread Shardul Mahadik (Jira)
Shardul Mahadik created SPARK-44379: --- Summary: Broadcast Joins taking up too much memory Key: SPARK-44379 URL: https://issues.apache.org/jira/browse/SPARK-44379 Project: Spark Issue Type:

[jira] [Commented] (SPARK-44279) Upgrade word-wrap

2023-07-11 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17742120#comment-17742120 ] Sean R. Owen commented on SPARK-44279: -- Is this a library that's used in spark? I couldn't find it

[jira] [Resolved] (SPARK-44304) Broadcast operation is not required when no parameters are specified

2023-07-11 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-44304. -- Resolution: Duplicate > Broadcast operation is not required when no parameters are specified

[jira] [Commented] (SPARK-44377) exclude junit5 deps from jersey-test-framework-provider-simple

2023-07-11 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17742105#comment-17742105 ] Sean R. Owen commented on SPARK-44377: -- Sure can you open a PR? > exclude junit5 deps from

[jira] [Commented] (SPARK-44376) Build using maven is broken using 2.13 and Java 11 and Java 17

2023-07-11 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17742104#comment-17742104 ] Sean R. Owen commented on SPARK-44376: -- Did you run dev/change-scala-version.sh 2.13 ? > Build

[jira] [Updated] (SPARK-44378) Jobs that have join & have .rdd calls get executed 2x when AQE is enabled.

2023-07-11 Thread Priyanka Raju (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyanka Raju updated SPARK-44378: -- Attachment: image2.png > Jobs that have join & have .rdd calls get executed 2x when AQE is

[jira] [Updated] (SPARK-44378) Jobs that have join & have .rdd calls get executed 2x when AQE is enabled.

2023-07-11 Thread Priyanka Raju (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyanka Raju updated SPARK-44378: -- Description: We have a few spark scala jobs that are currently running in production. Most

[jira] [Updated] (SPARK-44362) Use PartitionEvaluator API in AggregateInPandasExec,EvalPythonExec,AttachDistributedSequenceExec

2023-07-11 Thread Vinod KC (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod KC updated SPARK-44362: - Summary: Use PartitionEvaluator API in

[jira] [Updated] (SPARK-44362) Use PartitionEvaluator API in AggregateInPandasExec, WindowInPandasExec,EvalPythonExec,AttachDistributedSequenceExec

2023-07-11 Thread Vinod KC (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod KC updated SPARK-44362: - Description: Use  PartitionEvaluator API in AggregateInPandasExec EvalPythonExec

[jira] [Commented] (SPARK-44362) Use PartitionEvaluator API in AggregateInPandasExec, WindowInPandasExec,EvalPythonExec,AttachDistributedSequenceExec

2023-07-11 Thread Vinod KC (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17742099#comment-17742099 ] Vinod KC commented on SPARK-44362: -- yes, please go ahead > Use PartitionEvaluator API in

[jira] [Updated] (SPARK-44378) Jobs that have join & have .rdd calls get executed 2x when AQE is enabled.

2023-07-11 Thread Priyanka Raju (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyanka Raju updated SPARK-44378: -- Description: We have a few spark scala jobs that are currently running in production. Most

[jira] [Updated] (SPARK-44378) Jobs that have join & have .rdd calls get executed 2x when AQE is enabled.

2023-07-11 Thread Priyanka Raju (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyanka Raju updated SPARK-44378: -- Attachment: Screenshot 2023-07-11 at 9.36.19 AM.png > Jobs that have join & have .rdd calls

[jira] [Updated] (SPARK-44378) Jobs that have join & have .rdd calls get executed 2x when AQE is enabled.

2023-07-11 Thread Priyanka Raju (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyanka Raju updated SPARK-44378: -- Attachment: Screenshot 2023-07-11 at 9.36.14 AM.png > Jobs that have join & have .rdd calls

[jira] [Updated] (SPARK-44378) Jobs that have join & have .rdd calls get executed 2x when AQE is enabled.

2023-07-11 Thread Priyanka Raju (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyanka Raju updated SPARK-44378: -- Description: We have a few spark scala jobs that are currently running in production. Most

[jira] [Created] (SPARK-44378) Jobs that have join & have .rdd calls get executed 2x when AQE is enabled.

2023-07-11 Thread Priyanka Raju (Jira)
Priyanka Raju created SPARK-44378: - Summary: Jobs that have join & have .rdd calls get executed 2x when AQE is enabled. Key: SPARK-44378 URL: https://issues.apache.org/jira/browse/SPARK-44378

[jira] [Resolved] (SPARK-44360) Support schema pruning in delta-based MERGE operations

2023-07-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-44360. --- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41930

[jira] [Assigned] (SPARK-44360) Support schema pruning in delta-based MERGE operations

2023-07-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-44360: - Assignee: Anton Okolnychyi > Support schema pruning in delta-based MERGE operations >

[jira] [Updated] (SPARK-44377) exclude junit5 deps from jersey-test-framework-provider-simple

2023-07-11 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-44377: - Description: SPARK-44316 upgrade Jersey from 2.36 to 2.40. Jersey 2.38 start to use [Junit5 instead of

[jira] [Created] (SPARK-44377) exclude junit5 deps from jersey-test-framework-provider-simple

2023-07-11 Thread Yang Jie (Jira)
Yang Jie created SPARK-44377: Summary: exclude junit5 deps from jersey-test-framework-provider-simple Key: SPARK-44377 URL: https://issues.apache.org/jira/browse/SPARK-44377 Project: Spark

[jira] [Created] (SPARK-44376) Build using maven is broken using 2.13 and Java 11 and Java 17

2023-07-11 Thread Emil Ejbyfeldt (Jira)
Emil Ejbyfeldt created SPARK-44376: -- Summary: Build using maven is broken using 2.13 and Java 11 and Java 17 Key: SPARK-44376 URL: https://issues.apache.org/jira/browse/SPARK-44376 Project: Spark

[jira] [Comment Edited] (SPARK-33782) Place spark.files, spark.jars and spark.files under the current working directory on the driver in K8S cluster mode

2023-07-11 Thread Pratik Malani (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17742006#comment-17742006 ] Pratik Malani edited comment on SPARK-33782 at 7/11/23 1:33 PM: Hi

[jira] [Commented] (SPARK-33782) Place spark.files, spark.jars and spark.files under the current working directory on the driver in K8S cluster mode

2023-07-11 Thread Pratik Malani (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17742006#comment-17742006 ] Pratik Malani commented on SPARK-33782: --- Hi [~pralabhkumar]  The latest update in the

[jira] [Created] (SPARK-44375) Use PartitionEvaluator API in DebugExec

2023-07-11 Thread Jia Fan (Jira)
Jia Fan created SPARK-44375: --- Summary: Use PartitionEvaluator API in DebugExec Key: SPARK-44375 URL: https://issues.apache.org/jira/browse/SPARK-44375 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-44375) Use PartitionEvaluator API in DebugExec

2023-07-11 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17742004#comment-17742004 ] Jia Fan commented on SPARK-44375: - I'm working on it. > Use PartitionEvaluator API in DebugExec >

[jira] [Created] (SPARK-44374) Add example code

2023-07-11 Thread Weichen Xu (Jira)
Weichen Xu created SPARK-44374: -- Summary: Add example code Key: SPARK-44374 URL: https://issues.apache.org/jira/browse/SPARK-44374 Project: Spark Issue Type: Sub-task Components:

[jira] [Assigned] (SPARK-44374) Add example code

2023-07-11 Thread Weichen Xu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu reassigned SPARK-44374: -- Assignee: Weichen Xu > Add example code > > > Key:

[jira] [Assigned] (SPARK-42471) Distributed ML <> spark connect

2023-07-11 Thread Weichen Xu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu reassigned SPARK-42471: -- Assignee: Weichen Xu > Distributed ML <> spark connect > --- > >

[jira] [Updated] (SPARK-44341) Define the computing logic through PartitionEvaluator API and use it in WindowExec and WindowInPandasExec

2023-07-11 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-44341: --- Summary: Define the computing logic through PartitionEvaluator API and use it in WindowExec and

[jira] [Created] (SPARK-44373) Wrap withActive for Dataset API w/ parse logic

2023-07-11 Thread Kent Yao (Jira)
Kent Yao created SPARK-44373: Summary: Wrap withActive for Dataset API w/ parse logic Key: SPARK-44373 URL: https://issues.apache.org/jira/browse/SPARK-44373 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-38476) Use error classes in org.apache.spark.storage

2023-07-11 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-38476: Assignee: Bo Zhang > Use error classes in org.apache.spark.storage >

[jira] [Resolved] (SPARK-38476) Use error classes in org.apache.spark.storage

2023-07-11 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-38476. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41923

[jira] [Updated] (SPARK-44354) Cannot create dataframe with CharType/VarcharType column

2023-07-11 Thread Kai-Michael Roesner (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai-Michael Roesner updated SPARK-44354: Description: When trying to create a dataframe with a CharType or VarcharType

[jira] [Commented] (SPARK-44354) Cannot create dataframe with CharType/VarcharType column

2023-07-11 Thread Kai-Michael Roesner (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17741925#comment-17741925 ] Kai-Michael Roesner commented on SPARK-44354: - PS: I tried to work around the exception by

[jira] [Commented] (SPARK-44362) Use PartitionEvaluator API in AggregateInPandasExec, WindowInPandasExec,EvalPythonExec,AttachDistributedSequenceExec

2023-07-11 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17741921#comment-17741921 ] jiaan.geng commented on SPARK-44362: [~vinodkc] Because WindowInPandasExec related to WindowExec,

[jira] [Commented] (SPARK-43665) Enable PandasSQLStringFormatter.vformat to work with Spark Connect

2023-07-11 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17741916#comment-17741916 ] ASF GitHub Bot commented on SPARK-43665: User 'itholic' has created a pull request for this

[jira] [Commented] (SPARK-43665) Enable PandasSQLStringFormatter.vformat to work with Spark Connect

2023-07-11 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17741915#comment-17741915 ] ASF GitHub Bot commented on SPARK-43665: User 'itholic' has created a pull request for this

[jira] [Assigned] (SPARK-44263) Allow ChannelBuilder extensions -- Scala

2023-07-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-44263: Assignee: Alice Sayutina > Allow ChannelBuilder extensions -- Scala >

[jira] [Resolved] (SPARK-44263) Allow ChannelBuilder extensions -- Scala

2023-07-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-44263. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41880

[jira] [Resolved] (SPARK-44320) Assign names to the error class _LEGACY_ERROR_TEMP_[1067,1150,1220,1265,1277]

2023-07-11 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-44320. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41909

[jira] [Assigned] (SPARK-44320) Assign names to the error class _LEGACY_ERROR_TEMP_[1067,1150,1220,1265,1277]

2023-07-11 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-44320: Assignee: BingKun Pan > Assign names to the error class

[jira] [Created] (SPARK-44372) Enable KernelDensity within Spark Connect

2023-07-11 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-44372: --- Summary: Enable KernelDensity within Spark Connect Key: SPARK-44372 URL: https://issues.apache.org/jira/browse/SPARK-44372 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-43629) Enable RDD dependent tests with Spark Connect

2023-07-11 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-43629: Summary: Enable RDD dependent tests with Spark Connect (was: Enable RDD with Spark Connect) >

[jira] [Commented] (SPARK-44371) Define the computing logic through PartitionEvaluator API and use it in CollectLimitExec, CollectTailExec, LocalLimitExec and GlobalLimitExec

2023-07-11 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17741882#comment-17741882 ] jiaan.geng commented on SPARK-44371: I'm working on. > Define the computing logic through

[jira] [Created] (SPARK-44371) Define the computing logic through PartitionEvaluator API and use it in CollectLimitExec, CollectTailExec, LocalLimitExec and GlobalLimitExec

2023-07-11 Thread jiaan.geng (Jira)
jiaan.geng created SPARK-44371: -- Summary: Define the computing logic through PartitionEvaluator API and use it in CollectLimitExec, CollectTailExec, LocalLimitExec and GlobalLimitExec Key: SPARK-44371 URL:

[jira] [Created] (SPARK-44370) Migrate Buf remote generation alpha to remote plugins

2023-07-11 Thread Jia Fan (Jira)
Jia Fan created SPARK-44370: --- Summary: Migrate Buf remote generation alpha to remote plugins Key: SPARK-44370 URL: https://issues.apache.org/jira/browse/SPARK-44370 Project: Spark Issue Type: Bug