[jira] [Created] (SPARK-48676) Structured Logging Framework Scala Style Migration [Part 2]

2024-06-20 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-48676: -- Summary: Structured Logging Framework Scala Style Migration [Part 2] Key: SPARK-48676 URL: https://issues.apache.org/jira/browse/SPARK-48676 Project: Spark

[jira] [Resolved] (SPARK-48632) Remove unused LogKeys

2024-06-14 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu resolved SPARK-48632. Resolution: Not A Problem > Remove unused LogKeys > - > >

[jira] [Created] (SPARK-48632) Remove unused LogKeys

2024-06-14 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-48632: -- Summary: Remove unused LogKeys Key: SPARK-48632 URL: https://issues.apache.org/jira/browse/SPARK-48632 Project: Spark Issue Type: Sub-task Components:

[jira] [Created] (SPARK-48628) Add task peak on/off heap execution memory metrics

2024-06-14 Thread Ziqi Liu (Jira)
Ziqi Liu created SPARK-48628: Summary: Add task peak on/off heap execution memory metrics Key: SPARK-48628 URL: https://issues.apache.org/jira/browse/SPARK-48628 Project: Spark Issue Type:

[jira] [Created] (SPARK-48623) Structured Logging Framework Scala Style Migration

2024-06-13 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-48623: -- Summary: Structured Logging Framework Scala Style Migration Key: SPARK-48623 URL: https://issues.apache.org/jira/browse/SPARK-48623 Project: Spark Issue Type:

[jira] [Created] (SPARK-48610) Remove ExplainUtils.processPlan synchronize

2024-06-12 Thread Ziqi Liu (Jira)
Ziqi Liu created SPARK-48610: Summary: Remove ExplainUtils.processPlan synchronize Key: SPARK-48610 URL: https://issues.apache.org/jira/browse/SPARK-48610 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-48592) Add scala style check for logging message inline variables

2024-06-11 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-48592: -- Summary: Add scala style check for logging message inline variables Key: SPARK-48592 URL: https://issues.apache.org/jira/browse/SPARK-48592 Project: Spark Issue

[jira] [Created] (SPARK-48589) Add option snapshotStartBatchId and snapshotPartitionId to state data source

2024-06-11 Thread Yuchen Liu (Jira)
Yuchen Liu created SPARK-48589: -- Summary: Add option snapshotStartBatchId and snapshotPartitionId to state data source Key: SPARK-48589 URL: https://issues.apache.org/jira/browse/SPARK-48589 Project:

[jira] [Created] (SPARK-48588) Fine-grained State Data Source

2024-06-11 Thread Yuchen Liu (Jira)
Yuchen Liu created SPARK-48588: -- Summary: Fine-grained State Data Source Key: SPARK-48588 URL: https://issues.apache.org/jira/browse/SPARK-48588 Project: Spark Issue Type: Epic

[jira] [Created] (SPARK-48567) Pyspark StreamingQuery lastProgress and friend should return actual StreamingQueryProgress

2024-06-07 Thread Wei Liu (Jira)
Wei Liu created SPARK-48567: --- Summary: Pyspark StreamingQuery lastProgress and friend should return actual StreamingQueryProgress Key: SPARK-48567 URL: https://issues.apache.org/jira/browse/SPARK-48567

[jira] [Created] (SPARK-48542) Give snapshotStartBatchId and snapshotPartitionId to the state data source

2024-06-05 Thread Yuchen Liu (Jira)
Yuchen Liu created SPARK-48542: -- Summary: Give snapshotStartBatchId and snapshotPartitionId to the state data source Key: SPARK-48542 URL: https://issues.apache.org/jira/browse/SPARK-48542 Project:

[jira] [Created] (SPARK-48482) dropDuplicates and dropDuplicatesWithinWatermark should accept varargs

2024-05-30 Thread Wei Liu (Jira)
Wei Liu created SPARK-48482: --- Summary: dropDuplicates and dropDuplicatesWithinWatermark should accept varargs Key: SPARK-48482 URL: https://issues.apache.org/jira/browse/SPARK-48482 Project: Spark

[jira] [Created] (SPARK-48480) StreamingQueryListener thread should not be interruptable

2024-05-30 Thread Wei Liu (Jira)
Wei Liu created SPARK-48480: --- Summary: StreamingQueryListener thread should not be interruptable Key: SPARK-48480 URL: https://issues.apache.org/jira/browse/SPARK-48480 Project: Spark Issue Type:

[jira] [Created] (SPARK-48468) Add LogicalQueryStage interface in catalyst

2024-05-29 Thread Ziqi Liu (Jira)
Ziqi Liu created SPARK-48468: Summary: Add LogicalQueryStage interface in catalyst Key: SPARK-48468 URL: https://issues.apache.org/jira/browse/SPARK-48468 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-48465) Avoid no-op empty relation propagation in AQE

2024-05-29 Thread Ziqi Liu (Jira)
Ziqi Liu created SPARK-48465: Summary: Avoid no-op empty relation propagation in AQE Key: SPARK-48465 URL: https://issues.apache.org/jira/browse/SPARK-48465 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-48447) Check state store provider class before invoking the constructor

2024-05-28 Thread Yuchen Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuchen Liu updated SPARK-48447: --- Priority: Major (was: Minor) > Check state store provider class before invoking the constructor >

[jira] [Created] (SPARK-48447) Check state store provider class before invoking the constructor

2024-05-28 Thread Yuchen Liu (Jira)
Yuchen Liu created SPARK-48447: -- Summary: Check state store provider class before invoking the constructor Key: SPARK-48447 URL: https://issues.apache.org/jira/browse/SPARK-48447 Project: Spark

[jira] [Created] (SPARK-48446) Update SS Doc of dropDuplicatesWithinWatermark to use the right syntax

2024-05-28 Thread Yuchen Liu (Jira)
Yuchen Liu created SPARK-48446: -- Summary: Update SS Doc of dropDuplicatesWithinWatermark to use the right syntax Key: SPARK-48446 URL: https://issues.apache.org/jira/browse/SPARK-48446 Project: Spark

[jira] [Commented] (SPARK-48411) Add E2E test for DropDuplicateWithinWatermark

2024-05-24 Thread Yuchen Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17849356#comment-17849356 ] Yuchen Liu commented on SPARK-48411: I will work on this. > Add E2E test for

[jira] [Commented] (SPARK-48411) Add E2E test for DropDuplicateWithinWatermark

2024-05-24 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17849348#comment-17849348 ] Wei Liu commented on SPARK-48411: - Sorry I tagged the wrong Yuchen > Add E2E test for

[jira] [Created] (SPARK-48411) Add E2E test for DropDuplicateWithinWatermark

2024-05-24 Thread Wei Liu (Jira)
Wei Liu created SPARK-48411: --- Summary: Add E2E test for DropDuplicateWithinWatermark Key: SPARK-48411 URL: https://issues.apache.org/jira/browse/SPARK-48411 Project: Spark Issue Type: New Feature

[jira] [Commented] (SPARK-48411) Add E2E test for DropDuplicateWithinWatermark

2024-05-24 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17849202#comment-17849202 ] Wei Liu commented on SPARK-48411: - [~liuyuchen777] is going to work on this > Add E2E test for

[jira] [Resolved] (SPARK-47946) Nested field's nullable value could be invalid after extracted using GetStructField

2024-05-15 Thread Linhong Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Linhong Liu resolved SPARK-47946. - Resolution: Not A Problem > Nested field's nullable value could be invalid after extracted

[jira] [Commented] (SPARK-47946) Nested field's nullable value could be invalid after extracted using GetStructField

2024-05-15 Thread Linhong Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17846773#comment-17846773 ] Linhong Liu commented on SPARK-47946: - No, it's not an issue. think about this   ||key||value

[jira] [Created] (SPARK-48241) CSV parsing failure with char/varchar type columns

2024-05-11 Thread Jiayi Liu (Jira)
Jiayi Liu created SPARK-48241: - Summary: CSV parsing failure with char/varchar type columns Key: SPARK-48241 URL: https://issues.apache.org/jira/browse/SPARK-48241 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-48181) Unify StreamingPythonRunner and PythonPlannerRunner

2024-05-07 Thread Wei Liu (Jira)
Wei Liu created SPARK-48181: --- Summary: Unify StreamingPythonRunner and PythonPlannerRunner Key: SPARK-48181 URL: https://issues.apache.org/jira/browse/SPARK-48181 Project: Spark Issue Type: New

[jira] [Created] (SPARK-48147) Remove all client listeners when local spark session is deleted

2024-05-06 Thread Wei Liu (Jira)
Wei Liu created SPARK-48147: --- Summary: Remove all client listeners when local spark session is deleted Key: SPARK-48147 URL: https://issues.apache.org/jira/browse/SPARK-48147 Project: Spark Issue

[jira] [Created] (SPARK-48093) Add config to switch between client side listener and server side listener

2024-05-02 Thread Wei Liu (Jira)
Wei Liu created SPARK-48093: --- Summary: Add config to switch between client side listener and server side listener Key: SPARK-48093 URL: https://issues.apache.org/jira/browse/SPARK-48093 Project: Spark

[jira] [Created] (SPARK-48002) Add Observed metrics test in PySpark StreamingQueryListeners

2024-04-26 Thread Wei Liu (Jira)
Wei Liu created SPARK-48002: --- Summary: Add Observed metrics test in PySpark StreamingQueryListeners Key: SPARK-48002 URL: https://issues.apache.org/jira/browse/SPARK-48002 Project: Spark Issue

[jira] [Created] (SPARK-47877) Speed up test_parity_listener

2024-04-16 Thread Wei Liu (Jira)
Wei Liu created SPARK-47877: --- Summary: Speed up test_parity_listener Key: SPARK-47877 URL: https://issues.apache.org/jira/browse/SPARK-47877 Project: Spark Issue Type: New Feature

[jira] [Updated] (SPARK-47718) .sql() does not recognize watermark defined upstream

2024-04-09 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-47718: Labels: (was: pull-request-available) > .sql() does not recognize watermark defined upstream >

[jira] [Created] (SPARK-47722) Wait until RocksDB background work finish before closing

2024-04-03 Thread Wei Liu (Jira)
Wei Liu created SPARK-47722: --- Summary: Wait until RocksDB background work finish before closing Key: SPARK-47722 URL: https://issues.apache.org/jira/browse/SPARK-47722 Project: Spark Issue Type:

[jira] [Updated] (SPARK-47542) spark cannot hit oracle's index when column type is DATE

2024-03-25 Thread Danke Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danke Liu updated SPARK-47542: -- Description: When I use spark's jdbc to pull data from oracle, it will not hit the index if the

[jira] [Updated] (SPARK-47542) spark cannot hit oracle's index when column type is DATE

2024-03-25 Thread Danke Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danke Liu updated SPARK-47542: -- Description: When I use spark's jdbc to pull data from oracle, it will not hit the index if the

[jira] [Updated] (SPARK-47542) spark cannot hit oracle's index when column type is DATE

2024-03-25 Thread Danke Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danke Liu updated SPARK-47542: -- Description: When I use spark's jdbc to pull data from oracle, it will not hit the index if the

[jira] [Updated] (SPARK-47542) spark cannot hit oracle's index when column type is DATE

2024-03-25 Thread Danke Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danke Liu updated SPARK-47542: -- Description: When I use spark's jdbc to pull data from oracle, it will not hit the index if the

[jira] [Updated] (SPARK-47542) spark cannot hit oracle's index when column type is DATE

2024-03-25 Thread Danke Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danke Liu updated SPARK-47542: -- Description: When I use spark's jdbc to pull data from oracle, it will not hit the index if the

[jira] [Updated] (SPARK-47542) spark cannot hit oracle's index when column type is DATE

2024-03-25 Thread Danke Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danke Liu updated SPARK-47542: -- Description: When I use spark's jdbc to pull data from oracle, it will not hit the index if the

[jira] [Updated] (SPARK-47542) spark cannot hit oracle's index when column type is DATE

2024-03-25 Thread Danke Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danke Liu updated SPARK-47542: -- Description: When I use spark's jdbc to pull data from oracle, it will not hit the index if the

[jira] [Created] (SPARK-47542) spark cannot hit oracle's index when column type is DATE

2024-03-25 Thread Danke Liu (Jira)
Danke Liu created SPARK-47542: - Summary: spark cannot hit oracle's index when column type is DATE Key: SPARK-47542 URL: https://issues.apache.org/jira/browse/SPARK-47542 Project: Spark Issue

[jira] [Created] (SPARK-47332) StreamingPythonRunner don't need redundant logic for starting python process

2024-03-08 Thread Wei Liu (Jira)
Wei Liu created SPARK-47332: --- Summary: StreamingPythonRunner don't need redundant logic for starting python process Key: SPARK-47332 URL: https://issues.apache.org/jira/browse/SPARK-47332 Project: Spark

[jira] [Created] (SPARK-47292) safeMapToJValue should consider when map is null

2024-03-05 Thread Wei Liu (Jira)
Wei Liu created SPARK-47292: --- Summary: safeMapToJValue should consider when map is null Key: SPARK-47292 URL: https://issues.apache.org/jira/browse/SPARK-47292 Project: Spark Issue Type: New

[jira] [Created] (SPARK-47277) PySpark util function assertDataFrameEqual should not support streaming DF

2024-03-04 Thread Wei Liu (Jira)
Wei Liu created SPARK-47277: --- Summary: PySpark util function assertDataFrameEqual should not support streaming DF Key: SPARK-47277 URL: https://issues.apache.org/jira/browse/SPARK-47277 Project: Spark

[jira] [Updated] (SPARK-47177) Cached SQL plan do not display final AQE plan in explain string

2024-02-27 Thread Ziqi Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ziqi Liu updated SPARK-47177: - Description: AQE plan is expected to display final plan after execution. This is not true for cached

[jira] [Updated] (SPARK-47177) Cached SQL plan do not display final AQE plan in explain string

2024-02-27 Thread Ziqi Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ziqi Liu updated SPARK-47177: - Description: AQE plan is expected to display final plan after execution. This is not true for cached

[jira] [Updated] (SPARK-47177) Cached SQL plan do not display final AQE plan in explain string

2024-02-27 Thread Ziqi Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ziqi Liu updated SPARK-47177: - Description: AQE plan is expected to display final plan after execution. This is not true for cached

[jira] [Updated] (SPARK-47177) Cached SQL plan do not display final AQE plan in explain string

2024-02-26 Thread Ziqi Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ziqi Liu updated SPARK-47177: - Description: AQE plan is expected to display final plan after execution. This is not true for cached

[jira] [Created] (SPARK-47177) Cached SQL plan do not display final AQE plan in explain string

2024-02-26 Thread Ziqi Liu (Jira)
Ziqi Liu created SPARK-47177: Summary: Cached SQL plan do not display final AQE plan in explain string Key: SPARK-47177 URL: https://issues.apache.org/jira/browse/SPARK-47177 Project: Spark

[jira] [Created] (SPARK-47174) Client Side Listener - Server side implementation

2024-02-26 Thread Wei Liu (Jira)
Wei Liu created SPARK-47174: --- Summary: Client Side Listener - Server side implementation Key: SPARK-47174 URL: https://issues.apache.org/jira/browse/SPARK-47174 Project: Spark Issue Type:

[jira] [Commented] (SPARK-47174) Client Side Listener - Server side implementation

2024-02-26 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17820828#comment-17820828 ] Wei Liu commented on SPARK-47174: - im working on this > Client Side Listener - Server side

[jira] [Created] (SPARK-47173) fix typo in new streaming query listener explanation

2024-02-26 Thread Wei Liu (Jira)
Wei Liu created SPARK-47173: --- Summary: fix typo in new streaming query listener explanation Key: SPARK-47173 URL: https://issues.apache.org/jira/browse/SPARK-47173 Project: Spark Issue Type:

[jira] [Updated] (SPARK-46995) Allow AQE coalesce final stage in SQL cached plan

2024-02-06 Thread Ziqi Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ziqi Liu updated SPARK-46995: - Component/s: SQL > Allow AQE coalesce final stage in SQL cached plan >

[jira] [Created] (SPARK-46995) Allow AQE coalesce final stage in SQL cached plan

2024-02-06 Thread Ziqi Liu (Jira)
Ziqi Liu created SPARK-46995: Summary: Allow AQE coalesce final stage in SQL cached plan Key: SPARK-46995 URL: https://issues.apache.org/jira/browse/SPARK-46995 Project: Spark Issue Type:

[jira] [Created] (SPARK-46910) Eliminate JDK Requirement in PySpark Installation

2024-01-29 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-46910: -- Summary: Eliminate JDK Requirement in PySpark Installation Key: SPARK-46910 URL: https://issues.apache.org/jira/browse/SPARK-46910 Project: Spark Issue Type:

[jira] [Created] (SPARK-46873) PySpark spark.streams should not recreate new StreamingQueryManager

2024-01-25 Thread Wei Liu (Jira)
Wei Liu created SPARK-46873: --- Summary: PySpark spark.streams should not recreate new StreamingQueryManager Key: SPARK-46873 URL: https://issues.apache.org/jira/browse/SPARK-46873 Project: Spark

[jira] [Closed] (SPARK-44460) Pass user auth credential to Python workers for foreachBatch and listener

2024-01-23 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu closed SPARK-44460. --- > Pass user auth credential to Python workers for foreachBatch and listener >

[jira] [Updated] (SPARK-46627) Streaming UI hover-over shows incorrect value

2024-01-08 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-46627: Attachment: Screenshot 2024-01-08 at 15.06.24.png > Streaming UI hover-over shows incorrect value >

[jira] [Commented] (SPARK-46627) Streaming UI hover-over shows incorrect value

2024-01-08 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17804513#comment-17804513 ] Wei Liu commented on SPARK-46627: - Also batch percent doesn't add to 100% now: !Screenshot 2024-01-08

[jira] [Updated] (SPARK-46627) Streaming UI hover-over shows incorrect value

2024-01-08 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-46627: Attachment: Screenshot 2024-01-08 at 1.55.57 PM.png > Streaming UI hover-over shows incorrect value >

[jira] [Commented] (SPARK-46627) Streaming UI hover-over shows incorrect value

2024-01-08 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17804481#comment-17804481 ] Wei Liu commented on SPARK-46627: - Hi Kent : ) [~yao]  I was wondering if you have context in this

[jira] [Updated] (SPARK-46627) Streaming UI hover-over shows incorrect value

2024-01-08 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-46627: Description: Running a simple streaming query: val df =

[jira] [Created] (SPARK-46627) Streaming UI hover-over shows incorrect value

2024-01-08 Thread Wei Liu (Jira)
Wei Liu created SPARK-46627: --- Summary: Streaming UI hover-over shows incorrect value Key: SPARK-46627 URL: https://issues.apache.org/jira/browse/SPARK-46627 Project: Spark Issue Type: Task

[jira] [Updated] (SPARK-46384) Streaming UI doesn't display graph correctly

2023-12-12 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-46384: Summary: Streaming UI doesn't display graph correctly (was: Streaming UI doesn't show graph) >

[jira] [Updated] (SPARK-46384) Structured Streaming UI doesn't display graph correctly

2023-12-12 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-46384: Summary: Structured Streaming UI doesn't display graph correctly (was: Streaming UI doesn't display

[jira] [Created] (SPARK-46384) Streaming UI doesn't show graph

2023-12-12 Thread Wei Liu (Jira)
Wei Liu created SPARK-46384: --- Summary: Streaming UI doesn't show graph Key: SPARK-46384 URL: https://issues.apache.org/jira/browse/SPARK-46384 Project: Spark Issue Type: Task Components:

[jira] [Created] (SPARK-46279) Support write partition values to data files

2023-12-05 Thread fred liu (Jira)
fred liu created SPARK-46279: Summary: Support write partition values to data files Key: SPARK-46279 URL: https://issues.apache.org/jira/browse/SPARK-46279 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-46250) Deflake test_parity_listener

2023-12-04 Thread Wei Liu (Jira)
Wei Liu created SPARK-46250: --- Summary: Deflake test_parity_listener Key: SPARK-46250 URL: https://issues.apache.org/jira/browse/SPARK-46250 Project: Spark Issue Type: Task Components:

[jira] [Created] (SPARK-45845) Streaming UI add number of evicted state rows

2023-11-08 Thread Wei Liu (Jira)
Wei Liu created SPARK-45845: --- Summary: Streaming UI add number of evicted state rows Key: SPARK-45845 URL: https://issues.apache.org/jira/browse/SPARK-45845 Project: Spark Issue Type: Task

[jira] [Updated] (SPARK-45834) Fix Pearson correlation calculation more stable

2023-11-07 Thread Jiayi Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Liu updated SPARK-45834: -- Description: Spark uses the formula {{ck / sqrt(xMk * yMk)}} to calculate the Pearson Correlation

[jira] [Created] (SPARK-45729) Fix PySpark testing guide links

2023-10-30 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-45729: -- Summary: Fix PySpark testing guide links Key: SPARK-45729 URL: https://issues.apache.org/jira/browse/SPARK-45729 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-45637) Time window aggregation in separate streams followed by stream-stream join not returning results

2023-10-27 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-45637: Description: According to documentation update (SPARK-42591) resulting from SPARK-42376, Spark 3.5.0

[jira] [Created] (SPARK-45677) Observe API error logging

2023-10-26 Thread Wei Liu (Jira)
Wei Liu created SPARK-45677: --- Summary: Observe API error logging Key: SPARK-45677 URL: https://issues.apache.org/jira/browse/SPARK-45677 Project: Spark Issue Type: Task Components:

[jira] [Updated] (SPARK-45053) Improve python version mismatch logging

2023-09-01 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-45053: Description: Currently the syntax of the python version mismatching is a little bit confusing, it uses

[jira] [Created] (SPARK-45056) Add process termination tests for Python foreachBatch and StreamingQueryListener

2023-09-01 Thread Wei Liu (Jira)
Wei Liu created SPARK-45056: --- Summary: Add process termination tests for Python foreachBatch and StreamingQueryListener Key: SPARK-45056 URL: https://issues.apache.org/jira/browse/SPARK-45056 Project:

[jira] [Created] (SPARK-45053) Improve python version mismatch logging

2023-09-01 Thread Wei Liu (Jira)
Wei Liu created SPARK-45053: --- Summary: Improve python version mismatch logging Key: SPARK-45053 URL: https://issues.apache.org/jira/browse/SPARK-45053 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-44971) [BUG Fix] PySpark StreamingQuerProgress fromJson

2023-08-25 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-44971: Issue Type: Bug (was: New Feature) > [BUG Fix] PySpark StreamingQuerProgress fromJson >

[jira] [Created] (SPARK-44971) [BUG Fix] PySpark StreamingQuerProgress fromJson

2023-08-25 Thread Wei Liu (Jira)
Wei Liu created SPARK-44971: --- Summary: [BUG Fix] PySpark StreamingQuerProgress fromJson Key: SPARK-44971 URL: https://issues.apache.org/jira/browse/SPARK-44971 Project: Spark Issue Type: New

[jira] [Updated] (SPARK-44930) Deterministic ApplyFunctionExpression should be foldable

2023-08-23 Thread Xianyang Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyang Liu updated SPARK-44930: - Description: Currently, ApplyFunctionExpression is unfoldable because inherits the default

[jira] [Updated] (SPARK-44930) Deterministic ApplyFunctionExpression should be foldable

2023-08-23 Thread Xianyang Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyang Liu updated SPARK-44930: - Description: Currently, ApplyFunctionExpression is unfoldable because inherits the default

[jira] [Created] (SPARK-44930) Deterministic ApplyFunctionExpression should be foldable

2023-08-23 Thread Xianyang Liu (Jira)
Xianyang Liu created SPARK-44930: Summary: Deterministic ApplyFunctionExpression should be foldable Key: SPARK-44930 URL: https://issues.apache.org/jira/browse/SPARK-44930 Project: Spark

[jira] [Resolved] (SPARK-44917) PySpark Streaming DataStreamWriter table API

2023-08-22 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu resolved SPARK-44917. - Resolution: Not A Problem > PySpark Streaming DataStreamWriter table API >

[jira] [Created] (SPARK-44917) PySpark Streaming DataStreamWriter table API

2023-08-22 Thread Wei Liu (Jira)
Wei Liu created SPARK-44917: --- Summary: PySpark Streaming DataStreamWriter table API Key: SPARK-44917 URL: https://issues.apache.org/jira/browse/SPARK-44917 Project: Spark Issue Type: New Feature

[jira] [Created] (SPARK-44913) DS V2 supports push down V2 UDF that has magic method

2023-08-22 Thread Xianyang Liu (Jira)
Xianyang Liu created SPARK-44913: Summary: DS V2 supports push down V2 UDF that has magic method Key: SPARK-44913 URL: https://issues.apache.org/jira/browse/SPARK-44913 Project: Spark Issue

[jira] [Commented] (SPARK-44460) Pass user auth credential to Python workers for foreachBatch and listener

2023-08-21 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17757087#comment-17757087 ] Wei Liu commented on SPARK-44460: - [~rangadi] This seems to be a Databricks internal issue. See the

[jira] [Created] (SPARK-44839) Better error logging when user accesses spark session in foreachBatch and Listener

2023-08-16 Thread Wei Liu (Jira)
Wei Liu created SPARK-44839: --- Summary: Better error logging when user accesses spark session in foreachBatch and Listener Key: SPARK-44839 URL: https://issues.apache.org/jira/browse/SPARK-44839 Project:

[jira] [Commented] (SPARK-44808) refreshListener() API on StreamingQueryManager for spark connect

2023-08-14 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17754280#comment-17754280 ] Wei Liu commented on SPARK-44808: - This seems to be against the design principle of spark connect – We

[jira] [Resolved] (SPARK-44808) refreshListener() API on StreamingQueryManager for spark connect

2023-08-14 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu resolved SPARK-44808. - Resolution: Won't Do > refreshListener() API on StreamingQueryManager for spark connect >

[jira] [Updated] (SPARK-44808) refreshListener() API on StreamingQueryManager for spark connect

2023-08-14 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-44808: Description: I’m thinking of an improvement for connect python listener and foreachBatch. Currently if

[jira] [Updated] (SPARK-44808) refreshListener() API on StreamingQueryManager for spark connect

2023-08-14 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-44808: Description: I’m thinking of an improvement for connect python listener and foreachBatch. Currently if

[jira] [Updated] (SPARK-44808) refreshListener() API on StreamingQueryManager for spark connect

2023-08-14 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-44808: Description: I’m thinking of an improvement for connect python listener and foreachBatch. Currently if

[jira] [Updated] (SPARK-44808) refreshListener() API on StreamingQueryManager for spark connect

2023-08-14 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-44808: Description: I’m thinking of an improvement for connect python listener and foreachBatch. Currently if

[jira] [Created] (SPARK-44808) refreshListener() API on StreamingQueryManager for spark connect

2023-08-14 Thread Wei Liu (Jira)
Wei Liu created SPARK-44808: --- Summary: refreshListener() API on StreamingQueryManager for spark connect Key: SPARK-44808 URL: https://issues.apache.org/jira/browse/SPARK-44808 Project: Spark

[jira] [Updated] (SPARK-44808) refreshListener() API on StreamingQueryManager for spark connect

2023-08-14 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-44808: Description: I’m thinking of an improvement for python listener and foreachBatch. Currently if you

[jira] [Created] (SPARK-44764) Streaming process improvement

2023-08-10 Thread Wei Liu (Jira)
Wei Liu created SPARK-44764: --- Summary: Streaming process improvement Key: SPARK-44764 URL: https://issues.apache.org/jira/browse/SPARK-44764 Project: Spark Issue Type: New Feature

[jira] [Updated] (SPARK-44712) Migrate ‎test_timedelta_ops assert_eq to use assertDataFrameEqual

2023-08-07 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-44712: --- Description: Migrate assert_eq to assertDataFrameEqual in this file:

[jira] [Created] (SPARK-44712) Migrate ‎test_timedelta_ops assert_eq to use assertDataFrameEqual

2023-08-07 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-44712: -- Summary: Migrate ‎test_timedelta_ops assert_eq to use assertDataFrameEqual Key: SPARK-44712 URL: https://issues.apache.org/jira/browse/SPARK-44712 Project: Spark

[jira] [Updated] (SPARK-44711) Migrate test_series_conversion assert_eq to use assertDataFrameEqual

2023-08-07 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-44711: --- Description: Migrate assert_eq to assertDataFrameEqual in this file: 

[jira] [Created] (SPARK-44711) Migrate test_series_conversion assert_eq to use assertDataFrameEqual

2023-08-07 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-44711: -- Summary: Migrate test_series_conversion assert_eq to use assertDataFrameEqual Key: SPARK-44711 URL: https://issues.apache.org/jira/browse/SPARK-44711 Project: Spark

[jira] [Created] (SPARK-44708) Migrate test_reset_index assert_eq to use assertDataFrameEqual

2023-08-07 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-44708: -- Summary: Migrate test_reset_index assert_eq to use assertDataFrameEqual Key: SPARK-44708 URL: https://issues.apache.org/jira/browse/SPARK-44708 Project: Spark

[jira] [Updated] (SPARK-44597) Migrate test_sql assert_eq to use assertDataFrameEqual

2023-08-07 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-44597: --- Description: Migrate tests to new test utils in this file: python/pyspark/pandas/tests/test_sql.py

  1   2   3   4   5   6   7   8   9   10   >