[jira] [Updated] (SPARK-34037) Remove unnecessary upcasting for Avg & Sum which handle by themself internally

2021-01-15 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh updated SPARK-34037: Affects Version/s: (was: 3.1.0) 3.2.0 > Remove unnecessary upcasting fo

[jira] [Assigned] (SPARK-34037) Remove unnecessary upcasting for Avg & Sum which handle by themself internally

2021-01-15 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh reassigned SPARK-34037: --- Assignee: Kent Yao > Remove unnecessary upcasting for Avg & Sum which handle by themself in

[jira] [Resolved] (SPARK-34037) Remove unnecessary upcasting for Avg & Sum which handle by themself internally

2021-01-15 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh resolved SPARK-34037. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31079 [https://gith

[jira] [Created] (SPARK-34148) Move general StateStore tests to StateStoreSuiteBase

2021-01-17 Thread L. C. Hsieh (Jira)
L. C. Hsieh created SPARK-34148: --- Summary: Move general StateStore tests to StateStoreSuiteBase Key: SPARK-34148 URL: https://issues.apache.org/jira/browse/SPARK-34148 Project: Spark Issue Type

[jira] [Created] (SPARK-34187) Use available offset range obtainted during polling when checking offset validation

2021-01-20 Thread L. C. Hsieh (Jira)
L. C. Hsieh created SPARK-34187: --- Summary: Use available offset range obtainted during polling when checking offset validation Key: SPARK-34187 URL: https://issues.apache.org/jira/browse/SPARK-34187 Pro

[jira] [Updated] (SPARK-34187) Use available offset range obtainted during polling when checking offset validation

2021-01-20 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh updated SPARK-34187: Description: We support non-consecutive offsets for Kafka since 2.4.0. In `fetchRecord`, we do of

[jira] [Updated] (SPARK-34187) Use available offset range obtained during polling when checking offset validation

2021-01-20 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh updated SPARK-34187: Summary: Use available offset range obtained during polling when checking offset validation (was:

[jira] [Created] (SPARK-34198) Add RocksDB StateStore as external module

2021-01-21 Thread L. C. Hsieh (Jira)
L. C. Hsieh created SPARK-34198: --- Summary: Add RocksDB StateStore as external module Key: SPARK-34198 URL: https://issues.apache.org/jira/browse/SPARK-34198 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-34198) Add RocksDB StateStore as external module

2021-01-21 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17269737#comment-17269737 ] L. C. Hsieh commented on SPARK-34198: - cc [~dbtsai][~dongjoon][~hyukjin.kwon] > Add

[jira] [Updated] (SPARK-34198) Add RocksDB StateStore as external module

2021-01-21 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh updated SPARK-34198: Issue Type: New Feature (was: Bug) > Add RocksDB StateStore as external module >

[jira] [Resolved] (SPARK-34148) Move general StateStore tests to StateStoreSuiteBase

2021-01-21 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh resolved SPARK-34148. - Resolution: Resolved > Move general StateStore tests to StateStoreSuiteBase > --

[jira] [Updated] (SPARK-34148) Move general StateStore tests to StateStoreSuiteBase

2021-01-21 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh updated SPARK-34148: Parent: SPARK-34198 Issue Type: Sub-task (was: Test) > Move general StateStore tests to S

[jira] [Created] (SPARK-34205) Add pipe API to Dataset

2021-01-22 Thread L. C. Hsieh (Jira)
L. C. Hsieh created SPARK-34205: --- Summary: Add pipe API to Dataset Key: SPARK-34205 URL: https://issues.apache.org/jira/browse/SPARK-34205 Project: Spark Issue Type: New Feature Compo

[jira] [Updated] (SPARK-34205) Add pipe API to Dataset

2021-01-22 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh updated SPARK-34205: Description: Dataset doesn't have pipe API but RDD has it. Although for normal Dataset, user can c

[jira] [Updated] (SPARK-34187) Use available offset range obtained during polling when checking offset validation

2021-01-24 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh updated SPARK-34187: Labels: correctness (was: ) > Use available offset range obtained during polling when checking of

[jira] [Created] (SPARK-34270) Combine StateStoreMetrics should not override individual StateStoreCustomMetric

2021-01-27 Thread L. C. Hsieh (Jira)
L. C. Hsieh created SPARK-34270: --- Summary: Combine StateStoreMetrics should not override individual StateStoreCustomMetric Key: SPARK-34270 URL: https://issues.apache.org/jira/browse/SPARK-34270 Project

[jira] [Updated] (SPARK-34270) Combine StateStoreMetrics should not override individual StateStoreCustomMetric

2021-01-27 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh updated SPARK-34270: Affects Version/s: 3.1.2 2.3.4 2.4.7

[jira] [Updated] (SPARK-34270) Combine StateStoreMetrics should not override StateStoreCustomMetric

2021-01-27 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh updated SPARK-34270: Summary: Combine StateStoreMetrics should not override StateStoreCustomMetric (was: Combine State

[jira] [Created] (SPARK-34295) Allow option similar to mapreduce.job.hdfs-servers.token-renewal.exclude

2021-01-29 Thread L. C. Hsieh (Jira)
L. C. Hsieh created SPARK-34295: --- Summary: Allow option similar to mapreduce.job.hdfs-servers.token-renewal.exclude Key: SPARK-34295 URL: https://issues.apache.org/jira/browse/SPARK-34295 Project: Spark

[jira] [Created] (SPARK-34297) Add metrics for data loss and offset out range for KafkaMicroBatchStream

2021-01-29 Thread L. C. Hsieh (Jira)
L. C. Hsieh created SPARK-34297: --- Summary: Add metrics for data loss and offset out range for KafkaMicroBatchStream Key: SPARK-34297 URL: https://issues.apache.org/jira/browse/SPARK-34297 Project: Spark

[jira] [Updated] (SPARK-34297) Add metrics for data loss and offset out range for KafkaMicroBatchStream

2021-01-30 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh updated SPARK-34297: Component/s: SQL > Add metrics for data loss and offset out range for KafkaMicroBatchStream >

[jira] [Commented] (SPARK-34198) Add RocksDB StateStore as external module

2021-02-01 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17276863#comment-17276863 ] L. C. Hsieh commented on SPARK-34198: - If you are asking why adding it as an externa

[jira] [Assigned] (SPARK-34198) Add RocksDB StateStore as external module

2021-02-01 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh reassigned SPARK-34198: --- Assignee: L. C. Hsieh > Add RocksDB StateStore as external module > ---

[jira] [Created] (SPARK-34321) Fix the guarantee of foreachBatch

2021-02-01 Thread L. C. Hsieh (Jira)
L. C. Hsieh created SPARK-34321: --- Summary: Fix the guarantee of foreachBatch Key: SPARK-34321 URL: https://issues.apache.org/jira/browse/SPARK-34321 Project: Spark Issue Type: Documentation

[jira] [Resolved] (SPARK-34321) Fix the guarantee of foreachBatch

2021-02-01 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh resolved SPARK-34321. - Resolution: Invalid > Fix the guarantee of foreachBatch > - > >

[jira] [Commented] (SPARK-34321) Fix the guarantee of foreachBatch

2021-02-01 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17276865#comment-17276865 ] L. C. Hsieh commented on SPARK-34321: - Err...I made a mistake when reading the docum

[jira] [Assigned] (SPARK-34295) Allow option similar to mapreduce.job.hdfs-servers.token-renewal.exclude

2021-02-01 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh reassigned SPARK-34295: --- Assignee: L. C. Hsieh > Allow option similar to mapreduce.job.hdfs-servers.token-renewal.ex

[jira] [Created] (SPARK-34324) FileTable should not list TRUNCATE in capabilities by default

2021-02-01 Thread L. C. Hsieh (Jira)
L. C. Hsieh created SPARK-34324: --- Summary: FileTable should not list TRUNCATE in capabilities by default Key: SPARK-34324 URL: https://issues.apache.org/jira/browse/SPARK-34324 Project: Spark

[jira] [Commented] (SPARK-34198) Add RocksDB StateStore as external module

2021-02-01 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17276882#comment-17276882 ] L. C. Hsieh commented on SPARK-34198: - For external module here, I mean to put the r

[jira] [Commented] (SPARK-34198) Add RocksDB StateStore as external module

2021-02-01 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17276893#comment-17276893 ] L. C. Hsieh commented on SPARK-34198: - Thanks [~kabhwan] for your point. Besides th

[jira] [Commented] (SPARK-34198) Add RocksDB StateStore as external module

2021-02-02 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17276944#comment-17276944 ] L. C. Hsieh commented on SPARK-34198: - Raised the discussion: http://apache-spark-d

[jira] [Resolved] (SPARK-34324) FileTable should not list TRUNCATE in capabilities by default

2021-02-02 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh resolved SPARK-34324. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31432 [https://gith

[jira] [Created] (SPARK-34338) Report metrics from Datasource v2 scan

2021-02-02 Thread L. C. Hsieh (Jira)
L. C. Hsieh created SPARK-34338: --- Summary: Report metrics from Datasource v2 scan Key: SPARK-34338 URL: https://issues.apache.org/jira/browse/SPARK-34338 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-34338) Report metrics from Datasource v2 scan

2021-02-02 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh updated SPARK-34338: Description: This is related to SPARK-34297. In SPARK-34297, we want to add a couple of useful me

[jira] [Created] (SPARK-34366) Add metric interfaces to DS v2

2021-02-04 Thread L. C. Hsieh (Jira)
L. C. Hsieh created SPARK-34366: --- Summary: Add metric interfaces to DS v2 Key: SPARK-34366 URL: https://issues.apache.org/jira/browse/SPARK-34366 Project: Spark Issue Type: Sub-task C

[jira] [Updated] (SPARK-34338) Report metrics from Datasource v2 scan

2021-02-04 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh updated SPARK-34338: Issue Type: Umbrella (was: Improvement) > Report metrics from Datasource v2 scan > --

[jira] [Created] (SPARK-34367) Batch implementation for metrics from Datasource v2 scan

2021-02-04 Thread L. C. Hsieh (Jira)
L. C. Hsieh created SPARK-34367: --- Summary: Batch implementation for metrics from Datasource v2 scan Key: SPARK-34367 URL: https://issues.apache.org/jira/browse/SPARK-34367 Project: Spark Issue

[jira] [Created] (SPARK-34368) Streaming implementation for metrics from Datasource v2 scan

2021-02-04 Thread L. C. Hsieh (Jira)
L. C. Hsieh created SPARK-34368: --- Summary: Streaming implementation for metrics from Datasource v2 scan Key: SPARK-34368 URL: https://issues.apache.org/jira/browse/SPARK-34368 Project: Spark I

[jira] [Updated] (SPARK-34367) Batch implementation for metrics from Datasource v2 scan

2021-02-04 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh updated SPARK-34367: Description: Using metrics interface of DS v2 to report metrics for batch scan. > Batch implement

[jira] [Assigned] (SPARK-34347) CatalogImpl.uncacheTable should invalidate in cascade for temp views

2021-02-09 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh reassigned SPARK-34347: --- Assignee: Chao Sun > CatalogImpl.uncacheTable should invalidate in cascade for temp views

[jira] [Resolved] (SPARK-34347) CatalogImpl.uncacheTable should invalidate in cascade for temp views

2021-02-09 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh resolved SPARK-34347. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31462 [https://gith

[jira] [Created] (SPARK-34420) Throw exception if non-streaming Deduplicate is not replaced by aggregate

2021-02-10 Thread L. C. Hsieh (Jira)
L. C. Hsieh created SPARK-34420: --- Summary: Throw exception if non-streaming Deduplicate is not replaced by aggregate Key: SPARK-34420 URL: https://issues.apache.org/jira/browse/SPARK-34420 Project: Spar

[jira] [Assigned] (SPARK-34419) Move PartitionTransforms from java to scala directory

2021-02-10 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh reassigned SPARK-34419: --- Assignee: Chao Sun > Move PartitionTransforms from java to scala directory > --

[jira] [Resolved] (SPARK-34419) Move PartitionTransforms from java to scala directory

2021-02-10 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh resolved SPARK-34419. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31546 [https://gith

[jira] [Resolved] (SPARK-34420) Throw exception if non-streaming Deduplicate is not replaced by aggregate

2021-02-10 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh resolved SPARK-34420. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31547 [https://gith

[jira] [Created] (SPARK-34427) Session window support in SS

2021-02-12 Thread L. C. Hsieh (Jira)
L. C. Hsieh created SPARK-34427: --- Summary: Session window support in SS Key: SPARK-34427 URL: https://issues.apache.org/jira/browse/SPARK-34427 Project: Spark Issue Type: New Feature

[jira] [Commented] (SPARK-34427) Session window support in SS

2021-02-12 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284033#comment-17284033 ] L. C. Hsieh commented on SPARK-34427: - cc [~dbtsai] > Session window support in SS

[jira] [Commented] (SPARK-34427) Session window support in SS

2021-02-12 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284076#comment-17284076 ] L. C. Hsieh commented on SPARK-34427: - Thanks for the quick feedback. I didn't find

[jira] [Comment Edited] (SPARK-34427) Session window support in SS

2021-02-12 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284076#comment-17284076 ] L. C. Hsieh edited comment on SPARK-34427 at 2/13/21, 4:00 AM: ---

[jira] [Commented] (SPARK-34427) Session window support in SS

2021-02-13 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284348#comment-17284348 ] L. C. Hsieh commented on SPARK-34427: - If I don't miss anything, SPARK-10816 has no

[jira] [Commented] (SPARK-34427) Session window support in SS

2021-02-14 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284368#comment-17284368 ] L. C. Hsieh commented on SPARK-34427: - Please check the JIRA history and I don't thi

[jira] [Commented] (SPARK-34198) Add RocksDB StateStore as external module

2021-02-14 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284381#comment-17284381 ] L. C. Hsieh commented on SPARK-34198: - I'd tend to take as the baseline from [https

[jira] [Commented] (SPARK-34198) Add RocksDB StateStore as external module

2021-02-14 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284478#comment-17284478 ] L. C. Hsieh commented on SPARK-34198: - Thanks [~rxin]. Is the implementation used in

[jira] [Commented] (SPARK-34427) Session window support in SS

2021-02-14 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284504#comment-17284504 ] L. C. Hsieh commented on SPARK-34427: - Sigh...do you ever see that I say I want to i

[jira] [Commented] (SPARK-34295) Allow option similar to mapreduce.job.hdfs-servers.token-renewal.exclude

2021-02-14 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284588#comment-17284588 ] L. C. Hsieh commented on SPARK-34295: - To prevent other questioning about the assign

[jira] [Comment Edited] (SPARK-34295) Allow option similar to mapreduce.job.hdfs-servers.token-renewal.exclude

2021-02-14 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284588#comment-17284588 ] L. C. Hsieh edited comment on SPARK-34295 at 2/15/21, 7:39 AM: ---

[jira] [Commented] (SPARK-10816) EventTime based sessionization

2021-02-15 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-10816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284931#comment-17284931 ] L. C. Hsieh commented on SPARK-10816: - This feature is inactive in last two years. W

[jira] [Assigned] (SPARK-34198) Add RocksDB StateStore as external module

2021-02-15 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh reassigned SPARK-34198: --- Assignee: (was: L. C. Hsieh) > Add RocksDB StateStore as external module >

[jira] [Updated] (SPARK-10816) EventTime based sessionization (session window)

2021-02-15 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-10816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh updated SPARK-10816: Summary: EventTime based sessionization (session window) (was: EventTime based sessionization) >

[jira] [Updated] (SPARK-10816) EventTime based sessionization (session window)

2021-02-15 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-10816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh updated SPARK-10816: Description: Currently structured streaming supports two kinds of windows: tumbling window and sl

[jira] [Updated] (SPARK-10816) EventTime based sessionization (session window)

2021-02-15 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-10816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh updated SPARK-10816: Description: Currently structured streaming supports two kinds of windows: tumbling window and sl

[jira] [Updated] (SPARK-10816) EventTime based sessionization (session window)

2021-02-15 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-10816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh updated SPARK-10816: Description: Currently structured streaming supports two kinds of windows: tumbling window and sl

[jira] [Created] (SPARK-34474) Remove unnecessary Union under Distinct like operators

2021-02-19 Thread L. C. Hsieh (Jira)
L. C. Hsieh created SPARK-34474: --- Summary: Remove unnecessary Union under Distinct like operators Key: SPARK-34474 URL: https://issues.apache.org/jira/browse/SPARK-34474 Project: Spark Issue Ty

[jira] [Resolved] (SPARK-34502) Remove unused parameters in join methods

2021-02-23 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh resolved SPARK-34502. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31617 [https://gith

[jira] [Assigned] (SPARK-34502) Remove unused parameters in join methods

2021-02-23 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh reassigned SPARK-34502: --- Assignee: Huaxin Gao > Remove unused parameters in join methods > -

[jira] [Updated] (SPARK-33504) The application log in the Spark history server contains sensitive attributes such as password that should be redated instead of plain text

2021-02-24 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh updated SPARK-33504: Fix Version/s: 3.0.3 > The application log in the Spark history server contains sensitive attribut

[jira] [Commented] (SPARK-34198) Add RocksDB StateStore as external module

2021-02-25 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17290771#comment-17290771 ] L. C. Hsieh commented on SPARK-34198: - FYI, I ran a benchmark against two open sourc

[jira] [Resolved] (SPARK-34474) Remove unnecessary Union under Distinct like operators

2021-02-25 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh resolved SPARK-34474. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31595 [https://gith

[jira] [Created] (SPARK-34548) Remove unnecessary children from Union under Distince and Deduplicate

2021-02-25 Thread L. C. Hsieh (Jira)
L. C. Hsieh created SPARK-34548: --- Summary: Remove unnecessary children from Union under Distince and Deduplicate Key: SPARK-34548 URL: https://issues.apache.org/jira/browse/SPARK-34548 Project: Spark

[jira] [Created] (SPARK-34549) Upgrade aws kinesis to 1.14.0 and java sdk 1.11.844

2021-02-25 Thread L. C. Hsieh (Jira)
L. C. Hsieh created SPARK-34549: --- Summary: Upgrade aws kinesis to 1.14.0 and java sdk 1.11.844 Key: SPARK-34549 URL: https://issues.apache.org/jira/browse/SPARK-34549 Project: Spark Issue Type:

[jira] [Created] (SPARK-34592) Mark indeterminate RDD in Web UI

2021-03-02 Thread L. C. Hsieh (Jira)
L. C. Hsieh created SPARK-34592: --- Summary: Mark indeterminate RDD in Web UI Key: SPARK-34592 URL: https://issues.apache.org/jira/browse/SPARK-34592 Project: Spark Issue Type: Improvement

[jira] [Resolved] (SPARK-34548) Remove unnecessary children from Union under Distince and Deduplicate

2021-03-02 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh resolved SPARK-34548. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31656 [https://gith

[jira] [Commented] (SPARK-29721) Spark SQL reads unnecessary nested fields after using explode

2021-03-03 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17294794#comment-17294794 ] L. C. Hsieh commented on SPARK-29721: - Agree with Dongjoon. Looks like another type

[jira] [Created] (SPARK-34612) Whether to expose outputDeterministicLevel so custom RDDs can set deterministic level

2021-03-03 Thread L. C. Hsieh (Jira)
L. C. Hsieh created SPARK-34612: --- Summary: Whether to expose outputDeterministicLevel so custom RDDs can set deterministic level Key: SPARK-34612 URL: https://issues.apache.org/jira/browse/SPARK-34612 P

[jira] [Resolved] (SPARK-34592) Mark indeterminate RDD in Web UI

2021-03-05 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh resolved SPARK-34592. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31707 [https://gith

[jira] [Commented] (SPARK-34638) Spark SQL reads unnecessary nested fields (another type of pruning case)

2021-03-06 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17296782#comment-17296782 ] L. C. Hsieh commented on SPARK-34638: - Thanks [~yuryn] and [~hyukjin.kwon]. I will l

[jira] [Updated] (SPARK-34318) Dataset.colRegex should work with column names and qualifiers which contain newlines

2021-03-08 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh updated SPARK-34318: Fix Version/s: 2.4.8 > Dataset.colRegex should work with column names and qualifiers which contain

[jira] [Created] (SPARK-34672) Fix docker file for creating release

2021-03-09 Thread L. C. Hsieh (Jira)
L. C. Hsieh created SPARK-34672: --- Summary: Fix docker file for creating release Key: SPARK-34672 URL: https://issues.apache.org/jira/browse/SPARK-34672 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-34675) TimeZone inconsistencies when JVM and session timezones are different

2021-03-10 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17299059#comment-17299059 ] L. C. Hsieh commented on SPARK-34675: - Thanks for ping me [~dongjoon]. So looks lik

[jira] [Updated] (SPARK-34703) Fix pyspark test when using sort_values on Pandas

2021-03-10 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh updated SPARK-34703: Description: Three PySpark tests are currently failed in Jenkins 2.4 build: test_column_order, te

[jira] [Created] (SPARK-34703) Fix pyspark test when using sort_values on Pandas

2021-03-10 Thread L. C. Hsieh (Jira)
L. C. Hsieh created SPARK-34703: --- Summary: Fix pyspark test when using sort_values on Pandas Key: SPARK-34703 URL: https://issues.apache.org/jira/browse/SPARK-34703 Project: Spark Issue Type: B

[jira] [Assigned] (SPARK-34723) Subexpression elimination for whole-stage codegen should not fail for byte array

2021-03-11 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh reassigned SPARK-34723: --- Assignee: L. C. Hsieh > Subexpression elimination for whole-stage codegen should not fail f

[jira] [Created] (SPARK-34723) Subexpression elimination for whole-stage codegen should not fail for byte array

2021-03-11 Thread L. C. Hsieh (Jira)
L. C. Hsieh created SPARK-34723: --- Summary: Subexpression elimination for whole-stage codegen should not fail for byte array Key: SPARK-34723 URL: https://issues.apache.org/jira/browse/SPARK-34723 Projec

[jira] [Updated] (SPARK-34723) Correct parameter type for subexpression elimination under whole-stage

2021-03-11 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh updated SPARK-34723: Summary: Correct parameter type for subexpression elimination under whole-stage (was: Subexpressi

[jira] [Updated] (SPARK-34723) Correct parameter type for subexpression elimination under whole-stage

2021-03-11 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh updated SPARK-34723: Affects Version/s: 3.0.2 3.1.1 > Correct parameter type for subexpression e

[jira] [Resolved] (SPARK-34723) Correct parameter type for subexpression elimination under whole-stage

2021-03-13 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh resolved SPARK-34723. - Fix Version/s: 3.0.3 3.1.2 3.2.0 Resolution: Fixed

[jira] [Commented] (SPARK-34776) Catalyst error on on certain struct operation (Couldn't find _gen_alias_)

2021-03-18 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304622#comment-17304622 ] L. C. Hsieh commented on SPARK-34776: - Thanks for ping. Let me take a look. > Catal

[jira] [Resolved] (SPARK-31897) Enable codegen for GenerateExec

2021-03-18 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh resolved SPARK-31897. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 28715 [https://gith

[jira] [Assigned] (SPARK-31897) Enable codegen for GenerateExec

2021-03-18 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh reassigned SPARK-31897: --- Assignee: Karuppayya > Enable codegen for GenerateExec > --- >

[jira] [Updated] (SPARK-34776) Catalyst error on on certain struct operation (Couldn't find _gen_alias_)

2021-03-19 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh updated SPARK-34776: Affects Version/s: 3.2.0 > Catalyst error on on certain struct operation (Couldn't find _gen_alias

[jira] [Resolved] (SPARK-34726) Fix collectToPython timeouts

2021-03-22 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh resolved SPARK-34726. - Fix Version/s: 2.4.8 Resolution: Fixed Issue resolved by pull request 31818 [https://gith

[jira] [Assigned] (SPARK-34726) Fix collectToPython timeouts

2021-03-22 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh reassigned SPARK-34726: --- Assignee: Peter Toth > Fix collectToPython timeouts > > >

[jira] [Updated] (SPARK-34726) Fix collectToPython timeouts

2021-03-22 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh updated SPARK-34726: Description: One of our customers frequently encounters "serve-DataFrame" java.net.SocketTimeoutE

[jira] [Resolved] (SPARK-34295) Allow option similar to mapreduce.job.hdfs-servers.token-renewal.exclude

2021-03-24 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh resolved SPARK-34295. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31761 [https://gith

[jira] [Assigned] (SPARK-34596) NewInstance.doGenCode should not throw malformed class name error

2021-03-24 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh reassigned SPARK-34596: --- Assignee: Kris Mok > NewInstance.doGenCode should not throw malformed class name error > --

[jira] [Updated] (SPARK-34596) NewInstance.doGenCode should not throw malformed class name error

2021-03-24 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh updated SPARK-34596: Fix Version/s: 2.4.8 > NewInstance.doGenCode should not throw malformed class name error > ---

[jira] [Updated] (SPARK-34607) NewInstance.resolved should not throw malformed class name error

2021-03-25 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh updated SPARK-34607: Fix Version/s: 2.4.8 > NewInstance.resolved should not throw malformed class name error >

[jira] [Commented] (SPARK-36673) Incorrect Unions of struct with mismatched field name case

2021-09-15 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415921#comment-17415921 ] L. C. Hsieh commented on SPARK-36673: - The schema after union looks incorrect. By th

[jira] [Created] (SPARK-36797) Union should resolve nested columns as top-level columns

2021-09-18 Thread L. C. Hsieh (Jira)
L. C. Hsieh created SPARK-36797: --- Summary: Union should resolve nested columns as top-level columns Key: SPARK-36797 URL: https://issues.apache.org/jira/browse/SPARK-36797 Project: Spark Issue

[jira] [Commented] (SPARK-36776) Partition filter of DataSourceV2ScanRelation can not push down when select none dataSchema from FileScan

2021-09-18 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17417234#comment-17417234 ] L. C. Hsieh commented on SPARK-36776: - This is duplicated to SPARK-35985. We just ne

  1   2   3   4   5   6   7   8   9   10   >