[jira] [Updated] (SPARK-40617) Assertion failed in ExecutorMetricsPoller "task count shouldn't below 0"

2022-09-29 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-40617: --- Description: {noformat} 22/09/29 09:46:24 ERROR SparkUncaughtExceptionHandler:

[jira] [Updated] (SPARK-40617) Assertion failed in ExecutorMetricsPoller "task count shouldn't below 0"

2022-09-29 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-40617: --- Description: {noformat} 22/09/29 09:46:24 ERROR SparkUncaughtExceptionHandler:

[jira] [Assigned] (SPARK-40617) Assertion failed in ExecutorMetricsPoller "task count shouldn't below 0"

2022-09-29 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros reassigned SPARK-40617: -- Assignee: Attila Zsolt Piros > Assertion failed in ExecutorMetricsPoller

[jira] [Comment Edited] (SPARK-34779) ExecutorMetricsPoller should keep stage entry in stageTCMP until a heartbeat occurs

2022-09-29 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17611256#comment-17611256 ] Attila Zsolt Piros edited comment on SPARK-34779 at 9/29/22 10:05 PM:

[jira] [Comment Edited] (SPARK-34779) ExecutorMetricsPoller should keep stage entry in stageTCMP until a heartbeat occurs

2022-09-29 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17611256#comment-17611256 ] Attila Zsolt Piros edited comment on SPARK-34779 at 9/29/22 10:05 PM:

[jira] [Updated] (SPARK-40617) Assertion failed in ExecutorMetricsPoller "task count shouldn't below 0"

2022-09-29 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-40617: --- Description: Spurious failures because of the assert: {noformat} 22/09/29 09:46:24

[jira] [Commented] (SPARK-34779) ExecutorMetricsPoller should keep stage entry in stageTCMP until a heartbeat occurs

2022-09-29 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17611181#comment-17611181 ] Attila Zsolt Piros commented on SPARK-34779: [~praetp]

[jira] [Updated] (SPARK-40617) Assertion failed in ExecutorMetricsPoller "task count shouldn't below 0"

2022-09-29 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-40617: --- Description: Spurious failures regarding the assert: {noformat} 22/09/29 09:46:24

[jira] [Updated] (SPARK-40617) Assertion failed in ExecutorMetricsPoller "task count shouldn't below 0"

2022-09-29 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-40617: --- Description: Spurious failures beacuse of the assert: {noformat} 22/09/29 09:46:24

[jira] [Commented] (SPARK-34779) ExecutorMetricsPoller should keep stage entry in stageTCMP until a heartbeat occurs

2022-09-29 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17611217#comment-17611217 ] Attila Zsolt Piros commented on SPARK-34779: [~praetp] I have opened a new Jira and soon PR:

[jira] [Commented] (SPARK-34779) ExecutorMetricsPoller should keep stage entry in stageTCMP until a heartbeat occurs

2022-09-29 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17611256#comment-17611256 ] Attila Zsolt Piros commented on SPARK-34779: [~praetp] I have a working theory but an

[jira] [Updated] (SPARK-40617) Assertion failed in ExecutorMetricsPoller "task count shouldn't below 0"

2022-10-04 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-40617: --- Fix Version/s: 3.2.0 > Assertion failed in ExecutorMetricsPoller "task count

[jira] [Created] (SPARK-40612) On Kubernetes for long running app Spark using an invalid principal to renew the delegation token

2022-09-29 Thread Attila Zsolt Piros (Jira)
Attila Zsolt Piros created SPARK-40612: -- Summary: On Kubernetes for long running app Spark using an invalid principal to renew the delegation token Key: SPARK-40612 URL:

[jira] [Commented] (SPARK-34779) ExecutorMetricsPoller should keep stage entry in stageTCMP until a heartbeat occurs

2022-09-29 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17611353#comment-17611353 ] Attila Zsolt Piros commented on SPARK-34779: [~praetp] I managed to reproduce it in a unit

[jira] [Resolved] (SPARK-40617) Assertion failed in ExecutorMetricsPoller "task count shouldn't below 0"

2022-10-03 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros resolved SPARK-40617. Fix Version/s: 3.4.0 3.3.1 Resolution: Fixed Issue

[jira] [Updated] (SPARK-40039) Introducing a streaming checkpoint file manager based on Hadoop's Abortable interface

2022-08-10 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-40039: --- Summary: Introducing a streaming checkpoint file manager based on Hadoop's

[jira] [Commented] (SPARK-40039) Introducing checkpoint file manager based on Hadoop's Abortable interface

2022-08-10 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578189#comment-17578189 ] Attila Zsolt Piros commented on SPARK-40039: I am working on this. > Introducing checkpoint

[jira] [Created] (SPARK-40039) Introducing checkpoint file manager based on Hadoop's Abortable interface

2022-08-10 Thread Attila Zsolt Piros (Jira)
Attila Zsolt Piros created SPARK-40039: -- Summary: Introducing checkpoint file manager based on Hadoop's Abortable interface Key: SPARK-40039 URL: https://issues.apache.org/jira/browse/SPARK-40039

[jira] [Assigned] (SPARK-35084) [k8s] On Spark 3, jars listed in spark.jars and spark.jars.packages are not added to sparkContext

2023-01-14 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros reassigned SPARK-35084: -- Assignee: Attila Zsolt Piros > [k8s] On Spark 3, jars listed in spark.jars

[jira] [Resolved] (SPARK-35084) [k8s] On Spark 3, jars listed in spark.jars and spark.jars.packages are not added to sparkContext

2023-01-14 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros resolved SPARK-35084. Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-35084) [k8s] On Spark 3, jars listed in spark.jars and spark.jars.packages are not added to sparkContext

2023-01-14 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros reassigned SPARK-35084: -- Assignee: (was: Attila Zsolt Piros) > [k8s] On Spark 3, jars listed in

[jira] [Assigned] (SPARK-35084) [k8s] On Spark 3, jars listed in spark.jars and spark.jars.packages are not added to sparkContext

2023-01-14 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros reassigned SPARK-35084: -- Assignee: Keunhyun Oh > [k8s] On Spark 3, jars listed in spark.jars and

[jira] [Updated] (SPARK-42219) Make "SPARK-34674: Close SparkContext after the Main method has finished" configurable

2023-01-27 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-42219: --- Description: We run into an error after an upgrade from Spark 3.1 to Spark 3.2

[jira] [Assigned] (SPARK-42219) Make "SPARK-34674: Close SparkContext after the Main method has finished" configurable

2023-01-27 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros reassigned SPARK-42219: -- Assignee: Attila Zsolt Piros > Make "SPARK-34674: Close SparkContext after

[jira] [Updated] (SPARK-42219) Make "SPARK-34674: Close SparkContext after the Main method has finished" configurable

2023-01-27 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-42219: --- Description: We run into an error after an upgrade from Spark 3.1 to Spark 3.2

[jira] [Commented] (SPARK-42219) Make "SPARK-34674: Close SparkContext after the Main method has finished" configurable

2023-01-27 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17681454#comment-17681454 ] Attila Zsolt Piros commented on SPARK-42219: I will open a PR regarding this > Make

[jira] [Created] (SPARK-42219) Make "SPARK-34674: Close SparkContext after the Main method has finished" configurable

2023-01-27 Thread Attila Zsolt Piros (Jira)
Attila Zsolt Piros created SPARK-42219: -- Summary: Make "SPARK-34674: Close SparkContext after the Main method has finished" configurable Key: SPARK-42219 URL:

[jira] [Commented] (SPARK-29683) Job failed due to executor failures all available nodes are blacklisted

2022-10-31 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17626884#comment-17626884 ] Attila Zsolt Piros commented on SPARK-29683: [~srowen] I think we can close this as this

[jira] [Comment Edited] (SPARK-42068) Implicit conversion is not working with parallelization in scala with java 11 and spark3

2023-01-31 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682756#comment-17682756 ] Attila Zsolt Piros edited comment on SPARK-42068 at 1/31/23 9:10 PM: -

[jira] [Comment Edited] (SPARK-42068) Implicit conversion is not working with parallelization in scala with java 11 and spark3

2023-01-31 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682756#comment-17682756 ] Attila Zsolt Piros edited comment on SPARK-42068 at 1/31/23 9:09 PM: -

[jira] [Comment Edited] (SPARK-42068) Implicit conversion is not working with parallelization in scala with java 11 and spark3

2023-01-31 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682756#comment-17682756 ] Attila Zsolt Piros edited comment on SPARK-42068 at 1/31/23 9:08 PM: -

[jira] [Comment Edited] (SPARK-42068) Implicit conversion is not working with parallelization in scala with java 11 and spark3

2023-01-31 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682768#comment-17682768 ] Attila Zsolt Piros edited comment on SPARK-42068 at 1/31/23 9:40 PM: -

[jira] [Closed] (SPARK-42068) Implicit conversion is not working with parallelization in scala with java 11 and spark3

2023-01-31 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros closed SPARK-42068. -- Assignee: Attila Zsolt Piros > Implicit conversion is not working with

[jira] [Resolved] (SPARK-42068) Implicit conversion is not working with parallelization in scala with java 11 and spark3

2023-01-31 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros resolved SPARK-42068. Resolution: Workaround Resolving this issue as a workaround exits by using a

[jira] [Comment Edited] (SPARK-42068) Implicit conversion is not working with parallelization in scala with java 11 and spark3

2023-01-31 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682768#comment-17682768 ] Attila Zsolt Piros edited comment on SPARK-42068 at 1/31/23 9:42 PM: -

[jira] [Commented] (SPARK-42068) Implicit conversion is not working with parallelization in scala with java 11 and spark3

2023-01-31 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682756#comment-17682756 ] Attila Zsolt Piros commented on SPARK-42068: This issue is because of JDK bugfix

[jira] [Comment Edited] (SPARK-42068) Implicit conversion is not working with parallelization in scala with java 11 and spark3

2023-01-31 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682768#comment-17682768 ] Attila Zsolt Piros edited comment on SPARK-42068 at 1/31/23 9:39 PM: -

[jira] [Resolved] (SPARK-44198) Support propagation of the log level to the executors

2023-07-27 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros resolved SPARK-44198. Fix Version/s: 4.0.0 Assignee: Vinod KC Resolution: Fixed Issue

[jira] [Commented] (SPARK-44717) "pyspark.pandas.resample" is incorrect when DST is overlapped and setting "spark.sql.timestampType" to TIMESTAMP_NTZ does not applied

2023-08-07 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17751886#comment-17751886 ] Attila Zsolt Piros commented on SPARK-44717: cc [~itholic] [~gurwls223] >

[jira] [Updated] (SPARK-44717) "pyspark.pandas.resample" is incorrect when DST is overlapped and setting "spark.sql.timestampType" to TIMESTAMP_NTZ does not applied

2023-08-07 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-44717: --- Description: Use one of the existing test: - "11H" case of test_dataframe_resample

[jira] [Updated] (SPARK-44717) "pyspark.pandas.resample" is incorrect when DST is overlapped and setting "spark.sql.timestampType" to TIMESTAMP_NTZ does not help

2023-08-07 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-44717: --- Description: Use one of the existing test: - "11H" case of test_dataframe_resample

[jira] [Created] (SPARK-44717) "pyspark.pandas.resample" is incorrect when DST is overlapped and setting "spark.sql.timestampType" to TIMESTAMP_NTZ does not applied

2023-08-07 Thread Attila Zsolt Piros (Jira)
Attila Zsolt Piros created SPARK-44717: -- Summary: "pyspark.pandas.resample" is incorrect when DST is overlapped and setting "spark.sql.timestampType" to TIMESTAMP_NTZ does not applied Key: SPARK-44717 URL:

[jira] [Updated] (SPARK-44717) "pyspark.pandas.resample" is incorrect when DST is overlapped and setting "spark.sql.timestampType" to TIMESTAMP_NTZ does not applied

2023-08-07 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-44717: --- Description: Use one of the existing test: - "11H" case of test_dataframe_resample

[jira] [Updated] (SPARK-44717) "pyspark.pandas.resample" is incorrect when DST is overlapped and setting "spark.sql.timestampType" to TIMESTAMP_NTZ does not help

2023-08-07 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-44717: --- Summary: "pyspark.pandas.resample" is incorrect when DST is overlapped and setting

[jira] [Commented] (SPARK-44717) "pyspark.pandas.resample" is incorrect when DST is overlapped and setting "spark.sql.timestampType" to TIMESTAMP_NTZ does not help

2023-08-08 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17751962#comment-17751962 ] Attila Zsolt Piros commented on SPARK-44717: The TIMESTAMP_NTZ would work for sure. Here is

[jira] [Comment Edited] (SPARK-44717) "pyspark.pandas.resample" is incorrect when DST is overlapped and setting "spark.sql.timestampType" to TIMESTAMP_NTZ does not help

2023-08-08 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17751962#comment-17751962 ] Attila Zsolt Piros edited comment on SPARK-44717 at 8/8/23 8:56 AM:

[jira] [Updated] (SPARK-44253) Potential memory leak when temp views created from DF created by structured streaming

2023-06-29 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-44253: --- Description: If the user registers a temporary view from a dataframe created by

[jira] [Updated] (SPARK-44253) Potential memory leak when temp views created from DF created by structured streaming

2023-06-29 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-44253: --- Attachment: 1.png > Potential memory leak when temp views created from DF created

[jira] [Created] (SPARK-44253) Potential memory leak when temp views created from DF created by structured streaming

2023-06-29 Thread Attila Zsolt Piros (Jira)
Attila Zsolt Piros created SPARK-44253: -- Summary: Potential memory leak when temp views created from DF created by structured streaming Key: SPARK-44253 URL: https://issues.apache.org/jira/browse/SPARK-44253

[jira] [Updated] (SPARK-44253) Potential memory leak when temp views created from DF created by structured streaming

2023-06-29 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-44253: --- Description: If the user registers a temporary view from a dataframe created by

[jira] [Updated] (SPARK-44253) Potential memory leak when temp views created from DF created by structured streaming

2023-06-29 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-44253: --- Attachment: 2.png > Potential memory leak when temp views created from DF created

[jira] [Updated] (SPARK-44253) Potential memory leak when temp views created from DF created by structured streaming

2023-06-29 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-44253: --- Description: If the user registers a temporary view from a dataframe created by

<    1   2   3   4