[jira] [Commented] (SPARK-44717) "pyspark.pandas.resample" is incorrect when DST is overlapped and setting "spark.sql.timestampType" to TIMESTAMP_NTZ does not help

[ https://issues.apache.org/jira/browse/SPARK-44717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17751899#comment-17751899 ] Hyukjin Kwon commented on SPARK-44717: -- I think we should at least make this respects

[jira] [Updated] (SPARK-44717) "pyspark.pandas.resample" is incorrect when DST is overlapped and setting "spark.sql.timestampType" to TIMESTAMP_NTZ does not help

[ https://issues.apache.org/jira/browse/SPARK-44717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-44717: --- Description: Use one of the existing test: - "11H" case of test_dataframe_resample

[jira] [Commented] (SPARK-44717) "pyspark.pandas.resample" is incorrect when DST is overlapped and setting "spark.sql.timestampType" to TIMESTAMP_NTZ does not help

[ https://issues.apache.org/jira/browse/SPARK-44717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17751887#comment-17751887 ] Hyukjin Kwon commented on SPARK-44717: -- cc [~podongfeng] > "pyspark.pandas.resample" is incorrect

[jira] [Updated] (SPARK-44717) "pyspark.pandas.resample" is incorrect when DST is overlapped and setting "spark.sql.timestampType" to TIMESTAMP_NTZ does not help

[ https://issues.apache.org/jira/browse/SPARK-44717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-44717: --- Summary: "pyspark.pandas.resample" is incorrect when DST is overlapped and setting

[jira] [Commented] (SPARK-44717) "pyspark.pandas.resample" is incorrect when DST is overlapped and setting "spark.sql.timestampType" to TIMESTAMP_NTZ does not applied

[ https://issues.apache.org/jira/browse/SPARK-44717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17751886#comment-17751886 ] Attila Zsolt Piros commented on SPARK-44717: cc [~itholic] [~gurwls223] >

[jira] [Updated] (SPARK-44717) "pyspark.pandas.resample" is incorrect when DST is overlapped and setting "spark.sql.timestampType" to TIMESTAMP_NTZ does not applied

[ https://issues.apache.org/jira/browse/SPARK-44717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-44717: --- Description: Use one of the existing test: - "11H" case of test_dataframe_resample

[jira] [Updated] (SPARK-44717) "pyspark.pandas.resample" is incorrect when DST is overlapped and setting "spark.sql.timestampType" to TIMESTAMP_NTZ does not applied

[ https://issues.apache.org/jira/browse/SPARK-44717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-44717: --- Description: Use one of the existing test: - "11H" case of test_dataframe_resample

[jira] [Created] (SPARK-44717) "pyspark.pandas.resample" is incorrect when DST is overlapped and setting "spark.sql.timestampType" to TIMESTAMP_NTZ does not applied

Attila Zsolt Piros created SPARK-44717: -- Summary: "pyspark.pandas.resample" is incorrect when DST is overlapped and setting "spark.sql.timestampType" to TIMESTAMP_NTZ does not applied Key: SPARK-44717 URL:

[jira] [Deleted] (SPARK-44716) Add missing callUDF in Python Spark Connect client

[ https://issues.apache.org/jira/browse/SPARK-44716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon deleted SPARK-44716: - > Add missing callUDF in Python Spark Connect client >

[jira] [Created] (SPARK-44716) Add missing callUDF in Python Spark Connect client

Hyukjin Kwon created SPARK-44716: Summary: Add missing callUDF in Python Spark Connect client Key: SPARK-44716 URL: https://issues.apache.org/jira/browse/SPARK-44716 Project: Spark Issue

[jira] [Resolved] (SPARK-44703) Log eventLog rewrite duration when compact old event log files

[ https://issues.apache.org/jira/browse/SPARK-44703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-44703. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42378

[jira] [Assigned] (SPARK-44703) Log eventLog rewrite duration when compact old event log files

[ https://issues.apache.org/jira/browse/SPARK-44703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-44703: Assignee: shuyouZZ > Log eventLog rewrite duration when compact old event log files >

[jira] [Updated] (SPARK-44703) Log eventLog rewrite duration when compact old event log files

[ https://issues.apache.org/jira/browse/SPARK-44703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-44703: - Fix Version/s: (was: 3.5.0) > Log eventLog rewrite duration when compact old event log

[jira] [Resolved] (SPARK-44005) Improve error messages for regular Python UDTFs that return non-tuple values

[ https://issues.apache.org/jira/browse/SPARK-44005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-44005. --- Fix Version/s: 3.5.0 4.0.0 Resolution: Fixed Issue resolved by

[jira] [Assigned] (SPARK-44005) Improve error messages for regular Python UDTFs that return non-tuple values

[ https://issues.apache.org/jira/browse/SPARK-44005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-44005: - Assignee: Allison Wang > Improve error messages for regular Python UDTFs that return

[jira] [Resolved] (SPARK-44689) `UDFClassLoadingE2ESuite` failed in the daily test for Java 17.

[ https://issues.apache.org/jira/browse/SPARK-44689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie resolved SPARK-44689. -- Fix Version/s: 3.5.0 4.0.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-44689) `UDFClassLoadingE2ESuite` failed in the daily test for Java 17.

[ https://issues.apache.org/jira/browse/SPARK-44689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie reassigned SPARK-44689: Assignee: Yang Jie > `UDFClassLoadingE2ESuite` failed in the daily test for Java 17. >

[jira] [Resolved] (SPARK-44554) Install different Python linter dependencies for daily testing of different Spark versions

[ https://issues.apache.org/jira/browse/SPARK-44554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie resolved SPARK-44554. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42167

[jira] [Assigned] (SPARK-44554) Install different Python linter dependencies for daily testing of different Spark versions

[ https://issues.apache.org/jira/browse/SPARK-44554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie reassigned SPARK-44554: Assignee: Yang Jie > Install different Python linter dependencies for daily testing of different

[jira] [Resolved] (SPARK-44641) SPJ: Results duplicated when SPJ partial-cluster and pushdown enabled but conditions unmet

[ https://issues.apache.org/jira/browse/SPARK-44641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun resolved SPARK-44641. -- Fix Version/s: 3.4.2 3.5.0 Assignee: Chao Sun Resolution: Fixed >

[jira] [Updated] (SPARK-44683) Logging level isn't passed to RocksDB state store provider correctly

[ https://issues.apache.org/jira/browse/SPARK-44683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-44683: - Fix Version/s: 3.5.1 (was: 3.5.0) > Logging level isn't passed to

[jira] [Assigned] (SPARK-44683) Logging level isn't passed to RocksDB state store provider correctly

[ https://issues.apache.org/jira/browse/SPARK-44683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-44683: Assignee: Siying Dong > Logging level isn't passed to RocksDB state store provider

[jira] [Resolved] (SPARK-44683) Logging level isn't passed to RocksDB state store provider correctly

[ https://issues.apache.org/jira/browse/SPARK-44683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-44683. -- Fix Version/s: 3.5.0 4.0.0 Resolution: Fixed Issue resolved by pull

[jira] [Resolved] (SPARK-44694) Add Default & Active SparkSession for Python Client

[ https://issues.apache.org/jira/browse/SPARK-44694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-44694. -- Fix Version/s: 3.5.0 4.0.0 Resolution: Fixed Issue resolved by pull

[jira] [Assigned] (SPARK-44694) Add Default & Active SparkSession for Python Client

[ https://issues.apache.org/jira/browse/SPARK-44694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-44694: Assignee: Hyukjin Kwon > Add Default & Active SparkSession for Python Client >

[jira] [Created] (SPARK-44715) Add missing udf and callUdf functions

Herman van Hövell created SPARK-44715: - Summary: Add missing udf and callUdf functions Key: SPARK-44715 URL: https://issues.apache.org/jira/browse/SPARK-44715 Project: Spark Issue Type:

[jira] [Commented] (SPARK-44705) Make PythonRunner single-threaded

[ https://issues.apache.org/jira/browse/SPARK-44705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17751843#comment-17751843 ] Utkarsh Agarwal commented on SPARK-44705: - https://github.com/apache/spark/pull/42385 for this

[jira] [Updated] (SPARK-44714) Ease restriction of LCA resolution regarding queries with having

[ https://issues.apache.org/jira/browse/SPARK-44714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinyi Yu updated SPARK-44714: - Description: Current LCA resolution has a limitation, that it can't resolve the query, when it

[jira] [Created] (SPARK-44714) Ease restriction of LCA resolution regarding queries with having

Xinyi Yu created SPARK-44714: Summary: Ease restriction of LCA resolution regarding queries with having Key: SPARK-44714 URL: https://issues.apache.org/jira/browse/SPARK-44714 Project: Spark

[jira] [Created] (SPARK-44713) Deduplicate files between sql/core and Spark Connect Scala Client

Herman van Hövell created SPARK-44713: - Summary: Deduplicate files between sql/core and Spark Connect Scala Client Key: SPARK-44713 URL: https://issues.apache.org/jira/browse/SPARK-44713 Project:

[jira] [Updated] (SPARK-44712) Migrate ‎test_timedelta_ops assert_eq to use assertDataFrameEqual

[ https://issues.apache.org/jira/browse/SPARK-44712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-44712: --- Description: Migrate assert_eq to assertDataFrameEqual in this file:

[jira] [Created] (SPARK-44712) Migrate ‎test_timedelta_ops assert_eq to use assertDataFrameEqual

Amanda Liu created SPARK-44712: -- Summary: Migrate ‎test_timedelta_ops assert_eq to use assertDataFrameEqual Key: SPARK-44712 URL: https://issues.apache.org/jira/browse/SPARK-44712 Project: Spark

[jira] [Updated] (SPARK-44711) Migrate test_series_conversion assert_eq to use assertDataFrameEqual

[ https://issues.apache.org/jira/browse/SPARK-44711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-44711: --- Description: Migrate assert_eq to assertDataFrameEqual in this file: 

[jira] [Created] (SPARK-44711) Migrate test_series_conversion assert_eq to use assertDataFrameEqual

Amanda Liu created SPARK-44711: -- Summary: Migrate test_series_conversion assert_eq to use assertDataFrameEqual Key: SPARK-44711 URL: https://issues.apache.org/jira/browse/SPARK-44711 Project: Spark

[jira] [Created] (SPARK-44710) Support Dataset.dropDuplicatesWithinWatermark in Scala Client

Herman van Hövell created SPARK-44710: - Summary: Support Dataset.dropDuplicatesWithinWatermark in Scala Client Key: SPARK-44710 URL: https://issues.apache.org/jira/browse/SPARK-44710 Project:

[jira] [Resolved] (SPARK-44692) Move Trigger.scala to sql/api

[ https://issues.apache.org/jira/browse/SPARK-44692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hövell resolved SPARK-44692. --- Fix Version/s: 3.5.0 Resolution: Fixed > Move Trigger.scala to sql/api >

[jira] [Resolved] (SPARK-44132) nesting full outer joins confuses code generator

[ https://issues.apache.org/jira/browse/SPARK-44132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-44132. -- Fix Version/s: 3.5.0 4.0.0 Resolution: Fixed Issue resolved by pull

[jira] [Assigned] (SPARK-44132) nesting full outer joins confuses code generator

[ https://issues.apache.org/jira/browse/SPARK-44132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-44132: Assignee: Steven Aerts > nesting full outer joins confuses code generator >

[jira] [Created] (SPARK-44709) Fix flow control in ExecuteGrpcResponseSender

Juliusz Sompolski created SPARK-44709: - Summary: Fix flow control in ExecuteGrpcResponseSender Key: SPARK-44709 URL: https://issues.apache.org/jira/browse/SPARK-44709 Project: Spark

[jira] [Created] (SPARK-44708) Migrate test_reset_index assert_eq to use assertDataFrameEqual

Amanda Liu created SPARK-44708: -- Summary: Migrate test_reset_index assert_eq to use assertDataFrameEqual Key: SPARK-44708 URL: https://issues.apache.org/jira/browse/SPARK-44708 Project: Spark

[jira] [Assigned] (SPARK-44575) Implement Error Translation

[ https://issues.apache.org/jira/browse/SPARK-44575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-44575: Assignee: Yihong He > Implement Error Translation > --- > >

[jira] [Resolved] (SPARK-44575) Implement Error Translation

[ https://issues.apache.org/jira/browse/SPARK-44575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-44575. -- Fix Version/s: 3.5.0 4.0.0 Resolution: Fixed Issue resolved by pull

[jira] [Updated] (SPARK-44508) Add user guide for Python UDTFs

[ https://issues.apache.org/jira/browse/SPARK-44508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allison Wang updated SPARK-44508: - Priority: Major (was: Blocker) > Add user guide for Python UDTFs >

[jira] [Commented] (SPARK-44508) Add user guide for Python UDTFs

[ https://issues.apache.org/jira/browse/SPARK-44508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17751807#comment-17751807 ] Allison Wang commented on SPARK-44508: -- [~holden] Yup you're right. This isn't a blocker since we

[jira] [Assigned] (SPARK-43606) Enable IndexesTests.test_index_basic for pandas 2.0.0.

[ https://issues.apache.org/jira/browse/SPARK-43606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-43606: Assignee: Haejoon Lee > Enable IndexesTests.test_index_basic for pandas 2.0.0. >

[jira] [Resolved] (SPARK-43606) Enable IndexesTests.test_index_basic for pandas 2.0.0.

[ https://issues.apache.org/jira/browse/SPARK-43606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-43606. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42267

[jira] [Resolved] (SPARK-44707) Use INFO log in ExecutorPodsWatcher.onClose if SparkContext is stopped

[ https://issues.apache.org/jira/browse/SPARK-44707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-44707. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42381

[jira] [Assigned] (SPARK-44707) Use INFO log in ExecutorPodsWatcher.onClose if SparkContext is stopped

[ https://issues.apache.org/jira/browse/SPARK-44707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-44707: - Assignee: Dongjoon Hyun > Use INFO log in ExecutorPodsWatcher.onClose if SparkContext

[jira] [Commented] (SPARK-44050) Unable to Mount ConfigMap in Driver Pod - ConfigMap Creation Issue

[ https://issues.apache.org/jira/browse/SPARK-44050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17751796#comment-17751796 ] Holden Karau commented on SPARK-44050: -- Ah interesting, it sounds like the fix would be to retry

[jira] [Commented] (SPARK-44508) Add user guide for Python UDTFs

[ https://issues.apache.org/jira/browse/SPARK-44508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17751795#comment-17751795 ] Holden Karau commented on SPARK-44508: -- I'm not sure this should be a blocker. > Add user guide

[jira] [Updated] (SPARK-44597) Migrate test_sql assert_eq to use assertDataFrameEqual

[ https://issues.apache.org/jira/browse/SPARK-44597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-44597: --- Description: Migrate tests to new test utils in this file: python/pyspark/pandas/tests/test_sql.py

[jira] [Updated] (SPARK-44589) Migrate PySpark tests to use PySpark built-in test utils

[ https://issues.apache.org/jira/browse/SPARK-44589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-44589: --- Description: The Jira ticket SPARK-44042 SPIP: PySpark Test Framework introduces a new PySpark

[jira] [Resolved] (SPARK-44561) Fix AssertionError when converting UDTF output to a complex type

[ https://issues.apache.org/jira/browse/SPARK-44561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-44561. --- Fix Version/s: 3.5.0 Assignee: Takuya Ueshin Resolution: Fixed Issue

[jira] [Created] (SPARK-44707) Use INFO log in ExecutorPodsWatcher.onClose if SparkContext is stopped

Dongjoon Hyun created SPARK-44707: - Summary: Use INFO log in ExecutorPodsWatcher.onClose if SparkContext is stopped Key: SPARK-44707 URL: https://issues.apache.org/jira/browse/SPARK-44707 Project:

[jira] [Commented] (SPARK-24282) Add support for PMML export for the Standard Scaler Stage

[ https://issues.apache.org/jira/browse/SPARK-24282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17751758#comment-17751758 ] Holden Karau commented on SPARK-24282: -- I don't think were going to do this anymore. > Add support

[jira] [Resolved] (SPARK-28740) Add support for building with bloop

[ https://issues.apache.org/jira/browse/SPARK-28740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-28740. -- Resolution: Won't Fix > Add support for building with bloop >

[jira] [Commented] (SPARK-32111) Cleanup locks and docs in CoarseGrainedSchedulerBackend

[ https://issues.apache.org/jira/browse/SPARK-32111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17751754#comment-17751754 ] Holden Karau commented on SPARK-32111: -- I think this could be a good target for Spark 4.0 >

[jira] [Updated] (SPARK-32111) Cleanup locks and docs in CoarseGrainedSchedulerBackend

[ https://issues.apache.org/jira/browse/SPARK-32111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-32111: - Target Version/s: 4.0.0 Affects Version/s: 4.0.0 > Cleanup locks and docs in

[jira] [Resolved] (SPARK-38475) Use error classes in org.apache.spark.serializer

[ https://issues.apache.org/jira/browse/SPARK-38475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-38475. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42243

[jira] [Assigned] (SPARK-38475) Use error classes in org.apache.spark.serializer

[ https://issues.apache.org/jira/browse/SPARK-38475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-38475: Assignee: Bo Zhang > Use error classes in org.apache.spark.serializer >

[jira] [Commented] (SPARK-44706) ProductEncoder not working as expected within User Defined Fuction (UDF)

[ https://issues.apache.org/jira/browse/SPARK-44706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17751728#comment-17751728 ] Juan Carlos Blanco Martinez commented on SPARK-44706: - Using

[jira] [Resolved] (SPARK-44068) Support positional parameters in Scala connect client

[ https://issues.apache.org/jira/browse/SPARK-44068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-44068. -- Resolution: Duplicate > Support positional parameters in Scala connect client >

[jira] [Updated] (SPARK-44706) ProductEncoder not working as expected within User Defined Fuction (UDF)

[ https://issues.apache.org/jira/browse/SPARK-44706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Juan Carlos Blanco Martinez updated SPARK-44706: Description: Hi, When running the following code in Databricks'

[jira] [Updated] (SPARK-44706) ProductEncoder not working as expected within User Defined Fuction (UDF)

[ https://issues.apache.org/jira/browse/SPARK-44706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Juan Carlos Blanco Martinez updated SPARK-44706: Description: Hi, When running the following code in Databricks'

[jira] [Updated] (SPARK-44706) ProductEncoder not working as expected within User Defined Fuction (UDF)

[ https://issues.apache.org/jira/browse/SPARK-44706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Juan Carlos Blanco Martinez updated SPARK-44706: Description: Hi, When running the following code in Databricks'

[jira] [Updated] (SPARK-44706) ProductEncoder not working as expected within User Defined Fuction (UDF)

[ https://issues.apache.org/jira/browse/SPARK-44706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Juan Carlos Blanco Martinez updated SPARK-44706: Description: Hi, When running the following code in Databricks'

[jira] [Created] (SPARK-44706) ProductEncoder not working as expected within User Defined Fuction (UDF)

Juan Carlos Blanco Martinez created SPARK-44706: --- Summary: ProductEncoder not working as expected within User Defined Fuction (UDF) Key: SPARK-44706 URL:

[jira] [Created] (SPARK-44705) Make PythonRunner single-threaded

Utkarsh Agarwal created SPARK-44705: --- Summary: Make PythonRunner single-threaded Key: SPARK-44705 URL: https://issues.apache.org/jira/browse/SPARK-44705 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-44634) Encoders.bean does no longer support nested beans with type arguments

[ https://issues.apache.org/jira/browse/SPARK-44634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giambattista Bloisi resolved SPARK-44634. - Fix Version/s: 3.4.2 3.5.0 4.0.0

[jira] [Assigned] (SPARK-44697) Clean up the usage of org.apache.commons.lang3.RandomUtils

[ https://issues.apache.org/jira/browse/SPARK-44697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie reassigned SPARK-44697: Assignee: Yang Jie > Clean up the usage of org.apache.commons.lang3.RandomUtils >

[jira] [Resolved] (SPARK-44697) Clean up the usage of org.apache.commons.lang3.RandomUtils

[ https://issues.apache.org/jira/browse/SPARK-44697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie resolved SPARK-44697. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42370

[jira] [Commented] (SPARK-44686) Add option to create RowEncoder in Encoders helper class.

[ https://issues.apache.org/jira/browse/SPARK-44686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17751671#comment-17751671 ] Herman van Hövell commented on SPARK-44686: --- This has been merged to master/3.5. Preparing

[jira] [Commented] (SPARK-44704) Cleanup shuffle files from host node after migration due to graceful decommissioning

[ https://issues.apache.org/jira/browse/SPARK-44704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17751654#comment-17751654 ] Deependra Patel commented on SPARK-44704: - I will create for this soon. > Cleanup shuffle files

[jira] [Created] (SPARK-44704) Cleanup shuffle files from host node after migration due to graceful decommissioning

Deependra Patel created SPARK-44704: --- Summary: Cleanup shuffle files from host node after migration due to graceful decommissioning Key: SPARK-44704 URL: https://issues.apache.org/jira/browse/SPARK-44704

[jira] [Commented] (SPARK-44694) Add Default & Active SparkSession for Python Client

[ https://issues.apache.org/jira/browse/SPARK-44694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17751634#comment-17751634 ] Nikita Awasthi commented on SPARK-44694: User 'HyukjinKwon' has created a pull request for this

[jira] [Assigned] (SPARK-44701) Skip ClassificationTestsOnConnect when torch is not installed

[ https://issues.apache.org/jira/browse/SPARK-44701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-44701: Assignee: Ruifeng Zheng > Skip ClassificationTestsOnConnect when torch is not installed

[jira] [Resolved] (SPARK-44701) Skip ClassificationTestsOnConnect when torch is not installed

[ https://issues.apache.org/jira/browse/SPARK-44701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-44701. -- Fix Version/s: 3.5.0 4.0.0 Resolution: Fixed Issue resolved by pull

[jira] [Updated] (SPARK-20080) Spak streaming application do not throw serialisation exception in foreachRDD

[ https://issues.apache.org/jira/browse/SPARK-20080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Hryhoriev updated SPARK-20080: --- Description: Step to reproduce: 1)Use SPak Streaming application. 2) inside foreachRDD of

[jira] [Resolved] (SPARK-31427) Spark Structure streaming read data twice per every micro-batch.

[ https://issues.apache.org/jira/browse/SPARK-31427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Hryhoriev resolved SPARK-31427. Resolution: Invalid > Spark Structure streaming read data twice per every micro-batch. >

[jira] [Resolved] (SPARK-34204) When use input_file_name() func all column from file appeared in physical plan of query, not only projection.

[ https://issues.apache.org/jira/browse/SPARK-34204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Hryhoriev resolved SPARK-34204. Fix Version/s: 3.1.1 Resolution: Fixed > When use input_file_name() func all

[jira] [Commented] (SPARK-35094) Spark from_json(JsonToStruct) function return wrong value in permissive mode in case best effort

[ https://issues.apache.org/jira/browse/SPARK-35094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17751630#comment-17751630 ] Nick Hryhoriev commented on SPARK-35094: I miss understand, do the issue resolved? > Spark

[jira] [Created] (SPARK-44703) Log eventLog rewrite duration when compact old event log files

shuyouZZ created SPARK-44703: Summary: Log eventLog rewrite duration when compact old event log files Key: SPARK-44703 URL: https://issues.apache.org/jira/browse/SPARK-44703 Project: Spark

[jira] [Updated] (SPARK-40497) Upgrade Scala to 2.13.11

[ https://issues.apache.org/jira/browse/SPARK-40497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-40497: - Fix Version/s: (was: 3.5.0) > Upgrade Scala to 2.13.11 > > >

[jira] [Updated] (SPARK-40497) Upgrade Scala to 2.13.11

[ https://issues.apache.org/jira/browse/SPARK-40497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-40497: - Affects Version/s: 4.0.0 (was: 3.4.0) > Upgrade Scala to 2.13.11 >

[jira] [Reopened] (SPARK-40497) Upgrade Scala to 2.13.11

[ https://issues.apache.org/jira/browse/SPARK-40497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie reopened SPARK-40497: --  reopen this due to SPARK-44690 & SPARK-44376 downgrade to 2.13.8,  We need to upgrade again after

[jira] [Commented] (SPARK-44702) Cannot derive ExpressionEncoder when using types tagged by a trait

[ https://issues.apache.org/jira/browse/SPARK-44702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17751559#comment-17751559 ] Ben Hemsi commented on SPARK-44702: --- This is my first time working on spark. I'm happy to be assigned

[jira] [Updated] (SPARK-44702) Cannot derive ExpressionEncoder when using types tagged by a trait

[ https://issues.apache.org/jira/browse/SPARK-44702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Hemsi updated SPARK-44702: -- Priority: Minor (was: Major) > Cannot derive ExpressionEncoder when using types tagged by a trait >

[jira] [Commented] (SPARK-44700) Rule OptimizeCsvJsonExprs should not be applied to expression like from_json(regexp_replace)

[ https://issues.apache.org/jira/browse/SPARK-44700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17751555#comment-17751555 ] ASF GitHub Bot commented on SPARK-44700: User 'monkeyboy123' has created a pull request for this

[jira] [Updated] (SPARK-44702) Cannot derive ExpressionEncoder when using types tagged by a trait

[ https://issues.apache.org/jira/browse/SPARK-44702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Hemsi updated SPARK-44702: -- Description: Steps to reproduce: {code:java} trait Tag case class Foo(x: Int) case class Bar(x: Foo

[jira] [Created] (SPARK-44702) Cannot derive ExpressionEncoder when using types tagged by a trait

Ben Hemsi created SPARK-44702: - Summary: Cannot derive ExpressionEncoder when using types tagged by a trait Key: SPARK-44702 URL: https://issues.apache.org/jira/browse/SPARK-44702 Project: Spark

[jira] [Commented] (SPARK-44669) Parquet/ORC files written using Hive Serde should has file extension

[ https://issues.apache.org/jira/browse/SPARK-44669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17751548#comment-17751548 ] ASF GitHub Bot commented on SPARK-44669: User 'pan3793' has created a pull request for this

[jira] [Updated] (SPARK-44700) Rule OptimizeCsvJsonExprs should not be applied to expression like from_json(regexp_replace)

[ https://issues.apache.org/jira/browse/SPARK-44700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiahong.li updated SPARK-44700: --- Description: _SQL_ like below: select tmp.*   from  (select         device_id, ads_id,         

[jira] [Updated] (SPARK-44700) Rule OptimizeCsvJsonExprs should not be applied to expression like from_json(regexp_replace)

[ https://issues.apache.org/jira/browse/SPARK-44700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiahong.li updated SPARK-44700: --- Description: _SQL_ like below: select tmp.*   from  (select         device_id, ads_id,         

[jira] [Updated] (SPARK-44700) Rule OptimizeCsvJsonExprs should not be applied to expression like from_json(regexp_replace)

[ https://issues.apache.org/jira/browse/SPARK-44700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiahong.li updated SPARK-44700: --- Description: _SQL_ like below: ``` select tmp.*   from  (select         device_id, ads_id,       

[jira] [Updated] (SPARK-44700) Rule OptimizeCsvJsonExprs should not be applied to expression like from_json(regexp_replace)

[ https://issues.apache.org/jira/browse/SPARK-44700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiahong.li updated SPARK-44700: --- Description: _SQL_ like below: ``` select tmp.*   from  (select         device_id, ads_id,       

[jira] [Created] (SPARK-44701) Skip ClassificationTestsOnConnect when torch is not installed

Ruifeng Zheng created SPARK-44701: - Summary: Skip ClassificationTestsOnConnect when torch is not installed Key: SPARK-44701 URL: https://issues.apache.org/jira/browse/SPARK-44701 Project: Spark

[jira] [Updated] (SPARK-44700) Rule OptimizeCsvJsonExprs should not be applied to expression like from_json(regexp_replace)

[ https://issues.apache.org/jira/browse/SPARK-44700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiahong.li updated SPARK-44700: --- Component/s: SQL > Rule OptimizeCsvJsonExprs should not be applied to expression like >

[jira] [Updated] (SPARK-44700) Rule OptimizeCsvJsonExprs should not be applied to expression like from_json(regexp_replace)

[ https://issues.apache.org/jira/browse/SPARK-44700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiahong.li updated SPARK-44700: --- Component/s: (was: Spark Core) > Rule OptimizeCsvJsonExprs should not be applied to expression

[jira] [Created] (SPARK-44700) Rule OptimizeCsvJsonExprs should not be applied to expression like from_json(regexp_replace)

jiahong.li created SPARK-44700: -- Summary: Rule OptimizeCsvJsonExprs should not be applied to expression like from_json(regexp_replace) Key: SPARK-44700 URL: https://issues.apache.org/jira/browse/SPARK-44700

[jira] [Resolved] (SPARK-44698) Create table like other table should also copy table stats.

[ https://issues.apache.org/jira/browse/SPARK-44698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qi Zhu resolved SPARK-44698. Resolution: Not A Problem Sorry i misunderstand, the create table like don't need to copy data actually!

  1   2   >