[jira] [Assigned] (SPARK-43117) proto message abbreviation should support repeated fields
[ https://issues.apache.org/jira/browse/SPARK-43117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-43117: - Assignee: Ruifeng Zheng > proto message abbreviation should support repeated fields > - > > Key: SPARK-43117 > URL: https://issues.apache.org/jira/browse/SPARK-43117 > Project: Spark > Issue Type: Improvement > Components: Connect >Affects Versions: 3.5.0 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-43117) proto message abbreviation should support repeated fields
[ https://issues.apache.org/jira/browse/SPARK-43117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-43117. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45056 [https://github.com/apache/spark/pull/45056] > proto message abbreviation should support repeated fields > - > > Key: SPARK-43117 > URL: https://issues.apache.org/jira/browse/SPARK-43117 > Project: Spark > Issue Type: Improvement > Components: Connect >Affects Versions: 3.5.0 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-46615) Support s.c.immutable.ArraySeq in ArrowDeserializers
[ https://issues.apache.org/jira/browse/SPARK-46615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie reassigned SPARK-46615: Assignee: BingKun Pan > Support s.c.immutable.ArraySeq in ArrowDeserializers > > > Key: SPARK-46615 > URL: https://issues.apache.org/jira/browse/SPARK-46615 > Project: Spark > Issue Type: Improvement > Components: Connect >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Assignee: BingKun Pan >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-46615) Support s.c.immutable.ArraySeq in ArrowDeserializers
[ https://issues.apache.org/jira/browse/SPARK-46615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie resolved SPARK-46615. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44618 [https://github.com/apache/spark/pull/44618] > Support s.c.immutable.ArraySeq in ArrowDeserializers > > > Key: SPARK-46615 > URL: https://issues.apache.org/jira/browse/SPARK-46615 > Project: Spark > Issue Type: Improvement > Components: Connect >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Assignee: BingKun Pan >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-46997) Enable `spark.worker.cleanup.enabled` by default
[ https://issues.apache.org/jira/browse/SPARK-46997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-46997. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45055 [https://github.com/apache/spark/pull/45055] > Enable `spark.worker.cleanup.enabled` by default > > > Key: SPARK-46997 > URL: https://issues.apache.org/jira/browse/SPARK-46997 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47005) Refine docstring of `asc_nulls_first/asc_nulls_last/desc_nulls_first/desc_nulls_last`
[ https://issues.apache.org/jira/browse/SPARK-47005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie reassigned SPARK-47005: Assignee: Yang Jie > Refine docstring of > `asc_nulls_first/asc_nulls_last/desc_nulls_first/desc_nulls_last` > - > > Key: SPARK-47005 > URL: https://issues.apache.org/jira/browse/SPARK-47005 > Project: Spark > Issue Type: Sub-task > Components: Documentation, PySpark >Affects Versions: 4.0.0 >Reporter: Yang Jie >Assignee: Yang Jie >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47005) Refine docstring of `asc_nulls_first/asc_nulls_last/desc_nulls_first/desc_nulls_last`
[ https://issues.apache.org/jira/browse/SPARK-47005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie resolved SPARK-47005. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45066 [https://github.com/apache/spark/pull/45066] > Refine docstring of > `asc_nulls_first/asc_nulls_last/desc_nulls_first/desc_nulls_last` > - > > Key: SPARK-47005 > URL: https://issues.apache.org/jira/browse/SPARK-47005 > Project: Spark > Issue Type: Sub-task > Components: Documentation, PySpark >Affects Versions: 4.0.0 >Reporter: Yang Jie >Assignee: Yang Jie >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47005) Refine docstring of `asc_nulls_first/asc_nulls_last/desc_nulls_first/desc_nulls_last`
[ https://issues.apache.org/jira/browse/SPARK-47005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47005: --- Labels: pull-request-available (was: ) > Refine docstring of > `asc_nulls_first/asc_nulls_last/desc_nulls_first/desc_nulls_last` > - > > Key: SPARK-47005 > URL: https://issues.apache.org/jira/browse/SPARK-47005 > Project: Spark > Issue Type: Sub-task > Components: Documentation, PySpark >Affects Versions: 4.0.0 >Reporter: Yang Jie >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47005) Refine docstring of `asc_nulls_first/asc_nulls_last/desc_nulls_first/desc_nulls_last`
Yang Jie created SPARK-47005: Summary: Refine docstring of `asc_nulls_first/asc_nulls_last/desc_nulls_first/desc_nulls_last` Key: SPARK-47005 URL: https://issues.apache.org/jira/browse/SPARK-47005 Project: Spark Issue Type: Sub-task Components: Documentation, PySpark Affects Versions: 4.0.0 Reporter: Yang Jie -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-46994) Refactor PythonWrite to prepare for supporting python data source streaming write
[ https://issues.apache.org/jira/browse/SPARK-46994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-46994. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45049 [https://github.com/apache/spark/pull/45049] > Refactor PythonWrite to prepare for supporting python data source streaming > write > - > > Key: SPARK-46994 > URL: https://issues.apache.org/jira/browse/SPARK-46994 > Project: Spark > Issue Type: Improvement > Components: PySpark >Affects Versions: 4.0.0 >Reporter: Chaoqin Li >Assignee: Chaoqin Li >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > Move PythonBatchWrite out of PythonWrite. This is to prepare for supporting > python data source streaming write in the future. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-46994) Refactor PythonWrite to prepare for supporting python data source streaming write
[ https://issues.apache.org/jira/browse/SPARK-46994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-46994: Assignee: Chaoqin Li > Refactor PythonWrite to prepare for supporting python data source streaming > write > - > > Key: SPARK-46994 > URL: https://issues.apache.org/jira/browse/SPARK-46994 > Project: Spark > Issue Type: Improvement > Components: PySpark >Affects Versions: 4.0.0 >Reporter: Chaoqin Li >Assignee: Chaoqin Li >Priority: Major > Labels: pull-request-available > > Move PythonBatchWrite out of PythonWrite. This is to prepare for supporting > python data source streaming write in the future. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-46994) Refactor PythonWrite to prepare for supporting python data source streaming write
[ https://issues.apache.org/jira/browse/SPARK-46994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46994: --- Labels: pull-request-available (was: ) > Refactor PythonWrite to prepare for supporting python data source streaming > write > - > > Key: SPARK-46994 > URL: https://issues.apache.org/jira/browse/SPARK-46994 > Project: Spark > Issue Type: Improvement > Components: PySpark >Affects Versions: 4.0.0 >Reporter: Chaoqin Li >Priority: Major > Labels: pull-request-available > > Move PythonBatchWrite out of PythonWrite. This is to prepare for supporting > python data source streaming write in the future. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-46865) Add Batch Support for TransformWithState Operator
[ https://issues.apache.org/jira/browse/SPARK-46865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-46865. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44884 [https://github.com/apache/spark/pull/44884] > Add Batch Support for TransformWithState Operator > - > > Key: SPARK-46865 > URL: https://issues.apache.org/jira/browse/SPARK-46865 > Project: Spark > Issue Type: Improvement > Components: Structured Streaming >Affects Versions: 4.0.0 >Reporter: Eric Marnadi >Assignee: Eric Marnadi >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > Add Batch support for the TransformWithState operator to maintain parity > between the batch and streaming APIs -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-46865) Add Batch Support for TransformWithState Operator
[ https://issues.apache.org/jira/browse/SPARK-46865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-46865: Assignee: Eric Marnadi > Add Batch Support for TransformWithState Operator > - > > Key: SPARK-46865 > URL: https://issues.apache.org/jira/browse/SPARK-46865 > Project: Spark > Issue Type: Improvement > Components: Structured Streaming >Affects Versions: 4.0.0 >Reporter: Eric Marnadi >Assignee: Eric Marnadi >Priority: Major > Labels: pull-request-available > > Add Batch support for the TransformWithState operator to maintain parity > between the batch and streaming APIs -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-46998) The SQL config spark.sql.legacy.allowZeroIndexInFormatString doesn't work
[
https://issues.apache.org/jira/browse/SPARK-46998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun resolved SPARK-46998.
---
Fix Version/s: 4.0.0
Resolution: Fixed
Issue resolved by pull request 45057
[https://github.com/apache/spark/pull/45057]
> The SQL config spark.sql.legacy.allowZeroIndexInFormatString doesn't work
> -
>
> Key: SPARK-46998
> URL: https://issues.apache.org/jira/browse/SPARK-46998
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 4.0.0
>Reporter: Max Gekk
>Assignee: Max Gekk
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> The SQL config spark.sql.legacy.allowZeroIndexInFormatString doesn't allow to
> use the zero index. Even set it to true, users get the error:
> {code:sql}
> > select format_string('%0$s', 'Hello');
> Illegal format argument index = 0
> java.util.IllegalFormatArgumentIndexException: Illegal format argument index
> = 0
> at
> java.base/java.util.Formatter$FormatSpecifier.index(Formatter.java:2808)
> at
> java.base/java.util.Formatter$FormatSpecifier.(Formatter.java:2879)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47003) Detect and fail on invalid volume sizes (< 1KiB) in K8s
[ https://issues.apache.org/jira/browse/SPARK-47003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47003. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45061 [https://github.com/apache/spark/pull/45061] > Detect and fail on invalid volume sizes (< 1KiB) in K8s > --- > > Key: SPARK-47003 > URL: https://issues.apache.org/jira/browse/SPARK-47003 > Project: Spark > Issue Type: Bug > Components: Kubernetes >Affects Versions: 3.5.0, 4.0.0, 3.4.3 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47000) Use `getTotalMemorySize` in `WorkerArguments`
[ https://issues.apache.org/jira/browse/SPARK-47000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47000. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45060 [https://github.com/apache/spark/pull/45060] > Use `getTotalMemorySize` in `WorkerArguments` > - > > Key: SPARK-47000 > URL: https://issues.apache.org/jira/browse/SPARK-47000 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-46832) Collate and Collation expression support
[ https://issues.apache.org/jira/browse/SPARK-46832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46832: --- Labels: pull-request-available (was: ) > Collate and Collation expression support > > > Key: SPARK-46832 > URL: https://issues.apache.org/jira/browse/SPARK-46832 > Project: Spark > Issue Type: Task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Aleksandar Tomic >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-46832) Collate and Collation expression support
[ https://issues.apache.org/jira/browse/SPARK-46832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksandar Tomic updated SPARK-46832: - Summary: Collate and Collation expression support (was: Collation support in UTF8Strings) > Collate and Collation expression support > > > Key: SPARK-46832 > URL: https://issues.apache.org/jira/browse/SPARK-46832 > Project: Spark > Issue Type: Task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Aleksandar Tomic >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47002) Enforce that 'AnalyzeResult' 'orderBy' field is a list of pyspark.sql.functions.OrderingColumn
[ https://issues.apache.org/jira/browse/SPARK-47002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47002: --- Labels: pull-request-available (was: ) > Enforce that 'AnalyzeResult' 'orderBy' field is a list of > pyspark.sql.functions.OrderingColumn > -- > > Key: SPARK-47002 > URL: https://issues.apache.org/jira/browse/SPARK-47002 > Project: Spark > Issue Type: Sub-task > Components: PySpark >Affects Versions: 4.0.0 >Reporter: Daniel >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-46691) Support profiling on WindowInPandasExec
[ https://issues.apache.org/jira/browse/SPARK-46691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-46691. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45035 [https://github.com/apache/spark/pull/45035] > Support profiling on WindowInPandasExec > --- > > Key: SPARK-46691 > URL: https://issues.apache.org/jira/browse/SPARK-46691 > Project: Spark > Issue Type: Sub-task > Components: PySpark >Affects Versions: 4.0.0 >Reporter: Takuya Ueshin >Assignee: Xinrong Meng >Priority: Major > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-46688) Support profiling on AggregateInPandasExec
[ https://issues.apache.org/jira/browse/SPARK-46688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin reassigned SPARK-46688: - Assignee: Xinrong Meng > Support profiling on AggregateInPandasExec > -- > > Key: SPARK-46688 > URL: https://issues.apache.org/jira/browse/SPARK-46688 > Project: Spark > Issue Type: Sub-task > Components: PySpark >Affects Versions: 4.0.0 >Reporter: Takuya Ueshin >Assignee: Xinrong Meng >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-46688) Support profiling on AggregateInPandasExec
[ https://issues.apache.org/jira/browse/SPARK-46688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-46688. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45035 [https://github.com/apache/spark/pull/45035] > Support profiling on AggregateInPandasExec > -- > > Key: SPARK-46688 > URL: https://issues.apache.org/jira/browse/SPARK-46688 > Project: Spark > Issue Type: Sub-task > Components: PySpark >Affects Versions: 4.0.0 >Reporter: Takuya Ueshin >Assignee: Xinrong Meng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-46691) Support profiling on WindowInPandasExec
[ https://issues.apache.org/jira/browse/SPARK-46691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin reassigned SPARK-46691: - Assignee: Xinrong Meng > Support profiling on WindowInPandasExec > --- > > Key: SPARK-46691 > URL: https://issues.apache.org/jira/browse/SPARK-46691 > Project: Spark > Issue Type: Sub-task > Components: PySpark >Affects Versions: 4.0.0 >Reporter: Takuya Ueshin >Assignee: Xinrong Meng >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-46966) Create API for 'analyze' method to indicate subset of input table columns to select
[ https://issues.apache.org/jira/browse/SPARK-46966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin reassigned SPARK-46966: - Assignee: Daniel > Create API for 'analyze' method to indicate subset of input table columns to > select > --- > > Key: SPARK-46966 > URL: https://issues.apache.org/jira/browse/SPARK-46966 > Project: Spark > Issue Type: Sub-task > Components: PySpark >Affects Versions: 4.0.0 >Reporter: Daniel >Assignee: Daniel >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-46966) Create API for 'analyze' method to indicate subset of input table columns to select
[ https://issues.apache.org/jira/browse/SPARK-46966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-46966. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45007 [https://github.com/apache/spark/pull/45007] > Create API for 'analyze' method to indicate subset of input table columns to > select > --- > > Key: SPARK-46966 > URL: https://issues.apache.org/jira/browse/SPARK-46966 > Project: Spark > Issue Type: Sub-task > Components: PySpark >Affects Versions: 4.0.0 >Reporter: Daniel >Assignee: Daniel >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47003) Detect and fail on invalid volume sizes (< 1KiB) in K8s
[ https://issues.apache.org/jira/browse/SPARK-47003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47003: --- Labels: pull-request-available (was: ) > Detect and fail on invalid volume sizes (< 1KiB) in K8s > --- > > Key: SPARK-47003 > URL: https://issues.apache.org/jira/browse/SPARK-47003 > Project: Spark > Issue Type: Bug > Components: Kubernetes >Affects Versions: 3.5.0, 4.0.0, 3.4.3 >Reporter: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47003) Detect and fail on invalid volume sizes (< 1KiB) in K8s
[ https://issues.apache.org/jira/browse/SPARK-47003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47003: -- Affects Version/s: 3.5.0 3.4.3 > Detect and fail on invalid volume sizes (< 1KiB) in K8s > --- > > Key: SPARK-47003 > URL: https://issues.apache.org/jira/browse/SPARK-47003 > Project: Spark > Issue Type: Bug > Components: Kubernetes >Affects Versions: 3.5.0, 4.0.0, 3.4.3 >Reporter: Dongjoon Hyun >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47003) Detect and fail on invalid volume sizes (< 1KiB) in K8s
Dongjoon Hyun created SPARK-47003: - Summary: Detect and fail on invalid volume sizes (< 1KiB) in K8s Key: SPARK-47003 URL: https://issues.apache.org/jira/browse/SPARK-47003 Project: Spark Issue Type: Bug Components: Kubernetes Affects Versions: 4.0.0 Reporter: Dongjoon Hyun -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47002) Enforce that 'AnalyzeResult' 'orderBy' field is a list of pyspark.sql.functions.OrderingColumn
Daniel created SPARK-47002: -- Summary: Enforce that 'AnalyzeResult' 'orderBy' field is a list of pyspark.sql.functions.OrderingColumn Key: SPARK-47002 URL: https://issues.apache.org/jira/browse/SPARK-47002 Project: Spark Issue Type: Sub-task Components: PySpark Affects Versions: 4.0.0 Reporter: Daniel -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-46961) Adding processorHandle as a Context Variable
[ https://issues.apache.org/jira/browse/SPARK-46961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46961: --- Labels: pull-request-available (was: ) > Adding processorHandle as a Context Variable > > > Key: SPARK-46961 > URL: https://issues.apache.org/jira/browse/SPARK-46961 > Project: Spark > Issue Type: Improvement > Components: Structured Streaming >Affects Versions: 4.0.0 >Reporter: Eric Marnadi >Priority: Major > Labels: pull-request-available > > Instead of passing the StatefulProcessorHandle to the user in `init`, instead > embed it as a context variable, ProcessorContext, that the user can fetch -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-45869) Revisit and Improve Spark Standalone Cluster
[ https://issues.apache.org/jira/browse/SPARK-45869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-45869: -- Description: (was: This is an experimental internal configuration for advance users.) > Revisit and Improve Spark Standalone Cluster > > > Key: SPARK-45869 > URL: https://issues.apache.org/jira/browse/SPARK-45869 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Critical > Labels: releasenotes > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47000) Use `getTotalMemorySize` in `WorkerArguments`
[ https://issues.apache.org/jira/browse/SPARK-47000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-47000: - Assignee: Dongjoon Hyun > Use `getTotalMemorySize` in `WorkerArguments` > - > > Key: SPARK-47000 > URL: https://issues.apache.org/jira/browse/SPARK-47000 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47000) Use `getTotalMemorySize` in `WorkerArguments`
[ https://issues.apache.org/jira/browse/SPARK-47000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47000: --- Labels: pull-request-available (was: ) > Use `getTotalMemorySize` in `WorkerArguments` > - > > Key: SPARK-47000 > URL: https://issues.apache.org/jira/browse/SPARK-47000 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47000) Use `getTotalMemorySize` in `WorkerArguments`
Dongjoon Hyun created SPARK-47000: - Summary: Use `getTotalMemorySize` in `WorkerArguments` Key: SPARK-47000 URL: https://issues.apache.org/jira/browse/SPARK-47000 Project: Spark Issue Type: Sub-task Components: Spark Core Affects Versions: 4.0.0 Reporter: Dongjoon Hyun -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-46993) Allow session variables in more places such as from_json for schema
[ https://issues.apache.org/jira/browse/SPARK-46993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46993: --- Labels: pull-request-available (was: ) > Allow session variables in more places such as from_json for schema > --- > > Key: SPARK-46993 > URL: https://issues.apache.org/jira/browse/SPARK-46993 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 3.4.2 >Reporter: Serge Rielau >Priority: Major > Labels: pull-request-available > > It appears we do not allow session variables to provide a schema for > from_json(). > This is likely a generic restriction re constant folding. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-46922) Do not wrap runtime user-facing errors
[ https://issues.apache.org/jira/browse/SPARK-46922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-46922: --- Assignee: Wenchen Fan > Do not wrap runtime user-facing errors > -- > > Key: SPARK-46922 > URL: https://issues.apache.org/jira/browse/SPARK-46922 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Wenchen Fan >Assignee: Wenchen Fan >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-46922) Do not wrap runtime user-facing errors
[ https://issues.apache.org/jira/browse/SPARK-46922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-46922. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44953 [https://github.com/apache/spark/pull/44953] > Do not wrap runtime user-facing errors > -- > > Key: SPARK-46922 > URL: https://issues.apache.org/jira/browse/SPARK-46922 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Wenchen Fan >Assignee: Wenchen Fan >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-46999) ExpressionWithUnresolvedIdentifier should include other expressions in the expression tree
[ https://issues.apache.org/jira/browse/SPARK-46999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46999: --- Labels: pull-request-available (was: ) > ExpressionWithUnresolvedIdentifier should include other expressions in the > expression tree > -- > > Key: SPARK-46999 > URL: https://issues.apache.org/jira/browse/SPARK-46999 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.4.0 >Reporter: Wenchen Fan >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-46999) ExpressionWithUnresolvedIdentifier should include other expressions in the expression tree
Wenchen Fan created SPARK-46999: --- Summary: ExpressionWithUnresolvedIdentifier should include other expressions in the expression tree Key: SPARK-46999 URL: https://issues.apache.org/jira/browse/SPARK-46999 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 3.4.0 Reporter: Wenchen Fan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-46998) The SQL config spark.sql.legacy.allowZeroIndexInFormatString doesn't work
[
https://issues.apache.org/jira/browse/SPARK-46998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-46998:
---
Labels: pull-request-available (was: )
> The SQL config spark.sql.legacy.allowZeroIndexInFormatString doesn't work
> -
>
> Key: SPARK-46998
> URL: https://issues.apache.org/jira/browse/SPARK-46998
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 4.0.0
>Reporter: Max Gekk
>Assignee: Max Gekk
>Priority: Major
> Labels: pull-request-available
>
> The SQL config spark.sql.legacy.allowZeroIndexInFormatString doesn't allow to
> use the zero index. Even set it to true, users get the error:
> {code:sql}
> > select format_string('%0$s', 'Hello');
> Illegal format argument index = 0
> java.util.IllegalFormatArgumentIndexException: Illegal format argument index
> = 0
> at
> java.base/java.util.Formatter$FormatSpecifier.index(Formatter.java:2808)
> at
> java.base/java.util.Formatter$FormatSpecifier.(Formatter.java:2879)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-46998) The SQL config spark.sql.legacy.allowZeroIndexInFormatString doesn't work
Max Gekk created SPARK-46998:
Summary: The SQL config
spark.sql.legacy.allowZeroIndexInFormatString doesn't work
Key: SPARK-46998
URL: https://issues.apache.org/jira/browse/SPARK-46998
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.0.0
Reporter: Max Gekk
Assignee: Max Gekk
The SQL config spark.sql.legacy.allowZeroIndexInFormatString doesn't allow to
use the zero index. Even set it to true, users get the error:
{code:sql}
> select format_string('%0$s', 'Hello');
Illegal format argument index = 0
java.util.IllegalFormatArgumentIndexException: Illegal format argument index = 0
at
java.base/java.util.Formatter$FormatSpecifier.index(Formatter.java:2808)
at
java.base/java.util.Formatter$FormatSpecifier.(Formatter.java:2879)
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
