[jira] [Assigned] (SPARK-35022) Task Scheduling Plugin in Spark

[ https://issues.apache.org/jira/browse/SPARK-35022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh reassigned SPARK-35022: --- Assignee: L. C. Hsieh > Task Scheduling Plugin in Spark > --- >

[jira] [Commented] (SPARK-35022) Task Scheduling Plugin in Spark

[ https://issues.apache.org/jira/browse/SPARK-35022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17318666#comment-17318666 ] L. C. Hsieh commented on SPARK-35022: - Design doc:  https://docs.google.com/document

[jira] [Created] (SPARK-35022) Task Scheduling Plugin in Spark

L. C. Hsieh created SPARK-35022: --- Summary: Task Scheduling Plugin in Spark Key: SPARK-35022 URL: https://issues.apache.org/jira/browse/SPARK-35022 Project: Spark Issue Type: New Feature

[jira] [Updated] (SPARK-34897) Support reconcile schemas based on index after nested column pruning

[ https://issues.apache.org/jira/browse/SPARK-34897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34897: Summary: Support reconcile schemas based on index after nested column pruning (was: The given dat

[jira] [Issue Comment Deleted] (SPARK-34897) Support reconcile schemas based on index after nested column pruning

[ https://issues.apache.org/jira/browse/SPARK-34897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34897: Comment: was deleted (was: We can workaround this issue by setting spark.sql.optimizer.nestedSche

[jira] [Assigned] (SPARK-35019) Improve type hints on pyspark.sql.*

[ https://issues.apache.org/jira/browse/SPARK-35019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35019: Assignee: (was: Apache Spark) > Improve type hints on pyspark.sql.* > ---

[jira] [Assigned] (SPARK-35019) Improve type hints on pyspark.sql.*

[ https://issues.apache.org/jira/browse/SPARK-35019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35019: Assignee: Apache Spark > Improve type hints on pyspark.sql.* > --

[jira] [Commented] (SPARK-35019) Improve type hints on pyspark.sql.*

[ https://issues.apache.org/jira/browse/SPARK-35019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17318651#comment-17318651 ] Apache Spark commented on SPARK-35019: -- User 'Yikun' has created a pull request for

[jira] [Commented] (SPARK-35019) Improve type hints on pyspark.sql.*

[ https://issues.apache.org/jira/browse/SPARK-35019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17318652#comment-17318652 ] Apache Spark commented on SPARK-35019: -- User 'Yikun' has created a pull request for

[jira] [Commented] (SPARK-23443) Spark with Glue as external catalog

[ https://issues.apache.org/jira/browse/SPARK-23443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17318647#comment-17318647 ] Abhinav Kumar commented on SPARK-23443: --- [~e.klerks] were you able to work on it?

[jira] [Updated] (SPARK-34993) from_json() acts differently on created and literal strings with backslashes

[ https://issues.apache.org/jira/browse/SPARK-34993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lauri Koobas updated SPARK-34993: - Description: JSON string with the value that contains backslashes fails to be recovered by `fro

[jira] [Commented] (SPARK-34993) from_json() acts differently on created and literal strings with backslashes

[ https://issues.apache.org/jira/browse/SPARK-34993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17318644#comment-17318644 ] Lauri Koobas commented on SPARK-34993: -- Apologies, added a screenshot. Didn't reali

[jira] [Updated] (SPARK-34993) from_json() acts differently on created and literal strings with backslashes

[ https://issues.apache.org/jira/browse/SPARK-34993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lauri Koobas updated SPARK-34993: - Attachment: image-2021-04-11-07-21-02-750.png > from_json() acts differently on created and lite

[jira] [Commented] (SPARK-34993) from_json() acts differently on created and literal strings with backslashes

[ https://issues.apache.org/jira/browse/SPARK-34993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17318642#comment-17318642 ] Hyukjin Kwon commented on SPARK-34993: -- It would be great to share the current outp

[jira] [Commented] (SPARK-34992) Index Appending when Duplicate Column names

[ https://issues.apache.org/jira/browse/SPARK-34992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17318640#comment-17318640 ] Hyukjin Kwon commented on SPARK-34992: -- We could make it configurable. That referre

[jira] [Commented] (SPARK-34985) Different execution plans under jdbc and hdfs

[ https://issues.apache.org/jira/browse/SPARK-34985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17318639#comment-17318639 ] Hyukjin Kwon commented on SPARK-34985: -- [~ljzh], it sounds more like a question the

[jira] [Commented] (SPARK-34979) Failed to install pyspark[sql] (due to pyarrow error) on aarch64

[ https://issues.apache.org/jira/browse/SPARK-34979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17318638#comment-17318638 ] Hyukjin Kwon commented on SPARK-34979: -- We could document. I think changing minimum

[jira] [Commented] (SPARK-34978) Support time-travel SQL syntax

[ https://issues.apache.org/jira/browse/SPARK-34978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17318637#comment-17318637 ] Hyukjin Kwon commented on SPARK-34978: -- Why don't you use Delta? It's open source u

[jira] [Resolved] (SPARK-34967) Regression in spark 3.1.1 for window function and struct binding resolution

[ https://issues.apache.org/jira/browse/SPARK-34967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-34967. -- Resolution: Cannot Reproduce > Regression in spark 3.1.1 for window function and struct bindin

[jira] [Commented] (SPARK-34967) Regression in spark 3.1.1 for window function and struct binding resolution

[ https://issues.apache.org/jira/browse/SPARK-34967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17318636#comment-17318636 ] Hyukjin Kwon commented on SPARK-34967: -- I can't reproduce this in the current maste

[jira] [Updated] (SPARK-34967) Regression in spark 3.1.1 for window function and struct binding resolution

[ https://issues.apache.org/jira/browse/SPARK-34967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-34967: - Description: https://github.com/zulk666/SparkTestProject/blob/main/src/main/scala/FailStructWith

[jira] [Resolved] (SPARK-34964) Initial job has not accepted any resources

[ https://issues.apache.org/jira/browse/SPARK-34964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-34964. -- Resolution: Invalid Most likely network issue or configuration issues. For questions, it's bes

[jira] [Commented] (SPARK-34957) pyspark 2.4.7 in conda

[ https://issues.apache.org/jira/browse/SPARK-34957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17318634#comment-17318634 ] Hyukjin Kwon commented on SPARK-34957: -- That't not being maintained by Apache Spark

[jira] [Commented] (SPARK-34928) CTE Execution fails for Sql Server

[ https://issues.apache.org/jira/browse/SPARK-34928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17318633#comment-17318633 ] Hyukjin Kwon commented on SPARK-34928: -- It sounds most likely the issue from the dr

[jira] [Resolved] (SPARK-35008) Spark hangs after on doing multiple unions

[ https://issues.apache.org/jira/browse/SPARK-35008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-35008. -- Resolution: Invalid > Spark hangs after on doing multiple unions > ---

[jira] [Commented] (SPARK-35008) Spark hangs after on doing multiple unions

[ https://issues.apache.org/jira/browse/SPARK-35008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17318630#comment-17318630 ] Hyukjin Kwon commented on SPARK-35008: -- The reason is most likely the plans being g

[jira] [Commented] (SPARK-35008) Spark hangs after on doing multiple unions

[ https://issues.apache.org/jira/browse/SPARK-35008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17318629#comment-17318629 ] Hyukjin Kwon commented on SPARK-35008: -- Let's interact with mailing list for questi

[jira] [Resolved] (SPARK-35010) nestedSchemaPruning causes issue when reading hive generated Orc files

[ https://issues.apache.org/jira/browse/SPARK-35010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-35010. -- Resolution: Duplicate > nestedSchemaPruning causes issue when reading hive generated Orc files

[jira] [Commented] (SPARK-35013) Spark allows to set spark.driver.cores=0

[ https://issues.apache.org/jira/browse/SPARK-35013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17318628#comment-17318628 ] Hyukjin Kwon commented on SPARK-35013: -- {{spark.driver.cores}} might be allowed for

[jira] [Assigned] (SPARK-35011) Avoid Block Manager registerations when StopExecutor msg is in-flight.

[ https://issues.apache.org/jira/browse/SPARK-35011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35011: Assignee: Apache Spark > Avoid Block Manager registerations when StopExecutor msg is in-f

[jira] [Commented] (SPARK-35011) Avoid Block Manager registerations when StopExecutor msg is in-flight.

[ https://issues.apache.org/jira/browse/SPARK-35011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17318624#comment-17318624 ] Apache Spark commented on SPARK-35011: -- User 'sumeetgajjar' has created a pull requ

[jira] [Assigned] (SPARK-35011) Avoid Block Manager registerations when StopExecutor msg is in-flight.

[ https://issues.apache.org/jira/browse/SPARK-35011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35011: Assignee: (was: Apache Spark) > Avoid Block Manager registerations when StopExecutor

[jira] [Commented] (SPARK-35011) Avoid Block Manager registerations when StopExecutor msg is in-flight.

[ https://issues.apache.org/jira/browse/SPARK-35011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17318623#comment-17318623 ] Apache Spark commented on SPARK-35011: -- User 'sumeetgajjar' has created a pull requ

[jira] [Resolved] (SPARK-35007) Spark 2.4.x version does not support numeric

[ https://issues.apache.org/jira/browse/SPARK-35007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-35007. - Resolution: Won't Fix > Spark 2.4.x version does not support numeric > -

[jira] [Updated] (SPARK-35007) Spark 2.4.x version does not support numeric

[ https://issues.apache.org/jira/browse/SPARK-35007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-35007: Target Version/s: (was: 2.4.8) > Spark 2.4.x version does not support numeric >

[jira] [Created] (SPARK-35021) Group exception messages in connector/catalog

Allison Wang created SPARK-35021: Summary: Group exception messages in connector/catalog Key: SPARK-35021 URL: https://issues.apache.org/jira/browse/SPARK-35021 Project: Spark Issue Type: Sub

[jira] [Updated] (SPARK-35020) Group exception messages in catalyst/util

[ https://issues.apache.org/jira/browse/SPARK-35020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allison Wang updated SPARK-35020: - Description: 'sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util' || Filename

[jira] [Created] (SPARK-35020) Group exception messages in catalyst/util

Allison Wang created SPARK-35020: Summary: Group exception messages in catalyst/util Key: SPARK-35020 URL: https://issues.apache.org/jira/browse/SPARK-35020 Project: Spark Issue Type: Sub-tas

[jira] [Created] (SPARK-35019) Improve type hints on pyspark.sql.*

Yikun Jiang created SPARK-35019: --- Summary: Improve type hints on pyspark.sql.* Key: SPARK-35019 URL: https://issues.apache.org/jira/browse/SPARK-35019 Project: Spark Issue Type: Sub-task

[jira] [Assigned] (SPARK-35017) Transfer ANSI intervals via Hive Thrift server

[ https://issues.apache.org/jira/browse/SPARK-35017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35017: Assignee: Apache Spark (was: Max Gekk) > Transfer ANSI intervals via Hive Thrift server

[jira] [Assigned] (SPARK-35017) Transfer ANSI intervals via Hive Thrift server

[ https://issues.apache.org/jira/browse/SPARK-35017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35017: Assignee: Max Gekk (was: Apache Spark) > Transfer ANSI intervals via Hive Thrift server

[jira] [Commented] (SPARK-35017) Transfer ANSI intervals via Hive Thrift server

[ https://issues.apache.org/jira/browse/SPARK-35017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17318468#comment-17318468 ] Apache Spark commented on SPARK-35017: -- User 'MaxGekk' has created a pull request f

[jira] [Created] (SPARK-35018) Test transferring year-month interval via Hive Thrift server

Max Gekk created SPARK-35018: Summary: Test transferring year-month interval via Hive Thrift server Key: SPARK-35018 URL: https://issues.apache.org/jira/browse/SPARK-35018 Project: Spark Issue T

[jira] [Updated] (SPARK-35017) Transfer ANSI intervals via Hive Thrift server

[ https://issues.apache.org/jira/browse/SPARK-35017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk updated SPARK-35017: - Summary: Transfer ANSI intervals via Hive Thrift server (was: Transfer ANSI interval via Hive Thrift se

[jira] [Updated] (SPARK-34963) Nested column pruning fails to extract case-insensitive struct field from array

[ https://issues.apache.org/jira/browse/SPARK-34963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh updated SPARK-34963: Fix Version/s: 2.4.8 > Nested column pruning fails to extract case-insensitive struct field from

[jira] [Created] (SPARK-35017) Transfer ANSI interval via Hive Thrift server

Max Gekk created SPARK-35017: Summary: Transfer ANSI interval via Hive Thrift server Key: SPARK-35017 URL: https://issues.apache.org/jira/browse/SPARK-35017 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-35016) Format ANSI intervals in Hive style

[ https://issues.apache.org/jira/browse/SPARK-35016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17318435#comment-17318435 ] Apache Spark commented on SPARK-35016: -- User 'MaxGekk' has created a pull request f

[jira] [Assigned] (SPARK-35016) Format ANSI intervals in Hive style

[ https://issues.apache.org/jira/browse/SPARK-35016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35016: Assignee: Max Gekk (was: Apache Spark) > Format ANSI intervals in Hive style > -

[jira] [Assigned] (SPARK-35016) Format ANSI intervals in Hive style

[ https://issues.apache.org/jira/browse/SPARK-35016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35016: Assignee: Apache Spark (was: Max Gekk) > Format ANSI intervals in Hive style > -

[jira] [Updated] (SPARK-35016) Format ANSI intervals in Hive style

[ https://issues.apache.org/jira/browse/SPARK-35016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk updated SPARK-35016: - Description: Hive: {code:java} 0: jdbc:hive2://localhost:1/default> select timestamp'2021-01-01 01

[jira] [Updated] (SPARK-35016) Format ANSI intervals in Hive style

[ https://issues.apache.org/jira/browse/SPARK-35016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk updated SPARK-35016: - Description: Hive: {code:java} 0: jdbc:hive2://localhost:1/default> select timestamp'2021-01-01 01

[jira] [Created] (SPARK-35016) Format ANSI intervals in Hive style

Max Gekk created SPARK-35016: Summary: Format ANSI intervals in Hive style Key: SPARK-35016 URL: https://issues.apache.org/jira/browse/SPARK-35016 Project: Spark Issue Type: Sub-task Co