[
https://issues.apache.org/jira/browse/SPARK-46820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17928545#comment-17928545
]
Vinicius Errero commented on SPARK-46820:
-----------------------------------------
Hi [~gurwls223],
Thank you for working on this logging fix. I noticed this fix is scheduled for
Spark 4.0, and I wanted to understand the reasoning behind not backporting it
to Spark 3.5.x, given that:
# The logging issue affects current production deployments running Spark 3.5.x
# This appears to be a bugfix rather than a new feature
# Logging improvements typically have minimal risk of introducing regressions
Could you please help explain the technical constraints or compatibility
concerns that prevent this from being considered for a 3.5.x patch release?
This would help our team better plan our upgrade timeline and potential
workarounds.
We currently use Spark 3.5.3 at our clusters and this logging fix would improve
readability for our users and we won't be able to upgrade to Spark 4.0 for a
long time
Thank you!
> Fix error message regression by restoring new_msg
> -------------------------------------------------
>
> Key: SPARK-46820
> URL: https://issues.apache.org/jira/browse/SPARK-46820
> Project: Spark
> Issue Type: Sub-task
> Components: PySpark
> Affects Versions: 4.0.0
> Reporter: Haejoon Lee
> Assignee: Haejoon Lee
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> >>> from pyspark.sql.types import StructType, StructField, StringType,
> >>> IntegerType
> >>> schema = StructType([
> ... StructField("name", StringType(), nullable=True),
> ... StructField("age", IntegerType(), nullable=False)
> ... ])
> >>> df = spark.createDataFrame([("asd", None])], schema)
> pyspark.errors.exceptions.base.PySparkValueError: [CANNOT_BE_NONE] Argument
> `obj` cannot be None.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
