itholic opened a new pull request, #49054: URL: https://github.com/apache/spark/pull/49054
### What changes were proposed in this pull request? This PR followups https://github.com/apache/spark/pull/48964 to fix `with_origin_to_class` to be called when the `Column` actually initializing ### Why are the changes needed? We don't need call `with_origin_to_class` if the Column API is not used. Furthermore, the current way is not working properly from Spark Connect as it causes circular import error as mentioned from https://github.com/apache/spark/pull/48964#discussion_r1868569335 so this is more safer way to support `with_origin_to_class`. ### Does this PR introduce _any_ user-facing change? No API changes, but the Spark Connect Python client no longer issues circular import error when initializing. ### How was this patch tested? Manually tested **Before (Failed initializing Spark Connect Python client)** ``` % ./bin/pyspark --remote local Python 3.9.17 (main, Jul 5 2023, 15:35:09) [Clang 14.0.6 ] :: Anaconda, Inc. on darwin Type "help", "copyright", "credits" or "license" for more information. ... ImportError: cannot import name 'SparkSession' from partially initialized module 'pyspark.sql.connect.session' (most likely due to a circular import) (/Users/haejoon.lee/Desktop/git_repos/spark/python/pyspark/sql/connect/session.py) ``` **After (Successfully initializing Spark Connect Python client)** ``` % ./bin/pyspark --remote local Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /__ / .__/\_,_/_/ /_/\_\ version 4.0.0.dev0 /_/ Using Python version 3.9.17 (main, Jul 5 2023 15:35:09) Client connected to the Spark Connect server at localhost SparkSession available as 'spark'. >>> spark <pyspark.sql.connect.session.SparkSession object at 0x105c6e850> ``` ### Was this patch authored or co-authored using generative AI tooling? <!-- If generative AI tooling has been used in the process of authoring this patch, please include the phrase: 'Generated-by: ' followed by the name of the tool and its version. If no, write 'No'. Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details. --> No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
