HyukjinKwon commented on a change in pull request #35410:
URL: https://github.com/apache/spark/pull/35410#discussion_r810431358
##########
File path: python/pyspark/sql/dataframe.py
##########
@@ -116,13 +140,45 @@ def __init__(self, jdf: JavaObject, sql_ctx:
"SQLContext"):
# by __repr__ and _repr_html_ while eager evaluation opened.
self._support_repr_html = False
+ @property
+ def sql_ctx(self) -> "SQLContext":
+ from pyspark.sql.context import SQLContext
+
+ warnings.warn(
+ "DataFrame.sql_ctx is an internal property, and will be removed "
+ "in future releases. Use DataFrame.sparkSession instead."
+ )
+ if self._sql_ctx is None:
+ self._sql_ctx = SQLContext._get_or_create(self._sc)
+ return self._sql_ctx
+
+ @property # type: ignore[misc]
+ def sparkSession(self) -> "SparkSession":
+ """Returns Spark session that created this :class:`DataFrame`.
+
+ .. versionadded:: 3.3.0
+
+ Examples
+ --------
+ >>> df = spark.range(1)
+ >>> type(df.sparkSession)
+ <class 'pyspark.sql.session.SparkSession'>
+ """
+ from pyspark.sql.session import SparkSession
+
+ if self._session is None:
+ self._session = SparkSession._getActiveSessionOrCreate()
Review comment:
Yes, let me make a quick followup after double checking it (see
https://github.com/apache/spark/pull/35575).
For a bit of clarification, this code path will be triggered when thirdparty
libraries or external users directly create `DataFrame(..., SQLContext)` so
standard public APIs won't be affected. So the only case it gets affected is
when a user manually creates a `DataFrame(..., SQLContext)`, and they call this
new API `df.sparkSession`, which is It's pretty much less likely and rather
minor especially given that 1. I don't think `SQLContext` users would care
about `SparkSession`, 2. `SQLContext` is deprecated many years ago, 3.
DataFrame(...) constructor is subjected to be internal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
