HyukjinKwon commented on a change in pull request #33484:
URL: https://github.com/apache/spark/pull/33484#discussion_r675462634
##########
File path: python/pyspark/sql/dataframe.py
##########
@@ -1830,6 +1830,40 @@ def agg(self, *exprs):
"""
return self.groupBy().agg(*exprs)
+ @since(3.3)
+ def observe(self, observation, *exprs):
+ """Observe (named) metrics through an :class:`Observation` instance.
+
+ This method does not support streaming datasets.
+
+ .. versionadded:: 3.3.0
+
+ A user can retrieve the metrics by accessing `Observation.get`.
+
+ Example:
+ >>> from pyspark.sql.functions import col, count, lit, max
+ >>> from pyspark.sql.observation import Observation
+ >>> observation = Observation("my_metrics")
+ >>> observed_df = df.observe(observation, count(lit(1)),
max(col("age")))
+ >>> observed_df.count()
+ >>> metrics = observation.get
+
+ parameters
Review comment:
```suggestion
Parameters
```
BTW, there's a certain order of these Parameters, Examples, etc. to follow.
##########
File path: python/pyspark/sql/dataframe.py
##########
@@ -1830,6 +1830,40 @@ def agg(self, *exprs):
"""
return self.groupBy().agg(*exprs)
+ @since(3.3)
+ def observe(self, observation, *exprs):
+ """Observe (named) metrics through an :class:`Observation` instance.
+
+ This method does not support streaming datasets.
+
+ .. versionadded:: 3.3.0
+
+ A user can retrieve the metrics by accessing `Observation.get`.
+
+ Example:
+ >>> from pyspark.sql.functions import col, count, lit, max
+ >>> from pyspark.sql.observation import Observation
+ >>> observation = Observation("my_metrics")
+ >>> observed_df = df.observe(observation, count(lit(1)),
max(col("age")))
+ >>> observed_df.count()
+ >>> metrics = observation.get
+
+ parameters
+ ----------
+ observation : :class:`Observation`
+ an :class:`Observation` instance to obtain the metric.
+ exprs : list of :class:`Column`
+ column expressions (:class:`Column`).
+
+ returns
Review comment:
```suggestion
Returns
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
