hvanhovell closed pull request #45701: [SPARK-47545][CONNECT] Dataset `observe`
support for the Scala client
URL: https://github.com/apache/spark/pull/45701
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above
hvanhovell commented on PR #45701:
URL: https://github.com/apache/spark/pull/45701#issuecomment-2101304531
Merging!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To
xupefei commented on PR #45701:
URL: https://github.com/apache/spark/pull/45701#issuecomment-2088626203
> > @xupefei there is a genuine test failure. Can you check what is going on?
>
> It seems the test is flaky, even after the previous attempt to fix it:
#45173
I re-ran the
xupefei commented on PR #45701:
URL: https://github.com/apache/spark/pull/45701#issuecomment-2088228423
> @xupefei there is a genuine test failure. Can you check what is going on?
It seems the test is flaky, even after the previous attempt to fix it:
hvanhovell commented on PR #45701:
URL: https://github.com/apache/spark/pull/45701#issuecomment-2085537023
@xupefei there is a genuine test failure. Can you check what is going on?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to
hvanhovell commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1583392160
##
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/SparkResult.scala:
##
@@ -198,6 +206,29 @@ private[sql] class SparkResult[T](
hvanhovell commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1583350392
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/SparkSession.scala:
##
@@ -813,6 +823,23 @@ class SparkSession private[sql] (
* Set to false
hvanhovell commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1583349498
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/SparkSession.scala:
##
@@ -813,6 +823,23 @@ class SparkSession private[sql] (
* Set to false
xupefei commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1572265083
##
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/SparkResult.scala:
##
@@ -27,18 +27,22 @@ import
xupefei commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1572264833
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/SparkSession.scala:
##
@@ -813,6 +823,28 @@ class SparkSession private[sql] (
* Set to false to
xupefei commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1572263903
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/SparkSession.scala:
##
@@ -813,6 +823,28 @@ class SparkSession private[sql] (
* Set to false to
xupefei commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1572148735
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/SparkSession.scala:
##
@@ -813,6 +823,28 @@ class SparkSession private[sql] (
* Set to false to
xupefei commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1572148735
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/SparkSession.scala:
##
@@ -813,6 +823,28 @@ class SparkSession private[sql] (
* Set to false to
xupefei commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1572146508
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/SparkSession.scala:
##
@@ -813,6 +823,28 @@ class SparkSession private[sql] (
* Set to false to
hvanhovell commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1571122260
##
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/SparkResult.scala:
##
@@ -27,18 +27,22 @@ import
hvanhovell commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1571108161
##
connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/ClientE2ETestSuite.scala:
##
@@ -1511,6 +1514,46 @@ class ClientE2ETestSuite extends
hvanhovell commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1571105638
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/SparkSession.scala:
##
@@ -813,6 +823,28 @@ class SparkSession private[sql] (
* Set to false
hvanhovell commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1571102181
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/SparkSession.scala:
##
@@ -813,6 +823,28 @@ class SparkSession private[sql] (
* Set to false
xupefei commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1570681872
##
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/SparkResult.scala:
##
@@ -198,6 +206,29 @@ private[sql] class SparkResult[T](
hvanhovell commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1569265956
##
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/SparkResult.scala:
##
@@ -198,6 +206,29 @@ private[sql] class SparkResult[T](
xupefei commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1567676904
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/Observation.scala:
##
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF)
xupefei commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1567644036
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/Dataset.scala:
##
@@ -3397,7 +3488,11 @@ class Dataset[T] private[sql] (
xupefei commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1567173975
##
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/SparkResult.scala:
##
@@ -27,18 +27,21 @@ import
hvanhovell commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1566210772
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/Dataset.scala:
##
@@ -3397,7 +3488,11 @@ class Dataset[T] private[sql] (
hvanhovell commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1566170352
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/Observation.scala:
##
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF)
hvanhovell commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1566171078
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/Observation.scala:
##
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF)
hvanhovell commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1566169201
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/Observation.scala:
##
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF)
hvanhovell commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1566164128
##
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/SparkResult.scala:
##
@@ -79,6 +82,7 @@ private[sql] class SparkResult[T](
hvanhovell commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1566163349
##
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/SparkResult.scala:
##
@@ -27,18 +27,21 @@ import
hvanhovell commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1566146921
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/Observation.scala:
##
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF)
xupefei commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1561082565
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/Dataset.scala:
##
@@ -131,13 +131,25 @@ import org.apache.spark.util.SparkClassUtils
class
xupefei commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1559266569
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/Dataset.scala:
##
@@ -3397,7 +3488,11 @@ class Dataset[T] private[sql] (
xupefei commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1559262517
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/Observation.scala:
##
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF)
xupefei commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1559250767
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/Observation.scala:
##
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF)
xupefei commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1559250767
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/Observation.scala:
##
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF)
xupefei commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1559249905
##
connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/connect/client/CheckConnectJvmClientCompatibility.scala:
##
@@ -363,6 +363,8 @@ object
hvanhovell commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1558066618
##
connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/connect/client/CheckConnectJvmClientCompatibility.scala:
##
@@ -363,6 +363,8 @@ object
hvanhovell commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1557936456
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/Observation.scala:
##
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF)
hvanhovell commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1557935463
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/Dataset.scala:
##
@@ -3397,7 +3488,11 @@ class Dataset[T] private[sql] (
hvanhovell commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1557934162
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/Dataset.scala:
##
@@ -131,13 +131,25 @@ import org.apache.spark.util.SparkClassUtils
class
xupefei commented on PR #45701:
URL: https://github.com/apache/spark/pull/45701#issuecomment-2031447755
> So `df.collectObservations()` seems to be a new API available only in
Spark Connect Scala client?
Yes, similar to `df.attrs["observed_metrics"]` which is only in the Python
xupefei commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1547415263
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/Dataset.scala:
##
@@ -3338,7 +3358,25 @@ class Dataset[T] private[sql] (
}
def
ueshin commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1543431051
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/Dataset.scala:
##
@@ -3338,7 +3358,25 @@ class Dataset[T] private[sql] (
}
def observe(name:
ueshin commented on PR #45701:
URL: https://github.com/apache/spark/pull/45701#issuecomment-2025839011
So `df.collectObservations()` seems to be a new API available only in Spark
Connect Scala client?
--
This is an automated message from the Apache Git Service.
To respond to the message,
44 matches
Mail list logo
