amaliujia commented on code in PR #38460:
URL: https://github.com/apache/spark/pull/38460#discussion_r1010087722
##########
python/pyspark/sql/tests/connect/test_connect_plan_only.py:
##########
@@ -118,6 +118,15 @@ def test_relation_alias(self):
plan = df.alias("table_alias")._plan.to_proto(self.connect)
self.assertEqual(plan.root.common.alias, "table_alias")
+ def test_range(self):
+ plan = self.connect.range(start=10, end=20, step=3,
num_partitions=4)._plan.to_proto(
+ self.connect
+ )
+ self.assertEqual(plan.root.range.start, 10)
+ self.assertEqual(plan.root.range.end, 20)
+ self.assertEqual(plan.root.range.step.step, 3)
+ self.assertEqual(plan.root.range.num_partitions.num_partitions, 4)
+
Review Comment:
I added this test case but only test `step` and `num_partitions` is not set.
Right now there is a division between client and server such that:
1. Client take care of required fields, meaning that clients need to make
sure the required fields are set.
2. Server side take care of default values for optional fields.
This is to reduce load for both sides of implementation:
1. clients do not need to worry about default values for optional fields
unless the default value is exposed on the DataFrame API already.
2. Server side do not care for whether required field is set (clients
enforce it) but server side tracks the default value for optional fields. This
can also avoid that clients side to set different default value. The default
values are documented in proto:
https://github.com/apache/spark/blob/fb64041cea3e094b0807cb580deacc721b302408/connector/connect/src/main/protobuf/spark/connect/relations.proto#L232
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
