HyukjinKwon closed pull request #45232: [SPARK-46812][CONNECT][PYTHON] Make
mapInPandas / mapInArrow support ResourceProfile
URL: https://github.com/apache/spark/pull/45232
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use
HyukjinKwon commented on PR #45232:
URL: https://github.com/apache/spark/pull/45232#issuecomment-2030829152
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific
wbo4958 commented on PR #45232:
URL: https://github.com/apache/spark/pull/45232#issuecomment-2030792790
Hi @HyukjinKwon, Could you help merge it? Thx
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go
wbo4958 commented on PR #45232:
URL: https://github.com/apache/spark/pull/45232#issuecomment-2030753074
> Please make sure that the follow work does not get lost.
Sure, I will get it done.
--
This is an automated message from the Apache Git Service.
To respond to the message,
wbo4958 commented on PR #45232:
URL: https://github.com/apache/spark/pull/45232#issuecomment-2029532177
> In contrast to the regular spark API this implementation doesn't manage
the lifecycle of of the resource request. Can you create a follow up Jira that
removes the resource request from
wbo4958 commented on code in PR #45232:
URL: https://github.com/apache/spark/pull/45232#discussion_r1545872296
##
connector/connect/common/src/main/protobuf/spark/connect/base.proto:
##
@@ -375,6 +375,9 @@ message ExecutePlanResponse {
// Response type informing if the
grundprinzip commented on code in PR #45232:
URL: https://github.com/apache/spark/pull/45232#discussion_r1540499756
##
connector/connect/common/src/main/protobuf/spark/connect/base.proto:
##
@@ -375,6 +375,9 @@ message ExecutePlanResponse {
// Response type informing if
wbo4958 commented on PR #45232:
URL: https://github.com/apache/spark/pull/45232#issuecomment-2009170881
Hi @grundprinzip, @HyukjinKwon, @zhengruifeng, This PR has been there for a
while, could you help review/merge it? Thx
--
This is an automated message from the Apache Git Service.
To
wbo4958 commented on PR #45232:
URL: https://github.com/apache/spark/pull/45232#issuecomment-1996052736
Hi @grundprinzip, I would be grateful if you could kindly take another look
at this PR, Thx.
--
This is an automated message from the Apache Git Service.
To respond to the message,
wbo4958 commented on PR #45232:
URL: https://github.com/apache/spark/pull/45232#issuecomment-1989639850
Hi @grundprinzip, Could you help review it again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above
wbo4958 commented on code in PR #45232:
URL: https://github.com/apache/spark/pull/45232#discussion_r1519241310
##
connector/connect/common/src/main/protobuf/spark/connect/relations.proto:
##
@@ -892,6 +893,9 @@ message MapPartitions {
// (Optional) Whether to use barrier
wbo4958 commented on code in PR #45232:
URL: https://github.com/apache/spark/pull/45232#discussion_r1519241310
##
connector/connect/common/src/main/protobuf/spark/connect/relations.proto:
##
@@ -892,6 +893,9 @@ message MapPartitions {
// (Optional) Whether to use barrier
wbo4958 commented on code in PR #45232:
URL: https://github.com/apache/spark/pull/45232#discussion_r1519235937
##
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectBuildResourceProfileHandler.scala:
##
@@ -0,0 +1,75 @@
+/*
+ * Licensed to
wbo4958 commented on code in PR #45232:
URL: https://github.com/apache/spark/pull/45232#discussion_r1519232917
##
connector/connect/common/src/main/protobuf/spark/connect/base.proto:
##
@@ -1011,5 +1039,7 @@ service SparkConnectService {
// FetchErrorDetails retrieves the
wbo4958 commented on code in PR #45232:
URL: https://github.com/apache/spark/pull/45232#discussion_r1519232536
##
connector/connect/common/src/main/protobuf/spark/connect/relations.proto:
##
@@ -892,6 +893,9 @@ message MapPartitions {
// (Optional) Whether to use barrier
grundprinzip commented on code in PR #45232:
URL: https://github.com/apache/spark/pull/45232#discussion_r1518491571
##
connector/connect/common/src/main/protobuf/spark/connect/relations.proto:
##
@@ -892,6 +893,9 @@ message MapPartitions {
// (Optional) Whether to use
tgravescs commented on PR #45232:
URL: https://github.com/apache/spark/pull/45232#issuecomment-1985925035
changes look fine to me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific
wbo4958 commented on code in PR #45232:
URL: https://github.com/apache/spark/pull/45232#discussion_r1517085186
##
python/pyspark/resource/tests/test_connect_resources.py:
##
@@ -0,0 +1,46 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor
wbo4958 commented on code in PR #45232:
URL: https://github.com/apache/spark/pull/45232#discussion_r1517085041
##
python/pyspark/resource/tests/test_connect_resources.py:
##
@@ -0,0 +1,46 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor
wbo4958 commented on code in PR #45232:
URL: https://github.com/apache/spark/pull/45232#discussion_r1517084819
##
python/pyspark/resource/profile.py:
##
@@ -114,14 +122,23 @@ def id(self) -> int:
int
A unique id of this :class:`ResourceProfile`
wbo4958 commented on code in PR #45232:
URL: https://github.com/apache/spark/pull/45232#discussion_r1517084721
##
python/pyspark/resource/profile.py:
##
@@ -114,14 +122,23 @@ def id(self) -> int:
int
A unique id of this :class:`ResourceProfile`
tgravescs commented on code in PR #45232:
URL: https://github.com/apache/spark/pull/45232#discussion_r1516375405
##
python/pyspark/resource/tests/test_connect_resources.py:
##
@@ -0,0 +1,46 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+#
wbo4958 commented on PR #45232:
URL: https://github.com/apache/spark/pull/45232#issuecomment-1982322729
> > Does this PR introduce any user-facing change?
> > Yes, Users can pass ResourceProfile to mapInPandas/mapInArrow through
the connect pysprark client.
>
> I think you are
wbo4958 commented on code in PR #45232:
URL: https://github.com/apache/spark/pull/45232#discussion_r1515490084
##
dev/sparktestsupport/modules.py:
##
@@ -554,6 +554,7 @@ def __hash__(self):
"pyspark.resource.profile",
# unittests
wbo4958 commented on code in PR #45232:
URL: https://github.com/apache/spark/pull/45232#discussion_r1515485716
##
python/pyspark/sql/connect/resource/profile.py:
##
@@ -0,0 +1,69 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license
zhengruifeng commented on code in PR #45232:
URL: https://github.com/apache/spark/pull/45232#discussion_r1515455169
##
python/pyspark/sql/connect/resource/profile.py:
##
@@ -0,0 +1,69 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor
zhengruifeng commented on code in PR #45232:
URL: https://github.com/apache/spark/pull/45232#discussion_r1515448887
##
dev/sparktestsupport/modules.py:
##
@@ -554,6 +554,7 @@ def __hash__(self):
"pyspark.resource.profile",
# unittests
tgravescs commented on PR #45232:
URL: https://github.com/apache/spark/pull/45232#issuecomment-1980935581
>Does this PR introduce any user-facing change?
> Yes, Users can pass ResourceProfile to mapInPandas/mapInArrow through the
connect pysprark client.
I think you are adding the
HyukjinKwon commented on code in PR #45232:
URL: https://github.com/apache/spark/pull/45232#discussion_r1514004559
##
python/pyspark/resource/profile.py:
##
@@ -114,14 +122,26 @@ def id(self) -> int:
int
A unique id of this :class:`ResourceProfile`
wbo4958 commented on PR #45232:
URL: https://github.com/apache/spark/pull/45232#issuecomment-1980292725
Hi @HyukjinKwon, Could you help review again, thx very much.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
wbo4958 commented on code in PR #45232:
URL: https://github.com/apache/spark/pull/45232#discussion_r1513993480
##
python/pyspark/sql/connect/resource/profile.py:
##
@@ -0,0 +1,69 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license
wbo4958 commented on code in PR #45232:
URL: https://github.com/apache/spark/pull/45232#discussion_r1513991688
##
python/pyspark/resource/profile.py:
##
@@ -114,14 +122,26 @@ def id(self) -> int:
int
A unique id of this :class:`ResourceProfile`
wbo4958 commented on code in PR #45232:
URL: https://github.com/apache/spark/pull/45232#discussion_r1513987011
##
python/pyspark/resource/profile.py:
##
@@ -114,14 +122,26 @@ def id(self) -> int:
int
A unique id of this :class:`ResourceProfile`
HyukjinKwon commented on PR #45232:
URL: https://github.com/apache/spark/pull/45232#issuecomment-1976100589
Looks fine in general
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific
HyukjinKwon commented on code in PR #45232:
URL: https://github.com/apache/spark/pull/45232#discussion_r1510840968
##
python/pyspark/resource/profile.py:
##
@@ -114,14 +122,26 @@ def id(self) -> int:
int
A unique id of this :class:`ResourceProfile`
HyukjinKwon commented on code in PR #45232:
URL: https://github.com/apache/spark/pull/45232#discussion_r1510824227
##
python/pyspark/sql/connect/resource/profile.py:
##
@@ -0,0 +1,69 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor
HyukjinKwon commented on code in PR #45232:
URL: https://github.com/apache/spark/pull/45232#discussion_r1510818206
##
python/pyspark/resource/profile.py:
##
@@ -114,14 +122,26 @@ def id(self) -> int:
int
A unique id of this :class:`ResourceProfile`
HyukjinKwon commented on code in PR #45232:
URL: https://github.com/apache/spark/pull/45232#discussion_r1510814720
##
connector/connect/common/src/main/protobuf/spark/connect/base.proto:
##
@@ -967,6 +967,34 @@ message FetchErrorDetailsResponse {
}
}
+message
wbo4958 commented on PR #45232:
URL: https://github.com/apache/spark/pull/45232#issuecomment-1970509831
Hi @tgravescs, This PR changed ResourceProfile a little bit to support
connect, Could you help review it? Thx very much.
--
This is an automated message from the Apache Git Service.
To
wbo4958 commented on PR #45232:
URL: https://github.com/apache/spark/pull/45232#issuecomment-1963270400
Hi @tgravescs @WeichenXu123 @zhengruifeng @Ngone51, Could you also please
help review it. Thx.
--
This is an automated message from the Apache Git Service.
To respond to the message,
wbo4958 commented on code in PR #45232:
URL: https://github.com/apache/spark/pull/45232#discussion_r1502014261
##
python/pyspark/resource/profile.py:
##
@@ -99,6 +99,11 @@ def __init__(
_exec_req: Optional[Dict[str, ExecutorResourceRequest]] = None,
Review Comment:
wbo4958 commented on PR #45232:
URL: https://github.com/apache/spark/pull/45232#issuecomment-1963263080
## With dynamic allocation enabled.
``` bash
start-connect-server.sh --master spark://192.168.0.106:7077 \
--jars jars/spark-connect_2.13-4.0.0-SNAPSHOT.jar \
wbo4958 commented on PR #45232:
URL: https://github.com/apache/spark/pull/45232#issuecomment-1963261901
# Manual tests
The manual tests were conducted on a spark Standalone cluster with only 1
worker which has 6 cpu cores.
## With dynamic allocation disabled.
```
HyukjinKwon commented on code in PR #45232:
URL: https://github.com/apache/spark/pull/45232#discussion_r1501943441
##
python/pyspark/resource/profile.py:
##
@@ -99,6 +99,11 @@ def __init__(
_exec_req: Optional[Dict[str, ExecutorResourceRequest]] = None,
Review
wbo4958 opened a new pull request, #45232:
URL: https://github.com/apache/spark/pull/45232
### What changes were proposed in this pull request?
Support stage-level scheduling for PySpark connect DataFrame APIs
(mapInPandas and mapInArrow).
### Why are the changes needed?
45 matches
Mail list logo
