Github user superbobry commented on the issue:
https://github.com/apache/spark/pull/23008
Interestingly, `cloudpickle` adds overhead even if the namedtuple is
importable:
```bash
$ cat a.py
from collections import namedtuple
A = namedtuple("A", ["foo", "bar"])
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/23008
BTW, let.s test them in end-to-end. For instance,
`spark.range(1).rdd.map(lambda blabla).count()`
---
-
To unsubscribe,
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/23008
If the perf diff is big, let's don't change but document that we can use
`CloudPickleSerializer()` to avoid breaking change.
If the perf diff is rather trivial, let's check if we can
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/23008
Nope, it should be manually done.. should be great to have it FWIW.
I am not yet sure how we're going to measure the performance. I think you
can show the performance diff for
Github user superbobry commented on the issue:
https://github.com/apache/spark/pull/23008
Is there a benchmark suite for PySpark?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23008
**[Test build #98710 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98710/testReport)**
for PR 23008 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23008
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98710/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23008
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23008
**[Test build #98710 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98710/testReport)**
for PR 23008 at commit
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/23008
ok to test
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23008
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23008
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23008
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
13 matches
Mail list logo
