[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user gberger commented on the issue: https://github.com/apache/spark/pull/19792 Great! Thanks all --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/19792 Thanks! merging to master/2.3. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19792 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19792 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85784/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19792 **[Test build #85784 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85784/testReport)** for PR 19792 at commit [`6d171dd`](https://github.com/apache/spark/commit/6d171dda179ecdbe95dbc959c961397e08b8b537). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/19792 LGTM, pending Jenkins. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/19792 @HyukjinKwon Thanks, I'll take another look soon. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19792 To be clear (as I was reviewing this too), I am okay with going ahead, @ueshin if this looks good to you. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19792 **[Test build #85784 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85784/testReport)** for PR 19792 at commit [`6d171dd`](https://github.com/apache/spark/commit/6d171dda179ecdbe95dbc959c961397e08b8b537). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/19792 Jenkins, retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19792 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19792 **[Test build #85752 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85752/testReport)** for PR 19792 at commit [`6d171dd`](https://github.com/apache/spark/commit/6d171dda179ecdbe95dbc959c961397e08b8b537). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19792 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85752/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19792 **[Test build #85752 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85752/testReport)** for PR 19792 at commit [`6d171dd`](https://github.com/apache/spark/commit/6d171dda179ecdbe95dbc959c961397e08b8b537). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19792 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19792 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19792 **[Test build #84932 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84932/testReport)** for PR 19792 at commit [`6d171dd`](https://github.com/apache/spark/commit/6d171dda179ecdbe95dbc959c961397e08b8b537). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19792 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84932/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19792 **[Test build #84932 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84932/testReport)** for PR 19792 at commit [`6d171dd`](https://github.com/apache/spark/commit/6d171dda179ecdbe95dbc959c961397e08b8b537). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19792 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19792 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user gberger commented on the issue: https://github.com/apache/spark/pull/19792 @HyukjinKwon no worries, I understand. We gotta be 100% thorough here. Thanks for the help --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19792 Will take another look soon tomorrow. Sorry that is getting delayed again and again but I just realised this code path is a little bit tricky .. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19792 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19792 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84918/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19792 **[Test build #84918 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84918/testReport)** for PR 19792 at commit [`6d171dd`](https://github.com/apache/spark/commit/6d171dda179ecdbe95dbc959c961397e08b8b537). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19792 **[Test build #84918 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84918/testReport)** for PR 19792 at commit [`6d171dd`](https://github.com/apache/spark/commit/6d171dda179ecdbe95dbc959c961397e08b8b537). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19792 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84764/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19792 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19792 **[Test build #84764 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84764/testReport)** for PR 19792 at commit [`404fdbb`](https://github.com/apache/spark/commit/404fdbb1b5b265c6f1f651f01f42eb62d598b788). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19792 **[Test build #84764 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84764/testReport)** for PR 19792 at commit [`404fdbb`](https://github.com/apache/spark/commit/404fdbb1b5b265c6f1f651f01f42eb62d598b788). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19792 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84708/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19792 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19792 **[Test build #84708 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84708/testReport)** for PR 19792 at commit [`44a1879`](https://github.com/apache/spark/commit/44a1879919a61d732eea176e26ce6a79549984a0). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user gberger commented on the issue: https://github.com/apache/spark/pull/19792 Good catch @HyukjinKwon! I reverted those changes and added a test to cover this regression. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19792 **[Test build #84708 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84708/testReport)** for PR 19792 at commit [`44a1879`](https://github.com/apache/spark/commit/44a1879919a61d732eea176e26ce6a79549984a0). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19792 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84558/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19792 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19792 **[Test build #84558 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84558/testReport)** for PR 19792 at commit [`5131db2`](https://github.com/apache/spark/commit/5131db23bd48b3606c7c823cad9d5d376bae0d00). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19792 **[Test build #84558 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84558/testReport)** for PR 19792 at commit [`5131db2`](https://github.com/apache/spark/commit/5131db23bd48b3606c7c823cad9d5d376bae0d00). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19792 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84492/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19792 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19792 **[Test build #84492 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84492/testReport)** for PR 19792 at commit [`0103045`](https://github.com/apache/spark/commit/0103045f751e9f1c777673bdebd0632c4e781486). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19792 **[Test build #84492 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84492/testReport)** for PR 19792 at commit [`0103045`](https://github.com/apache/spark/commit/0103045f751e9f1c777673bdebd0632c4e781486). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19792 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19792 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84487/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19792 **[Test build #84487 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84487/testReport)** for PR 19792 at commit [`8dd5db6`](https://github.com/apache/spark/commit/8dd5db6f24314a50f7e647875d1c0d8be3a5a5f1). * This patch **fails Python style tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19792 **[Test build #84487 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84487/testReport)** for PR 19792 at commit [`8dd5db6`](https://github.com/apache/spark/commit/8dd5db6f24314a50f7e647875d1c0d8be3a5a5f1). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user gberger commented on the issue:
https://github.com/apache/spark/pull/19792
@HyukjinKwon done, with test added.
```
>>> spark.createDataFrame(spark.sparkContext.parallelize([[None, 1], ["a",
None], [1, 1]]), schema=["a", "b"], samplingRatio=0.99)
Traceback (most recent call last):
File "", line 1, in
File "/Users/gberger/Projects/spark/python/pyspark/sql/session.py", line
644, in createDataFrame
rdd, schema = self._createFromRDD(data.map(prepare), schema,
samplingRatio)
File "/Users/gberger/Projects/spark/python/pyspark/sql/session.py", line
383, in _createFromRDD
struct = self._inferSchema(rdd, samplingRatio, names=schema)
File "/Users/gberger/Projects/spark/python/pyspark/sql/session.py", line
375, in _inferSchema
schema = rdd.map(lambda row: _infer_schema(row,
names)).reduce(_merge_type)
File "/Users/gberger/Projects/spark/python/pyspark/rdd.py", line 852, in
reduce
return reduce(f, vals)
File "/Users/gberger/Projects/spark/python/pyspark/sql/types.py", line
1133, in _merge_type
for f in a.fields]
File "/Users/gberger/Projects/spark/python/pyspark/sql/types.py", line
1126, in _merge_type
raise TypeError(new_msg("Can not merge type %s and %s" % (type(a),
type(b
TypeError: field a: Can not merge type and
```
Also, with this last change, I could simplify the code in `_createFromRDD`.
Since I pass the field names down to `_inferSchema` (and to `_infer_schema`
from there), the inferred schema already comes with field names, so no need to
set them again in `_createFromRDD`. Tests for this still pass. Let me know if
you can think of any edge case not covered by tests that would break.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19792
Can we set the `names` for `_createFromRDD` -> `_inferSchema` too?:
```python
>>> spark.createDataFrame(spark.sparkContext.parallelize([[None, 1], ["a",
None], [1, 1]]), schema=["a", "b"], samplingRatio=0.99)
```
```
Traceback (most recent call last):
File "", line 1, in
File
"/Users/hyukjinkwon/Desktop/workspace/repos/forked/spark/python/pyspark/sql/session.py",
line 644, in createDataFrame
rdd, schema = self._createFromRDD(data.map(prepare), schema,
samplingRatio)
File
"/Users/hyukjinkwon/Desktop/workspace/repos/forked/spark/python/pyspark/sql/session.py",
line 383, in _createFromRDD
struct = self._inferSchema(rdd, samplingRatio)
File
"/Users/hyukjinkwon/Desktop/workspace/repos/forked/spark/python/pyspark/sql/session.py",
line 375, in _inferSchema
schema = rdd.map(_infer_schema).reduce(_merge_type)
File "/.../spark/python/pyspark/rdd.py", line 852, in reduce
return reduce(f, vals)
File "/.../spark/python/pyspark/sql/types.py", line 1133, in _merge_type
for f in a.fields]
File "/.../spark/python/pyspark/sql/types.py", line 1126, in _merge_type
raise TypeError(new_msg("Can not merge type %s and %s" % (type(a),
type(b
TypeError: field _1: Can not merge type and
```
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19792 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84429/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19792 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19792 **[Test build #84429 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84429/testReport)** for PR 19792 at commit [`41766fa`](https://github.com/apache/spark/commit/41766fa875b987fecf910b7fa8bd9429e27ce88e). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19792 **[Test build #84429 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84429/testReport)** for PR 19792 at commit [`41766fa`](https://github.com/apache/spark/commit/41766fa875b987fecf910b7fa8bd9429e27ce88e). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user gberger commented on the issue: https://github.com/apache/spark/pull/19792 Fixed --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19792 Oh, will double check too for sure shortly. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user gberger commented on the issue: https://github.com/apache/spark/pull/19792 Friendly ping -- I've fixed that @ueshin. Is there anything else I should look at to get this to be merged? /cc @HyukjinKwon --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19792 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19792 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84335/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19792 **[Test build #84335 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84335/testReport)** for PR 19792 at commit [`2240a42`](https://github.com/apache/spark/commit/2240a42847ed8155375a99c30a54d0749c655966). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19792 **[Test build #84335 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84335/testReport)** for PR 19792 at commit [`2240a42`](https://github.com/apache/spark/commit/2240a42847ed8155375a99c30a54d0749c655966). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19792 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84305/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19792 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19792 **[Test build #84305 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84305/testReport)** for PR 19792 at commit [`c603251`](https://github.com/apache/spark/commit/c6032518774022f026d33656787a6387d6f83b5c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19792 **[Test build #84305 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84305/testReport)** for PR 19792 at commit [`c603251`](https://github.com/apache/spark/commit/c6032518774022f026d33656787a6387d6f83b5c). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user gberger commented on the issue:
https://github.com/apache/spark/pull/19792
Hi all,
I have changed the error message to be like #18521. Here are some examples:
```
_merge_type(
StructType([StructField("f1", ArrayType(MapType(StringType(),
LongType(]),
StructType([StructField("f1", ArrayType(MapType(DoubleType(),
LongType(])
)
# TypeError: key of map element in array field f1: Can not merge type
and
_merge_type(
StructType([
StructField("f1", MapType(StringType(), LongType())),
StructField("f2", StringType())]),
StructType([
StructField("f1", MapType(StringType(), DoubleType())),
StructField("f2", StringType())]))
# TypeError: value of map field f1: Can not merge type and
_merge_type(
StructType([
StructField("f1", ArrayType(LongType())),
StructField("f2", StringType())]),
StructType([
StructField("f1", ArrayType(DoubleType())),
StructField("f2", StringType())]))
# TypeError: element in array field f1: Can not merge type and
_merge_type(
StructType([StructField("f1", LongType()), StructField("f2",
StringType())]),
StructType([StructField("f1", DoubleType()), StructField("f2",
StringType())]))
# TypeError: field f1: Can not merge type and
_merge_type(
StructType([StructField("f1", StructType([StructField("f2",
LongType())]))]),
StructType([StructField("f1", StructType([StructField("f2",
StringType())]))]))
# TypeError: field f2 in field f1: Can not merge type and
```
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19792 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84272/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19792 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19792 **[Test build #84272 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84272/testReport)** for PR 19792 at commit [`8665115`](https://github.com/apache/spark/commit/8665115a524b4b2c7f9e1cd1cb69f8038fb1c904). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19792 **[Test build #84272 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84272/testReport)** for PR 19792 at commit [`8665115`](https://github.com/apache/spark/commit/8665115a524b4b2c7f9e1cd1cb69f8038fb1c904). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19792 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84140/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19792 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19792 **[Test build #84140 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84140/testReport)** for PR 19792 at commit [`3346a6c`](https://github.com/apache/spark/commit/3346a6ca65cc46a4912d81fd9a5086a406e875c5). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19792 **[Test build #84140 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84140/testReport)** for PR 19792 at commit [`3346a6c`](https://github.com/apache/spark/commit/3346a6ca65cc46a4912d81fd9a5086a406e875c5). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user gberger commented on the issue:
https://github.com/apache/spark/pull/19792
Maybe a more performant way to do the path in the error message would be to
propagate it *up* the stack via try/catching the errors and adding the paths as
it goes.
But this way seems really weird to me...
```
elif isinstance(a, ArrayType):
try:
return ArrayType(_merge_type(a.elementType, b.elementType),
True)
catch TypeError as e:
raise TypeError("arrayElement." + str(e))
```
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19792 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19792 **[Test build #84139 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84139/testReport)** for PR 19792 at commit [`61ace28`](https://github.com/apache/spark/commit/61ace285bdbe6bff36ef5e0d8c8e10f9bc5a227b). * This patch **fails Python style tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19792 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84139/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user gberger commented on the issue:
https://github.com/apache/spark/pull/19792
Hey all,
## Error message
I revamped the error message and made it "recursive" similar to
@HyukjinKwon. Here's an example:
```
>>> _merge_type(
... StructType([StructField("f1",
ArrayType(MapType(StringType(), LongType(]),
... StructType([StructField("f1",
ArrayType(MapType(DoubleType(), LongType(])
... )
Traceback (most recent call last):
File "", line 3, in
File "/Users/gberger/Projects/spark/python/pyspark/sql/types.py", line
1129, in _merge_type
for f in a.fields]
File "/Users/gberger/Projects/spark/python/pyspark/sql/types.py", line
1129, in
for f in a.fields]
File "/Users/gberger/Projects/spark/python/pyspark/sql/types.py", line
1137, in _merge_type
return ArrayType(_merge_type(a.elementType, b.elementType,
field=field+'.arrayElement'), True)
File "/Users/gberger/Projects/spark/python/pyspark/sql/types.py", line
1140, in _merge_type
return MapType(_merge_type(a.keyType, b.keyType, field=field+'.mapKey'),
File "/Users/gberger/Projects/spark/python/pyspark/sql/types.py", line
1122, in _merge_type
raise TypeError("%s: Can not merge type %s and %s" % (field, type(a),
type(b)))
TypeError: .structField("f1").arrayElement.mapKey: Can not merge type
and
```
Happy to iterate on the exact formatting or wording of the path shown.
## Tests
I wrote a bunch of tests too, hope they are comprehensive enough but happy
to add more if not. @ueshin
## Benchmark
It seems that the time it takes for a nested _merge_type on my machine has
increased for ~2.75 microseconds to ~2.85 microseconds, around a 3% increase.
This can be attributed to the string concatenation that goes on every time
_merge_type goes one level down from a StructType, ArrayType or MapType. I'm
not sure if there's a better way to propagate this information down the stack,
maybe a tuple?
Code used:
```
from pyspark.sql.types import *
from pyspark.sql.types import _merge_type
import time
def test_f():
return _merge_type(
StructType([StructField("f1", ArrayType(MapType(StringType(),
LongType(]),
StructType([StructField("f1", ArrayType(MapType(StringType(),
LongType(])
)
def timing(f):
def wrap(*args):
time1 = time.time()
for __ in range(10):
ret = f(*args)
time2 = time.time()
print('took %0.3f ms' % ((time2-time1)*1000.0))
return ret
return wrap
for _ in range(10):
timing(test_f)()
```
Before:
> took 2701.337 ms
> took 2905.867 ms
> took 2725.119 ms
> took 2796.098 ms
> took 2718.981 ms
> took 2773.560 ms
> took 2717.995 ms
> took 2796.466 ms
> took 2716.173 ms
> took 2744.121 ms
After:
> took 2865.038 ms
> took 2836.403 ms
> took 2871.871 ms
> took 2827.625 ms
> took 2820.170 ms
> took 2873.976 ms
> took 2833.609 ms
> took 2909.599 ms
> took 3162.108 ms
> took 2940.864 ms
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19792 D'oh, you mean performance regression test. Manual tests should be fine. When you share some codes you ran, maybe we can double check. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19792 I think you can add a new test case somewhere around here - https://github.com/apache/spark/blob/master/python/pyspark/sql/tests.py#L1724 maybe dealing with some combinations of `StructType`, `MapType`, `ArrayType` and one of non-nested type (e.g., `IntegerType`), checking if the error message is really improved. For running tests, you could refer https://spark.apache.org/docs/latest/building-spark.html#pyspark-tests-with-maven. I personally test them locally first and then add and run some tests. Maybe running `./python/run-tests --modules=pyspark-sql` could be enough. Also, make sure running `./dev/lint-python` to check if it complies PEP8 and etc. When you finish them, commit and push which will trigger the build via Jenkins here. See also "Pull Request" in http://spark.apache.org/contributing.html --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user gberger commented on the issue: https://github.com/apache/spark/pull/19792 For sure @ueshin, I will add tests. @HyukjinKwon understood! How do we go about testing performance regression? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19792 @gberger, BTW, just to be clear, IIRC the type inference and merging code path here are shared for other data types, for example, dict, namedtuple, row and etc. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19792 Thanks @ueshin. Yup, +1 for adding some tests. I just wonder if we could have a similar form of error message in type verification. I remember I fixed a similar issue for type verification - https://github.com/apache/spark/pull/18521 (see the links in "Before" and "After"), for example: ``` field c in field b in field a: can not merge type IntegerType and StringType element in array element in array field a: can not merge type IntegerType and StringType ``` Let's make sure there is no performance regression as well (even I was about to make the mistake before). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/19792 Can you add tests for this? Maybe we need some tests for simple case and some nested type cases. I'd also cc @HyukjinKwon. Thanks. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19792 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19792 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84108/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19792 **[Test build #84108 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84108/testReport)** for PR 19792 at commit [`6aa9963`](https://github.com/apache/spark/commit/6aa99631154cc0a6809832a1333ee46959efcbfb). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19792 **[Test build #84108 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84108/testReport)** for PR 19792 at commit [`6aa9963`](https://github.com/apache/spark/commit/6aa99631154cc0a6809832a1333ee46959efcbfb). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/19792 Jenkins, retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user gberger commented on the issue: https://github.com/apache/spark/pull/19792 @ueshin I think this build fail was an outage, can we retest? Could not load hsdis-amd64.so; library not loadable; PrintAssembly is disabled --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user gberger commented on the issue: https://github.com/apache/spark/pull/19792 Jenkins, retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user gberger commented on the issue: https://github.com/apache/spark/pull/19792 Jenkins, please retest --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19792 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19792 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84106/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19792 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84105/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19792 **[Test build #84106 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84106/testReport)** for PR 19792 at commit [`6aa9963`](https://github.com/apache/spark/commit/6aa99631154cc0a6809832a1333ee46959efcbfb). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19792 **[Test build #84105 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84105/testReport)** for PR 19792 at commit [`b29434e`](https://github.com/apache/spark/commit/b29434e939cbb8e5eb3f3fc3e36e33bc8eab2cf1). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19792 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19792: [SPARK-22566][PYTHON] Better error message for `_merge_t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19792 **[Test build #84106 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84106/testReport)** for PR 19792 at commit [`6aa9963`](https://github.com/apache/spark/commit/6aa99631154cc0a6809832a1333ee46959efcbfb). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
