Github user gberger commented on the issue:
https://github.com/apache/spark/pull/19792
Great! Thanks all
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/19792
Thanks! merging to master/2.3.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands,
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19792
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19792
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85784/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19792
**[Test build #85784 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85784/testReport)**
for PR 19792 at commit
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/19792
LGTM, pending Jenkins.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/19792
@HyukjinKwon Thanks, I'll take another look soon.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19792
To be clear (as I was reviewing this too), I am okay with going ahead,
@ueshin if this looks good to you.
---
-
To
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19792
**[Test build #85784 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85784/testReport)**
for PR 19792 at commit
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/19792
Jenkins, retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19792
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19792
**[Test build #85752 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85752/testReport)**
for PR 19792 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19792
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85752/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19792
**[Test build #85752 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85752/testReport)**
for PR 19792 at commit
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19792
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19792
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19792
**[Test build #84932 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84932/testReport)**
for PR 19792 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19792
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84932/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19792
**[Test build #84932 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84932/testReport)**
for PR 19792 at commit
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19792
ok to test
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19792
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user gberger commented on the issue:
https://github.com/apache/spark/pull/19792
@HyukjinKwon no worries, I understand. We gotta be 100% thorough here.
Thanks for the help
---
-
To unsubscribe, e-mail:
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19792
Will take another look soon tomorrow. Sorry that is getting delayed again
and again but I just realised this code path is a little bit tricky ..
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19792
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19792
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84918/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19792
**[Test build #84918 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84918/testReport)**
for PR 19792 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19792
**[Test build #84918 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84918/testReport)**
for PR 19792 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19792
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84764/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19792
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19792
**[Test build #84764 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84764/testReport)**
for PR 19792 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19792
**[Test build #84764 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84764/testReport)**
for PR 19792 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19792
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84708/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19792
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19792
**[Test build #84708 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84708/testReport)**
for PR 19792 at commit
Github user gberger commented on the issue:
https://github.com/apache/spark/pull/19792
Good catch @HyukjinKwon! I reverted those changes and added a test to cover
this regression.
---
-
To unsubscribe, e-mail:
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19792
**[Test build #84708 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84708/testReport)**
for PR 19792 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19792
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84558/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19792
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19792
**[Test build #84558 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84558/testReport)**
for PR 19792 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19792
**[Test build #84558 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84558/testReport)**
for PR 19792 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19792
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84492/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19792
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19792
**[Test build #84492 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84492/testReport)**
for PR 19792 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19792
**[Test build #84492 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84492/testReport)**
for PR 19792 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19792
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19792
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84487/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19792
**[Test build #84487 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84487/testReport)**
for PR 19792 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19792
**[Test build #84487 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84487/testReport)**
for PR 19792 at commit
Github user gberger commented on the issue:
https://github.com/apache/spark/pull/19792
@HyukjinKwon done, with test added.
```
>>> spark.createDataFrame(spark.sparkContext.parallelize([[None, 1], ["a",
None], [1, 1]]), schema=["a", "b"], samplingRatio=0.99)
Traceback
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19792
Can we set the `names` for `_createFromRDD` -> `_inferSchema` too?:
```python
>>> spark.createDataFrame(spark.sparkContext.parallelize([[None, 1], ["a",
None], [1, 1]]),
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19792
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84429/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19792
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19792
**[Test build #84429 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84429/testReport)**
for PR 19792 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19792
**[Test build #84429 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84429/testReport)**
for PR 19792 at commit
Github user gberger commented on the issue:
https://github.com/apache/spark/pull/19792
Fixed
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19792
Oh, will double check too for sure shortly.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user gberger commented on the issue:
https://github.com/apache/spark/pull/19792
Friendly ping -- I've fixed that @ueshin.
Is there anything else I should look at to get this to be merged?
/cc @HyukjinKwon
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19792
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19792
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84335/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19792
**[Test build #84335 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84335/testReport)**
for PR 19792 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19792
**[Test build #84335 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84335/testReport)**
for PR 19792 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19792
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84305/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19792
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19792
**[Test build #84305 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84305/testReport)**
for PR 19792 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19792
**[Test build #84305 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84305/testReport)**
for PR 19792 at commit
Github user gberger commented on the issue:
https://github.com/apache/spark/pull/19792
Hi all,
I have changed the error message to be like #18521. Here are some examples:
```
_merge_type(
StructType([StructField("f1", ArrayType(MapType(StringType(),
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19792
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84272/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19792
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19792
**[Test build #84272 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84272/testReport)**
for PR 19792 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19792
**[Test build #84272 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84272/testReport)**
for PR 19792 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19792
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84140/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19792
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19792
**[Test build #84140 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84140/testReport)**
for PR 19792 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19792
**[Test build #84140 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84140/testReport)**
for PR 19792 at commit
Github user gberger commented on the issue:
https://github.com/apache/spark/pull/19792
Maybe a more performant way to do the path in the error message would be to
propagate it *up* the stack via try/catching the errors and adding the paths as
it goes.
But this way seems
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19792
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19792
**[Test build #84139 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84139/testReport)**
for PR 19792 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19792
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84139/
Test FAILed.
---
Github user gberger commented on the issue:
https://github.com/apache/spark/pull/19792
Hey all,
## Error message
I revamped the error message and made it "recursive" similar to
@HyukjinKwon. Here's an example:
```
>>> _merge_type(
...
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19792
D'oh, you mean performance regression test. Manual tests should be fine.
When you share some codes you ran, maybe we can double check.
---
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19792
I think you can add a new test case somewhere around here -
https://github.com/apache/spark/blob/master/python/pyspark/sql/tests.py#L1724
maybe dealing with some combinations of `StructType`,
Github user gberger commented on the issue:
https://github.com/apache/spark/pull/19792
For sure @ueshin, I will add tests.
@HyukjinKwon understood!
How do we go about testing performance regression?
---
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19792
@gberger, BTW, just to be clear, IIRC the type inference and merging code
path here are shared for other data types, for example, dict, namedtuple, row
and etc.
---
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19792
Thanks @ueshin. Yup, +1 for adding some tests.
I just wonder if we could have a similar form of error message in type
verification. I remember I fixed a similar issue for type
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/19792
Can you add tests for this?
Maybe we need some tests for simple case and some nested type cases.
I'd also cc @HyukjinKwon. Thanks.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19792
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19792
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84108/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19792
**[Test build #84108 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84108/testReport)**
for PR 19792 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19792
**[Test build #84108 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84108/testReport)**
for PR 19792 at commit
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/19792
Jenkins, retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user gberger commented on the issue:
https://github.com/apache/spark/pull/19792
@ueshin I think this build fail was an outage, can we retest?
Could not load hsdis-amd64.so; library not loadable; PrintAssembly is
disabled
---
Github user gberger commented on the issue:
https://github.com/apache/spark/pull/19792
Jenkins, retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user gberger commented on the issue:
https://github.com/apache/spark/pull/19792
Jenkins, please retest
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19792
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19792
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84106/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19792
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84105/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19792
**[Test build #84106 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84106/testReport)**
for PR 19792 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19792
**[Test build #84105 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84105/testReport)**
for PR 19792 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19792
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19792
**[Test build #84106 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84106/testReport)**
for PR 19792 at commit
1 - 100 of 110 matches
Mail list logo
