Github user dilipbiswal commented on the issue:
https://github.com/apache/spark/pull/20579
Thanks a lot @cloud-fan @gatorsmile
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/20579
thanks, merging to master!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88485/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20579
**[Test build #88485 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88485/testReport)**
for PR 20579 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1691/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20579
**[Test build #88485 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88485/testReport)**
for PR 20579 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88458/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20579
**[Test build #88458 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88458/testReport)**
for PR 20579 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1670/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20579
**[Test build #88458 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88458/testReport)**
for PR 20579 at commit
Github user dilipbiswal commented on the issue:
https://github.com/apache/spark/pull/20579
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88450/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88455/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20579
**[Test build #88455 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88455/testReport)**
for PR 20579 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20579
**[Test build #88450 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88450/testReport)**
for PR 20579 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1668/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20579
**[Test build #88455 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88455/testReport)**
for PR 20579 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1665/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20579
**[Test build #88450 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88450/testReport)**
for PR 20579 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user dilipbiswal commented on the issue:
https://github.com/apache/spark/pull/20579
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88440/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20579
**[Test build #88440 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88440/testReport)**
for PR 20579 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88436/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20579
**[Test build #88436 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88436/testReport)**
for PR 20579 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1656/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20579
**[Test build #88440 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88440/testReport)**
for PR 20579 at commit
Github user dilipbiswal commented on the issue:
https://github.com/apache/spark/pull/20579
@cloud-fan ok.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/20579
LGTM, can we add something to the migration guide?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user dilipbiswal commented on the issue:
https://github.com/apache/spark/pull/20579
@cloud-fan Thank you. I assumed (wrongly) that we don't want to change the
behaviour for an external file based datasource and we wanted to scope our
check only to spark built in data sources.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20579
**[Test build #88436 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88436/testReport)**
for PR 20579 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1652/
Github user dilipbiswal commented on the issue:
https://github.com/apache/spark/pull/20579
@gatorsmile When you get a chance, could you please see if the check for
internal datasource looks reasonable ?
---
-
To
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88396/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20579
**[Test build #88396 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88396/testReport)**
for PR 20579 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20579
**[Test build #88396 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88396/testReport)**
for PR 20579 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1621/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user dilipbiswal commented on the issue:
https://github.com/apache/spark/pull/20579
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88388/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20579
**[Test build #88388 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88388/testReport)**
for PR 20579 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20579
**[Test build #88388 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88388/testReport)**
for PR 20579 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1618/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user dilipbiswal commented on the issue:
https://github.com/apache/spark/pull/20579
@gatorsmile Thank you Sean. I will follow your suggestion.
---
-
To unsubscribe, e-mail:
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/20579
Check whether the format is file-based data sources? Then, we do not need
to check the same thing for different file sources.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88327/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20579
**[Test build #88327 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88327/testReport)**
for PR 20579 at commit
Github user dilipbiswal commented on the issue:
https://github.com/apache/spark/pull/20579
@cloud-fan OK.. i was thinking of adding this check in each built in
datasource like Text, CSV, Parquet, ORC, JSON etc. Just like we check it in
Parquet with this PR. Would you have any
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/20579
After some more thoughts, we should try our best to not introduce behavior
change to existing data sources. How about we only add this check for
file-based data sources(all of them are built-in)?
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1572/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20579
**[Test build #88327 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88327/testReport)**
for PR 20579 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88279/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20579
**[Test build #88279 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88279/testReport)**
for PR 20579 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20579
**[Test build #88279 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88279/testReport)**
for PR 20579 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1541/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88273/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20579
**[Test build #88273 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88273/testReport)**
for PR 20579 at commit
Github user dilipbiswal commented on the issue:
https://github.com/apache/spark/pull/20579
@cloud-fan @rdblue Thank you for clarification. I am sorry, i hadn't seen
your comments before i pushed the last change which targets only parquet. I
will adjust the fix to target all formats
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20579
**[Test build #88273 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88273/testReport)**
for PR 20579 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1537/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/20579
I agree, we should probably add a check for storing a DataFrame with no
columns for now. This is normally caught by the pre-insert rules, but since the
table is getting "created" in this case there
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/20579
I think this should be applied to all data sources not only parquet. I
can't think of any cases that a data source needs to write data with empty
schema, cc @rdblue for confirmation.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87341/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20579
**[Test build #87341 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87341/testReport)**
for PR 20579 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/816/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20579
**[Test build #87341 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87341/testReport)**
for PR 20579 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/815/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/20579
Retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/20579
@dilipbiswal this is a nice improvement. I left a few comments.
---
-
To unsubscribe, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87317/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20579
**[Test build #87317 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87317/testReport)**
for PR 20579 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/801/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20579
**[Test build #87317 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87317/testReport)**
for PR 20579 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20579
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87314/
Test FAILed.
---
1 - 100 of 104 matches
Mail list logo
