Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/20387
Thanks for all your help getting this committed, @cloud-fan!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/20387
thanks, merging to master!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20387
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20387
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87532/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20387
**[Test build #87532 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87532/testReport)**
for PR 20387 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20387
**[Test build #87532 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87532/testReport)**
for PR 20387 at commit
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/20387
Thanks for the update! Enjoy your vacation, and thanks for letting me know.
---
-
To unsubscribe, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20387
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20387
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/951/
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/20387
I'm on vacation and will be back next week, will have a more thorough
review at that time, sorry for the inconvenience!
---
-
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/20387
@cloud-fan, is there anything else that needs to be updated, or is this
ready to be merged?
---
-
To unsubscribe, e-mail:
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/20387
@cloud-fan, can you have a look at this? I've made the requested changes
and tests are passing.
---
-
To unsubscribe, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20387
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20387
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87434/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20387
**[Test build #87434 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87434/testReport)**
for PR 20387 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20387
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87432/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20387
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20387
**[Test build #87432 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87432/testReport)**
for PR 20387 at commit
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/20387
Okay, I rebased again after SPARK-23303 was reverted.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20387
**[Test build #87434 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87434/testReport)**
for PR 20387 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20387
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/882/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20387
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20387
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87427/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20387
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20387
**[Test build #87427 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87427/testReport)**
for PR 20387 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20387
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20387
**[Test build #87432 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87432/testReport)**
for PR 20387 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20387
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/880/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20387
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87426/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20387
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20387
**[Test build #87426 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87426/testReport)**
for PR 20387 at commit
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/20387
@cloud-fan, I've rebased and made the requested changes.
#20603 reverts the last commit that adds back support for user-supplied
schemas that are identical to the source schema.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20387
**[Test build #87427 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87427/testReport)**
for PR 20387 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20387
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/876/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20387
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20387
**[Test build #87426 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87426/testReport)**
for PR 20387 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20387
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/875/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20387
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20387
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87274/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20387
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20387
**[Test build #87274 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87274/testReport)**
for PR 20387 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20387
**[Test build #87274 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87274/testReport)**
for PR 20387 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20387
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20387
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/760/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20387
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20387
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87268/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20387
**[Test build #87268 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87268/testReport)**
for PR 20387 at commit
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/20387
> Sorry, what do you want to change?
Nothing, just a potential use case to support creating `DataSourceOptions`
in `DataSourceV2Relation`. If there are a lot of places like this, it's
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20387
**[Test build #87268 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87268/testReport)**
for PR 20387 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20387
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/755/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20387
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/20387
> See FindDataSourceTable.readDataSourceTable about how we handle the path
option.
Sorry, what do you want to change?
---
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/20387
> We've added a resolution rule from UnresolvedRelation to
DataSourceV2Relation that uses our implementation. UnresolvedRelation needs to
pass its TableIdentifier to the v2 relation, which is why
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20387
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87170/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20387
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20387
**[Test build #87170 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87170/testReport)**
for PR 20387 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20387
**[Test build #87170 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87170/testReport)**
for PR 20387 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20387
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/674/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20387
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/20387
@cloud-fan: Rebased and removed path.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands,
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20387
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20387
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87123/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20387
**[Test build #87123 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87123/testReport)**
for PR 20387 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20387
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/637/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20387
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20387
**[Test build #87123 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87123/testReport)**
for PR 20387 at commit
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/20387
Will do.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/20387
I think there will be a lot of discussion about data source v2 table
support. For now how about we remove the table/path stuff and get this PR in
ASAP?
---
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/20387
> Can you give a use case about this?
We've added a resolution rule from `UnresolvedRelation` to
`DataSourceV2Relation` that uses our implementation. `UnresolvedRelation` needs
to pass its
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/20387
> The reader is to be created and configured by the relation, then the
relation needs to be able to set the table, path, and other properties. This
adds necessary data to the relation that is no
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/20387
The reader is to be created and configured by the relation, then the
relation needs to be able to set the table, path, and other properties. This
adds necessary data to the relation that is no
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/20387
This PR does 3 things:
1. make `DataSourceV2Relation` immutable. This extends the constructor of
`DataSourceV2Relation` to include pushed filters and pruned columns.
2. carry some standard
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/20387
@cloud-fan, this is a single commit on purpose because predicate push-down
makes plan changes. I think it's best to do these at once to avoid unnecessary
work. That's why I started looking more
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/20387
I'm OK to replace the new push down implementation with
`PhysicalOperation`, but please do that in an individual PR. If we do find the
new implementation is necessary, it's easier for us to bring
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/20387
> For safety, I wanna keep it unchanged, and start something new for data
source v2 only.
I disagree.
* **#20476 addresses a bug caused by the new implementation that is not a
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/20387
For doing pushdown at logical or physical phase, I don't have a strong
preference. I think at logical phase we should try our best to push down
data-size-reduction operators(like filter,
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20387
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87007/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20387
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20387
**[Test build #87007 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87007/testReport)**
for PR 20387 at commit
Github user marmbrus commented on the issue:
https://github.com/apache/spark/pull/20387
Regarding, `computeStats`, the logical plan seems like it might not be the
right place. As we move towards more CBO it seems like we are going to need to
pick physical operators before we can
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20387
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87004/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20387
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20387
**[Test build #87004 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87004/testReport)**
for PR 20387 at commit
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/20387
> Why pushdown is happening in logical optimization and not during query
planning. My first instinct would be to have the optimizer get operators as
close to the leaves as possible and then fuse (or
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20387
**[Test build #87007 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87007/testReport)**
for PR 20387 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20387
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20387
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/543/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20387
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/541/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20387
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20387
**[Test build #87004 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87004/testReport)**
for PR 20387 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20387
**[Test build #87002 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87002/testReport)**
for PR 20387 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20387
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87002/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20387
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20387
**[Test build #87002 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87002/testReport)**
for PR 20387 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20387
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/539/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20387
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20387
**[Test build #87001 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87001/testReport)**
for PR 20387 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20387
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87001/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20387
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20387
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
1 - 100 of 103 matches
Mail list logo
