Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/15998
thanks, merging to master/2.1!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/15998
Yea this one seems like directly related to the work we did in 2.1 for
large table handling.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/15998
LGTM, @rxin shall we backport this? Although it's not a bug, it's a
scalability issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15998
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69691/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15998
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15998
**[Test build #69691 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69691/consoleFull)**
for PR 15998 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15998
**[Test build #69691 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69691/consoleFull)**
for PR 15998 at commit
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/15998
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/15998
I suspect this is a spurious, unrelated test failure. Can we get a rebuild,
please?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15998
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69683/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15998
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15998
**[Test build #69683 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69683/consoleFull)**
for PR 15998 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15998
**[Test build #69683 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69683/consoleFull)**
for PR 15998 at commit
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/15998
@gatorsmile I've applied your patch and reverted the change I made in the
previous commit to workaround that defect. The failed test now passes for me.
Let's see what Jenkins says.
---
If your
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/15998
> #15998 (comment) found a bug. If this PR will not be merged to Spark 2.1
branch, I think we need to submit a separate PR for resolving the bug.
I would like to get this patch into Spark
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/15998
@gatorsmile I can't find your commit:
```
[msa@ip-10-0-8-34 spark-master]$ git fetch origin
remote: Counting objects: 114, done.
remote: Compressing objects: 100% (53/53), done.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/15998
@mallman ,
https://github.com/apache/spark/commit/f4c48e1d61dc4d1fc581b78dd9e5c948965acad5
might be what @cloud-fan wants.
---
If your project is set up for it, you can reply to this email
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/15998
https://github.com/apache/spark/pull/15998#issuecomment-264761754 found a
bug. If this PR will not be merged to Spark 2.1 branch, I think we need to
submit a separate PR for resolving the bug.
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/15998
@cloud-fan I'm not familiar enough with that code to be comfortable making
that change. Can you submit a PR against
`VideoAmp:spark-18572-list_partition_names` with the necessary changes to this
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/15998
I think we can just fix it in this PR, it's only several lines change.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/15998
@cloud-fan That's unfortunate if it's going to block this PR. How do we
proceed?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/15998
@mallman ah it's a mistake, check
https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala#L228
, we need to move
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15998
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15998
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69652/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15998
**[Test build #69652 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69652/consoleFull)**
for PR 15998 at commit
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/15998
> @mallman do you know which tests fail the partition spec checking? It
looks to me that before we call partition related API in SessionCatalog, the
partition column names should be normalized
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/15998
@mallman do you know which tests fail the partition spec checking? It looks
to me that before we call partition related API in `SessionCatalog`, the
partition column names should be normalized
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15998
**[Test build #69652 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69652/consoleFull)**
for PR 15998 at commit
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/15998
I've modified the behavior of the partition spec checking methods in
`SessionCatalog` to test for case-sensitive analysis.
---
If your project is set up for it, you can reply to this email and
Github user ericl commented on the issue:
https://github.com/apache/spark/pull/15998
Spark SQL is case insensitive by default. As @gatorsmile mentioned it might
be good to check that flag, though I don't know how well case sensitive mode
works when enabled.
---
If your project is
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/15998
This depends on `spark.sql.caseSensitive`, right? cc @ericl @cloud-fan
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/15998
@ericl @gatorsmile Please see test failure here:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15998
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15998
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69581/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15998
**[Test build #69581 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69581/consoleFull)**
for PR 15998 at commit
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/15998
@gatorsmile I enhanced the coverage of `SessionCatalog.listPartitions` and
`SessionCatalog.listPartitionNames` to include tests for invalid partial
partition specs.
---
If your project is set up
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15998
**[Test build #69581 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69581/consoleFull)**
for PR 15998 at commit
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/15998
> can you also address this comment? #15998 (comment)
Addressed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/15998
What is the expected output? In the ExternalCatalogSuite, it sounds like we
do not issue any error.
```
catalog.listPartitionNames("db2", "tbl2", Some(Map("unknown" -> "unknown")))
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/15998
can you also address this comment?
https://github.com/apache/spark/pull/15998#discussion_r90181774
LGTM except a few comments.
---
If your project is set up for it, you can reply to
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/15998
> Like the other partition ExternalCatalog APIs, could you also add the
negative test cases to ExternalCatalogSuite.scala?
I'm sorry, I don't understand what you're asking for. Can you be
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/15998
Like the other partition ExternalCatalog APIs, could you also add the
negative test cases to `ExternalCatalogSuite.scala`?
---
If your project is set up for it, you can reply to this email and
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15998
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15998
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69501/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15998
**[Test build #69501 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69501/consoleFull)**
for PR 15998 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15998
**[Test build #69501 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69501/consoleFull)**
for PR 15998 at commit
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/15998
The test that failed is definitely related to this PR, however it passes
for me locally. I'll investigate...
---
If your project is set up for it, you can reply to this email and have your
reply
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15998
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15998
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69491/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15998
**[Test build #69491 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69491/consoleFull)**
for PR 15998 at commit
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/15998
LMK if there's anything else you'd like me to address, otherwiseâassuming
the tests passâplease merge to master. Also, it would be great if we can back
port this into 2.1 as well.
---
If your
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15998
**[Test build #69491 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69491/consoleFull)**
for PR 15998 at commit
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/15998
Added a couple of unit tests and rebased.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/15998
I think this PR is for performance improvement, why we update the
`HiveCommandSuite` here? Otherwise LGTM
---
If your project is set up for it, you can reply to this email and have your
reply
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15998
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69378/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15998
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15998
**[Test build #69378 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69378/consoleFull)**
for PR 15998 at commit
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/15998
I will work on additional unit test coverage tomorrow.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15998
**[Test build #69378 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69378/consoleFull)**
for PR 15998 at commit
Github user ericl commented on the issue:
https://github.com/apache/spark/pull/15998
* looks good once InMemoryCatalog is fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user ericl commented on the issue:
https://github.com/apache/spark/pull/15998
@mallman I'll take a look today
On Tue, Nov 29, 2016, 9:45 AM Michael Allman
wrote:
> Hi Guys,
>
> Repeating my comment/query for @ericl
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/15998
Hi Guys,
Repeating my comment/query for @ericl. I'm hoping someone can provide
affirmation/refutation to my question before I proceed with new unit tests.
I've run some tests to
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/15998
> where is the speed-up come from? Is it because the hive API
getPartitionNames is faster than getPartitions? Or is it because we generate
the partition string(a=1/b=2/c=3) at hive side and it's
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/15998
where is the speed-up come from? Is it because the hive API
`getPartitionNames` is faster than `getPartitions`? Or is it because we
generate the partition string(`a=1/b=2/c=3`) at hive side and
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15998
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69195/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15998
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15998
**[Test build #69195 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69195/consoleFull)**
for PR 15998 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15998
**[Test build #69195 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69195/consoleFull)**
for PR 15998 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15998
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15998
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69174/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15998
**[Test build #69174 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69174/consoleFull)**
for PR 15998 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15998
**[Test build #69174 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69174/consoleFull)**
for PR 15998 at commit
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/15998
CC @ericl @cloud-fan
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15998
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69089/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15998
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15998
**[Test build #69089 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69089/consoleFull)**
for PR 15998 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15998
**[Test build #69089 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69089/consoleFull)**
for PR 15998 at commit
77 matches
Mail list logo