[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[ericl]

Github user ericl commented on the issue:

https://github.com/apache/spark/pull/14690
  
Fyi, I'm working on some of the mentioned refactoring and adding back
metadata caching. Will file JIRAs for those when I get a chance.

On Fri, Oct 14, 2016, 6:29 PM asfgit  wrote:

> Closed #14690  via 6ce1b67
> 

> .
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> , or mute the
> thread
> 

> .
>



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[rxin]

Github user rxin commented on the issue:

https://github.com/apache/spark/pull/14690
  
Thanks a lot -- going to merge this. We should continue clean it up as 
@cloud-fan suggested.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14690
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66992/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14690
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14690
  
**[Test build #66992 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66992/consoleFull)**
 for PR 14690 at commit 
[`014c998`](https://github.com/apache/spark/commit/014c9989cb132041bce45215450a85661c3202e2).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[mallman]

Github user mallman commented on the issue:

https://github.com/apache/spark/pull/14690
  
>> Hm, I haven't seen that with my test queries. Would adding your 
workaround to SparkILoopInit work?

> It does not, unfortunately.

I believe this impacts people with parquet files written from an older 
version of Spark using an older version of the parquet-mr libraries. We're 
using parquet-mr 1.7.0 in production. It's apparently not writing the "build" 
part of the version string that the 1.8.1 parquet reader is looking for.

I'll work on fixing this, but unless I can get it done really quickly I 
probably won't make progress today. I just have too many things on my plate 
right now.

Cheers.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[mallman]

Github user mallman commented on the issue:

https://github.com/apache/spark/pull/14690
  
> Hm, I haven't seen that with my test queries. Would adding your 
workaround to SparkILoopInit work?

It does not, unfortunately.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[ericl]

Github user ericl commented on the issue:

https://github.com/apache/spark/pull/14690
  
Hm, I haven't seen that with my test queries. Would adding your workaround 
to SparkILoopInit work?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[mallman]

Github user mallman commented on the issue:

https://github.com/apache/spark/pull/14690
  
> btw, what's the parquet log redirection issue? I don't see anything 
unusual in spark shell.

Whenever I run a query on a Hive parquet table I get

```
spark-sql> select eid from event.bid_request where ds=20160915 limit 10;
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".

SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further 
details.
Oct 14, 2016 10:07:03 PM WARNING: org.apache.parquet.CorruptStatistics: 
Ignoring statistics because created_by could not be parsed (see PARQUET-251): 
parquet-mr version 1.6.0
org.apache.parquet.VersionParser$VersionParseException: Could not parse 
created_by: parquet-mr version 1.6.0 using format: (.+) version ((.*) )?\(build 
?(.*)\)
at org.apache.parquet.VersionParser.parse(VersionParser.java:112)
at 
org.apache.parquet.CorruptStatistics.shouldIgnoreStatistics(CorruptStatistics.java:60)
at 
org.apache.parquet.format.converter.ParquetMetadataConverter.fromParquetStatistics(ParquetMetadataConverter.java:263)
at 
org.apache.parquet.format.converter.ParquetMetadataConverter.fromParquetMetadata(ParquetMetadataConverter.java:567)
at 
org.apache.parquet.format.converter.ParquetMetadataConverter.readParquetMetadata(ParquetMetadataConverter.java:544)
at 
org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:431)
at 
org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:386)
at 
org.apache.spark.sql.execution.datasources.parquet.SpecificParquetRecordReaderBase.initialize(SpecificParquetRecordReaderBase.java:107)
at 
org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.initialize(VectorizedParquetRecordReader.java:109)
at 
org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anonfun$buildReader$1.apply(ParquetFileFormat.scala:368)
at 
org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anonfun$buildReader$1.apply(ParquetFileFormat.scala:342)
at 
org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:149)
at 
org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:102)
at 
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.scan_nextBatch$(Unknown
 Source)
at 
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown
 Source)
at 
org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at 
org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:372)
at 
org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:231)
at 
org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:225)
at 
org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:803)
at 
org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:803)
at 
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Oct 14, 2016 10:07:03 PM WARNING: org.apache.parquet.CorruptStatistics: 
Ignoring statistics because created_by could not be parsed (see PARQUET-251): 
parquet-mr version 1.6.0
org.apache.parquet.VersionParser$VersionParseException: Could not parse 
created_by: parquet-mr version 1.6.0 using format: (.+) version ((.*) )?\(build 
?(.*)\)
at org.apache.parquet.VersionParser.parse(VersionParser.java:112)
at 
org.apache.parquet.CorruptStatistics.shouldIgnoreStatistics(CorruptStatistics.java:60)
at 
org.apache.parquet.format.converter.ParquetMetadataConverter.fromParquetStatistics(ParquetMetadataConverter.java:263)
at 
org.apache.parquet.format.converter.ParquetMetadataConverter.fromParquetMetadata(ParquetMetadataConverter.java:567)
...
```

This is really just a small fraction of the lines of these warnings I get.

I looked into this a couple months back. The underlying problem is detailed 
in 

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14690
  
**[Test build #66992 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66992/consoleFull)**
 for PR 14690 at commit 
[`014c998`](https://github.com/apache/spark/commit/014c9989cb132041bce45215450a85661c3202e2).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[mallman]

Github user mallman commented on the issue:

https://github.com/apache/spark/pull/14690
  
I just pushed the rebase. It was really hairy, but I tried hard to ensure I 
got essentially all three branches' changes in.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[rxin]

Github user rxin commented on the issue:

https://github.com/apache/spark/pull/14690
  
We can also merge this first and then do clean-ups next week BTW.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[ericl]

Github user ericl commented on the issue:

https://github.com/apache/spark/pull/14690
  
btw, what's the parquet log redirection issue? I don't see anything unusual 
in spark shell.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[ericl]

Github user ericl commented on the issue:

https://github.com/apache/spark/pull/14690
  
> My main concern is that, the new FileCatalog class hierarchy is really 
complex now. Can we simplify it a little? or at least let's put a class 
hierarchy chart(or text explanation) in PR description to make other reviewers 
easier to understand this new hierarchy.

One low-hanging fruit here is to unify SessionFileCatalog with 
PartitionAwareFileCatalog. It seems that after some of the changes here no 
longer need them to be separate. This is a PR I can add on after the rebase is 
finished.

I also think we should rename some of the classes, but since this would be 
an invasive change we can handle it in a follow-up.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[ericl]

Github user ericl commented on the issue:

https://github.com/apache/spark/pull/14690
  
https://github.com/VideoAmp/spark-public/pull/7


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[ericl]

Github user ericl commented on the issue:

https://github.com/apache/spark/pull/14690
  
Yes, I'll look at that today.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[mallman]

Github user mallman commented on the issue:

https://github.com/apache/spark/pull/14690
  
I'm still working on the rebase. It's very complex—there are two other 
commits involved.

>> 1. Do we need a workaround for ORC like we made for Parquet?

> 1) yes

@ericl Do you have time to contribute this?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[davies]

Github user davies commented on the issue:

https://github.com/apache/spark/pull/14690
  
1) yes
2) it's OK for now.
3) could be done by follow up PR
4) should fix it if it's not hard


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[mallman]

Github user mallman commented on the issue:

https://github.com/apache/spark/pull/14690
  
I will work on a rebase. Meanwhile, I've revisited the open issues in the 
PR description. To summarize:

1. Do we need a workaround for ORC like we made for Parquet?
1. What's the impact of omitting an implementation of 
`listPartitionsByFilter` from `InMemoryCatalog`?
1. What's the performance impact of omitting partition metadata caching 
from this PR?
1. Parquet logging output redirection is still broken.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14690
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14690
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66938/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14690
  
**[Test build #66938 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66938/consoleFull)**
 for PR 14690 at commit 
[`71049d1`](https://github.com/apache/spark/commit/71049d130e89aedba75e8875d8fde7620d6a55e2).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/14690
  
My main concern is that, the new `FileCatalog` class hierarchy is really 
complex now. Can we simplify it a little? or at least let's put a class 
hierarchy chart(or text explanation) in PR description to make other reviewers 
easier to understand this new hierarchy.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14690
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14690
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66931/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14690
  
**[Test build #66931 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66931/consoleFull)**
 for PR 14690 at commit 
[`e1635e4`](https://github.com/apache/spark/commit/e1635e4570c0e4b892b93d1ac1e71d52d5a4f66b).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14690
  
**[Test build #66938 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66938/consoleFull)**
 for PR 14690 at commit 
[`71049d1`](https://github.com/apache/spark/commit/71049d130e89aedba75e8875d8fde7620d6a55e2).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14690
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66923/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14690
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14690
  
**[Test build #66923 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66923/consoleFull)**
 for PR 14690 at commit 
[`765f93c`](https://github.com/apache/spark/commit/765f93ce664ef33c1c62bf80b678ff5ba2992b85).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `case class ParquetDataWithKey(pQ: Int, intField: Int, stringField: 
String)`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14690
  
**[Test build #66931 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66931/consoleFull)**
 for PR 14690 at commit 
[`e1635e4`](https://github.com/apache/spark/commit/e1635e4570c0e4b892b93d1ac1e71d52d5a4f66b).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[ericl]

Github user ericl commented on the issue:

https://github.com/apache/spark/pull/14690
  
https://github.com/VideoAmp/spark-public/pull/5/files fixes the jenkins 
failure and adds test for expected metadata io cost.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14690
  
**[Test build #66923 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66923/consoleFull)**
 for PR 14690 at commit 
[`765f93c`](https://github.com/apache/spark/commit/765f93ce664ef33c1c62bf80b678ff5ba2992b85).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14690
  
**[Test build #3339 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3339/consoleFull)**
 for PR 14690 at commit 
[`dc9e613`](https://github.com/apache/spark/commit/dc9e61348364646c939e85e79a08495e27d46c58).
 * This patch **fails Spark unit tests**.
 * This patch **does not merge cleanly**.
 * This patch adds the following public classes _(experimental)_:
  * `case class ParquetDataWithKey(pQ: Int, intField: Int, stringField: 
String)`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[mallman]

Github user mallman commented on the issue:

https://github.com/apache/spark/pull/14690
  
> Oops there is a conflict now.

NP. I'm working on the rebase.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14690
  
**[Test build #3339 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3339/consoleFull)**
 for PR 14690 at commit 
[`dc9e613`](https://github.com/apache/spark/commit/dc9e61348364646c939e85e79a08495e27d46c58).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[rxin]

Github user rxin commented on the issue:

https://github.com/apache/spark/pull/14690
  
Oops there is a conflict now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[rxin]

Github user rxin commented on the issue:

https://github.com/apache/spark/pull/14690
  
The mima failure is just mima flaky.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14690
  
**[Test build #66912 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66912/consoleFull)**
 for PR 14690 at commit 
[`dc9e613`](https://github.com/apache/spark/commit/dc9e61348364646c939e85e79a08495e27d46c58).
 * This patch **fails MiMa tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `case class ParquetDataWithKey(pQ: Int, intField: Int, stringField: 
String)`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14690
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14690
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66912/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14690
  
**[Test build #66912 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66912/consoleFull)**
 for PR 14690 at commit 
[`dc9e613`](https://github.com/apache/spark/commit/dc9e61348364646c939e85e79a08495e27d46c58).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14690
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14690
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66906/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14690
  
**[Test build #66906 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66906/consoleFull)**
 for PR 14690 at commit 
[`3a072bd`](https://github.com/apache/spark/commit/3a072bd92fe95468c75804f605b46016be25b6d8).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[ericl]

Github user ericl commented on the issue:

https://github.com/apache/spark/pull/14690
  
> Ok. I'm looking into the org.apache.spark.sql.hive.ParquetSourceSuite 
failure.

I think that was caused by the p => pQ change


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[mallman]

Github user mallman commented on the issue:

https://github.com/apache/spark/pull/14690
  
> Btw, I noticed that this suite was failing in jenkins only.
>
> [info] - partitioned pruned table reports only selected files *** FAILED 
*** (610 milliseconds)
>
> That's pretty weird, so I'll look into that (having acquired ssh access 
to jenkins).

Ok. I'm looking into the `org.apache.spark.sql.hive.ParquetSourceSuite` 
failure.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14690
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66903/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14690
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14690
  
**[Test build #66903 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66903/consoleFull)**
 for PR 14690 at commit 
[`d48ff10`](https://github.com/apache/spark/commit/d48ff104433490ea38e3799698197cecd0dfdbcd).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14690
  
Build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14690
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66899/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14690
  
**[Test build #66899 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66899/consoleFull)**
 for PR 14690 at commit 
[`adeddbf`](https://github.com/apache/spark/commit/adeddbf4d92fde71ee4421430e31056e11bdf3f0).
 * This patch **fails Spark unit tests**.
 * This patch **does not merge cleanly**.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14690
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66898/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14690
  
Build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14690
  
**[Test build #66898 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66898/consoleFull)**
 for PR 14690 at commit 
[`afb3340`](https://github.com/apache/spark/commit/afb3340360969af4bb19a03083dc0cad8c6f74e2).
 * This patch **fails Spark unit tests**.
 * This patch **does not merge cleanly**.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14690
  
**[Test build #66906 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66906/consoleFull)**
 for PR 14690 at commit 
[`3a072bd`](https://github.com/apache/spark/commit/3a072bd92fe95468c75804f605b46016be25b6d8).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[ericl]

Github user ericl commented on the issue:

https://github.com/apache/spark/pull/14690
  
MiMa is supposed to detect binary incompatible changes. Looking at the 
console output though, it seems like MiMa just crashed.

Running `./dev/mima` locally worked though, so maybe it was some transient 
jenkins issue.

Btw, I noticed that this suite was failing in jenkins only.

```
[info] - partitioned pruned table reports only selected files *** FAILED 
*** (610 milliseconds)
[info]   
Array("file:/home/jenkins/workspace/SparkPullRequestBuilder@2/target/tmp/spark-e7d44560-5cc4-4d4c-b72c-7d68c1bd9de9/f1=2/f2=2",
 
"file:/home/jenkins/workspace/SparkPullRequestBuilder@2/target/tmp/spark-e7d44560-5cc4-4d4c-b72c-7d68c1bd9de9/f1=1/f2=1",
 
"file:/home/jenkins/workspace/SparkPullRequestBuilder@2/target/tmp/spark-e7d44560-5cc4-4d4c-b72c-7d68c1bd9de9/f1=4/f2=4",
 
"file:/home/jenkins/workspace/SparkPullRequestBuilder@2/target/tmp/spark-e7d44560-5cc4-4d4c-b72c-7d68c1bd9de9/f1=0/f2=0",
 
"file:/home/jenkins/workspace/SparkPullRequestBuilder@2/target/tmp/spark-e7d44560-5cc4-4d4c-b72c-7d68c1bd9de9/f1=3/f2=3")
 had length 5 instead of expected length 2 (HiveDataFrameSuite.scala:60)
```

That's pretty weird, so I'll look into that (having acquired ssh access to 
jenkins).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[mallman]

Github user mallman commented on the issue:

https://github.com/apache/spark/pull/14690
  
> This patch fails MiMa tests.

I've never seen this before. What does this mean?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14690
  
**[Test build #66903 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66903/consoleFull)**
 for PR 14690 at commit 
[`d48ff10`](https://github.com/apache/spark/commit/d48ff104433490ea38e3799698197cecd0dfdbcd).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14690
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66900/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14690
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14690
  
**[Test build #66900 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66900/consoleFull)**
 for PR 14690 at commit 
[`bf6f46f`](https://github.com/apache/spark/commit/bf6f46f5d407451fe53b7cd6fbb7d164ed2059cf).
 * This patch **fails MiMa tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14690
  
**[Test build #66900 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66900/consoleFull)**
 for PR 14690 at commit 
[`bf6f46f`](https://github.com/apache/spark/commit/bf6f46f5d407451fe53b7cd6fbb7d164ed2059cf).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[mallman]

Github user mallman commented on the issue:

https://github.com/apache/spark/pull/14690
  
I've pushed an update to `ParquetMetastoreSuite` that illustrates the bug 
(or "limitation") WRT support for mixed-case partition columns I discovered 
yesterday. To reiterate, this is not a regression—this test fails in master 
as well.

Also, I will push a rebase shortly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14690
  
**[Test build #66899 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66899/consoleFull)**
 for PR 14690 at commit 
[`adeddbf`](https://github.com/apache/spark/commit/adeddbf4d92fde71ee4421430e31056e11bdf3f0).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14690
  
**[Test build #66898 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66898/consoleFull)**
 for PR 14690 at commit 
[`afb3340`](https://github.com/apache/spark/commit/afb3340360969af4bb19a03083dc0cad8c6f74e2).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14690
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66861/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14690
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14690
  
**[Test build #66861 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66861/consoleFull)**
 for PR 14690 at commit 
[`59fecdf`](https://github.com/apache/spark/commit/59fecdf1e889c218ac81cdf73ba3e46142d052e6).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14690
  
**[Test build #66861 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66861/consoleFull)**
 for PR 14690 at commit 
[`59fecdf`](https://github.com/apache/spark/commit/59fecdf1e889c218ac81cdf73ba3e46142d052e6).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[ericl]

Github user ericl commented on the issue:

https://github.com/apache/spark/pull/14690
  
Conf flag here: https://github.com/VideoAmp/spark-public/pull/3


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14690
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66833/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14690
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14690
  
**[Test build #66833 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66833/consoleFull)**
 for PR 14690 at commit 
[`14ca612`](https://github.com/apache/spark/commit/14ca61266290ad8b4231ab0355cb0e848f2acfbe).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[mallman]

Github user mallman commented on the issue:

https://github.com/apache/spark/pull/14690
  
I determined the performance regression was introduced by a commit I hadn't 
pushed to this PR. Sorry for the false alarm. 😞 Needless to say, I'm not 
pushing that commit.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14690
  
**[Test build #66833 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66833/consoleFull)**
 for PR 14690 at commit 
[`14ca612`](https://github.com/apache/spark/commit/14ca61266290ad8b4231ab0355cb0e848f2acfbe).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[mallman]

Github user mallman commented on the issue:

https://github.com/apache/spark/pull/14690
  
>> Btw I've noticed a significant performance difference between 
ListingFileCatalog and TableFileCatalog's implementation of ListFiles. The 
difference seems to be that ListingFileCatalog parallelizes file listing (which 
is slower locally, but presumably faster when listing against remote storage on 
a larger cluster).

> I'm aware of that and have created a patch to address that, I just 
haven't pushed it yet.

I just pushed this commit. LMK what you think.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[mallman]

Github user mallman commented on the issue:

https://github.com/apache/spark/pull/14690
  
> Btw I've noticed a significant performance difference between 
ListingFileCatalog and TableFileCatalog's implementation of ListFiles. The 
difference seems to be that ListingFileCatalog parallelizes file listing (which 
is slower locally, but presumably faster when listing against remote storage on 
a larger cluster).

I'm aware of that and have created a patch to address that, I just haven't 
pushed it yet.

Even with the parallelized implementation in `TableFileCatalog`, 
performance is still bad. Still investigating...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[ericl]

Github user ericl commented on the issue:

https://github.com/apache/spark/pull/14690
  
Btw I've noticed a significant performance difference between 
ListingFileCatalog and TableFileCatalog's implementation of ListFiles. The 
difference seems to be that ListingFileCatalog parallelizes file listing (which 
is slower locally, but presumably faster when listing against remote storage on 
a larger cluster).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[mallman]

Github user mallman commented on the issue:

https://github.com/apache/spark/pull/14690
  
I'm testing this patch on a couple of tables internally with on the order 
of 10k partitions. Performance is much slower than it should be. I'm 
investigating.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[davies]

Github user davies commented on the issue:

https://github.com/apache/spark/pull/14690
  
Workaround should be reasonable to unblock this PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14690
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66828/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14690
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14690
  
**[Test build #66828 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66828/consoleFull)**
 for PR 14690 at commit 
[`2a696d7`](https://github.com/apache/spark/commit/2a696d7b28ebb4df5789b8b62af48fd57c6a29cc).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14690
  
**[Test build #66828 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66828/consoleFull)**
 for PR 14690 at commit 
[`2a696d7`](https://github.com/apache/spark/commit/2a696d7b28ebb4df5789b8b62af48fd57c6a29cc).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[mallman]

Github user mallman commented on the issue:

https://github.com/apache/spark/pull/14690
  
I updated the description of this PR to reflect the workaround for the 
Hive/Parquet case-sensitivity issue.

Do we need a similar workaround for ORC?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14690
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14690
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66772/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14690
  
**[Test build #66772 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66772/consoleFull)**
 for PR 14690 at commit 
[`10e9e8a`](https://github.com/apache/spark/commit/10e9e8a08661aa53347bccfecbc88aad8e89adb8).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14690
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66764/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14690
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14690
  
**[Test build #66764 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66764/consoleFull)**
 for PR 14690 at commit 
[`175c268`](https://github.com/apache/spark/commit/175c2684eb515a1d0def8cf6a72011aa9a48625d).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14690
  
**[Test build #66772 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66772/consoleFull)**
 for PR 14690 at commit 
[`10e9e8a`](https://github.com/apache/spark/commit/10e9e8a08661aa53347bccfecbc88aad8e89adb8).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[ericl]

Github user ericl commented on the issue:

https://github.com/apache/spark/pull/14690
  
Btw I also made https://github.com/VideoAmp/spark-public/pull/2/files, to 
fix inputFiles.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14690
  
**[Test build #66764 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66764/consoleFull)**
 for PR 14690 at commit 
[`175c268`](https://github.com/apache/spark/commit/175c2684eb515a1d0def8cf6a72011aa9a48625d).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14690
  
**[Test build #66751 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66751/consoleFull)**
 for PR 14690 at commit 
[`2762efd`](https://github.com/apache/spark/commit/2762efd925325a9ab732b34e6672b704dc514062).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14690
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14690
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66751/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

[mallman]

Github user mallman commented on the issue:

https://github.com/apache/spark/pull/14690
  
>> Finally, this would require us to read the schema files. That's 
something I'm trying to avoid in this patch.

> Not sure what you mean here, but the parquet change should be execution 
time only. I'll submit a pr here for that.

Okay. I look forward to seeing that.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org