[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-14 Thread ericl
Github user ericl commented on the issue: https://github.com/apache/spark/pull/14690 Fyi, I'm working on some of the mentioned refactoring and adding back metadata caching. Will file JIRAs for those when I get a chance. On Fri, Oct 14, 2016, 6:29 PM asfgit

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-14 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/14690 Thanks a lot -- going to merge this. We should continue clean it up as @cloud-fan suggested. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14690 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66992/ Test PASSed. ---

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14690 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14690 **[Test build #66992 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66992/consoleFull)** for PR 14690 at commit

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-14 Thread mallman
Github user mallman commented on the issue: https://github.com/apache/spark/pull/14690 >> Hm, I haven't seen that with my test queries. Would adding your workaround to SparkILoopInit work? > It does not, unfortunately. I believe this impacts people with parquet files

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-14 Thread mallman
Github user mallman commented on the issue: https://github.com/apache/spark/pull/14690 > Hm, I haven't seen that with my test queries. Would adding your workaround to SparkILoopInit work? It does not, unfortunately. --- If your project is set up for it, you can reply to

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-14 Thread ericl
Github user ericl commented on the issue: https://github.com/apache/spark/pull/14690 Hm, I haven't seen that with my test queries. Would adding your workaround to SparkILoopInit work? --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-14 Thread mallman
Github user mallman commented on the issue: https://github.com/apache/spark/pull/14690 > btw, what's the parquet log redirection issue? I don't see anything unusual in spark shell. Whenever I run a query on a Hive parquet table I get ``` spark-sql> select eid

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14690 **[Test build #66992 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66992/consoleFull)** for PR 14690 at commit

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-14 Thread mallman
Github user mallman commented on the issue: https://github.com/apache/spark/pull/14690 I just pushed the rebase. It was really hairy, but I tried hard to ensure I got essentially all three branches' changes in. --- If your project is set up for it, you can reply to this email and

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-14 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/14690 We can also merge this first and then do clean-ups next week BTW. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-14 Thread ericl
Github user ericl commented on the issue: https://github.com/apache/spark/pull/14690 btw, what's the parquet log redirection issue? I don't see anything unusual in spark shell. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-14 Thread ericl
Github user ericl commented on the issue: https://github.com/apache/spark/pull/14690 > My main concern is that, the new FileCatalog class hierarchy is really complex now. Can we simplify it a little? or at least let's put a class hierarchy chart(or text explanation) in PR description

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-14 Thread ericl
Github user ericl commented on the issue: https://github.com/apache/spark/pull/14690 https://github.com/VideoAmp/spark-public/pull/7 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-14 Thread ericl
Github user ericl commented on the issue: https://github.com/apache/spark/pull/14690 Yes, I'll look at that today. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-14 Thread mallman
Github user mallman commented on the issue: https://github.com/apache/spark/pull/14690 I'm still working on the rebase. It's very complex—there are two other commits involved. >> 1. Do we need a workaround for ORC like we made for Parquet? > 1) yes @ericl

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-14 Thread davies
Github user davies commented on the issue: https://github.com/apache/spark/pull/14690 1) yes 2) it's OK for now. 3) could be done by follow up PR 4) should fix it if it's not hard --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-14 Thread mallman
Github user mallman commented on the issue: https://github.com/apache/spark/pull/14690 I will work on a rebase. Meanwhile, I've revisited the open issues in the PR description. To summarize: 1. Do we need a workaround for ORC like we made for Parquet? 1. What's the impact

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14690 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14690 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66938/ Test FAILed. ---

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14690 **[Test build #66938 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66938/consoleFull)** for PR 14690 at commit

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-13 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/14690 My main concern is that, the new `FileCatalog` class hierarchy is really complex now. Can we simplify it a little? or at least let's put a class hierarchy chart(or text explanation) in PR

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14690 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14690 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66931/ Test FAILed. ---

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14690 **[Test build #66931 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66931/consoleFull)** for PR 14690 at commit

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14690 **[Test build #66938 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66938/consoleFull)** for PR 14690 at commit

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14690 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66923/ Test FAILed. ---

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14690 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14690 **[Test build #66923 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66923/consoleFull)** for PR 14690 at commit

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14690 **[Test build #66931 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66931/consoleFull)** for PR 14690 at commit

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-13 Thread ericl
Github user ericl commented on the issue: https://github.com/apache/spark/pull/14690 https://github.com/VideoAmp/spark-public/pull/5/files fixes the jenkins failure and adds test for expected metadata io cost. --- If your project is set up for it, you can reply to this email and

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14690 **[Test build #66923 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66923/consoleFull)** for PR 14690 at commit

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14690 **[Test build #3339 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3339/consoleFull)** for PR 14690 at commit

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-13 Thread mallman
Github user mallman commented on the issue: https://github.com/apache/spark/pull/14690 > Oops there is a conflict now. NP. I'm working on the rebase. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14690 **[Test build #3339 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3339/consoleFull)** for PR 14690 at commit

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-13 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/14690 Oops there is a conflict now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-13 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/14690 The mima failure is just mima flaky. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14690 **[Test build #66912 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66912/consoleFull)** for PR 14690 at commit

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14690 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14690 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66912/ Test FAILed. ---

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14690 **[Test build #66912 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66912/consoleFull)** for PR 14690 at commit

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14690 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14690 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66906/ Test FAILed. ---

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14690 **[Test build #66906 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66906/consoleFull)** for PR 14690 at commit

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-13 Thread ericl
Github user ericl commented on the issue: https://github.com/apache/spark/pull/14690 > Ok. I'm looking into the org.apache.spark.sql.hive.ParquetSourceSuite failure. I think that was caused by the p => pQ change --- If your project is set up for it, you can reply to this

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-13 Thread mallman
Github user mallman commented on the issue: https://github.com/apache/spark/pull/14690 > Btw, I noticed that this suite was failing in jenkins only. > > [info] - partitioned pruned table reports only selected files *** FAILED *** (610 milliseconds) > > That's pretty

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14690 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66903/ Test FAILed. ---

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14690 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14690 **[Test build #66903 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66903/consoleFull)** for PR 14690 at commit

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14690 Build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14690 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66899/ Test FAILed. ---

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14690 **[Test build #66899 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66899/consoleFull)** for PR 14690 at commit

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14690 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66898/ Test FAILed. ---

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14690 Build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14690 **[Test build #66898 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66898/consoleFull)** for PR 14690 at commit

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14690 **[Test build #66906 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66906/consoleFull)** for PR 14690 at commit

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-13 Thread ericl
Github user ericl commented on the issue: https://github.com/apache/spark/pull/14690 MiMa is supposed to detect binary incompatible changes. Looking at the console output though, it seems like MiMa just crashed. Running `./dev/mima` locally worked though, so maybe it was some

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-13 Thread mallman
Github user mallman commented on the issue: https://github.com/apache/spark/pull/14690 > This patch fails MiMa tests. I've never seen this before. What does this mean? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14690 **[Test build #66903 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66903/consoleFull)** for PR 14690 at commit

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14690 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66900/ Test FAILed. ---

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14690 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14690 **[Test build #66900 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66900/consoleFull)** for PR 14690 at commit

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14690 **[Test build #66900 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66900/consoleFull)** for PR 14690 at commit

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-13 Thread mallman
Github user mallman commented on the issue: https://github.com/apache/spark/pull/14690 I've pushed an update to `ParquetMetastoreSuite` that illustrates the bug (or "limitation") WRT support for mixed-case partition columns I discovered yesterday. To reiterate, this is not a

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14690 **[Test build #66899 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66899/consoleFull)** for PR 14690 at commit

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14690 **[Test build #66898 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66898/consoleFull)** for PR 14690 at commit

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14690 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66861/ Test FAILed. ---

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14690 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14690 **[Test build #66861 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66861/consoleFull)** for PR 14690 at commit

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14690 **[Test build #66861 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66861/consoleFull)** for PR 14690 at commit

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-12 Thread ericl
Github user ericl commented on the issue: https://github.com/apache/spark/pull/14690 Conf flag here: https://github.com/VideoAmp/spark-public/pull/3 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14690 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66833/ Test FAILed. ---

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14690 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14690 **[Test build #66833 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66833/consoleFull)** for PR 14690 at commit

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-12 Thread mallman
Github user mallman commented on the issue: https://github.com/apache/spark/pull/14690 I determined the performance regression was introduced by a commit I hadn't pushed to this PR. Sorry for the false alarm. 😞 Needless to say, I'm not pushing that commit. --- If your project is

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14690 **[Test build #66833 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66833/consoleFull)** for PR 14690 at commit

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-12 Thread mallman
Github user mallman commented on the issue: https://github.com/apache/spark/pull/14690 >> Btw I've noticed a significant performance difference between ListingFileCatalog and TableFileCatalog's implementation of ListFiles. The difference seems to be that ListingFileCatalog

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-12 Thread mallman
Github user mallman commented on the issue: https://github.com/apache/spark/pull/14690 > Btw I've noticed a significant performance difference between ListingFileCatalog and TableFileCatalog's implementation of ListFiles. The difference seems to be that ListingFileCatalog

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-12 Thread ericl
Github user ericl commented on the issue: https://github.com/apache/spark/pull/14690 Btw I've noticed a significant performance difference between ListingFileCatalog and TableFileCatalog's implementation of ListFiles. The difference seems to be that ListingFileCatalog parallelizes

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-12 Thread mallman
Github user mallman commented on the issue: https://github.com/apache/spark/pull/14690 I'm testing this patch on a couple of tables internally with on the order of 10k partitions. Performance is much slower than it should be. I'm investigating. --- If your project is set up for it,

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-12 Thread davies
Github user davies commented on the issue: https://github.com/apache/spark/pull/14690 Workaround should be reasonable to unblock this PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14690 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66828/ Test FAILed. ---

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14690 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14690 **[Test build #66828 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66828/consoleFull)** for PR 14690 at commit

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14690 **[Test build #66828 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66828/consoleFull)** for PR 14690 at commit

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-12 Thread mallman
Github user mallman commented on the issue: https://github.com/apache/spark/pull/14690 I updated the description of this PR to reflect the workaround for the Hive/Parquet case-sensitivity issue. Do we need a similar workaround for ORC? --- If your project is set up for it,

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14690 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14690 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66772/ Test FAILed. ---

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-11 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14690 **[Test build #66772 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66772/consoleFull)** for PR 14690 at commit

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14690 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66764/ Test PASSed. ---

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14690 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-11 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14690 **[Test build #66764 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66764/consoleFull)** for PR 14690 at commit

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-11 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14690 **[Test build #66772 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66772/consoleFull)** for PR 14690 at commit

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-11 Thread ericl
Github user ericl commented on the issue: https://github.com/apache/spark/pull/14690 Btw I also made https://github.com/VideoAmp/spark-public/pull/2/files, to fix inputFiles. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-11 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14690 **[Test build #66764 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66764/consoleFull)** for PR 14690 at commit

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-11 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14690 **[Test build #66751 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66751/consoleFull)** for PR 14690 at commit

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14690 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14690 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66751/ Test FAILed. ---

[GitHub] spark issue #14690: [SPARK-16980][SQL] Load only catalog table partition met...

2016-10-11 Thread mallman
Github user mallman commented on the issue: https://github.com/apache/spark/pull/14690 >> Finally, this would require us to read the schema files. That's something I'm trying to avoid in this patch. > Not sure what you mean here, but the parquet change should be execution

  1   2   >