Github user rajeshbalamohan commented on the issue:
https://github.com/apache/spark/pull/14537
Thanks @gatorsmile . Removed the changes related to OrcFileFormat
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14537
**[Test build #64449 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64449/consoleFull)**
for PR 14537 at commit
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/14537
You might forget this comment
https://github.com/apache/spark/pull/14537#discussion_r76189474
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14537
**[Test build #64446 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64446/consoleFull)**
for PR 14537 at commit
Github user rajeshbalamohan commented on the issue:
https://github.com/apache/spark/pull/14537
Fixed the test case name. I haven't changed the parquet code path as I
wasn't sure on whether it would break any backward compatibility.
---
If your project is set up for it, you can reply
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14537
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14537
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64399/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14537
**[Test build #64399 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64399/consoleFull)**
for PR 14537 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14537
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64397/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14537
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14537
**[Test build #64397 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64397/consoleFull)**
for PR 14537 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14537
**[Test build #64399 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64399/consoleFull)**
for PR 14537 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14537
**[Test build #64397 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64397/consoleFull)**
for PR 14537 at commit
Github user rajeshbalamohan commented on the issue:
https://github.com/apache/spark/pull/14537
Thanks @gatorsmile, it would be good to retain the change in
OrcFileInputFormat's inferschema (just in case it is referenced later).
---
If your project is set up for it, you can reply to
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14537
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14537
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64386/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14537
**[Test build #64386 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64386/consoleFull)**
for PR 14537 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14537
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14537
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64384/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14537
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64385/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14537
**[Test build #64384 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64384/consoleFull)**
for PR 14537 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14537
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14537
**[Test build #64385 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64385/consoleFull)**
for PR 14537 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14537
**[Test build #64386 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64386/consoleFull)**
for PR 14537 at commit
Github user rajeshbalamohan commented on the issue:
https://github.com/apache/spark/pull/14537
ok, reverted the changes related to physical schema changes. In both cases,
it returns metastoreschema, and mismatches can be handled separately.
---
If your project is set up for it, you
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14537
**[Test build #64385 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64385/consoleFull)**
for PR 14537 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14537
**[Test build #64384 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64384/consoleFull)**
for PR 14537 at commit
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/14537
I quickly checked the latest fixes. It is based on a few assumptions. For
example, the column order and length of metastore schema are the same as the
physical schema. I am not sure whether we
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14537
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14537
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64341/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14537
**[Test build #64341 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64341/consoleFull)**
for PR 14537 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14537
**[Test build #64341 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64341/consoleFull)**
for PR 14537 at commit
Github user rajeshbalamohan commented on the issue:
https://github.com/apache/spark/pull/14537
For non-partitioned ORC, it is currently using the metastore schema and is
not inferring the schema currently in HiveMetastoreCatalog, and hence not an
issue. But the problem of wrong
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/14537
Based on what you replied to @cloud-fan 's question, my follow-up question
is:
How about the non-partitioned empty ORC table?
---
If your project is set up for it, you can reply to
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14537
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14537
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64314/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14537
**[Test build #64314 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64314/consoleFull)**
for PR 14537 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14537
**[Test build #64314 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64314/consoleFull)**
for PR 14537 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14537
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14537
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64285/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14537
**[Test build #64285 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64285/consoleFull)**
for PR 14537 at commit
Github user rajeshbalamohan commented on the issue:
https://github.com/apache/spark/pull/14537
Thanks @gatorsmile. Addressed review comments
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14537
**[Test build #64285 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64285/consoleFull)**
for PR 14537 at commit
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/14537
uh, I missed this ping. Will review it tonight. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user rajeshbalamohan commented on the issue:
https://github.com/apache/spark/pull/14537
For latest ORC, if the data was written out by Hive, it would have the same
mapping.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/14537
@rajeshbalamohan So for Orc 2.x files, would schema inference be
unnecessary?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If
Github user rajeshbalamohan commented on the issue:
https://github.com/apache/spark/pull/14537
Right, for Parquet this could be part of initial codebase (from Spark-1251
I believe) which merges any metastore conflicts with parq files. But in the
case of ORC, this inference is still
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/14537
why do we infer schema for tables? Table schema should be persisted to
metastore when it was created.
---
If your project is set up for it, you can reply to this email and have your
reply appear
Github user rajeshbalamohan commented on the issue:
https://github.com/apache/spark/pull/14537
Thanks @rxin . Incorporated review comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14537
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14537
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64183/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14537
**[Test build #64183 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64183/consoleFull)**
for PR 14537 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14537
**[Test build #64183 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64183/consoleFull)**
for PR 14537 at commit
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/14537
cc @cloud-fan @gatorsmile can you also take a look at this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user rajeshbalamohan commented on the issue:
https://github.com/apache/spark/pull/14537
@rxin Can you please review when you find time?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user rajeshbalamohan commented on the issue:
https://github.com/apache/spark/pull/14537
Thank you thejas and @mallman
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/14537
@rajeshbalamohan We'll need a committer to review your patch.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user tejasapatil commented on the issue:
https://github.com/apache/spark/pull/14537
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the
Github user rajeshbalamohan commented on the issue:
https://github.com/apache/spark/pull/14537
@tejasapatil, @mallman - Can you please review when you find time?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14537
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63474/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14537
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14537
**[Test build #63474 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63474/consoleFull)**
for PR 14537 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14537
**[Test build #63474 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63474/consoleFull)**
for PR 14537 at commit
Github user rajeshbalamohan commented on the issue:
https://github.com/apache/spark/pull/14537
Thanks @mallman . Fixed review comments in latest commit.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14537
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63441/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14537
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14537
**[Test build #63441 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63441/consoleFull)**
for PR 14537 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14537
**[Test build #63441 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63441/consoleFull)**
for PR 14537 at commit
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/14537
@rajeshbalamohan, the changes to `HiveMetastoreCatalog.scala` look
reasonable. This mirrors the behavior of this method before the `if
(fileType.equals("parquet"))` expression was introduced in
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14537
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63352/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14537
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14537
**[Test build #63352 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63352/consoleFull)**
for PR 14537 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14537
**[Test build #63352 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63352/consoleFull)**
for PR 14537 at commit
73 matches
Mail list logo
