Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14365
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14365
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70336/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14365
**[Test build #70336 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70336/testReport)**
for PR 14365 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14365
**[Test build #70336 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70336/testReport)**
for PR 14365 at commit
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/14365
ping @cloud-fan @yhuai May you take a look? Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/14365
It looks good to me. Thank you for updating, @viirya !
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14365
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69856/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14365
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14365
**[Test build #69856 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69856/consoleFull)**
for PR 14365 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14365
**[Test build #69856 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69856/consoleFull)**
for PR 14365 at commit
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/14365
retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so,
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14365
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14365
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69852/
Test FAILed.
---
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/14365
@cloud-fan @yhuai @dongjoon-hyun I've updated this as:
* Assume metastore schema matches with physical Orc schema by column,
disregarding column names.
* Mapping required schema to
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14365
**[Test build #69852 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69852/consoleFull)**
for PR 14365 at commit
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/14365
We have two options. First one is to map metastore schema to physical Orc
schema like this. But we don't infer physical schema of Orc file now. I will
update this to have this mapping in
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/14365
@dongjoon-hyun yeah, I see. Because we directly use metastore schema of
converted Orc table, when the physical schema in Orc file and metastore schema
mismatch, this issue happens.
---
If your
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/14365
It seems to exist some cases like the following (on the current master).
On hive,
```sql
CREATE TABLE t1 (a string) PARTITIONED BY (b string) STORED AS ORC;
INSERT INTO
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/14365
As we replace schema inferring with metastore schema completely by #14690
for converted Hive tables, we may not have the chance to detect the mismatch
between Orc file's physical schema and
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/14365
Hi, @viirya , @cloud-fan , @yhuai .
#14690 and the parent issue SPARK-17861 seems to be resolved?
What about restarting this issue?
---
If your project is set up for it, you can reply
Github user yhuai commented on the issue:
https://github.com/apache/spark/pull/14365
@cloud-fan actually, this conversion was disabled because of this bug.
btw, pr that @cloud-fan mentioned is
https://github.com/apache/spark/pull/14690.
I think it is better to hold
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/14365
I noticed that `spark.sql.hive.convertMetastoreOrc` is false by default, so
this bug is not so critical.
And we may stop inferring file schema when reading ORC tables, for
performance
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/14365
ping @cloud-fan Can you have a decision about this? Are we going to have
complex logic for this issue? Or just disable it?
---
If your project is set up for it, you can reply to this email and have
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/14365
@cloud-fan So do we have decision on this? A simpler approach to disable
the conversion if the schema is inconsistent, or a complex one to work around
this Hive bug?
---
If your project is set up
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14365
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63214/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14365
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14365
**[Test build #63214 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63214/consoleFull)**
for PR 14365 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14365
**[Test build #63214 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63214/consoleFull)**
for PR 14365 at commit
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/14365
@cloud-fan I've submitted a PR #14282 previously that disables the
conversion if the schema is inconsistent.
---
If your project is set up for it, you can reply to this email and have your
reply
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/14365
@rxin @yhuai , is it worth adding this complex logic to work around this
hive bug? This feature(convert Metastore Orc tables to data source table) is
not a major feature and the broken case seems
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/14365
cc @yhuai @liancheng @rxin
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14365
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62993/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14365
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14365
**[Test build #62993 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62993/consoleFull)**
for PR 14365 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14365
**[Test build #62993 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62993/consoleFull)**
for PR 14365 at commit
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/14365
cc @cloud-fan @yhuai @liancheng @rxin Please review this change. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14365
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14365
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62880/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14365
**[Test build #62880 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62880/consoleFull)**
for PR 14365 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14365
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14365
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62875/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14365
**[Test build #62875 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62875/consoleFull)**
for PR 14365 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14365
**[Test build #62880 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62880/consoleFull)**
for PR 14365 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14365
**[Test build #62875 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62875/consoleFull)**
for PR 14365 at commit
44 matches
Mail list logo
