Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22197
thanks, merging to master!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22197
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22197
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95524/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22197
**[Test build #95524 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95524/testReport)**
for PR 22197 at commit
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/22197
One minor comment that can be addressed in a follow-up PR. LGTM.
---
-
To unsubscribe, e-mail:
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22197
**[Test build #95524 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95524/testReport)**
for PR 22197 at commit
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22197
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22197
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22197
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95517/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22197
**[Test build #95517 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95517/testReport)**
for PR 22197 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22197
**[Test build #95517 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95517/testReport)**
for PR 22197 at commit
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22197
Seems fine to me too.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user yucai commented on the issue:
https://github.com/apache/spark/pull/22197
@cloud-fan, tests have passed. And I will use a followup PR to make it
cleaner.
---
-
To unsubscribe, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22197
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22197
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95462/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22197
**[Test build #95462 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95462/testReport)**
for PR 22197 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22197
**[Test build #95462 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95462/testReport)**
for PR 22197 at commit
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22197
LGTM
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22197
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22197
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22197
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95456/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22197
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95455/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22197
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22197
**[Test build #95455 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95455/testReport)**
for PR 22197 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22197
**[Test build #95456 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95456/testReport)**
for PR 22197 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22197
**[Test build #95456 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95456/testReport)**
for PR 22197 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22197
**[Test build #95455 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95455/testReport)**
for PR 22197 at commit
Github user yucai commented on the issue:
https://github.com/apache/spark/pull/22197
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22197
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22197
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95449/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22197
**[Test build #95449 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95449/testReport)**
for PR 22197 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22197
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95454/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22197
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22197
**[Test build #95454 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95454/testReport)**
for PR 22197 at commit
Github user yucai commented on the issue:
https://github.com/apache/spark/pull/22197
@cloud-fan I reverted to the previous version.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user yucai commented on the issue:
https://github.com/apache/spark/pull/22197
@dongjoon-hyun Sorry for the late response, description is changed to:
> Although filter "ID < 100L" is generated by Spark, it fails to pushdown
into parquet actually, Spark still does the
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22197
**[Test build #95454 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95454/testReport)**
for PR 22197 at commit
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22197
> Is it acceptable?
apparently not...
OK let's just check duplicated filed names twice: one in filter pushdown,
one in column pruning. And clean it up in followup PRs.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22197
**[Test build #95449 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95449/testReport)**
for PR 22197 at commit
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/22197
```
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Duplicate
column name c1 in the table definition.
at
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22197
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95428/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22197
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22197
**[Test build #95428 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95428/testReport)**
for PR 22197 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22197
**[Test build #95428 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95428/testReport)**
for PR 22197 at commit
Github user yucai commented on the issue:
https://github.com/apache/spark/pull/22197
I will treat the above case as acceptable and will add a duplicated field
check for the parquet schema.
---
-
To unsubscribe,
Github user yucai commented on the issue:
https://github.com/apache/spark/pull/22197
Both `catalystRequestedSchema` and `parquetSchema` are recursive structure,
is there the easy way to find duplicated fields? Thanks!
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22197
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95410/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22197
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22197
**[Test build #95410 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95410/testReport)**
for PR 22197 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22197
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95409/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22197
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22197
**[Test build #95409 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95409/testReport)**
for PR 22197 at commit
Github user yucai commented on the issue:
https://github.com/apache/spark/pull/22197
@cloud-fan I also think my way changes too much in this PR.
> go through the parquet schema and find duplicated field names
If user query only query non-duplicated field, this way
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22197
I agree moving the `clipSchema` to the beginning is cleaner, but I feel it
goes too far away from our original target: making parquet filter pushdown case
insensitive when spark is case
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22197
**[Test build #95410 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95410/testReport)**
for PR 22197 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22197
**[Test build #95409 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95409/testReport)**
for PR 22197 at commit
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/22197
Thanks. I got it. Definitely, it's irrelevant to this and an intentional
regression due to that reverting.
---
-
To
Github user yucai commented on the issue:
https://github.com/apache/spark/pull/22197
@dongjoon-hyun In the **schema matched case** as you listed, it is expected
behavior in current master.
```
spark.sparkContext.hadoopConfiguration.setInt("parquet.block.size", 8 *
1024 *
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/22197
@gatorsmile . I don't think so we have this regression on ORC data source.
However, there was another JIRA report,
[SPARK-25175](https://issues.apache.org/jira/browse/SPARK-25175) 5 days
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/22197
@yucai . If you don't mind, could you update the PR description? This PR
doesn't generate new filters here. This only changes `field resolution` logic.
With and without this PR, there exists
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22197
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22197
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95264/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22197
**[Test build #95264 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95264/testReport)**
for PR 22197 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22197
**[Test build #95264 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95264/testReport)**
for PR 22197 at commit
Github user yucai commented on the issue:
https://github.com/apache/spark/pull/22197
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22197
**[Test build #95258 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95258/testReport)**
for PR 22197 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22197
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95258/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22197
**[Test build #95257 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95257/testReport)**
for PR 22197 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22197
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95257/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22197
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22197
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22197
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95256/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22197
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22197
**[Test build #95256 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95256/testReport)**
for PR 22197 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22197
**[Test build #95258 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95258/testReport)**
for PR 22197 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22197
**[Test build #95257 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95257/testReport)**
for PR 22197 at commit
Github user yucai commented on the issue:
https://github.com/apache/spark/pull/22197
@gatorsmile I can help check `spark.sql.caseSensitive` for all the built-in
data sources.
---
-
To unsubscribe, e-mail:
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/22197
This PR is basically trying to resolve case sensitivity when the logical
schema and physical schema do not match. This sounds like a general issue in
all the data sources. Could any of you do us
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/22197
@dongjoon-hyun Do you think we face the same issue in ORC?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22197
**[Test build #95256 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95256/testReport)**
for PR 22197 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22197
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22197
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95253/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22197
**[Test build #95253 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95253/testReport)**
for PR 22197 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22197
**[Test build #95253 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95253/testReport)**
for PR 22197 at commit
Github user yucai commented on the issue:
https://github.com/apache/spark/pull/22197
@cloud-fan @HyukjinKwon Seem cannot simply add `originalName` into
`ParquetSchemaType`.
Because we need exact ParquetSchemaType info for type match, like:
```
private val
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22197
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22197
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95247/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22197
**[Test build #95247 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95247/testReport)**
for PR 22197 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22197
**[Test build #95247 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95247/testReport)**
for PR 22197 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22197
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95149/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22197
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22197
**[Test build #95149 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95149/testReport)**
for PR 22197 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22197
**[Test build #95149 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95149/testReport)**
for PR 22197 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22197
**[Test build #95147 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95147/testReport)**
for PR 22197 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22197
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95147/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22197
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22197
**[Test build #95147 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95147/testReport)**
for PR 22197 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22197
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22197
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
99 matches
Mail list logo
