Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/20580
LGTM.
Since this is a regression + blocker of Spark 2.3 release, I am merging it
now. Please submit a follow-up PR to add the tests. Thanks!
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20580
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20580
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87335/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20580
**[Test build #87335 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87335/testReport)**
for PR 20580 at commit
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/20580
You don't need to generate the parquet file manually, just write a parquet
file using Spark and read it back. We can probably add this test in
`ParquetFileFormatSuite`.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20580
**[Test build #87335 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87335/testReport)**
for PR 20580 at commit
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/20580
ok to test
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user jamesthomp commented on the issue:
https://github.com/apache/spark/pull/20580
I'll see if I can generate a parquet file with the right schema to add for
a test, but probably cannot look at this till tomorrow.
---
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/20580
It was an accident, thanks for the fix!
Can we add a test? It's always good to have a test for a bug fix, even the
bug was introduced recently.
---
Github user a10y commented on the issue:
https://github.com/apache/spark/pull/20580
If you did add a test it should probably generate the Parquet file
programmatically rather than checking it in. Some examples in
Github user jamesthomp commented on the issue:
https://github.com/apache/spark/pull/20580
@kiszk - I've changed the implementation to no longer use
`column.isArray()` and instead just inline the decimal type check (so no
changes needed to the public api). I don't think you could
Github user a10y commented on the issue:
https://github.com/apache/spark/pull/20580
As far as we can tell this is an accidental breaking change, as dropping
support for this in vectorized Parquet reader was never called out. We have
Parquet datasets with binary columns with logical
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/20580
@cloud-fan Is there any reason that the above PR removed to support some
types such as `Array`?
---
-
To unsubscribe, e-mail:
Github user jamesthomp commented on the issue:
https://github.com/apache/spark/pull/20580
Yeah I believe it will add support for the array type too. Spark actually
previously supported these types but the support was removed in this PR:
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/20580
IIUC, does this PR support array type, too?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20580
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20580
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
17 matches
Mail list logo
