dongjoon-hyun commented on issue #26133: [SPARK-29376][SQL][PYTHON] Upgrade Apache Arrow to version 0.15.1 URL: https://github.com/apache/spark/pull/26133#issuecomment-554656715 Hi, @shaneknapp , @felixcheung , @shivaram , @BryanCutler , @HyukjinKwon , @srowen . It turns out that our Jenkins jobs (including PR builder) ignores Arrow test. Arrow 0.15 has a breaking API changes at [ARROW-5505](https://issues.apache.org/jira/browse/ARROW-5505), and AppVeyor was the only having SparkR Arrow tests but it's broken now. **Jenkins** ``` Skipped ------------------------------------------------------------------------ 1. createDataFrame/collect Arrow optimization (@test_sparkSQL_arrow.R#25) - arrow not installed 2. createDataFrame/collect Arrow optimization - many partitions (partition order test) (@test_sparkSQL_arrow.R#48) - arrow not installed 3. createDataFrame/collect Arrow optimization - type specification (@test_sparkSQL_arrow.R#64) - arrow not installed 4. dapply() Arrow optimization (@test_sparkSQL_arrow.R#94) - arrow not installed 5. dapply() Arrow optimization - type specification (@test_sparkSQL_arrow.R#134) - arrow not installed 6. dapply() Arrow optimization - type specification (date and timestamp) (@test_sparkSQL_arrow.R#169) - arrow not installed 7. gapply() Arrow optimization (@test_sparkSQL_arrow.R#188) - arrow not installed 8. gapply() Arrow optimization - type specification (@test_sparkSQL_arrow.R#237) - arrow not installed 9. gapply() Arrow optimization - type specification (date and timestamp) (@test_sparkSQL_arrow.R#276) - arrow not installed 10. Arrow optimization - unsupported types (@test_sparkSQL_arrow.R#297) - arrow not installed 11. sparkJars tag in SparkContext (@test_Windows.R#22) - This test is only for Windows, skipped ``` For AppVeyor, currently, it failed with OOM due to protocol mismatch between versions. After I matched the version, we hit ARROW-5505 finally. In short, SparkR Arrow integration is broken now. We need to update it with new Arrow R interface.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
