BryanCutler edited a comment on issue #27383: [SPARK-29367][DOC][2.4] Add compatibility note for Arrow 0.15.0 to SQL guide URL: https://github.com/apache/spark/pull/27383#issuecomment-579971851 >3.0.0 document is different because we start to use 0.15 as a minimum and that means it's a breaking change from 2.3.x and 2.4.x. 3.0 won't need this env var, so it could be take out of that doc but it still could be helpful for people to see it there too. There have already been a couple JIRAs and mailing list questions about this topic. >For 2.3.x and 2.4.x, we are still providing Arrow 0.10.0 by default, aren't we? For Java, yes. But users can upgrade pyarrow on their own, and then run into this problem. >This document aims to help the 2.3.x or 2.4.x users to use Arrow 0.15.x. So, I feel that it's a little weird to advertise 2.3.x. I guess this PR is specifically for branch 2.4, but if someone on Spark 2.3 hits this problem, it's not really documented anywhere so they might not know how to fix it. Is there a better place to put it for 2.3 users?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
