PS you are correct; with Guava 27 and my recent changes, and Hadoop
3.2.1 + Hive 2.3, I still see ...
*** RUN ABORTED ***
java.lang.IllegalAccessError: tried to access method
com.google.common.collect.Iterators.emptyIterator()Lcom/google/common/collect/UnmodifiableIterator;
from class
Yeah that won't be the last problem I bet. Here's a proposal for just
directly reducing exposure to Guava in Spark itself though:
https://github.com/apache/spark/pull/26911
On Mon, Dec 16, 2019 at 11:36 AM Marcelo Vanzin wrote:
>
> Great that Hadoop has done it (which, btw, probably means that
Great that Hadoop has done it (which, btw, probably means that Spark
won't work with that version of Hadoop yet), but Hive also depends on
Guava, and last time I tried, even Hive 3.x did not work with Guava
27.
(Newer Hadoop versions also have a new artifact that shades a lot of
dependencies,
See for example:
https://github.com/apache/spark/pull/25932#issuecomment-565822573
https://issues.apache.org/jira/browse/SPARK-23897
This is a dicey dependency that we have been reluctant to update as a)
Hadoop used an old version and b) Guava versions are incompatible
after a few releases.
But
