2.x changes are usually compatible but still we have to test it a bit. The target in Hadoop dependencies is always the oldest because the full ecosystem is still tight to those e.g. Spark/Flink.
It would be awesome if we could get tests to be run with earlier versions too at least with 2.7.x the oldest target at the present. On Fri, Oct 30, 2020 at 12:21 PM Piotr Szuberski <[email protected]> wrote: > > Ok I see. I'll update Hadoop to the recent 2.10.1. > > On 2020/10/28 16:48:57, Tyson Hamilton <[email protected]> wrote: > > > > > > On 2020/10/22 07:35:47, Piotr Szuberski <[email protected]> wrote: > > > I'm trying to update Hadoop dependencies to the recent 3.3.0 and I've > > > encountered a problem - the Hadoop related checks seem to work without > > > any further changes but Hcatalog requires to be bumped to 3.x.y versions > > > as well (2.x.y versions require Hadoop 2.x.y). > > > > Are there any backwards incompatible changes with the Hadoop major version > > bump? If so it may be better to push this off until the next major Beam > > release and just to a minor version bump for Hadoop. > > > > > > > > When I use Hadoop 3.3.0 there is a guava jar versions related exception > > > [1] which I tried to solve by enforcing Guava 27.0-jre which is used by > > > Hadoop 3.3.0 - without success. > > > > > > Then I used Hadoop 3.2.0 which doesn't have guava updated and Hive 3.1.2. > > > I also replaced hive-site.xml with the recent default one from Hive's > > > master. Then 4 tests from io/hcatalog are failing: > > > testWriteThenReadSuccess - with exception [2] > > > testWriteThenUnboundedReadSuccess - with the same exception. > > > > > > As far as I deduced it's a bit misleading because setOutput indeed is > > > called in HCatalogIO.Write's writerContext = masterWriter.prepareWrite() > > > - which under the hood tries to call setOutput and fails. > > > > > > The probable cause could be Hcatalog configuration. But I definitely lack > > > knowledge how to set it up, especially the Hcatalog's version 3.x > > > documentation really doesn't help. > > > > > > Do we have anyone with some knowledge about HCatalog that could help me > > > with this? > > > > timrobertson100@ is listed as the owner of this dependency [1]. > > > > [1] > > https://github.com/apache/beam/blob/master/ownership/JAVA_DEPENDENCY_OWNERS.yaml > > > > > > > > > > > [1] NoSuchMethodError: > > > com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;J)V > > > [2] org.apache.beam.sdk.util.UserCodeException: > > > org.apache.hive.hcatalog.common.HCatException : 2004 : HCatOutputFormat > > > not initialized, setOutput has to be called. Cause : > > > org.apache.hive.hcatalog.common.HCatException : 2001 : Error setting > > > output information. Cause : java.lang.NullPointerException > > > > >
