Okay! Together with the update to 2.10.1 I'll make the tests for 2.7.7 and later for 3.2.1.
On 2020/10/30 14:49:51, Ismaël Mejía <[email protected]> wrote: > 2.x changes are usually compatible but still we have to test it a bit. > > The target in Hadoop dependencies is always the oldest because the full > ecosystem is still tight to those e.g. Spark/Flink. > > It would be awesome if we could get tests to be run with earlier versions too > at > least with 2.7.x the oldest target at the present. > > On Fri, Oct 30, 2020 at 12:21 PM Piotr Szuberski > <[email protected]> wrote: > > > > Ok I see. I'll update Hadoop to the recent 2.10.1. > > > > On 2020/10/28 16:48:57, Tyson Hamilton <[email protected]> wrote: > > > > > > > > > On 2020/10/22 07:35:47, Piotr Szuberski <[email protected]> > > > wrote: > > > > I'm trying to update Hadoop dependencies to the recent 3.3.0 and I've > > > > encountered a problem - the Hadoop related checks seem to work without > > > > any further changes but Hcatalog requires to be bumped to 3.x.y > > > > versions as well (2.x.y versions require Hadoop 2.x.y). > > > > > > Are there any backwards incompatible changes with the Hadoop major > > > version bump? If so it may be better to push this off until the next > > > major Beam release and just to a minor version bump for Hadoop. > > > > > > > > > > > When I use Hadoop 3.3.0 there is a guava jar versions related exception > > > > [1] which I tried to solve by enforcing Guava 27.0-jre which is used by > > > > Hadoop 3.3.0 - without success. > > > > > > > > Then I used Hadoop 3.2.0 which doesn't have guava updated and Hive > > > > 3.1.2. I also replaced hive-site.xml with the recent default one from > > > > Hive's master. Then 4 tests from io/hcatalog are failing: > > > > testWriteThenReadSuccess - with exception [2] > > > > testWriteThenUnboundedReadSuccess - with the same exception. > > > > > > > > As far as I deduced it's a bit misleading because setOutput indeed is > > > > called in HCatalogIO.Write's writerContext = > > > > masterWriter.prepareWrite() - which under the hood tries to call > > > > setOutput and fails. > > > > > > > > The probable cause could be Hcatalog configuration. But I definitely > > > > lack knowledge how to set it up, especially the Hcatalog's version 3.x > > > > documentation really doesn't help. > > > > > > > > Do we have anyone with some knowledge about HCatalog that could help me > > > > with this? > > > > > > timrobertson100@ is listed as the owner of this dependency [1]. > > > > > > [1] > > > https://github.com/apache/beam/blob/master/ownership/JAVA_DEPENDENCY_OWNERS.yaml > > > > > > > > > > > > > > > [1] NoSuchMethodError: > > > > com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;J)V > > > > [2] org.apache.beam.sdk.util.UserCodeException: > > > > org.apache.hive.hcatalog.common.HCatException : 2004 : HCatOutputFormat > > > > not initialized, setOutput has to be called. Cause : > > > > org.apache.hive.hcatalog.common.HCatException : 2001 : Error setting > > > > output information. Cause : java.lang.NullPointerException > > > > > > > >
