MartijnVisser commented on PR #21529: URL: https://github.com/apache/flink/pull/21529#issuecomment-1358263883
@luoyuxia I could use your help with this. This is a PR to validate that after the update of Hadoop from 2.8.5 to 2.10.2 everything would still work. As part of the upgrade for Hadoop 2, we've concluded that we also need to update our Hadoop 3 support from Hadoop 3.1.3 to Hadoop 3.2.3. This PR was just to check that everything would work if we would run the Hadoop 3 profile (via `-Dflink.hadoop.version=3.2.3 -Phadoop3-tests,hive3`). Everything works except that now the `HiveRunnerITCase` fails with: ``` 2022-12-19T19:42:44.1764995Z Dec 19 19:42:44 [ERROR] org.apache.flink.connectors.hive.HiveRunnerITCase.testOrcSchemaEvol Time elapsed: 0.882 s <<< ERROR! 2022-12-19T19:42:44.1766652Z Dec 19 19:42:44 java.lang.IllegalArgumentException: Failed to executeQuery Hive query insert into table db1.src values (1,100),(2,200): Error while processing statement: FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. org/apache/hadoop/metrics/Updater 2022-12-19T19:42:44.1767729Z Dec 19 19:42:44 at com.klarna.hiverunner.HiveServerContainer.executeStatement(HiveServerContainer.java:143) 2022-12-19T19:42:44.1768485Z Dec 19 19:42:44 at com.klarna.hiverunner.builder.HiveShellBase.executeStatementsWithCommandShellEmulation(HiveShellBase.java:121) 2022-12-19T19:42:44.1769312Z Dec 19 19:42:44 at com.klarna.hiverunner.builder.HiveShellBase.executeScriptWithCommandShellEmulation(HiveShellBase.java:110) 2022-12-19T19:42:44.1770263Z Dec 19 19:42:44 at com.klarna.hiverunner.builder.HiveShellBase.execute(HiveShellBase.java:129) 2022-12-19T19:42:44.1770974Z Dec 19 19:42:44 at org.apache.flink.connectors.hive.HiveRunnerITCase.testOrcSchemaEvol(HiveRunnerITCase.java:540) 2022-12-19T19:42:44.1771599Z Dec 19 19:42:44 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 2022-12-19T19:42:44.1772276Z Dec 19 19:42:44 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 2022-12-19T19:42:44.1773807Z Dec 19 19:42:44 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 2022-12-19T19:42:44.1775029Z Dec 19 19:42:44 at java.lang.reflect.Method.invoke(Method.java:498) 2022-12-19T19:42:44.1776298Z Dec 19 19:42:44 at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) 2022-12-19T19:42:44.1777597Z Dec 19 19:42:44 at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) 2022-12-19T19:42:44.1778841Z Dec 19 19:42:44 at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) 2022-12-19T19:42:44.1780098Z Dec 19 19:42:44 at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) 2022-12-19T19:42:44.1781377Z Dec 19 19:42:44 at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) 2022-12-19T19:42:44.1782663Z Dec 19 19:42:44 at org.apache.flink.connectors.hive.FlinkEmbeddedHiveRunner.runTestMethod(FlinkEmbeddedHiveRunner.java:128) 2022-12-19T19:42:44.1784165Z Dec 19 19:42:44 at org.apache.flink.connectors.hive.FlinkEmbeddedHiveRunner.runChild(FlinkEmbeddedHiveRunner.java:116) 2022-12-19T19:42:44.1785673Z Dec 19 19:42:44 at org.apache.flink.connectors.hive.FlinkEmbeddedHiveRunner.runChild(FlinkEmbeddedHiveRunner.java:71) 2022-12-19T19:42:44.1786903Z Dec 19 19:42:44 at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) 2022-12-19T19:42:44.1787956Z Dec 19 19:42:44 at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) 2022-12-19T19:42:44.1789050Z Dec 19 19:42:44 at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) 2022-12-19T19:42:44.1790152Z Dec 19 19:42:44 at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) 2022-12-19T19:42:44.1791230Z Dec 19 19:42:44 at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) 2022-12-19T19:42:44.1792405Z Dec 19 19:42:44 at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) 2022-12-19T19:42:44.1793627Z Dec 19 19:42:44 at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) 2022-12-19T19:42:44.1794944Z Dec 19 19:42:44 at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54) 2022-12-19T19:42:44.1796202Z Dec 19 19:42:44 at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54) 2022-12-19T19:42:44.1797250Z Dec 19 19:42:44 at org.junit.rules.RunRules.evaluate(RunRules.java:20) 2022-12-19T19:42:44.1797836Z Dec 19 19:42:44 at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) 2022-12-19T19:42:44.1798432Z Dec 19 19:42:44 at org.junit.runners.ParentRunner.run(ParentRunner.java:413) 2022-12-19T19:42:44.1799097Z Dec 19 19:42:44 at org.junit.runner.JUnitCore.run(JUnitCore.java:137) 2022-12-19T19:42:44.1799624Z Dec 19 19:42:44 at org.junit.runner.JUnitCore.run(JUnitCore.java:115) 2022-12-19T19:42:44.1800303Z Dec 19 19:42:44 at org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:42) 2022-12-19T19:42:44.1801066Z Dec 19 19:42:44 at org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:80) 2022-12-19T19:42:44.1801849Z Dec 19 19:42:44 at org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:72) 2022-12-19T19:42:44.1802653Z Dec 19 19:42:44 at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:147) 2022-12-19T19:42:44.1803493Z Dec 19 19:42:44 at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:127) 2022-12-19T19:42:44.1804509Z Dec 19 19:42:44 at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:90) 2022-12-19T19:42:44.1805313Z Dec 19 19:42:44 at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:55) 2022-12-19T19:42:44.1806355Z Dec 19 19:42:44 at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:102) 2022-12-19T19:42:44.1807456Z Dec 19 19:42:44 at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:54) 2022-12-19T19:42:44.1808297Z Dec 19 19:42:44 at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114) 2022-12-19T19:42:44.1809110Z Dec 19 19:42:44 at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86) 2022-12-19T19:42:44.1810021Z Dec 19 19:42:44 at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86) 2022-12-19T19:42:44.1811428Z Dec 19 19:42:44 at org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:53) 2022-12-19T19:42:44.1812854Z Dec 19 19:42:44 at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.execute(JUnitPlatformProvider.java:188) 2022-12-19T19:42:44.1814427Z Dec 19 19:42:44 at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:154) 2022-12-19T19:42:44.1815989Z Dec 19 19:42:44 at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:124) 2022-12-19T19:42:44.1817292Z Dec 19 19:42:44 at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:428) 2022-12-19T19:42:44.1818582Z Dec 19 19:42:44 at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162) 2022-12-19T19:42:44.1819786Z Dec 19 19:42:44 at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:562) 2022-12-19T19:42:44.1820961Z Dec 19 19:42:44 at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:548) 2022-12-19T19:42:44.1823519Z Dec 19 19:42:44 Caused by: org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. org/apache/hadoop/metrics/Updater 2022-12-19T19:42:44.1825352Z Dec 19 19:42:44 at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:335) 2022-12-19T19:42:44.1826754Z Dec 19 19:42:44 at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:226) 2022-12-19T19:42:44.1828034Z Dec 19 19:42:44 at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:263) 2022-12-19T19:42:44.1829540Z Dec 19 19:42:44 at org.apache.hive.service.cli.operation.Operation.run(Operation.java:247) 2022-12-19T19:42:44.1830869Z Dec 19 19:42:44 at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:541) 2022-12-19T19:42:44.1832258Z Dec 19 19:42:44 at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:510) 2022-12-19T19:42:44.1833561Z Dec 19 19:42:44 at org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:267) 2022-12-19T19:42:44.1834910Z Dec 19 19:42:44 at com.klarna.hiverunner.HiveServerContainer.executeStatement(HiveServerContainer.java:116) 2022-12-19T19:42:44.1835988Z Dec 19 19:42:44 ... 50 more 2022-12-19T19:42:44.1836801Z Dec 19 19:42:44 Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/metrics/Updater 2022-12-19T19:42:44.1837771Z Dec 19 19:42:44 at java.lang.ClassLoader.defineClass1(Native Method) 2022-12-19T19:42:44.1838648Z Dec 19 19:42:44 at java.lang.ClassLoader.defineClass(ClassLoader.java:756) 2022-12-19T19:42:44.1839968Z Dec 19 19:42:44 at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) 2022-12-19T19:42:44.1841033Z Dec 19 19:42:44 at java.net.URLClassLoader.defineClass(URLClassLoader.java:468) 2022-12-19T19:42:44.1842108Z Dec 19 19:42:44 at java.net.URLClassLoader.access$100(URLClassLoader.java:74) 2022-12-19T19:42:44.1843164Z Dec 19 19:42:44 at java.net.URLClassLoader$1.run(URLClassLoader.java:369) 2022-12-19T19:42:44.1844223Z Dec 19 19:42:44 at java.net.URLClassLoader$1.run(URLClassLoader.java:363) 2022-12-19T19:42:44.1845243Z Dec 19 19:42:44 at java.security.AccessController.doPrivileged(Native Method) 2022-12-19T19:42:44.1846522Z Dec 19 19:42:44 at java.net.URLClassLoader.findClass(URLClassLoader.java:362) 2022-12-19T19:42:44.1847535Z Dec 19 19:42:44 at java.lang.ClassLoader.loadClass(ClassLoader.java:418) 2022-12-19T19:42:44.1848594Z Dec 19 19:42:44 at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) 2022-12-19T19:42:44.1849658Z Dec 19 19:42:44 at java.lang.ClassLoader.loadClass(ClassLoader.java:351) 2022-12-19T19:42:44.1850679Z Dec 19 19:42:44 at org.apache.hadoop.mapred.LocalJobRunner.<init>(LocalJobRunner.java:714) 2022-12-19T19:42:44.1851327Z Dec 19 19:42:44 at org.apache.hadoop.mapred.LocalJobRunner.<init>(LocalJobRunner.java:707) 2022-12-19T19:42:44.1852026Z Dec 19 19:42:44 at org.apache.hadoop.mapred.LocalClientProtocolProvider.create(LocalClientProtocolProvider.java:42) 2022-12-19T19:42:44.1852687Z Dec 19 19:42:44 at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:130) 2022-12-19T19:42:44.1853281Z Dec 19 19:42:44 at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:109) 2022-12-19T19:42:44.1853870Z Dec 19 19:42:44 at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:102) 2022-12-19T19:42:44.1854495Z Dec 19 19:42:44 at org.apache.hadoop.mapred.JobClient.init(JobClient.java:475) 2022-12-19T19:42:44.1855079Z Dec 19 19:42:44 at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:454) 2022-12-19T19:42:44.1855858Z Dec 19 19:42:44 at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:381) 2022-12-19T19:42:44.1856496Z Dec 19 19:42:44 at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:149) 2022-12-19T19:42:44.1857117Z Dec 19 19:42:44 at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205) 2022-12-19T19:42:44.1857741Z Dec 19 19:42:44 at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97) 2022-12-19T19:42:44.1858344Z Dec 19 19:42:44 at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2664) 2022-12-19T19:42:44.1858930Z Dec 19 19:42:44 at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2335) 2022-12-19T19:42:44.1859522Z Dec 19 19:42:44 at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2011) 2022-12-19T19:42:44.1860081Z Dec 19 19:42:44 at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1709) 2022-12-19T19:42:44.1860640Z Dec 19 19:42:44 at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1703) 2022-12-19T19:42:44.1861377Z Dec 19 19:42:44 at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157) 2022-12-19T19:42:44.1862026Z Dec 19 19:42:44 at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:224) 2022-12-19T19:42:44.1862534Z Dec 19 19:42:44 ... 56 more 2022-12-19T19:42:44.1862993Z Dec 19 19:42:44 Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.metrics.Updater 2022-12-19T19:42:44.1863559Z Dec 19 19:42:44 at java.net.URLClassLoader.findClass(URLClassLoader.java:382) 2022-12-19T19:42:44.1864194Z Dec 19 19:42:44 at java.lang.ClassLoader.loadClass(ClassLoader.java:418) 2022-12-19T19:42:44.1864767Z Dec 19 19:42:44 at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) 2022-12-19T19:42:44.1865328Z Dec 19 19:42:44 at java.lang.ClassLoader.loadClass(ClassLoader.java:351) 2022-12-19T19:42:44.1866495Z Dec 19 19:42:44 ... 87 more ``` Any idea what might cause that and more important, how I could fix it? Thanks in advance. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
