[
https://issues.apache.org/jira/browse/TEZ-3884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16371812#comment-16371812
]
Thai Bui edited comment on TEZ-3884 at 2/21/18 6:49 PM:
--------------------------------------------------------
[~ewohlstadter] I was working on getting tez 0.9.1 + Hive 3.x to use hadoop
3.0.0 and encountered a dependency conflict between guava 11.x and hadoop 3
guava 19. I suppose we should upgrade Guava as well?
Here's the dependency conflict detail causing DAGAppMaster to shutdown
immediately after it had started
{noformat}
2018-02-21 06:51:00,829 [FATAL]
[org.apache.hadoop.hive.common.JvmPauseMonitor$Monitor@3cec92b6]
|yarn.YarnUncaughtExceptionHandler|: Thread
Thread[org.apache.hadoop.hive.common.JvmPauseMonitor$Monitor@3cec92b6,5,main]
threw an Error. Shutting down now...
java.lang.NoSuchMethodError:
com.google.common.base.Stopwatch.createUnstarted()Lcom/google/common/base/Stopwatch;
at
org.apache.hadoop.hive.common.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:176)
at java.lang.Thread.run(Thread.java:748)
2018-02-21 06:51:00,831 [DEBUG]
[org.apache.hadoop.hive.common.JvmPauseMonitor$Monitor@3cec92b6]
|util.ExitUtil|: Exiting with status -1: ExitException
-1: ExitException
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:304)
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:292)
at
org.apache.hadoop.yarn.YarnUncaughtExceptionHandler.uncaughtException(YarnUncaughtExceptionHandler.java:65)
at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1057)
at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1052)
at java.lang.Thread.dispatchUncaughtException(Thread.java:1959)
2018-02-21 06:51:00,831 [INFO]
[org.apache.hadoop.hive.common.JvmPauseMonitor$Monitor@3cec92b6]
|util.ExitUtil|: Exiting with status -1: ExitException
2018-02-21 06:51:00,832 [DEBUG]
[ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager]
|impl.MetricsSystemImpl|: JvmMetrics, JVM related metrics etc.
2018-02-21 06:51:00,833 [DEBUG]
[ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager]
|impl.MetricsConfig|: poking parent 'PropertiesConfiguration' for key:
source.source.start_mbeans
2018-02-21 06:51:00,833 [DEBUG]
[ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager]
|impl.MetricsConfig|: poking parent 'MetricsConfig' for key: source.start_mbeans
2018-02-21 06:51:00,833 [DEBUG]
[ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager]
|impl.MetricsConfig|: poking parent 'PropertiesConfiguration' for key:
*.source.start_mbeans
2018-02-21 06:51:00,833 [INFO] [pool-1-thread-1] |app.DAGAppMaster|:
DAGAppMasterShutdownHook invoked
2018-02-21 06:51:00,833 [INFO] [pool-1-thread-1] |app.DAGAppMaster|:
DAGAppMaster received a signal. Signaling TaskScheduler
2018-02-21 06:51:00,833 [INFO] [pool-1-thread-1] |rm.TaskSchedulerManager|:
TaskScheduler notified that iSignalled was : true{noformat}
Noticed that DAGAppMaster received a KILL signal due to Guava dependency
conflict in JvmPauseMonitor. This caused the DAGAppMaster to shutdown
immediately after it had started
{noformat}
2018-02-21 06:51:36,280 [DEBUG] [main] |service.AbstractService|: Service
org.apache.tez.dag.app.DAGAppMaster is started
2018-02-21 06:51:36,280 [DEBUG] [pool-1-thread-1] |service.AbstractService|:
Service: org.apache.tez.dag.app.DAGAppMaster entered state STOPPED
2018-02-21 06:51:36,280 [DEBUG] [IPC Parameter Sending Thread #0] |ipc.Client|:
IPC Client (484199463) connection to ip-10-8-118-171/10.8.118.171:18030 from
root sending #11 org.apache.hadoop.yarn.api.ApplicationMasterProtocolPB.allocate
2018-02-21 06:51:36,280 [DEBUG] [IPC Client (484199463) connection to
ip-10-8-118-171/10.8.118.171:18030 from root] |ipc.Client|: IPC Client
(484199463) connection to ip-10-8-118-171/10.8.118.171:18030 from root:
starting, having connections 1
2018-02-21 06:51:36,280 [INFO] [pool-1-thread-1] |rm.YarnTaskSchedulerService|:
Initiating stop of YarnTaskScheduler
2018-02-21 06:51:36,281 [INFO] [pool-1-thread-1] |rm.YarnTaskSchedulerService|:
Releasing held containers
2018-02-21 06:51:36,281 [INFO] [pool-1-thread-1] |rm.YarnTaskSchedulerService|:
Removing all pending taskRequests{noformat}
I'm working on upgrading Guava and perhaps various other issue and will keep it
posted here. Let me know what you guys think. Thanks!
was (Author: thai.bui):
[~ewohlstadter] I was working on getting tez 0.9.1 + Hive 3.x to use hadoop
3.0.0 and encountered a dependency conflict between guava 11.x and hadoop 3
guava 19. I supposed we should upgrade Guava as well?
Here's the dependency conflict detail causing DAGAppMaster to shutdown
immediately after it had started
{noformat}
2018-02-21 06:51:00,829 [FATAL]
[org.apache.hadoop.hive.common.JvmPauseMonitor$Monitor@3cec92b6]
|yarn.YarnUncaughtExceptionHandler|: Thread
Thread[org.apache.hadoop.hive.common.JvmPauseMonitor$Monitor@3cec92b6,5,main]
threw an Error. Shutting down now...
java.lang.NoSuchMethodError:
com.google.common.base.Stopwatch.createUnstarted()Lcom/google/common/base/Stopwatch;
at
org.apache.hadoop.hive.common.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:176)
at java.lang.Thread.run(Thread.java:748)
2018-02-21 06:51:00,831 [DEBUG]
[org.apache.hadoop.hive.common.JvmPauseMonitor$Monitor@3cec92b6]
|util.ExitUtil|: Exiting with status -1: ExitException
-1: ExitException
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:304)
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:292)
at
org.apache.hadoop.yarn.YarnUncaughtExceptionHandler.uncaughtException(YarnUncaughtExceptionHandler.java:65)
at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1057)
at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1052)
at java.lang.Thread.dispatchUncaughtException(Thread.java:1959)
2018-02-21 06:51:00,831 [INFO]
[org.apache.hadoop.hive.common.JvmPauseMonitor$Monitor@3cec92b6]
|util.ExitUtil|: Exiting with status -1: ExitException
2018-02-21 06:51:00,832 [DEBUG]
[ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager]
|impl.MetricsSystemImpl|: JvmMetrics, JVM related metrics etc.
2018-02-21 06:51:00,833 [DEBUG]
[ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager]
|impl.MetricsConfig|: poking parent 'PropertiesConfiguration' for key:
source.source.start_mbeans
2018-02-21 06:51:00,833 [DEBUG]
[ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager]
|impl.MetricsConfig|: poking parent 'MetricsConfig' for key: source.start_mbeans
2018-02-21 06:51:00,833 [DEBUG]
[ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager]
|impl.MetricsConfig|: poking parent 'PropertiesConfiguration' for key:
*.source.start_mbeans
2018-02-21 06:51:00,833 [INFO] [pool-1-thread-1] |app.DAGAppMaster|:
DAGAppMasterShutdownHook invoked
2018-02-21 06:51:00,833 [INFO] [pool-1-thread-1] |app.DAGAppMaster|:
DAGAppMaster received a signal. Signaling TaskScheduler
2018-02-21 06:51:00,833 [INFO] [pool-1-thread-1] |rm.TaskSchedulerManager|:
TaskScheduler notified that iSignalled was : true{noformat}
Noticed that DAGAppMaster received a KILL signal due to Guava dependency
conflict in JvmPauseMonitor. This caused the DAGAppMaster to shutdown
immediately after it had started
{noformat}
2018-02-21 06:51:36,280 [DEBUG] [main] |service.AbstractService|: Service
org.apache.tez.dag.app.DAGAppMaster is started
2018-02-21 06:51:36,280 [DEBUG] [pool-1-thread-1] |service.AbstractService|:
Service: org.apache.tez.dag.app.DAGAppMaster entered state STOPPED
2018-02-21 06:51:36,280 [DEBUG] [IPC Parameter Sending Thread #0] |ipc.Client|:
IPC Client (484199463) connection to ip-10-8-118-171/10.8.118.171:18030 from
root sending #11 org.apache.hadoop.yarn.api.ApplicationMasterProtocolPB.allocate
2018-02-21 06:51:36,280 [DEBUG] [IPC Client (484199463) connection to
ip-10-8-118-171/10.8.118.171:18030 from root] |ipc.Client|: IPC Client
(484199463) connection to ip-10-8-118-171/10.8.118.171:18030 from root:
starting, having connections 1
2018-02-21 06:51:36,280 [INFO] [pool-1-thread-1] |rm.YarnTaskSchedulerService|:
Initiating stop of YarnTaskScheduler
2018-02-21 06:51:36,281 [INFO] [pool-1-thread-1] |rm.YarnTaskSchedulerService|:
Releasing held containers
2018-02-21 06:51:36,281 [INFO] [pool-1-thread-1] |rm.YarnTaskSchedulerService|:
Removing all pending taskRequests{noformat}
I'm working on upgrading Guava and perhaps various other issue and will keep it
posted here. Let me know what you guys think. Thanks!
> Hadoop3-beta1 fixes for Tez tests
> ---------------------------------
>
> Key: TEZ-3884
> URL: https://issues.apache.org/jira/browse/TEZ-3884
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.9.1
> Reporter: Gopal V
> Priority: Minor
> Attachments: TEZ-3884.patch.1
>
>
> {code}
> [ERROR]
> /grid/5/dev/gopalv/llap-autobuild/tez/tez-api/src/test/java/org/apache/tez/client/TestTezClientUtils.java:[48,30]
> cannot find symbol
> [ERROR] symbol: class DistributedFileSystem
> [ERROR] location: package org.apache.hadoop.hdfs
> [ERROR]
> /grid/5/dev/gopalv/llap-autobuild/tez/tez-api/src/test/java/org/apache/tez/client/TestTezClientUtils.java:[680,50]
> cannot find symbol
> [ERROR] symbol: class DistributedFileSystem
> [ERROR] location: class org.apache.tez.client.TestTezClientUtils
> [ERROR]
> /grid/5/dev/gopalv/llap-autobuild/tez/tez-api/src/test/java/org/apache/tez/common/TestTezCommonUtils.java:[62,42]
> cannot access org.apache.hadoop.hdfs.DistributedFileSystem
> [ERROR] class file for org.apache.hadoop.hdfs.DistributedFileSystem not found
> [ERROR] -> [Help 1]
> [ERROR]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
