This may likely be TaskAttemptListenerImpTezDag used for communication between the AM and the task containers. There are no configuration knobs for it currently as it is always internal to a cluster.
If you don’t mind, can you file a jira and contribute a patch to add a clarifying log message to help other folks down the line? — Hitesh On Jun 17, 2015, at 9:45 AM, Johannes Zillmann <[email protected]> wrote: > Hey guys, > > i’m trying to hunt down some port usage from the tez AM. > Log looks like > > 2015-06-17 18:33:30,104 INFO [main] history.HistoryEventHandler: Initializing > HistoryEventHandler > 2015-06-17 18:33:30,110 INFO [main] impl.SimpleHistoryLoggingService: Log > file location for SimpleHistoryLoggingService not specified, defaulting to > containerLogDir=/Users/jz/Desktop/hadoop/hadoop-2.4.0/logs/userlogs/application_1434557883115_0003/container_1434557883115_0003_01_000001 > 2015-06-17 18:33:30,111 INFO [main] impl.SimpleHistoryLoggingService: > Initializing SimpleHistoryLoggingService, > logFileLocation=/Users/jz/Desktop/hadoop/hadoop-2.4.0/logs/userlogs/application_1434557883115_0003/container_1434557883115_0003_01_000001/history.txt.appattempt_1434557883115_0003_000001, > maxErrors=10 > 2015-06-17 18:33:30,111 INFO [main] recovery.RecoveryService: Initializing > RecoveryService > 2015-06-17 18:33:30,115 INFO [main] history.HistoryEventHandler: > [HISTORY][DAG:N/A][Event:APP_LAUNCHED]: > applicationId=application_1434557883115_0003, appSubmitTime=1434558806972, > launchTime=1434558808608 > 2015-06-17 18:33:30,128 INFO [main] history.HistoryEventHandler: > [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: > appAttemptId=appattempt_1434557883115_0003_000001, > appSubmitTime=1434558806972, launchTime=1434558808608 > 2015-06-17 18:33:30,132 INFO > [ServiceThread:org.apache.tez.dag.history.HistoryEventHandler] > history.HistoryEventHandler: Starting HistoryEventHandler > 2015-06-17 18:33:30,132 INFO > [ServiceThread:org.apache.tez.dag.history.HistoryEventHandler] > impl.SimpleHistoryLoggingService: Starting SimpleHistoryLoggingService > 2015-06-17 18:33:30,134 INFO > [ServiceThread:org.apache.tez.dag.app.launcher.ContainerLauncherImpl] > impl.ContainerManagementProtocolProxy: yarn.client.max-nodemanagers-proxies : > 500 > 2015-06-17 18:33:30,140 INFO > [ServiceThread:org.apache.tez.dag.app.TaskAttemptListenerImpTezDag] > ipc.CallQueueManager: Using callQueue class > java.util.concurrent.LinkedBlockingQueue > 2015-06-17 18:33:30,140 INFO [ServiceThread:DAGClientRPCServer] > ipc.CallQueueManager: Using callQueue class > java.util.concurrent.LinkedBlockingQueue > 2015-06-17 18:33:30,157 INFO > [ServiceThread:org.apache.tez.dag.history.HistoryEventHandler] > recovery.RecoveryService: Starting RecoveryService > 2015-06-17 18:33:30,157 INFO [HistoryEventHandlingThread] > impl.SimpleHistoryLoggingService: Writing event APP_LAUNCHED to history file > 2015-06-17 18:33:30,162 INFO [Socket Reader #1 for port 60875] ipc.Server: > Starting Socket Reader #1 for port 60875 > 2015-06-17 18:33:30,163 INFO [Socket Reader #1 for port 41111] ipc.Server: > Starting Socket Reader #1 for port 41111 > 2015-06-17 18:33:30,189 INFO [IPC Server Responder] ipc.Server: IPC Server > Responder: starting > 2015-06-17 18:33:30,189 INFO [IPC Server listener on 41111] ipc.Server: IPC > Server listener on 41111: starting > 2015-06-17 18:33:30,193 INFO [IPC Server Responder] ipc.Server: IPC Server > Responder: starting > 2015-06-17 18:33:30,193 INFO [IPC Server listener on 60875] ipc.Server: IPC > Server listener on 60875: starting > 2015-06-17 18:33:30,195 INFO [ServiceThread:DAGClientRPCServer] > client.DAGClientServer: Instantiated DAGClientRPCServer at > jzillmann.lan/192.168.178.28:41111 > > Port 41111 is controlled by tez.am.client.am.port-range , you’ve any idea > what service 60875 could be ? And if so is there a way to configure it ? > It does seem to vary for each AM, usually about 60xxx. > > Any ideas ?\ > Johannes
