Chen Xi created TEZ-3728:
----------------------------
Summary: When using HDFS federation, setting
tez.simple.history.logging.dir to viewfs:// causes AM to fail
Key: TEZ-3728
URL: https://issues.apache.org/jira/browse/TEZ-3728
Project: Apache Tez
Issue Type: Bug
Affects Versions: 0.8.5
Environment: hive1.1.0
hadoop2.6.0
Reporter: Chen Xi
tez.simple.history.logging.dir does not support viewfs schema.
In tez-site.xml we have:
{noformat}
<property>
<name>tez.simple.history.logging.dir</name>
<value>viewfs://clusterX/history-tez</value>
</property>
{noformat}
TezAM throws such exception:
{noformat}
2017-05-15 15:34:00,809 [INFO] [main] |impl.SimpleHistoryLoggingService|: Using
configured log file location for SimpleHistoryLoggingService
logDirPath=viewfs://clusterX/history-tez
2017-05-15 15:34:00,812 [INFO] [main] |impl.SimpleHistoryLoggingService|:
Initializing SimpleHistoryLoggingService,
logFileLocation=hdfs://ns2/history-tez/history.txt.appattempt_1494315174208_1108_000001,
maxErrors=10
2017-05-15 15:34:00,812 [INFO] [main] |recovery.RecoveryService|:
RecoveryService initialized with
recoveryPath=hdfs://ns2/tmp/hive/op/_tez_session_dir/6b76d2f7-2f00-4f0c-9a15-4ad8478cf32e/.tez/application_1494315174208_1108/recovery/1,
bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2017-05-15 15:34:00,814 [INFO] [main] |history.HistoryEventHandler|:
[HISTORY][DAG:N/A][Event:APP_LAUNCHED]:
applicationId=application_1494315174208_1108, appSubmitTime=1494833615647,
launchTime=1494833639649
2017-05-15 15:34:00,815 [INFO] [main] |history.HistoryEventHandler|:
[HISTORY][DAG:N/A][Event:AM_LAUNCHED]:
appAttemptId=appattempt_1494315174208_1108_000001, appSubmitTime=1494833615647,
launchTime=1494833639649
2017-05-15 15:34:00,818 [INFO]
[ServiceThread:org.apache.tez.dag.history.HistoryEventHandler]
|impl.SimpleHistoryLoggingService|: Starting SimpleHistoryLoggingService
2017-05-15 15:34:00,819 [INFO]
[ServiceThread:org.apache.tez.dag.history.HistoryEventHandler]
|service.AbstractService|: Service
org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService failed in
state STARTED; cause: java.lang.IllegalArgumentException: Wrong FS:
hdfs://ns2/history-tez/history.txt.appattempt_1494315174208_1108_000001,
expected: viewfs://clusterX/
java.lang.IllegalArgumentException: Wrong FS:
hdfs://ns2/history-tez/history.txt.appattempt_1494315174208_1108_000001,
expected: viewfs://clusterX/
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:645)
at
org.apache.hadoop.fs.viewfs.ViewFileSystem.getUriPath(ViewFileSystem.java:117)
at
org.apache.hadoop.fs.viewfs.ViewFileSystem.getDefaultReplication(ViewFileSystem.java:615)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:786)
at
org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService.serviceStart(SimpleHistoryLoggingService.java:96)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
at
org.apache.tez.dag.history.HistoryEventHandler.serviceStart(HistoryEventHandler.java:105)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.tez.dag.app.DAGAppMaster$ServiceWithDependency.start(DAGAppMaster.java:1838)
at
org.apache.tez.dag.app.DAGAppMaster$ServiceThread.run(DAGAppMaster.java:1859)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
