[
https://issues.apache.org/jira/browse/TEZ-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated TEZ-4737:
--------------------------------
Description:
The serialized DAG plan file (tez-dag.pb<counter>) written for plans exceeding
the IPC limit is named by a per-TezClient counter and never deleted. When the
session and its staging directory outlive the TezClient instances, a new client
generation recomputes the same file name and fails with
FileAlreadyExistsException, permanently blocking oversized submissions on that
session.
{code:java}
org.apache.hadoop.fs.FileAlreadyExistsException: Operation failed: "The
specified path already exists.", 409, PUT,
.../.tez/application_xxx_yyyy/tez-dag.pb1?resource=file
at
org.apache.tez.client.TezClient.submitDAGSession(TezClient.java:700) {code}
was:
The serialized DAG plan file (tez-dag.pb<counter>) written for plans exceeding
the IPC limit is named by a per-TezClient counter and never deleted. When the
session and its staging directory outlive the TezClient instances, a new client
generation recomputes the same file name and fails with
FileAlreadyExistsException, permanently blocking oversized submissions on that
session.
{code}
"The specified path already exists.
RequestId:469932fb-401f-00c4-0734-0df45e000000
Time:2026-07-06T10:46:55.2828655Z" stack trace:
[org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.checkException(AzureBlobFileSystem.java:1501),
org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.create(AzureBlobFileSystem.java:341),
org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1179),
org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1159),
org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1048),
org.apache.tez.client.TezClient.submitDAGSession(TezClient.java:700),
org.apache.tez.client.TezClient.submitDAG(TezClient.java:616),
org.apache.hadoop.hive.ql.exec.tez.TezTask.submitInternal(TezTask.java:708),
org.apache.hadoop.hive.ql.exec.tez.TezTask.submit(TezTask.java:674),
org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:267),
org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213),
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105),
org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:356),
org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:329),
org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246),
org.apache.hadoop.hive.ql.Executor.execute(Executor.java:107),
org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:804),
org.apache.hadoop.hive.ql.Driver.run(Driver.java:539),
org.apache.hadoop.hive.ql.Driver.run(Driver.java:533),
org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:190),
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:235),
org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:91),
org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:342),
java.base/java.security.AccessController.doPrivileged(Native Method),
java.base/javax.security.auth.Subject.doAs(Subject.java:423),
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1910),
org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:362),
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515),
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264),
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515),
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264),
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128),
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628),
java.base/java.lang.Thread.run(Thread.java:829)] retrying...
14225:<14>1 2026-07-06T10:46:55.282Z
hiveserver2-0.hiveserver2-service.compute-1782912973-6s7m.svc.cluster.local
hiveserver2 1 8e2b8b48-b7d7-40cb-bc90-88d495865c36 [mdc@38374 class="exec.Task"
dagId="dag_1783334749548_0001_2" level="INFO" operationLogLevel="EXECUTION"
operationLogLocation="/tmp/hive/operation_logs"
queryId="hive_20260706104554_f05ee797-3d66-4544-879b-b309dd7b4487"
sessionId="a86a7ab2-54f0-43dd-b376-6c05d8bfc69a"
thread="HiveServer2-Background-Pool: Thread-81280"] Dag submit failed due to
Operation failed:
{code}
> Session DAG submission fails with FileAlreadyExistsException when the session
> outlives the TezClient
> ----------------------------------------------------------------------------------------------------
>
> Key: TEZ-4737
> URL: https://issues.apache.org/jira/browse/TEZ-4737
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Denys Kuzmenko
> Priority: Major
>
> The serialized DAG plan file (tez-dag.pb<counter>) written for plans
> exceeding the IPC limit is named by a per-TezClient counter and never
> deleted. When the session and its staging directory outlive the TezClient
> instances, a new client generation recomputes the same file name and fails
> with FileAlreadyExistsException, permanently blocking oversized submissions
> on that session.
> {code:java}
> org.apache.hadoop.fs.FileAlreadyExistsException: Operation failed: "The
> specified path already exists.", 409, PUT,
> .../.tez/application_xxx_yyyy/tez-dag.pb1?resource=file
> at
> org.apache.tez.client.TezClient.submitDAGSession(TezClient.java:700)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
