[
https://issues.apache.org/jira/browse/HIVE-27023?focusedWorklogId=843592&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-843592
]
ASF GitHub Bot logged work on HIVE-27023:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 03/Feb/23 22:15
Start Date: 03/Feb/23 22:15
Worklog Time Spent: 10m
Work Description: maswin opened a new pull request, #4015:
URL: https://github.com/apache/hive/pull/4015
### What changes were proposed in this pull request?
A setting to disable tez session being preemptively opened during startup of
hive cli.
### Why are the changes needed?
DML only operations won't need a tez session to be opened. Oozie action that
launches hive cli for DML only operations need this setting. Sometimes before
the tez session thread could completed, the oozie hive action completes its DML
operations and clears the scratch folders causing the tez session to fail with
following exception:
Added a setting disable Tez session being opened during the startup of Hive
cli. This is particularly needed when Hive action is being launched by Oozie to
perform DML only operations. In that cases before the tez session thread could
complete the launcher gets completed and clears all the scratch directories.
This leads to a failure with following stack trace:
```
Added a setting disable Tez session being opened during the startup of Hive
cli. This is particularly needed when Hive action is being launched by Oozie to
perform DML only operations. In that cases before the tez session thread could
complete the launcher gets completed and clears all the scratch directories.
This leads to a failure with following stack trace:
Application application_1667416881396_24229473 failed 3 times due to AM
Container for appattempt_1667416881396_24229473_000003 exited with exitCode:
-1000
Failing this attempt.Diagnostics: [2023-02-02 19:02:12.139]File does not
exist:
hdfs://<name_node>/tmp/<db>/_tez_session_dir/4050c4b0-b7af-4eda-832b-399c954eb576/.tez/application_1667416881396_24229473/tez.session.local-resources.pbjava.io.FileNotFoundException:
File does not exist:
hdfs://<name_node>/tmp/<db>/_tez_session_dir/4050c4b0-b7af-4eda-832b-399c954eb576/.tez/application_1667416881396_24229473/tez.session.local-resources.pbat
org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1529)at
org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1522)at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1537)at
```
### Does this PR introduce _any_ user-facing change?
New config value, otherwise no.
### How was this patch tested?
Manually tested
Issue Time Tracking
-------------------
Worklog Id: (was: 843592)
Remaining Estimate: 0h
Time Spent: 10m
> Add setting to disable hive session being started during Hive Cli startup
> -------------------------------------------------------------------------
>
> Key: HIVE-27023
> URL: https://issues.apache.org/jira/browse/HIVE-27023
> Project: Hive
> Issue Type: Improvement
> Reporter: Alagappan Maruthappan
> Assignee: Alagappan Maruthappan
> Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Added a setting disable Tez session being opened during the startup of Hive
> cli. This is particularly needed when Hive action is being launched by Oozie
> to perform DML only operations. In that cases before the tez session thread
> could complete the launcher gets completed and clears all the scratch
> directories. This leads to a failure with following stack trace:
> {code:java}
> Application application_1667416881396_24229473 failed 3 times due to AM
> Container for appattempt_1667416881396_24229473_000003 exited with exitCode:
> -1000
> Failing this attempt.Diagnostics: [2023-02-02 19:02:12.139]File does not
> exist:
> hdfs://<name_node>/tmp/<db>/_tez_session_dir/4050c4b0-b7af-4eda-832b-399c954eb576/.tez/application_1667416881396_24229473/tez.session.local-resources.pbjava.io.FileNotFoundException:
> File does not exist:
> hdfs://<name_node>/tmp/<db>/_tez_session_dir/4050c4b0-b7af-4eda-832b-399c954eb576/.tez/application_1667416881396_24229473/tez.session.local-resources.pbat
>
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1529)at
>
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1522)at
>
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)at
>
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1537)at
> org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:253)at
> org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:63)at
> org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:366)at
> org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:364)at
> java.security.AccessController.doPrivileged(Native Method)at
> javax.security.auth.Subject.doAs(Subject.java:422)at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:2136)at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:2116)at
> org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:364)at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.doDownloadCall(ContainerLocalizer.java:249)at
>
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.call(ContainerLocalizer.java:242)at
>
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.call(ContainerLocalizer.java:230)at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at
> java.lang.Thread.run(Thread.java:748) {code}
> Also opening up of tez session in such cases is not needed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
