[
https://issues.apache.org/jira/browse/HIVE-21446?focusedWorklogId=214603&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-214603
]
ASF GitHub Bot logged work on HIVE-21446:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 18/Mar/19 07:18
Start Date: 18/Mar/19 07:18
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #569: HIVE-21446 :
Hive Server going OOM during hive external table replications
URL: https://github.com/apache/hive/pull/569#discussion_r266315969
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
##########
@@ -72,33 +72,40 @@ public CopyUtils(String distCpDoAsUser, HiveConf hiveConf)
{
public void copyAndVerify(FileSystem destinationFs, Path destRoot,
List<ReplChangeManager.FileInfo> srcFiles) throws
IOException, LoginException, HiveFatalException {
Map<FileSystem, Map< Path, List<ReplChangeManager.FileInfo>>> map =
fsToFileMap(srcFiles, destRoot);
- for (Map.Entry<FileSystem, Map<Path, List<ReplChangeManager.FileInfo>>>
entry : map.entrySet()) {
- FileSystem sourceFs = entry.getKey();
- Map<Path, List<ReplChangeManager.FileInfo>> destMap = entry.getValue();
- for (Map.Entry<Path, List<ReplChangeManager.FileInfo>> destMapEntry :
destMap.entrySet()) {
- Path destination = destMapEntry.getKey();
- List<ReplChangeManager.FileInfo> fileInfoList =
destMapEntry.getValue();
- boolean useRegularCopy = regularCopy(destinationFs, sourceFs,
fileInfoList);
-
- if (!destinationFs.exists(destination)
- && !FileUtils.mkdir(destinationFs, destination, hiveConf)) {
- LOG.error("Failed to create destination directory: " + destination);
- throw new IOException("Destination directory creation failed");
- }
+ UserGroupInformation proxyUser = getProxyUser();
+ try {
+ for (Map.Entry<FileSystem, Map<Path, List<ReplChangeManager.FileInfo>>>
entry : map.entrySet()) {
+ FileSystem sourceFs = entry.getKey();
Review comment:
The hive retry logic is for very limited local functionality ..for all other
places ..in case of file system error ..dlm will retry
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 214603)
Time Spent: 1h (was: 50m)
> Hive Server going OOM during hive external table replications
> -------------------------------------------------------------
>
> Key: HIVE-21446
> URL: https://issues.apache.org/jira/browse/HIVE-21446
> Project: Hive
> Issue Type: Bug
> Components: repl
> Affects Versions: 4.0.0
> Reporter: mahesh kumar behera
> Assignee: mahesh kumar behera
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21446.01.patch
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> The file system objects opened using proxy users are not closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
