[
https://issues.apache.org/jira/browse/HIVE-24862?focusedWorklogId=563142&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-563142
]
ASF GitHub Bot logged work on HIVE-24862:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 09/Mar/21 15:38
Start Date: 09/Mar/21 15:38
Worklog Time Spent: 10m
Work Description: zchovan commented on pull request #2053:
URL: https://github.com/apache/hive/pull/2053#issuecomment-794059800
failed test "dynpart_sort_opt_bucketing" succeeds locally, rerunning the
tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 563142)
Time Spent: 0.5h (was: 20m)
> Fix race condition causing NPE during dynamic partition loading
> ---------------------------------------------------------------
>
> Key: HIVE-24862
> URL: https://issues.apache.org/jira/browse/HIVE-24862
> Project: Hive
> Issue Type: Improvement
> Reporter: Rajesh Balamohan
> Assignee: Zoltan Chovan
> Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Following properties default to 15 threads.
> {noformat}
> hive.load.dynamic.partitions.thread
> hive.mv.files.thread
> {noformat}
> During loadDynamicPartitions, it ends ups initializing {{newFiles}} without
> synchronization (HIVE-20661, HIVE-24738).
>
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L2871]
> This causes race condition when dynamic partition thread internally makes use
> of {{hive.mv.files.threads}} in copyFiles/replaceFiles.
> This causes "NPE" during retrieval in {{addInsertFileInformation()}}.
>
> e.g stacktrace
> {noformat}
> Caused by: java.lang.NullPointerException
> at org.apache.hadoop.fs.FileSystem.fixRelativePart(FileSystem.java:2734)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.fixRelativePart(DistributedFileSystem.java:3396)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1740)
> at org.apache.hadoop.fs.FileSystem.isDirectory(FileSystem.java:1740)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.addInsertFileInformation(Hive.java:3566)
> at org.apache.hadoop.hive.ql.metadata.Hive.fireInsertEvent(Hive.java:3540)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.loadPartitionInternal(Hive.java:2414)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.lambda$loadDynamicPartitions$4(Hive.java:2909)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
