[jira] [Commented] (HIVE-22247) HiveHFileOutputFormat throws FileNotFoundException when partition's task output empty
[ https://issues.apache.org/jira/browse/HIVE-22247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17411865#comment-17411865 ] Aasha Medhi commented on HIVE-22247: Thank you for the patch [~ayushtkn]. Committed to master. > HiveHFileOutputFormat throws FileNotFoundException when partition's task > output empty > - > > Key: HIVE-22247 > URL: https://issues.apache.org/jira/browse/HIVE-22247 > Project: Hive > Issue Type: Bug > Components: HBase Handler >Affects Versions: 2.2.0, 3.0.0 >Reporter: xiepengjie >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > When partition's task output empty, HiveHFileOutputFormat throws > FileNotFoundException like this: > {code:java} > 2019-09-24 19:15:55,886 INFO [main] > org.apache.hadoop.hive.ql.exec.FileSinkOperator: 1 finished. closing... > 2019-09-24 19:15:55,886 INFO [main] > org.apache.hadoop.hive.ql.exec.FileSinkOperator: FS[1]: records written - 0 > 2019-09-24 19:15:55,886 INFO [main] > org.apache.hadoop.hive.ql.exec.FileSinkOperator: Final Path: FS > hdfs://Hdptest-mini-nmg/tmp/hive-staging/hadoop_hive_2019-09-24_19-15-26_453_1697529445006435790-5/_tmp.-ext-10002/02_0 > 2019-09-24 19:15:55,886 INFO [main] > org.apache.hadoop.hive.ql.exec.FileSinkOperator: Writing to temp file: FS > hdfs://Hdptest-mini-nmg/tmp/hive-staging/hadoop_hive_2019-09-24_19-15-26_453_1697529445006435790-5/_task_tmp.-ext-10002/_tmp.02_0 > 2019-09-24 19:15:55,886 INFO [main] > org.apache.hadoop.hive.ql.exec.FileSinkOperator: New Final Path: FS > hdfs://Hdptest-mini-nmg/tmp/hive-staging/hadoop_hive_2019-09-24_19-15-26_453_1697529445006435790-5/_tmp.-ext-10002/02_0 > 2019-09-24 19:15:55,915 INFO [main] > org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output > Committer Algorithm version is 1 > 2019-09-24 19:15:55,954 INFO [main] > org.apache.hadoop.conf.Configuration.deprecation: hadoop.native.lib is > deprecated. Instead, use io.native.lib.available > 2019-09-24 19:15:56,089 ERROR [main] ExecReducer: Hit error while closing > operators - failing tree > 2019-09-24 19:15:56,090 WARN [main] org.apache.hadoop.mapred.YarnChild: > Exception running child : java.lang.RuntimeException: Hive Runtime Error > while closing operators: java.io.FileNotFoundException: File > hdfs://Hdptest-mini-nmg/tmp/hive-staging/hadoop_hive_2019-09-24_19-15-26_453_1697529445006435790-5/_task_tmp.-ext-10002/_tmp.02_0 > does not exist. > at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:287) > at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:453) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1923) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > java.io.FileNotFoundException: File > hdfs://Hdptest-mini-nmg/tmp/hive-staging/hadoop_hive_2019-09-24_19-15-26_453_1697529445006435790-5/_task_tmp.-ext-10002/_tmp.02_0 > does not exist. > at > org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.closeWriters(FileSinkOperator.java:200) > at > org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1016) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:617) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:631) > at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:278) > ... 7 more > Caused by: java.io.FileNotFoundException: File > hdfs://Hdptest-mini-nmg/tmp/hive-staging/hadoop_hive_2019-09-24_19-15-26_453_1697529445006435790-5/_task_tmp.-ext-10002/_tmp.02_0 > does not exist. > at > org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:880) > at > org.apache.hadoop.hdfs.DistributedFileSystem.access$700(DistributedFileSystem.java:109) > at > org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:938) > at > org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:934) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:945) > at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1592) > at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1632) >
[jira] [Commented] (HIVE-22247) HiveHFileOutputFormat throws FileNotFoundException when partition's task output empty
[ https://issues.apache.org/jira/browse/HIVE-22247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17388021#comment-17388021 ] Ayush Saxena commented on HIVE-22247: - [~xiepengjie] Are you planning to post a fix? If not I can take over!!! > HiveHFileOutputFormat throws FileNotFoundException when partition's task > output empty > - > > Key: HIVE-22247 > URL: https://issues.apache.org/jira/browse/HIVE-22247 > Project: Hive > Issue Type: Bug > Components: HBase Handler >Affects Versions: 2.2.0, 3.0.0 >Reporter: xiepengjie >Assignee: xiepengjie >Priority: Major > > When partition's task output empty, HiveHFileOutputFormat throws > FileNotFoundException like this: > {code:java} > 2019-09-24 19:15:55,886 INFO [main] > org.apache.hadoop.hive.ql.exec.FileSinkOperator: 1 finished. closing... > 2019-09-24 19:15:55,886 INFO [main] > org.apache.hadoop.hive.ql.exec.FileSinkOperator: FS[1]: records written - 0 > 2019-09-24 19:15:55,886 INFO [main] > org.apache.hadoop.hive.ql.exec.FileSinkOperator: Final Path: FS > hdfs://Hdptest-mini-nmg/tmp/hive-staging/hadoop_hive_2019-09-24_19-15-26_453_1697529445006435790-5/_tmp.-ext-10002/02_0 > 2019-09-24 19:15:55,886 INFO [main] > org.apache.hadoop.hive.ql.exec.FileSinkOperator: Writing to temp file: FS > hdfs://Hdptest-mini-nmg/tmp/hive-staging/hadoop_hive_2019-09-24_19-15-26_453_1697529445006435790-5/_task_tmp.-ext-10002/_tmp.02_0 > 2019-09-24 19:15:55,886 INFO [main] > org.apache.hadoop.hive.ql.exec.FileSinkOperator: New Final Path: FS > hdfs://Hdptest-mini-nmg/tmp/hive-staging/hadoop_hive_2019-09-24_19-15-26_453_1697529445006435790-5/_tmp.-ext-10002/02_0 > 2019-09-24 19:15:55,915 INFO [main] > org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output > Committer Algorithm version is 1 > 2019-09-24 19:15:55,954 INFO [main] > org.apache.hadoop.conf.Configuration.deprecation: hadoop.native.lib is > deprecated. Instead, use io.native.lib.available > 2019-09-24 19:15:56,089 ERROR [main] ExecReducer: Hit error while closing > operators - failing tree > 2019-09-24 19:15:56,090 WARN [main] org.apache.hadoop.mapred.YarnChild: > Exception running child : java.lang.RuntimeException: Hive Runtime Error > while closing operators: java.io.FileNotFoundException: File > hdfs://Hdptest-mini-nmg/tmp/hive-staging/hadoop_hive_2019-09-24_19-15-26_453_1697529445006435790-5/_task_tmp.-ext-10002/_tmp.02_0 > does not exist. > at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:287) > at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:453) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1923) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > java.io.FileNotFoundException: File > hdfs://Hdptest-mini-nmg/tmp/hive-staging/hadoop_hive_2019-09-24_19-15-26_453_1697529445006435790-5/_task_tmp.-ext-10002/_tmp.02_0 > does not exist. > at > org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.closeWriters(FileSinkOperator.java:200) > at > org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1016) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:617) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:631) > at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:278) > ... 7 more > Caused by: java.io.FileNotFoundException: File > hdfs://Hdptest-mini-nmg/tmp/hive-staging/hadoop_hive_2019-09-24_19-15-26_453_1697529445006435790-5/_task_tmp.-ext-10002/_tmp.02_0 > does not exist. > at > org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:880) > at > org.apache.hadoop.hdfs.DistributedFileSystem.access$700(DistributedFileSystem.java:109) > at > org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:938) > at > org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:934) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:945) > at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1592) > at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1632) > at > org.apache.hadoop.hive.hbase.HiveHFileOutputFormat$1.close(HiveHFileOutputFo