[jira] [Updated] (HIVE-8771) Abstract merge file operator does not move/rename incompatible files correctly
[ https://issues.apache.org/jira/browse/HIVE-8771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-8771: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to trunk and .14. Thanks [~prasanth_j]! Abstract merge file operator does not move/rename incompatible files correctly -- Key: HIVE-8771 URL: https://issues.apache.org/jira/browse/HIVE-8771 Project: Hive Issue Type: Bug Affects Versions: 0.14.0 Reporter: Prasanth J Assignee: Prasanth J Priority: Critical Fix For: 0.14.0 Attachments: HIVE-8771.1.patch AbstractFileMergeOperator moves incompatible files (files which cannot be merged) to final destination. The destination path must be directory instead of file. This causes orc_merge_incompat2.q to fail under CentOS with IOException failing to rename/move files. Stack trace: {code} 2014-11-05 02:38:56,588 DEBUG fs.FileSystem (RawLocalFileSystem.java:rename(337)) - Falling through to a copy of file:/home/prasanth/hive/itests/qtest/target/warehouse/orc_merge5a/st=80.0/00_0 to file:/home/prasanth/hive/itests/qtest/target/tmp/scratchdir/prasanth/0de64e52-6615-4c5a-bdfb-c3b2c28131f6/hive_2014-11-05_02-38-55_511_7578595409877157627-1/_tmp.-ext-1/00_0/00_0 2014-11-05 02:38:56,589 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete. 2014-11-05 02:38:56,590 WARN mapred.LocalJobRunner (LocalJobRunner.java:run(560)) - job_local1144733438_0036 java.lang.Exception: java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to close AbstractFileMergeOperator at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522) Caused by: java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to close AbstractFileMergeOperator at org.apache.hadoop.hive.ql.io.merge.MergeFileMapper.close(MergeFileMapper.java:100) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:679) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to close AbstractFileMergeOperator at org.apache.hadoop.hive.ql.exec.AbstractFileMergeOperator.closeOp(AbstractFileMergeOperator.java:233) at org.apache.hadoop.hive.ql.exec.OrcFileMergeOperator.closeOp(OrcFileMergeOperator.java:220) at org.apache.hadoop.hive.ql.io.merge.MergeFileMapper.close(MergeFileMapper.java:98) ... 10 more Caused by: java.io.FileNotFoundException: Destination exists and is not a directory: /home/prasanth/hive/itests/qtest/target/tmp/scratchdir/prasanth/0de64e52-6615-4c5a-bdfb-c3b2c28131f6/hive_2014-11-05_02-38-55_511_7578595409877157627-1/_tmp.-ext-1/00_0 at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:423) at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:267) at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:257) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:887) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:784) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:365) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:338) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:289) at org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:339) at org.apache.hadoop.fs.ChecksumFileSystem.rename(ChecksumFileSystem.java:507) at org.apache.hadoop.fs.FilterFileSystem.rename(FilterFileSystem.java:214) at org.apache.hadoop.fs.ProxyFileSystem.rename(ProxyFileSystem.java:177) at org.apache.hadoop.fs.FilterFileSystem.rename(FilterFileSystem.java:214) at org.apache.hadoop.hive.ql.exec.Utilities.renameOrMoveFiles(Utilities.java:1589) at
[jira] [Updated] (HIVE-8771) Abstract merge file operator does not move/rename incompatible files correctly
[ https://issues.apache.org/jira/browse/HIVE-8771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth J updated HIVE-8771: - Status: Patch Available (was: Open) Abstract merge file operator does not move/rename incompatible files correctly -- Key: HIVE-8771 URL: https://issues.apache.org/jira/browse/HIVE-8771 Project: Hive Issue Type: Bug Affects Versions: 0.14.0 Reporter: Prasanth J Assignee: Prasanth J Priority: Critical Fix For: 0.14.0 Attachments: HIVE-8771.1.patch AbstractFileMergeOperator moves incompatible files (files which cannot be merged) to final destination. The destination path must be directory instead of file. This causes orc_merge_incompat2.q to fail under CentOS with IOException failing to rename/move files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8771) Abstract merge file operator does not move/rename incompatible files correctly
[ https://issues.apache.org/jira/browse/HIVE-8771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth J updated HIVE-8771: - Attachment: HIVE-8771.1.patch Abstract merge file operator does not move/rename incompatible files correctly -- Key: HIVE-8771 URL: https://issues.apache.org/jira/browse/HIVE-8771 Project: Hive Issue Type: Bug Affects Versions: 0.14.0 Reporter: Prasanth J Assignee: Prasanth J Priority: Critical Fix For: 0.14.0 Attachments: HIVE-8771.1.patch AbstractFileMergeOperator moves incompatible files (files which cannot be merged) to final destination. The destination path must be directory instead of file. This causes orc_merge_incompat2.q to fail under CentOS with IOException failing to rename/move files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8771) Abstract merge file operator does not move/rename incompatible files correctly
[ https://issues.apache.org/jira/browse/HIVE-8771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth J updated HIVE-8771: - Description: AbstractFileMergeOperator moves incompatible files (files which cannot be merged) to final destination. The destination path must be directory instead of file. This causes orc_merge_incompat2.q to fail under CentOS with IOException failing to rename/move files. Stack trace: {code} 2014-11-05 02:38:56,588 DEBUG fs.FileSystem (RawLocalFileSystem.java:rename(337)) - Falling through to a copy of file:/home/prasanth/hive/itests/qtest/target/warehouse/orc_merge5a/st=80.0/00_0 to file:/home/prasanth/hive/itests/qtest/target/tmp/scratchdir/prasanth/0de64e52-6615-4c5a-bdfb-c3b2c28131f6/hive_2014-11-05_02-38-55_511_7578595409877157627-1/_tmp.-ext-1/00_0/00_0 2014-11-05 02:38:56,589 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete. 2014-11-05 02:38:56,590 WARN mapred.LocalJobRunner (LocalJobRunner.java:run(560)) - job_local1144733438_0036 java.lang.Exception: java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to close AbstractFileMergeOperator at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522) Caused by: java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to close AbstractFileMergeOperator at org.apache.hadoop.hive.ql.io.merge.MergeFileMapper.close(MergeFileMapper.java:100) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:679) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to close AbstractFileMergeOperator at org.apache.hadoop.hive.ql.exec.AbstractFileMergeOperator.closeOp(AbstractFileMergeOperator.java:233) at org.apache.hadoop.hive.ql.exec.OrcFileMergeOperator.closeOp(OrcFileMergeOperator.java:220) at org.apache.hadoop.hive.ql.io.merge.MergeFileMapper.close(MergeFileMapper.java:98) ... 10 more Caused by: java.io.FileNotFoundException: Destination exists and is not a directory: /home/prasanth/hive/itests/qtest/target/tmp/scratchdir/prasanth/0de64e52-6615-4c5a-bdfb-c3b2c28131f6/hive_2014-11-05_02-38-55_511_7578595409877157627-1/_tmp.-ext-1/00_0 at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:423) at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:267) at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:257) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:887) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:784) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:365) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:338) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:289) at org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:339) at org.apache.hadoop.fs.ChecksumFileSystem.rename(ChecksumFileSystem.java:507) at org.apache.hadoop.fs.FilterFileSystem.rename(FilterFileSystem.java:214) at org.apache.hadoop.fs.ProxyFileSystem.rename(ProxyFileSystem.java:177) at org.apache.hadoop.fs.FilterFileSystem.rename(FilterFileSystem.java:214) at org.apache.hadoop.hive.ql.exec.Utilities.renameOrMoveFiles(Utilities.java:1589) at org.apache.hadoop.hive.ql.exec.AbstractFileMergeOperator.closeOp(AbstractFileMergeOperator.java:218) ... 12 more {code} was:AbstractFileMergeOperator moves incompatible files (files which cannot be merged) to final destination. The destination path must be directory instead of file. This causes orc_merge_incompat2.q to fail under CentOS with IOException failing to rename/move files. Abstract merge file operator does not move/rename incompatible files correctly -- Key: HIVE-8771 URL: https://issues.apache.org/jira/browse/HIVE-8771 Project: Hive Issue Type: Bug Affects Versions: 0.14.0