[jira] [Updated] (HIVE-17774) compaction may start with 0 splits and fail

2017-10-13 Thread Sergey Shelukhin (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-17774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-17774:

   Resolution: Fixed
Fix Version/s: 2.4.0
   Status: Resolved  (was: Patch Available)

The failures appear to be existing ones on branch-2. Committed to branch-2, 
thanks for the review.

> compaction may start with 0 splits and fail
> ---
>
> Key: HIVE-17774
> URL: https://issues.apache.org/jira/browse/HIVE-17774
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 2.4.0
>
> Attachments: HIVE-17774-branch-2.patch
>
>
> {noformat}
> 2017-09-26 10:36:01,979 INFO  [...]: compactor.CompactorMR 
> (CompactorMR.java:launchCompactionJob(295)) - 
> Submitting MINOR compaction job 
>  (current delta dirs count=0, obsolete delta dirs count=0. 
> TxnIdRange[9223372036854775807,-9223372036854775808]
> ...
> 2017-09-26 10:36:02,350 INFO  [...]: mapreduce.JobSubmitter 
> (JobSubmitter.java:submitJobInternal(198)) - number of splits:0
> ...
> 2017-09-26 10:36:08,637 INFO  [...]: mapreduce.Job 
> (Job.java:monitorAndPrintJob(1380)) - 
> Job job_1503950256860_15982 failed with state FAILED due to: No of maps and 
> reduces are 0 job_1503950256860_15982
> Job commit failed: java.io.FileNotFoundException: File 
> .../hello_acid/load_date=2016-03-03/_tmp_a95346ad-bd89-4e66-9b05-e60fdfa11858 
> does not exist.
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:904)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:113)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:966)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:962)
>   at 
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:962)
>   at 
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorOutputCommitter.commitJob(CompactorMR.java:776)
>   at 
> org.apache.hadoop.mapred.OutputCommitter.commitJob(OutputCommitter.java:291)
>   at 
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:285)
>   at 
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:237)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> {noformat}
> Looks like the MR job should not have been attempted in this case.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17774) compaction may start with 0 splits and fail

2017-10-12 Thread Sergey Shelukhin (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-17774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-17774:

Assignee: Sergey Shelukhin  (was: Eugene Koifman)
  Status: Patch Available  (was: Open)

> compaction may start with 0 splits and fail
> ---
>
> Key: HIVE-17774
> URL: https://issues.apache.org/jira/browse/HIVE-17774
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-17774-branch-2.patch
>
>
> {noformat}
> 2017-09-26 10:36:01,979 INFO  [...]: compactor.CompactorMR 
> (CompactorMR.java:launchCompactionJob(295)) - 
> Submitting MINOR compaction job 
>  (current delta dirs count=0, obsolete delta dirs count=0. 
> TxnIdRange[9223372036854775807,-9223372036854775808]
> ...
> 2017-09-26 10:36:02,350 INFO  [...]: mapreduce.JobSubmitter 
> (JobSubmitter.java:submitJobInternal(198)) - number of splits:0
> ...
> 2017-09-26 10:36:08,637 INFO  [...]: mapreduce.Job 
> (Job.java:monitorAndPrintJob(1380)) - 
> Job job_1503950256860_15982 failed with state FAILED due to: No of maps and 
> reduces are 0 job_1503950256860_15982
> Job commit failed: java.io.FileNotFoundException: File 
> .../hello_acid/load_date=2016-03-03/_tmp_a95346ad-bd89-4e66-9b05-e60fdfa11858 
> does not exist.
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:904)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:113)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:966)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:962)
>   at 
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:962)
>   at 
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorOutputCommitter.commitJob(CompactorMR.java:776)
>   at 
> org.apache.hadoop.mapred.OutputCommitter.commitJob(OutputCommitter.java:291)
>   at 
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:285)
>   at 
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:237)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> {noformat}
> Looks like the MR job should not have been attempted in this case.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17774) compaction may start with 0 splits and fail

2017-10-12 Thread Sergey Shelukhin (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-17774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-17774:

Attachment: HIVE-17774-branch-2.patch

[~ekoifman] can you take a look?

> compaction may start with 0 splits and fail
> ---
>
> Key: HIVE-17774
> URL: https://issues.apache.org/jira/browse/HIVE-17774
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Eugene Koifman
> Attachments: HIVE-17774-branch-2.patch
>
>
> {noformat}
> 2017-09-26 10:36:01,979 INFO  [...]: compactor.CompactorMR 
> (CompactorMR.java:launchCompactionJob(295)) - 
> Submitting MINOR compaction job 
>  (current delta dirs count=0, obsolete delta dirs count=0. 
> TxnIdRange[9223372036854775807,-9223372036854775808]
> ...
> 2017-09-26 10:36:02,350 INFO  [...]: mapreduce.JobSubmitter 
> (JobSubmitter.java:submitJobInternal(198)) - number of splits:0
> ...
> 2017-09-26 10:36:08,637 INFO  [...]: mapreduce.Job 
> (Job.java:monitorAndPrintJob(1380)) - 
> Job job_1503950256860_15982 failed with state FAILED due to: No of maps and 
> reduces are 0 job_1503950256860_15982
> Job commit failed: java.io.FileNotFoundException: File 
> .../hello_acid/load_date=2016-03-03/_tmp_a95346ad-bd89-4e66-9b05-e60fdfa11858 
> does not exist.
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:904)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:113)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:966)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:962)
>   at 
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:962)
>   at 
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorOutputCommitter.commitJob(CompactorMR.java:776)
>   at 
> org.apache.hadoop.mapred.OutputCommitter.commitJob(OutputCommitter.java:291)
>   at 
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:285)
>   at 
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:237)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> {noformat}
> Looks like the MR job should not have been attempted in this case.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17774) compaction may start with 0 splits and fail

2017-10-11 Thread Eugene Koifman (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-17774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-17774:
--
Component/s: Transactions

> compaction may start with 0 splits and fail
> ---
>
> Key: HIVE-17774
> URL: https://issues.apache.org/jira/browse/HIVE-17774
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Eugene Koifman
>
> {noformat}
> 2017-09-26 10:36:01,979 INFO  [...]: compactor.CompactorMR 
> (CompactorMR.java:launchCompactionJob(295)) - 
> Submitting MINOR compaction job 
>  (current delta dirs count=0, obsolete delta dirs count=0. 
> TxnIdRange[9223372036854775807,-9223372036854775808]
> ...
> 2017-09-26 10:36:02,350 INFO  [...]: mapreduce.JobSubmitter 
> (JobSubmitter.java:submitJobInternal(198)) - number of splits:0
> ...
> 2017-09-26 10:36:08,637 INFO  [...]: mapreduce.Job 
> (Job.java:monitorAndPrintJob(1380)) - 
> Job job_1503950256860_15982 failed with state FAILED due to: No of maps and 
> reduces are 0 job_1503950256860_15982
> Job commit failed: java.io.FileNotFoundException: File 
> .../hello_acid/load_date=2016-03-03/_tmp_a95346ad-bd89-4e66-9b05-e60fdfa11858 
> does not exist.
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:904)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:113)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:966)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:962)
>   at 
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:962)
>   at 
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorOutputCommitter.commitJob(CompactorMR.java:776)
>   at 
> org.apache.hadoop.mapred.OutputCommitter.commitJob(OutputCommitter.java:291)
>   at 
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:285)
>   at 
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:237)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> {noformat}
> Looks like the MR job should not have been attempted in this case.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)