[
https://issues.apache.org/jira/browse/PIG-3081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohini Palaniswamy updated PIG-3081:
------------------------------------
Attachment: PIG-3081.patch
calculateProgress=(mapprog+reduceprog)/2
prog=calculateProgress/totalMRJobs (Pig displays progress as a percentage of
all the jobs combined together and not per job)
mapprog=0.6962389945983887, reduceprog=0.0
numMRJobsCompl=0,totalMRJobs=3,calculateProgress=0.34811949729919434
prog=0.11603983243306477,lastProg=0.11490649978319804
mapprog=0.7032797336578369, reduceprog=0.0
numMRJobsCompl=0,totalMRJobs=3,calculateProgress=0.35163986682891846
prog=0.11721328894297282,lastProg=0.11603983243306477
.....
mapprog=0.7190613150596619, reduceprog=0.0
numMRJobsCompl=0,totalMRJobs=3,calculateProgress=0.35953065752983093
prog=0.11984355250994365,lastProg=0.1190514365832011
mapprog=0.725161075592041, reduceprog=0.0
numMRJobsCompl=0,totalMRJobs=3,calculateProgress=0.3625805377960205
prog=0.12086017926534016,lastProg=0.11984355250994365
if(prog>=(lastProg+0.01)){
int perCom = (int)(prog * 100);
if(perCom!=100) {
log.info( perCom + "% complete");
}
The job progress is checked every 500ms. Since the frequency of checking is
very often and the prog is never > lastProg by more than 0.01 , it stays at 0%
and goes directly to 33% when the first job completed.
> Pig progress stays at 0% for the first job in hadoop 23
> -------------------------------------------------------
>
> Key: PIG-3081
> URL: https://issues.apache.org/jira/browse/PIG-3081
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.10.0
> Reporter: Rohini Palaniswamy
> Fix For: 0.12
>
> Attachments: PIG-3081.patch
>
>
> We are seeing that for many scripts if there are multiple jobs in the job
> graph, progress stays at 0% for the first job and jumps to 33% when the first
> job completes. There is no intermediate progress. After that intermediate
> progress gets reported for the subsequent jobs. Noticed this with jobs that
> do filtering and order by.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
