[jira] Commented: (PIG-1641) Incorrect counters in local mode
[ https://issues.apache.org/jira/browse/PIG-1641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12915408#action_12915408 ] Ashutosh Chauhan commented on PIG-1641: --- Tested manually for local mode. Messages were same as proposed above. +1 for the commit. One minor suggestion is to put a line at the start saying something like: "Detected Local mode. Stats reported below may be incomplete." This will reinforce the message to users that stats reporting is not transparent across different modes (local Vs map-reduce). > Incorrect counters in local mode > > > Key: PIG-1641 > URL: https://issues.apache.org/jira/browse/PIG-1641 > Project: Pig > Issue Type: Bug >Affects Versions: 0.8.0 >Reporter: Ashutosh Chauhan >Assignee: Richard Ding > Fix For: 0.8.0 > > Attachments: PIG-1641.patch > > > User report, not verified. > > HadoopVersionPigVersionUserIdStartedAtFinishedAtFeatures > 0.20.20.8.0-SNAPSHOTuser2010-09-21 19:25:582010-09-21 > 21:58:42ORDER_BY > Success! > Job Stats (time in seconds): > JobIdMapsReducesMaxMapTimeMinMapTImeAvgMapTime > MaxReduceTimeMinReduceTimeAvgReduceTimeAliasFeatureOutputs > job_local_000100000000rawMAP_ONLY > job_local_000200000000rank_sort > SAMPLER > job_local_000300000000rank_sort > ORDER_BYProcessed/user_visits_table, > Input(s): > Successfully read 0 records from: "Data/Raw/UserVisits.dat" > Output(s): > Successfully stored 0 records in: "Processed/user_visits_table" > However, when I look in the output: > $ ls -lh Processed/user_visits_table/CG0/ > total 15250760 > -rwxrwxrwx 1 user _lpoperator 7.3G Sep 21 21:58 part-0* > It read a 20G input file and generated some output... > > Is it that in local mode counters are not available? If so, instead of > printing zeros we should print "Information Unavailable" or some such. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1641) Incorrect counters in local mode
[
https://issues.apache.org/jira/browse/PIG-1641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12913736#action_12913736
]
Richard Ding commented on PIG-1641:
---
Hadoop counters are not available in local mode (PIG-1286).
So for now I propose that, in local mode, Pig stats output is changed to
something like the following:
{code}
Job Stats (time in seconds):
JobId Alias Feature Outputs
job_local_0001 raw MAP_ONLY
job_local_0002 rank_sort SAMPLER
job_local_0003 rank_sort ORDER_BY Processed/user_visits_table,
Input(s):
Successfully read records from: "Data/Raw/UserVisits.dat"
Output(s):
Successfully stored records in: "Processed/user_visits_table"
{code}
> Incorrect counters in local mode
>
>
> Key: PIG-1641
> URL: https://issues.apache.org/jira/browse/PIG-1641
> Project: Pig
> Issue Type: Bug
>Affects Versions: 0.8.0
>Reporter: Ashutosh Chauhan
>
> User report, not verified.
>
> HadoopVersionPigVersionUserIdStartedAtFinishedAtFeatures
> 0.20.20.8.0-SNAPSHOTuser2010-09-21 19:25:582010-09-21
> 21:58:42ORDER_BY
> Success!
> Job Stats (time in seconds):
> JobIdMapsReducesMaxMapTimeMinMapTImeAvgMapTime
> MaxReduceTimeMinReduceTimeAvgReduceTimeAliasFeatureOutputs
> job_local_000100000000rawMAP_ONLY
> job_local_000200000000rank_sort
> SAMPLER
> job_local_000300000000rank_sort
> ORDER_BYProcessed/user_visits_table,
> Input(s):
> Successfully read 0 records from: "Data/Raw/UserVisits.dat"
> Output(s):
> Successfully stored 0 records in: "Processed/user_visits_table"
> However, when I look in the output:
> $ ls -lh Processed/user_visits_table/CG0/
> total 15250760
> -rwxrwxrwx 1 user _lpoperator 7.3G Sep 21 21:58 part-0*
> It read a 20G input file and generated some output...
>
> Is it that in local mode counters are not available? If so, instead of
> printing zeros we should print "Information Unavailable" or some such.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
