[ 
https://issues.apache.org/jira/browse/PIG-943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12751560#action_12751560
 ] 

Pradeep Kamath commented on PIG-943:
------------------------------------

I am wondering if we should report that we were unable to determine the number 
of records/bytes written - 
 {code}
           if (stats.getRecordsWritten()!=-1)                                   
                                                                                
                                                                          
                log.info("Records written : " + stats.getRecordsWritten());     
                                                                                
                                                                           
            if (stats.getBytesWritten()!=-1)                                    
                                                                                
                                                                           
                log.info("Bytes written : " + stats.getBytesWritten());         
                                                                                
                                                                           
        }                              
 {code}
could change to
{code}
        if(stats.getRecordsWritten()==-1) {
        log.info("Records written : Unable to determine number of records 
written");
    } else {
        log.info("Records written : stats.getRecordsWritten()");
    }
        if(stats.getBytesWritten()==-1) {
        log.info("Bytes written : Unable to determine number of bytes written");
    } else {
        log.info("Bytes written : stats.getBytesWritten()");
    }
{code}

If we are using -1 to indicate we did not get counter information but the 
warning has occured some number of times (maybe in one of the
jobs where we did get counter information),  the following function's output 
message should change
:
{code}
public static void logAggregate(Map<Enum, Long> aggMap, MessageType 
messageType, Log log) {
        for(Enum e: aggMap.keySet()) {
                Long count = aggMap.get(e);
                if(count != null && count > 0) {
                        String message = "Encountered " + messageType + " " + 
e.toString() + " " + count + " time(s).";
                        logMessage(message, messageType, log);
                }
        }       
    }
{code}
to
{code}
public static void logAggregate(Map<Enum, Long> aggMap, MessageType 
messageType, Log log) {
        for(Enum e: aggMap.keySet()) {
                Long count = aggMap.get(e);
                if(count != null && (count == -1 || count > 0)) {
                        String message = "Encountered " + messageType + " " + 
e.toString() + " " + (count == -1 ? "unknown number of ": count.toString()) + " 
time(s).";
                        logMessage(message, messageType, log);
                }
        }       
    }
{code}

This way we still warn the user about the warning message which got aggregated 
but just not report the number of occurences.

To enable the above change the code in MapReduceLauncher.java should also 
change to ensure we report 3 types of counts:
0 -> the warning did not occur
-1 ->the warning occured in some of the jobs but in one or more jobs we get a 
null counter and hence we cannot report an accurate count
>0 -> we can accurately report the count of occurences of the warning.


> Pig crash when it cannot get counter from hadoop
> ------------------------------------------------
>
>                 Key: PIG-943
>                 URL: https://issues.apache.org/jira/browse/PIG-943
>             Project: Pig
>          Issue Type: Bug
>          Components: impl
>    Affects Versions: 0.3.0
>            Reporter: Daniel Dai
>            Assignee: Daniel Dai
>             Fix For: 0.4.0
>
>         Attachments: PIG-943-1.patch
>
>
> We see following call stacks in Pig:
> Case 1:
> Caused by: java.lang.NullPointerException
>         at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.computeWarningAggregate(MapReduceLauncher.java:390)
>         at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:238)
>         at 
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:265)
> Case 2:
> Caused by: java.lang.NullPointerException
>         at 
> org.apache.pig.tools.pigstats.PigStats.accumulateMRStats(PigStats.java:150)
>         at 
> org.apache.pig.tools.pigstats.PigStats.accumulateStats(PigStats.java:91)
>         at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:192)
>         at 
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:265)
> In both cases, hadoop jobs finishes without error. The cause of both problems 
> is RunningJob.getCounters() returns a null, and Pig do not currently check 
> for that. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply via email to