Re: Removing local job traces in local mode

Fri, 26 Aug 2011 07:57:16 -0700

Here is how I solved this issue (it was only related to log4j configuration): 

      /* Deactivate most traces from Hadoop and PIG (keep ERROR) */
      props.setProperty("log4j.logger.org.apache.hadoop", "ERROR");
      props.setProperty("log4j.logger.org.apache.zookeeper", "ERROR");
      props.setProperty("log4j.logger.org.apache.pig", "ERROR");

      /*
* Keep traces from M/R launcher (percentage of advancement) and job statistics (nb of record 
       * written etc.)
       */
      props.setProperty(
"log4j.logger.org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher", 
        "INFO");
props.setProperty("log4j.logger.org.apache.pig.tools.pigstats.PigStats", "INFO"); 


Le 27/06/11 13:21, Vincent Barat a écrit :

Thanks, but it is actually not what I'm looking for.
First, I use PIG from Java, not from the pig shell, so the "-d" option cannot be used. Second, I want to be able to choose between which trace I want, and which I don't, depending on the origin of the trace. I guess log4j can be configured for that, but I don't manage to configure it correctly. 

Any hint ?

Le 27/06/11 12:49, Vincent a écrit :

You can try -d option:

For example pig -d WARN -x local myscript.pig
On Mon, Jun 27, 2011 at 2:46 PM, Vincent Barat<[email protected]>wrote: 


Hi,
I'm using PIG in local mode during development phases, and I get bored 
about traces outputted by local jobs (the one attached).
Is there a way to get rid of them?

Thanks for your help.

2011-06-27 12:26:56,289 | WARN  | Thread-55 | JobClient  | Use
GenericOptionsParser for parsing the arguments. Applications should
implement Tool for the same.
2011-06-27 12:26:56,423 | INFO  | Thread-55 | JvmMetrics | Cannot
initialize JVM Metrics with processName=JobTracker, sessionId= - already 
initialized
2011-06-27 12:26:56,427 | INFO  | Thread-55 | JvmMetrics | Cannot
initialize JVM Metrics with processName=JobTracker, sessionId= - already 
initialized
2011-06-27 12:26:56,429 | INFO | Thread-55 | FileInputFormat | Total input 
paths to process : 1
2011-06-27 12:26:56,429 | INFO | Thread-55 | MapRedUtil | Total input 
paths to process : 1
2011-06-27 12:26:56,429 | INFO | Thread-55 | MapRedUtil | Total input 
paths (combined) to process : 1
2011-06-27 12:26:56,614 | INFO  | Thread-64 | JvmMetrics | Cannot
initialize JVM Metrics with processName=JobTracker, sessionId= - already 
initialized
2011-06-27 12:26:56,616 | INFO | Thread-64 | FileInputFormat | Total input 
paths to process : 1
2011-06-27 12:26:56,616 | INFO | Thread-64 | MapRedUtil | Total input 
paths to process : 1
2011-06-27 12:26:56,616 | INFO | Thread-64 | MapRedUtil | Total input 
paths (combined) to process : 1
2011-06-27 12:26:56,653 | INFO  | Thread-64 | JvmMetrics | Cannot
initialize JVM Metrics with processName=JobTracker, sessionId= - already 
initialized
2011-06-27 12:26:56,657 | INFO  | Thread-64 | JvmMetrics | Cannot
initialize JVM Metrics with processName=JobTracker, sessionId= - already 
initialized
2011-06-27 12:26:56,658 | INFO | Thread-64 | MapTask | io.sort.mb = 100 2011-06-27 12:26:56,751 | INFO | Thread-64 | MapTask | data buffer = 
79691776/99614720
2011-06-27 12:26:56,751 | INFO | Thread-64 | MapTask | record buffer = 
262144/327680
2011-06-27 12:26:56,775 | INFO | Thread-64 | MapTask | Starting flush 
of map output
2011-06-27 12:26:56,785 | INFO | Thread-64 | MapTask | Finished spill 0 
2011-06-27 12:26:56,790 | INFO  | Thread-64 | LocalJobRunner |
2011-06-27 12:26:56,790 | INFO  | Thread-64 | TaskRunner | Task
'attempt_local_0004_m_000000_**0' done.
2011-06-27 12:26:56,795 | INFO  | Thread-64 | JvmMetrics | Cannot
initialize JVM Metrics with processName=JobTracker, sessionId= - already 
initialized
2011-06-27 12:26:56,797 | INFO  | Thread-64 | JvmMetrics | Cannot
initialize JVM Metrics with processName=JobTracker, sessionId= - already 
initialized
2011-06-27 12:26:56,797 | INFO  | Thread-64 | LocalJobRunner |
2011-06-27 12:26:56,798 | INFO | Thread-64 | Merger | Merging 1 sorted 
segments
2011-06-27 12:26:56,799 | INFO | Thread-64 | Merger | Down to the last 
merge-pass, with 1 segments left of total size: 350 bytes
2011-06-27 12:26:56,799 | INFO  | Thread-64 | LocalJobRunner |
2011-06-27 12:26:56,800 | INFO  | Thread-64 | JvmMetrics | Cannot
initialize JVM Metrics with processName=JobTracker, sessionId= - already 
initialized
2011-06-27 12:26:56,847 | INFO  | Thread-64 | TaskRunner |
Task:attempt_local_0004_r_**000000_0 is done. And is in the process of 
commiting
2011-06-27 12:26:56,848 | INFO  | Thread-64 | JvmMetrics | Cannot
initialize JVM Metrics with processName=JobTracker, sessionId= - already 
initialized
2011-06-27 12:26:56,849 | INFO  | Thread-64 | LocalJobRunner |
2011-06-27 12:26:56,849 | INFO  | Thread-64 | TaskRunner | Task
attempt_local_0004_r_000000_0 is allowed to commit now
2011-06-27 12:26:56,850 | INFO  | Thread-64 | JvmMetrics | Cannot
initialize JVM Metrics with processName=JobTracker, sessionId= - already 
initialized
2011-06-27 12:26:56,853 | INFO | Thread-64 | FileOutputCommitter | Saved 
output of task 'attempt_local_0004_r_000000_**0' to
file:/tmp/temp-791598942/**tmp1907259615
2011-06-27 12:26:56,853 | INFO | Thread-64 | LocalJobRunner | reduce> 
reduce
2011-06-27 12:26:56,854 | INFO  | Thread-64 | TaskRunner | Task
'attempt_local_0004_r_000000_**0' done.

Reply via email to