Walter Ray-Dulany commented on PIRK-65:

I've been digging around on this. 

First I thought that perhaps we weren't setting the right property file 
variables to ensure the log level we wanted. I did a bunch of things, but began 
to suspect this wasn't the problem when, after eliminating the entire contents 
of log4j2.xml, the distributed tests produced exactly the same logs as before.

Then I thought the problem was some incompatibility between hadoop (which only 
knows log4j1) and use (using log4j2), and so I rolled my local branch back to 
log4j1, and tried a bunch of different options suggested by various internet 
search hits in a log4j.properties file. This, frustratingly, also did nothing.

Finally, I found some posts that explained things. [this 
post|http://stackoverflow.com/a/29117078] describes how the log4j levels are 
determined in a hadoop/yarn system: for the driver, by a system-wide 
log4j.properties file, and for yarn, by either a data node-by-data node config 
file container-log4j.properties or, if that doesn't exist, by the values 
hard-coded into the hadoop-yarn-server-nodemanager jar file. The author of that 
post assures the reader that we cannot change these no matter how many 
log4j.properties files we add because the classpath used by hadoop will 
pre-empt them. I did some reading to know why/how hadoop was influencing the 
classpath: I found [this explanation|http://stackoverflow.com/a/28260868] 
sufficiently illuminating, and essentially confirmed what the first-linked post 
above was saying.

I haven't yet tested this in a system where I was anything but a tenant, which 
means I can't test changing the global values, or using log4j2 (for us) and...I 
guess a log4j.properties file (to affect hadoop?).

Unless I'm seriously misunderstanding things,
 - In a tenant system without the ability to change global configuration files 
for hadoop and yarn, there is no way to change the logging level.
 - In a system where you have the ability to change the global configuration 
files, you can make global configuration changes to change log levels.
 - I think this ticket should be closed as "Won't Fix" or something similar.

I greatly desire the community's thoughts, and would love nothing more than to 
be wrong, b/c right now I suspect all we can do to debug the distributed 
portions is to, as needed, change logger.debug statements to logger.info 

> Debug Logging Not Working within Distributed Applications
> ---------------------------------------------------------
>                 Key: PIRK-65
>                 URL: https://issues.apache.org/jira/browse/PIRK-65
>             Project: PIRK
>          Issue Type: Bug
>            Reporter: Ellison Anne Williams
>            Assignee: Walter Ray-Dulany
> It appears that debug logging is not working within distributed applications 
> - only within standalone components. 

This message was sent by Atlassian JIRA

Reply via email to