ok for whatever reason, log4j seems to fail to do default initialization in
the shell. Perhaps we are tweaking something, or spark is (not) doing some
of its own logging magic setup.
anyhow, calling explicitly
PropertyConfigurator.configure(getMahoutHome() +
"/conf/log4j.properties")
seems to do the trick for the time being. I've added
$MAHOUT_HOME/conf/log4j.properties as default template so that shell would
not spit any INFO. Until there's a better ("right") solution for this
logging nonsense, i suppose that should do.
it is also better to have mahout's own log4j since it only applies for
front-end but any standalone spark workers (in case they are runnin on the
same node) will still be using spark's setup. which is good.
On Fri, Aug 15, 2014 at 5:17 PM, Dmitriy Lyubimov <[email protected]> wrote:
> one more bit of info
>
> i also seem to have tracked down that spark is using slf4j and
> StaticLoggerBinder, so it would seem it is bypassing log4j factory for
> slf4j; but then still, log4j.properties are being picked up.
>
> On the other when i force and hack slf4j to use log4j with mahout shell,
> the log4j.properties still are not being picked up. I have very little
> experience with slf4j, so this is all purplexing to no end.
>
>
>
> On Fri, Aug 15, 2014 at 4:59 PM, Ted Dunning <[email protected]>
> wrote:
>
>> One of our guys just had a very similar problem. I will step over and
>> ask.
>>
>>
>>
>> On Fri, Aug 15, 2014 at 4:51 PM, Dmitriy Lyubimov <[email protected]>
>> wrote:
>>
>> > with spark shell. Trying to clamp down on the INFO logging to console.
>> > Can't seem to solve it.
>> >
>> > Since spark itself seems to add $SPARK_HOME/conf classpath (to verify
>> what
>> > mahout ends up with in this case, run bin/mahout -spark classpath | sed
>> > 's/:/\n/g'), one obvious solution is to modify spark's
>> > conf/log4j.properties itself. When i do that, debug output is clamped
>> down
>> > with spark shell, but not with Mahout's.
>> >
>> > I also did a hack and added log4j.properties into mahout's own
>> classpath,
>> > to no avail either.
>> >
>> > So I can't seem to be able to track down this damn logger initialization
>> > issue, no matter what i do. it would seem Spark's elements of mahout
>> shell
>> > start identically, but for some reason log4j just refuses to initialize
>> > differently.
>> >
>> > this INFO stuff coming from spark is quite annoying in the shell, we
>> need
>> > to be able to regulate console output of the log4j. any help with the
>> issue
>> > is appreciated.
>> >
>> > d
>> >
>>
>
>
