Hi,
I had the same problem with pyspark. Here's how I resolved it: What I've found in python (not sure about scala) is that if the function being serialized was written in the same python module as the main function, then logging fails. If the serialized function is in a separate module, then logging does not fail. I just created this gist to demo the situation and (python) solution. Is there a similar way to do this in scala? https://gist.github.com/adgaudio/0191e14717af68bbba81 Alex On Mon, Jun 2, 2014 at 7:18 PM, Shivani Rao <raoshiv...@gmail.com> wrote: > Hello Spark fans, > > I am trying to log messages from my spark application. When the main() > function attempts to log, using log.info() it works great, but when I try > the same command from the code that probably runs on the worker, I > initially got an serialization error. To solve that, I created a new logger > in the code that operates on the data, which solved the serialization issue > but now there is no output in the console or on the worker node logs. I > don't see any application level log messages in the spark logs either. When > I use println() instead, I do see console output being generated. > > I tried the following and none of them works > > a) pass log4j.properties by using -Dlog4j.properties in my java command > line initiation of the spark application > b) setting the properties within the worker by calling log.addAppender(new > ConsoleAppender) > > None of them work. > > What am i missing? > > > Thanks, > Shivani > -- > Software Engineer > Analytics Engineering Team@ Box > Mountain View, CA >
