Ah if its manual ad-hoc logging of the 100 to 200 lines then thats probably OK.
On Mon, Feb 1, 2016 at 3:48 PM, Utkarsh Sengar <utkarsh2...@gmail.com> wrote: > Not alot of string messages, I need it mostly for debugging purposed which > I will use on an ahdoc basis - manually add debug statements which returns > info about the dataset etc. > I would assume the strings will vary from 100-200lines max, that would be > about 50-100KB if they are really long lines. > > -Utkarsh > > On Mon, Feb 1, 2016 at 3:40 PM, Holden Karau <hol...@pigscanfly.ca> wrote: > >> I wouldn't use accumulators for things which could get large, they can >> become kind of a bottle neck. Do you have a lot of string messages you want >> to bring back or only a few? >> >> On Mon, Feb 1, 2016 at 3:24 PM, Utkarsh Sengar <utkarsh2...@gmail.com> >> wrote: >> >>> I am trying to debug code executed in executors by logging. Even when I >>> add log4j's LOG.info(..) inside .map() I don't see it in mesos task logs in >>> the corresponding slaves. >>> Its anyway inefficient to keep checking multiple slaves for logs. >>> >>> One way to deal with this is to push logs to a central location. >>> >>> Another way (for debugging purposes) is to use accumulators . Is it >>> advisable to use accumulators to push string from executors to driver? >>> It will simplify things when I am debugging datasets, bugs which is hard >>> to reproduce locally etc. >>> >>> Suggestions/comments? >>> >>> -Utkarsh >>> >> >> >> >> -- >> Cell : 425-233-8271 >> Twitter: https://twitter.com/holdenkarau >> > > > > -- > Thanks, > -Utkarsh > -- Cell : 425-233-8271 Twitter: https://twitter.com/holdenkarau
