Github user squito commented on the issue:
    I agree with @jiangxb1987 ... we already have issues with event logs being 
too big, as it the driver gets backlogged even writing them out, and then the 
history server takes a long time to parse those files.  There have been recent 
improvements to that, but doesn't mean we should reintroduce the problem.
    I'm not saying this doesn't have a use, I'd just like to figure out if this 
the best way to do it.  If it only has one very specific use case for 
@LantaoJin , then maybe they have an alternative still using public apis, with 
a custom listener as I suggested.  I worry a user might turn this on (why not, 
more data is better) and then later on hit other scaling challenges and not 
realize this was the problem.
    Or if this does have some general use case for all users, then maybe its 
fine, but I haven't seen that yet.  And maybe there is a better way to do that 
... do we need another way to get detailed output metrics from executors, that 
doesn't have some of the scaling challenges of the event log? 


To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to