tgravescs commented on issue #23310: [SPARK-26363][WebUI] Avoid duplicated KV store lookups for task table URL: https://github.com/apache/spark/pull/23310#issuecomment-447409194 I'm not against the change, one can argue both ways whether it should be in the scope of task data or not. I personally don't see that as a problem based on how we are trying to do server side stuff here. In many ways it makes sense for the rest api to return exactly what you want for your UI so you don't have to do joins or lookups on other tables. logs are directly related to tasks so from a logical perspective they do belong there. Actually I hate how when we stop tracking executors to save memory the log links go away. Its very annoying from a debugging point of view. Reducing the # of lookups should be good. I was just wanting to know if you actually saw a performance issue with this or not. I can change any code I want because I think its better but unless I measure it to prove that it doesn't mean it does or is necessary. In this case since we don't do the executor table on the server side I think this is ok, theoretically that could get out of sync with the task table since its doing server side lookups and not reloading the entire page. This change could make that slightly worse if you get new executors not in that table. But until/if we convert everything to server side I think that is ok.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
