vanzin commented on issue #26508: [SPARK-26260][Core]For disk store tasks summary table should show only successful tasks summary URL: https://github.com/apache/spark/pull/26508#issuecomment-554540314 So, to solve this problem, you're changing the format of the data on disk. That breaks backwards compatibility (you'll run into problems if you run with an old disk store). So you have to update `AppStatusStore.CURRENT_VERSION`. Given that I don't see a way to fix this without changing the disk format, I wonder if there are better alternatives than basically doubling the amount of disk space needed to store metrics. What if you use the same existing fields and indices, but record successful tasks with positive numbers, and in progress or failed tasks as negative? You need some slight adjustments (e.g. `TaskDataWrapper.hasMetrics` needs to be a field now instead of computed from metrics, you need to use `math.abs` when returning metrics, etc), but wouldn't it result in the same thing?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
