HeartSaVioR edited a comment on issue #25943: [WIP][SPARK-29261][SQL][CORE] Support recover live entities from KVStore for (SQL)AppStatusListener URL: https://github.com/apache/spark/pull/25943#issuecomment-545668898 @Ngone51 Sorry I revisit too late. We've had some sort of a stuck moment since compatibility was raised as a major problem. I've discussed with @vanzin and @squito offline, and we all agreed about allowing no-compatibility on live entities/snapshots for now. (The goal of SPARK-28870 is changed to not allow deleting old event logs.) Based on the agreement, I feel we could just try to serialize live entities by itself (although we should have to recover references between them) instead of recreating them from KVStore entities which seem to be too complicated and likely lossy. Could you update your patch to simplify based on this agreement? Actually I have a POC commit to deal with this, and got a feedback, too verbose (like just let jackson works for me). https://github.com/HeartSaVioR/spark/commit/946bca40f66aa5a1709f04e67a5d61b6b7351a6b I'd be really appreciate if you could take this up and simplify the code change. No need to retain authorship - it's yours if you are willing to take up. Please also let me know if you are not willing to work on. I'll take up the issue. Apologize of the change of direction. I'm really sorry about this for missing critical one to check and leading wrong direction. Please let me know if you have any question or uncertain. Thanks in advance.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
