HeartSaVioR commented on issue #25943: [WIP][SPARK-29261][SQL][CORE] Support recover live entities from KVStore for (SQL)AppStatusListener URL: https://github.com/apache/spark/pull/25943#issuecomment-538838751 @Ngone51 I've reviewed only a small part yet, but feel that supporting bi-directional conversions between LiveEntity and instance to be stored to KVStore wouldn't be easy, as LiveEntity has more information than stored instance to deal with incoming events. I've found a missing spot on converting JobDataWrapper to LiveJob (https://github.com/apache/spark/pull/25943#discussion_r331841963), and I'm guessing it would be due to technical reason. I'm not sure it is viable, but it might be pretty much easier if we find a way to store/load live entities to KVStore, with keeping the feature to convert live entities to instance to be stored and update. (may need to do POC - I would be happy to experiment on this) Then we just need to store live entities to KVStore once we flush the (SQL)AppStatusListener (to perform snapshot, or close AppStatusListener) and restore live entities from KVStore once initializing (SQL)AppStatusListener.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
