jonathan-albrecht-ibm commented on PR #49811: URL: https://github.com/apache/spark/pull/49811#issuecomment-2640906566
@HeartSaVioR thanks for reviewing. I understand that a schema change could break backwards compatibility. I'd just like to mention in case it helps that all of the tests, except the two in FlatMapGroupsWithStateExecHelperSuite.scala that explicitly assert on the schema, continue to pass. That includes tests that load flatmapgroup checkpoint files from 2.x era. Would it be possible to write a test or something that would show this change is safe? I'd be happy to work on it with some guidance. > So let's move on to the latest version and see whether there is a problem. I'm not sure what `latest version` you mean above but if you mean the v2 version of the state manager then I can say from my testing that the v2 state manager passes all tests on big endian platforms and I don't know of any problems with it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
