naveenp2708 commented on PR #57644: URL: https://github.com/apache/spark/pull/57644#issuecomment-6488841099
Thanks @szehon-ho. Yeah the auto-review had a point on that two-run test. It was making a new MemoryStream for each run, so the second run just picked up offset 0 again from the checkpoint and didn’t actually merge anything — the final check was basically passing for free. Changed it to use one stream across both runs (same as AutoCdcScd1SchemaEvolutionSuite) so the older event in run 2 shows up as a real second microbatch and gets merged against the newer row. I also reverted the engine-order rebuild to make sure it still catches the bug — it does, run 2 fails with SEQUENCING_TYPE_DRIFT. Everything else is just tests and comments, no logic changed. Added a test so validation uses the flow’s resolver and not the session one, and fixed the stale comments you pointed out. I kept the reserved column nullable on purpose since that’s what keeps the declared and omitted paths building the same shape — added a note in the scaladoc. Can switch it if you’d rather hold onto a declared NOT NULL. scalastyle’s clean locally and CI’s green now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
