HeartSaVioR opened a new pull request, #37368: URL: https://github.com/apache/spark/pull/37368
Credit to @pranavanand on figuring out the issue and providing the broken test code! ### What changes were proposed in this pull request? This PR proposes to refresh the destination catalog table when streaming query is writing to the catalog table via DSv1 sink. ### Why are the changes needed? It has been long standing issue that streaming query is not aware of catalog table (not the table brought by DSv2), hence no way for streaming query to refresh the catalog table after updating it. As a side effect of SPARK-39564, Spark brings up the information of destination catalog table into MicrobatchExecution, which enables to refresh table. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? New UT. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
