ahshahid opened a new issue, #5935: URL: https://github.com/apache/iceberg/issues/5935
Some of the functions of Spark3Util which are internally invoking the catalog being used by the current SparkSession, do not get the right Catalog Instance, as the current session is not being set as the active one , in the ThreadLocal of SparkSession. As a result , the catalog instance being retrieved is the wrong one ( of the default spark session which got created first). The bug shows up if a new session is created and iceberg's SparkSessionCatalog is set in the new session instead of the original Session. The fix could be 1) Either we document that some methods of Spark3Util be invoked only after setting the current session as active one Or 2) The Spark3Util's code sets the session as active one. Creating a PR based on #2 The PR has the bug test -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
