Spenserrrr commented on PR #57407: URL: https://github.com/apache/spark/pull/57407#issuecomment-5040175298
CI found an error in my fix. For `StreamingQueryListener._set_spark_session`'s annotation, I originally pinned it to the classic SparkSession. CI reports a type error here, and I found out that itsonly callers are in Spark Connect (connect/streaming/query.py, connect/streaming/worker/listener_worker.py), which pass a connect SparkSession. Thus, I typed all three as ConnectSparkSession instead. Union["SparkSession", "ConnectSparkSession"] would also work, and I see that this is a base class that is used by both classic and connect. I can switch to this if you prefer to keep the base class backed-agnostic. @HyukjinKwon -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
