HeartSaVioR commented on issue #26127: [SPARK-29348][SQL] Add observable Metrics for Streaming queries URL: https://github.com/apache/spark/pull/26127#issuecomment-542994970 1) StreamingQueryListener doesn't work with continuous mode. Please refer #24537 which proposes to support StreamingQueryListener in continuous mode. (I haven't had a look in details though... I'd spend my time on continuous mode once we are clear it's still making progress.) 2) In continuous mode, the task is never completed - the patch seems to do something when task is completed which will never happen. ``` // Only publish the value of the accumulator when the task has completed. This is done by // updating a task local accumulator ('updater') which will be merged with the actual // accumulator as soon as the task completes. This avoids the following problems during the // heartbeat: // - Correctness issues due to partially completed/visible updates. // - Performance issues due to excessive serialization. ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
