[
https://issues.apache.org/jira/browse/SPARK-51731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-51731:
----------------------------------
Target Version/s: (was: 4.1.0)
> Introduce a new callback "onQueryTriggerStart" to StreamingQueryListener
> ------------------------------------------------------------------------
>
> Key: SPARK-51731
> URL: https://issues.apache.org/jira/browse/SPARK-51731
> Project: Spark
> Issue Type: Improvement
> Components: Connect, Java API, PySpark, Spark Core, Structured
> Streaming
> Affects Versions: 4.1.0
> Reporter: Jevon Cowell
> Priority: Minor
>
> Currently, {{{}StreamingQueryListener{}}}s does not notify users when a query
> is triggered; it only provides updates when a query starts, progresses,
> becomes idle, or terminates. By introducing a a new event callback that is
> emitted whenever a query is triggered, users will be informed of this
> occurrence, allowing for a variety of use cases.
> In my use case, I plan to use this new event along with {{QueryProgress}} and
> {{QueryIdle}} to develop a Maintenance Job Orchestration System. This system
> will programmatically submit and execute Spark commands, determining the
> eligibility of maintenance job executions based on the activity status of
> streaming queries.
> I've already started an initial implementation on my own
> [fork|https://github.com/JevonCowell/spark/pull/1] but before proceeding with
> writing test to verify functionality, I want to ensure this is something
> apache folks are interested in having added in the first place.
> Note that for a feature like this I'm:
> * Ensuring there is backwards compatibility so that any folks with their own
> custom SparkQueryListener won't break upon upgrading.
> * Disabling by default to not add unnecessary load to folks who are just
> upgrading their dependencies
> * Aiming to set a sane interval to determine how often an event like this
> should be emitted per query
> I tired my best to see if a request like this had come up before but was
> unable to. If it has please link me back to that ticket.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
