alexanderbianchi opened a new issue, #27169: URL: https://github.com/apache/datafusion/issues/27169
### Is your feature request related to a problem or challenge? Applications embedding DataFusion can benefit from a common place to add custom observability around data source execution. `DataSourceExec` provides a natural boundary for observing partition opening and the resulting stream across the sources that use it. Use cases include: - Collecting a consistent set of application-defined scan metrics across data sources, such as output rows and batches, opening failures, stream failures, and completed or cancelled scans. - Choosing metric names and labels in the application, including grouping multiple source implementations under the same metric definitions. - Attaching tracing context while opening a partition and polling its stream, so work performed by the source can participate in the application's traces. - Applying instrumentation selectively based on the source and task context, while retaining source-specific metrics alongside application metrics. The aim is to make shared instrumentation easy to reuse as new sources are added, while keeping application-specific metric definitions and export policy outside DataFusion. ### Describe the solution you'd like An opt-in extension point around `DataSourceExec` execution. There are at least two possible API directions: **1. A flexible execution hook** Pass application code the source, partition, task context, a persistent metrics set, and a callback that opens the partition. The application can establish context around the callback and wrap the returned stream to observe its lifecycle. This leaves the instrumentation vocabulary open: applications can implement counters, timing, tracing, and cancellation handling without DataFusion defining each event. The tradeoff is that implementations write their own stream wrappers and must preserve the stream's schema, output, errors, and polling behavior. **2. A defined observer API** Have DataFusion own the stream wrapper and expose explicit observation points, for example partition opening, opening failure, output batch, stream error, completion, and cancellation. A factory could create an observer for each partition, with access to the source and task context. This would give applications common lifecycle semantics and less wrapper code. It would also mean deciding which events and measurements belong in the API and how to evolve them. Tracing context around opening and each poll deserves particular attention: notifications after an operation alone cannot establish context for work inside that operation. Both designs could use a session extension for registration and expose application-created metrics through the execution plan's existing metrics interface. ### Describe alternatives you've considered Instrumentation can also live in individual data sources or in custom execution-plan wrappers. Those remain useful for source-specific details or broader operator instrumentation. A shared `DataSourceExec` extension point would allow applications to reuse instrumentation across the sources that already execute through this node. Questions to settle include: - Should the public API offer a general execution hook, defined lifecycle callbacks, or a small hook with reusable observer helpers? - Where should observation happen relative to source limits and output batch splitting? - What lifecycle semantics should apply to opening errors, stream errors, cancellation, and plan resets? - How should application metrics coexist with source metrics, including sources that do not retain their own metric set? Timing semantics should be explicit: a scan stream's lifetime includes downstream backpressure and is different from time spent polling the source or servicing storage requests. ### Additional context Draft implementation: https://github.com/apache/datafusion/pull/27167 The draft explores the flexible hook design with session registration, observation after batch splitting, and executor-owned storage for application metrics. It is a concrete starting point for discussing the API rather than a decision to prefer that design. The proposed scope is sources executed through `DataSourceExec`. Direct source opening and custom execution plans are outside that boundary. A session-based hook would be installed in each executing process rather than serialized with the plan. ### Willingness to contribute I would like to contribute to this feature, but request guidance from the DataFusion community. An initial implementation is available in the linked draft PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
