waynr commented on issue #7135: URL: https://github.com/apache/arrow-rs/issues/7135#issuecomment-2660582057
> I'm not familiar with how the OSS InfluxDb handles this, but the closed source Influx at least used to construct a SessionContext for each query. This would be an ideal opportunity to construct the instrumented MemCachedObjectStore While I'm still learning my way around this code, this suggestion is somewhat at odds with my understanding of how spans ideally work. You're saying that at the bottom level of a query call stack we should create a SessionContext (or rather, a [SpanContext](https://github.com/influxdata/influxdb_iox/blob/7419ce7dc877d9073344e6c77221780142753322/trace/src/ctx.rs#L44) since that's what we're interested in reporting spans to a trace collector like Jaeger) and pass that in to an `ObjectStore` impl then initialize the datafusion and query execution runtimes per-query (whereas these things are currently initialized once at program startup). The problem with this is that the span for any of the `ObjectStore` methods called using this would exist in own separate tree of spans rather than being the child of the context which makes the method calls. I'm also doubtful that it would be possible or efficient to refactor Influxdb3 to do all this object store and datafusion runtime initialization on a per-query basis. To be clear, when I'm talking about spans this is what I mean: https://www.jaegertracing.io/docs/2.3/terminology/#span > TBC designing and then stabilising a session abstraction for object_store is a non-trivial piece of work Yeah, I realize it's a big ask. I definitely appreciate your time discussing this! > it isn't immediately obvious to me that a generic abstraction is even possible I was trying to illustrate how it might be possible by using the existing [`datafusion::execution::SessionConfig` custom configuration](https://docs.rs/datafusion/latest/datafusion/execution/config/struct.SessionConfig.html#custom-configuration) as an example. Spans are being passed to arbitrary implementations of the `TableProvider` trait using this approach in the closed source Influx. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
