Brian, I would expect a PR to OTEL's java auto instrumentation project over the > next few months that adds NiFi to its list of instrumentations. If the NiFi > committers would like a demo / tech exchange to go over the current state > of the tracing agent, we'd be happy to accommodate. As it stands, the agent > utilizes flowfile attributes to pass along the tracestate so trace > propagation can occur across NiFi to NiFi boundaries. >
I think a lot of us would love to get on a call for a demo and discuss this work. It sounds great! How would you like to proceed? Le mar. 23 mai 2023 à 12:19, Michael Hogue <michael.p.hogu...@gmail.com> a écrit : > Related & complementary to Brian's tracing work would be modeling > provenance/lineage in OTEL. I chatted with some OTEL maintainers about > modeling provenance natively as a top-level Signal [1] at KubeCon and they > advised raising an issue in the opentelemetry-specification repo for > discussion: > https://github.com/open-telemetry/opentelemetry-specification/issues/3447 > > Their in-person response was largely that OTEL events could be used to > model provenance, but I argued that the model should be standardized for > use across tooling. This way, the provenance model wouldn't need to be > maintained separately. > > My original thought was that we could have a standard provenance model in > OTEL and shift NiFi to emit these instead of its own provenance events. > Then we can tie provenance together across heterogeneous tooling using log > visualization through something like Loki & Grafana, but there are many > options in this space. > > Thanks, > Mike > > [1] https://opentelemetry.io/docs/concepts/signals/ > > On Tue, May 23, 2023 at 3:56 AM Brian Putt <puttbr...@gmail.com> wrote: > > > Hello Joe / All, > > > > Jaeger or Grafana (w/ tempo) offer comparable tools to visualize the > trace > > data. I believe additional tools will be needed to get the most out of > the > > trace data. We've been experimenting with a number of open source > products > > to see what works best for the amount of trace data that NiFi emits. So > > far, Grafana Tempo, Victoria Metrics, and Clickhouse seem to offer a good > > set of features to cover searching / viewing the traces along with > > summarizing certain flowfile attributes. As long as the trace data is in > > OTEL's format, the collector offers flexibility in exporting the data to > a > > number of services with ease. > > > > I would expect a PR to OTEL's java auto instrumentation project over the > > next few months that adds NiFi to its list of instrumentations. If the > NiFi > > committers would like a demo / tech exchange to go over the current state > > of the tracing agent, we'd be happy to accommodate. As it stands, the > agent > > utilizes flowfile attributes to pass along the tracestate so trace > > propagation can occur across NiFi to NiFi boundaries. > > > > Thanks, > > > > Brian > > > > On Wed, May 17, 2023 at 1:05 PM Joe Witt <joe.w...@gmail.com> wrote: > > > > > Brian Putt, All > > > > > > Are you aware of any good tools/services that can ingest the traces and > > > provide an interesting view/story/reporting on it? > > > > > > I could see us emitting otel events instead of our current provenance > > > mechanism and using that both internally to do what we already do but > > also > > > have a clear/spec friendly way of exporting it to others. > > > > > > Thanks > > > > > > On Sat, Jul 30, 2022 at 7:43 AM u...@moosheimer.com <u...@moosheimer.com > > > > > wrote: > > > > > > > Hello Brian, Bryan, Greg, NiFi devs, > > > > > > > > Integrating OpenTelemetry is a very good idea, especially since the > > major > > > > cloud providers also rely on it. This could also be interesting for > > > > Stateless NiFi. > > > > > > > > I have a suggestion that I would like to put up for discussion. > > > > > > > > Would it be useful to make a list of what extensions or new > development > > > > would be helpful for a complete integration of OpenTelemetry? > > > > > > > > I'm thinking of ConsumeMQTT and PublishMQTT, for example. Currently > > these > > > > can do max. MQTT version 3.11, but since version 5 the User > Properties > > > > exist, which are similar to the HTTP header fields. > > > > Thus one could implement OpenTelemetry in the MQTT processors > similarly > > > as > > > > in HTTP. > > > > > > > > With a list we could make an overview of the "necessary" adjustments > > and > > > > advertise for support. > > > > > > > > If what I write is nonsense, then I may not have understood something > > and > > > > I take it all back :) > > > > > > > > Mit freundlichen Grüßen / best regards > > > > Kay-Uwe Moosheimer > > > > > > > > > Am 29.07.2022 um 05:09 schrieb Brian Putt <puttbr...@gmail.com>: > > > > > > > > > > Hello Bryan / Greg / NiFi devs, > > > > > > > > > > Distributed tracing (DT) is similar to provenance in that it shows > > the > > > > path > > > > > a particular flowfile travels, but its core selling point is that > it > > > > > supports tracing across multiple systems/services regardless of > > what's > > > > > receiving the data. Provenance is a fantastic feature and there are > > > > > instances where one might want to draw that bigger picture of > > > identifying > > > > > bottlenecks as data flows from one system to another and that > system > > > > > may/may not be using NiFi. > > > > > > > > > > DT utilizes three ids: traceId, parentId, and spanId. While a tree > > can > > > be > > > > > built using two ids, the third id (traceId) helps bring all of the > > > > relevant > > > > > information out of a datastore more easily. > > > > > DT is focused more on performance and identifying bottlenecks in > one > > or > > > > > more systems. Imagine if NiFi were receiving data from various > > sources > > > > > (i.e. HTTP, Kafka, SQS) and NiFi egressed to other sources (HTTP, > > > Kafka, > > > > > NiFi). > > > > > DT provides a spec that we'd be able to follow and correlate the > data > > > as > > > > it > > > > > traverses from system to system. Each system that participates in > the > > > DT > > > > > ecosystem would simply emit information (a trace is made up of one > or > > > > more > > > > > spans) and there'd be a collection system which would aggregate all > > of > > > > > these spans and would draw a bigger picture of the path that data > > went > > > > > through and could help identify key bottlenecks. > > > > > > > > > > OpenTelemetry (OTEL) provides clients (across many languages, > > including > > > > > java) where developers can instrument their library's APIs and > > > > participate > > > > > in a DT ecosystem as it adheres to the tracing spec. Egressing > trace > > > data > > > > > is possible without using OTEL, but then we may find ourselves > having > > > to > > > > > recreate the wheel, but could be optimized for NiFi. > > > > > > > > > > Creating a reporting task could certainly be a path, mainly have a > > few > > > > > concerns with that: > > > > > > > > > > 1. If provenance is disabled, will provenance events still be > emitted > > > and > > > > > be collected by a new reporting task? > > > > > 2. There'll be an impact on performance, how much is unknown. OTEL > is > > > > > gaining traction across industry and there are ways to mitigate > > > > > performance, mainly sampling and the fact that *tracing is best > > > effort*. > > > > > Spans would be emitted from NiFi via UDP to a collector on the same > > > > network > > > > > 3. Would there be any issues with appending a flowfile attribute > that > > > is > > > > > carried throughout the flow where it maintains the traceId, > > > parentSpanId, > > > > > and trace flags? See below for more details > > > > > > > > > > There's a W3C spec (Trace context) which includes a formatted > string > > > that > > > > > would be propagated to services (HTTP, Kafka, etc...). So if NiFi > > were > > > to > > > > > put information onto kafka, any consumers of that data would be > able > > to > > > > > continue the trace and help draw the bigger picture. > > > > > > > > > > W3C Spec: https://www.w3.org/TR/trace-context/#traceparent-header > > > > > > > > > > For #2, since DT is focused on performance, sampling can help > > alleviate > > > > > chatter over the wire and ideally, 0.01% would draw the same > picture > > as > > > > 1% > > > > > or 10%+. This is certainly different from provenance as DT is > focused > > > on > > > > > performance over quality of the data and should not be thought of > as > > > > > auditing. > > > > > > > > > > > > > > > https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/sdk.md#sampler > > > > > > > > > >> On Thu, Jul 28, 2022 at 5:01 PM Bryan Bende <bbe...@gmail.com> > > wrote: > > > > >> > > > > >> Hi Greg, > > > > >> > > > > >> I don't really know anything about OpenTelemetry, but from the > > > > >> perspective of integrating something into the framework, some > things > > > > >> to consider... > > > > >> > > > > >> Is there some way to piggy-back on provenance and use a > > ReportingTask > > > > >> to process provenance events and report something to > OpenTelemetry? > > > > >> > > > > >> If something new does need to be added, it should probably be an > > > > >> extension point where there is an interface in the framework-api > and > > > > >> different implementations can be plugged in. > > > > >> Ideally the framework itself wouldn't have any knowledge of > > > > >> OpenTelemetry specifically, it would only be reporting some > > > > >> information, which could then be used in some way by the > > OpenTelemetry > > > > >> implementation. > > > > >> > > > > >> How does NiFi actually communicate with OpenTelemetry? Are you > > > > >> expecting to send data to OpenTelemetry in this new method you are > > > > >> suggesting? > > > > >> That would likely have a significant impact on the performance of > > the > > > > flow. > > > > >> > > > > >> Thanks, > > > > >> > > > > >> Bryan > > > > >> > > > > >>> On Thu, Jul 28, 2022 at 3:17 PM glma...@uwe.nsa.gov < > > > > glma...@uwe.nsa.gov> > > > > >>> wrote: > > > > >>> > > > > >>> Nifi Devs, > > > > >>> > > > > >>> My team and I are looking for guidance on how we can extend > Apache > > > > >> Nifi's capabilities. Specifically we're looking to include > > distributed > > > > >> tracing. We'll approach this effort as if we're the tracing > experts > > > and > > > > >> simply seeking implementation guidance. Our developers have good > > > > exposure > > > > >> to working with Nifi and creating custom processors. We plan to > fork > > > the > > > > >> project to begin this effort but want to make sure we approach > this > > > with > > > > >> the best possible direction for community adoption. > > > > >>> > > > > >>> Our initial thoughts on this approach would be to piggyback on > how > > > > >> Provenance was implemented. We essentially want to include a > > > subroutine > > > > or > > > > >> method that gets implicitly invoked upon a processors 'onTrigger' > > > > method. > > > > >> From there we would analyze the FlowFiles attributes to check for > > the > > > > >> existence of 'traceId' and/or propagate one if found. > > > > >>> > > > > >>> We can expound upon all of these tracing/observability details if > > > that > > > > >> helps by any means. We're able to provide more detailed scope of > > this > > > > task > > > > >> as well but for now we just want to get feed back for our overall > > goal > > > > and > > > > >> proposed approach. > > > > >>> > > > > >>> Thanks, > > > > >>> Greg Marshall > > > > >> > > > > > > > > > > > > > >
