Hi Prashant, Thanks for clarifying that pair-wise correlation of multiple before/after event pairs per request is _not_ required.
That was my only concern in this thread. I do not have any additional use cases in mind. I was just trying to achieve clarity on what was actually required and what was collateral :) Cheers, Dmitri. On Mon, Nov 3, 2025 at 10:48 PM Adnan Hemani <[email protected]> wrote: > Hi Dmitri, > > Thanks for creating this thread! If what has sparked this thought was my > input earlier that users would like to correlate their events together, I'd > like to make a clarification: I believe that users would like to correlate > events coming from the same request to Polaris together. In which case, I > think Request ID is sufficient to do what we need. If there is other input > that we should be able to correlate pairs of Before/After events, then I > think we should discuss either adding "Correlation ID" (which I really > don't like, as it makes things quite confusing IMO) or putting a "Span ID" > inside the `additionalProperties` of the event, like Yufei suggested (and > we may have to generate that "Span ID"). > > As to Yufei's point, I don't think we need to explicitly store OTel context > information. If it is present through Quarkus' integration with OTel, I > still believe the OTel Trace ID should be the Request ID, if the user has > not defined a Request ID (sorry to mix the lines with the other threads' > topics on this). > > WDYT? > > Best, > Adnan Hemani > > On Fri, Oct 31, 2025 at 11:47 AM Yufei Gu <[email protected]> wrote: > > > Agreed with Dmitri. > > > > X-Request-Id (or Request-Id) is not a standard HTTP header but is widely > > used across systems, including Polaris, for logging, events, and metrics. > > Its primary purpose is debuggability. For example, it lets us correlate > > logs and events for the same HTTP call. > > Example: > > Log: {timestamp: ..., request-id: 123, message: "Update table > > successfully!"} > > Event: {type: BeforeUpdateTable, request-id: 123}, {type: > AfterUpdateTable, > > request-id: 123} > > > > *Request ID* should remain the canonical identifier for every request > > handled by Polaris. > > > > - If missing, Polaris generates one (e.g., UUIDv7 or brings back the > one > > Adnan built). > > - All logs, events, and metrics include it for consistent internal and > > downstream correlation. No need to invent another concept like > > "correlated ID" for event only, which introduce inconsistency and make > > correlation harder. > > > > *OTel context* is OPTIONAL. > > > > - If present, persist it alongside for tracing. Put it in > > `additionalProperties` for events if people need it. > > - If absent, users decide to start a new trace (nice to have) or not. > > > > > > Yufei > > > > > > On Fri, Oct 31, 2025 at 10:37 AM Dmitri Bourlatchkov <[email protected]> > > wrote: > > > > > Hi All, > > > > > > I'd like to broaden the events / OTel discussion [1][2] a bit. > Apologies > > > for forking the email threads even more, but I think this aspect > > deserves a > > > focused discussion. > > > > > > From previous messages my impression is that there are use cases for > > > strongly correlating "before" and "after" events. In that situation the > > > full OTel context may be too volatile. For example, nothing guarantees > > that > > > the "before" and "after" events are produced under the same OTel span > in > > > Polaris. > > > > > > Another aspect of this is multiple before/after event pairs may be > > produced > > > under the same API request. Assuming event consumers do want to > > > re-establish correct event pairs, I believe using only the request ID > is > > > not sufficient for that purpose. > > > > > > We probably need to restore the Polaris request ID generator but use it > > to > > > produce an ID per group of events (before/after) and publish / persist > > that > > > ID with the event data. > > > > > > WDYT? > > > > > > [1] https://lists.apache.org/thread/bb1qyxjt827t3tomv2xp0s1kovxjsp94 > > > [2] https://lists.apache.org/thread/5dpyo0nn2jbnjtkgv0rm1dz8mpt132j9 > > > > > > Thanks, > > > Dmitri. > > > > > >
