This is an automated email from the ASF dual-hosted git repository.
pcongiusti pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel.git
The following commit(s) were added to refs/heads/main by this push:
new 61a51722c57 chore(doc): MDC documentation and proposal
61a51722c57 is described below
commit 61a51722c57526b67e6969e5f050808fb92dac23
Author: Pasquale Congiusti <[email protected]>
AuthorDate: Fri Mar 7 13:09:08 2025 +0100
chore(doc): MDC documentation and proposal
---
docs/user-manual/modules/ROOT/pages/mdc.adoc | 36 +++++++++++
proposals/mdc.adoc | 95 ++++++++++++++++++++++++++++
2 files changed, 131 insertions(+)
diff --git a/docs/user-manual/modules/ROOT/pages/mdc.adoc
b/docs/user-manual/modules/ROOT/pages/mdc.adoc
new file mode 100644
index 00000000000..ff98814edeb
--- /dev/null
+++ b/docs/user-manual/modules/ROOT/pages/mdc.adoc
@@ -0,0 +1,36 @@
+= Mapped Diagnostic Context (MDC)
+
+The Mapped Diagnostic Context is a technology used in Java to provide a set of
customized information into each log trace. The major logging frameworks
implements it, and, although it may have certain limitations, this technology
is used to enhance the logging and monitoring of a Java application (Camel
applications included).
+
+The main limitation of this technology is the fact that it stores values on a
context that is available at thread level. Since Camel is an application that
manages multiple thread, when it deals with asynchronous calls, the context
propagation may not work correctly.
+
+NOTE: the framework should generally handle MDC correctly. However, there
could be components (eg, tracing components) and other asynchronous parts of
the system that still require the implementation of the context propagation:
please report if you notice anything wrong.
+
+== How to configure in Camel application
+
+The first thing you need to do is to enable the
`camel.main.useMdcLogging=true`. This flag will automatically include in the
MDC context the following Exchange information:
+
+* camel.breadcrumbId
+* camel.exchangeId
+* camel.messageId
+* camel.correlationId
+* camel.routeId
+* camel.stepId
+* camel.contextId
+* camel.transactionKey
+
+You can use the above variables for MDC depending on the logging framework
you're using. For example, if you're using log4j2, then, the variable will be
like `%X{camel.exchangeId}`. Other logging frameworks should have a similar
approach, just check its specific documentation.
+
+== User values
+
+If you're using Java DSL you can include any customized information by adding
that using low level MDC API:
+
+```java
+ org.slf4j.MDC.put("myKey", "myValue");
+```
+
+Each MDC should be now able to include that information.
+
+== Context propagation
+
+If you're using some asynchronous component, then, you may need to configure
the application to enable the MDC context propagation. For that reason you need
to add the `camel.main.mdcLoggingKeysPattern` configuration. This configuration
will drive the process of copying the MDC context on the thread that will
execute your Exchange asynchronously.
diff --git a/proposals/mdc.adoc b/proposals/mdc.adoc
new file mode 100644
index 00000000000..2ddd054aa84
--- /dev/null
+++ b/proposals/mdc.adoc
@@ -0,0 +1,95 @@
+---
+title: MDC logging
+authors:
+ - "@squakez"
+reviewers: []
+approvers: []
+creation-date: 2025-03-07
+last-updated: 2025-03-07
+status: draft
+see-also: []
+replaces: []
+superseded-by: []
+---
+
+== Summary
+
+Mapped Diagnostic Context (MDC) is a logging technique employed by most of the
Java logging framework to store arbitrary properties and show these information
along with any trace logged to the output stream. This technology has turned
useful as many Apache Camel users are leveraging it to implement a finer
logging and monitoring around their applications (ie, based on custom
parameters).
+
+== Motivation
+
+The technology works fine when running in a single thread and the execution of
Camel route does not swap between different threads. It has however certain
downsides when it overlaps with the asynchronous execution in Camel due to how
the framework manages multithreading and does not propagate the context out of
the box.
+
+== Goals
+
+Goal of this proposal is to analyze the actual design, the challenges we are
facing and provide an alternative design to have a simpler long term
maintenance and a better user experience.
+
+== Context
+
+The MDC feature is not actually an organic part of the framework. We have
certain features around MDC (above all on tracing components) that can be
configured. We do miss the context propagation, so, whenever an asynchronous
execution come in place, the result may not be in line with the user
expectation (given the limitation of MDC when dealing with multi-threading).
+
+=== Actual design
+
+The core can creates a `MDCUnitOfWork` when the user explicitly asks for the
usage of MDC (via `camel.main.useMdcLogging`). This is in charge to include a
predefined set of variables into the MDC context. The user can include these
information in the logging. Once the "Unit of work" is completed, then, the
core clear the context.
+
+==== Inconsistent implementation
+
+Each asynchronous component or asynchronous part of the system is required to
explicitly manage the context propagation, leading to a situation of possible
inconsistency. At the moment we have certain components that can handle that,
and others that don't. The necessity to explicitly include such context
propagation is a maintenance problem we need to take in consideration.
+
+==== Fixed variables
+
+The problems we have with this approach are the fixed number of variables we
are able to include.
+
+==== Low level of abstraction
+
+If the user wants to include an additional parameter he must access directly
the low level MDC API via an additional Processor. And this would only be
available if the user is programming in Java DSL. We miss a higher level
abstraction to be able to expose and use the feature regardless of the DSL of
choice.
+
+==== Obsolescence of the technology
+
+We must also consider that the MDC is an old technology that may not
necessarily evolve favorably in the future. It is clear that there is space for
certain use case, but, having it embedded directly in different core
dependencies and other components, may pose challenges from a maintenance
perspective if, in the future, this technology is deprecated.
+
+== Proposal
+
+We can leverage the component "plugin" approach in order to develop a sort of
"MDC component" whose goal is to provide the major features expected by this
technology. It would take care to set the values, let the logging system use
them in their lifecycle and clear the values when the logging is over. We have
something similar in place for telemetry components, where we extend the
`LogListener` to capture the logging events:
+
+```java
+public interface LogListener {
+
+ /**
+ * Invoked right before Log component or Log EIP logs. Note that {@link
CamelLogger} holds the {@link LoggingLevel}
+ * and {@link org.slf4j.Marker}. The listener can check {@link
CamelLogger#getLevel()} to see in which log level
+ * this is going to be logged.
+ *
+ * @param exchange camel exchange
+ * @param camelLogger {@link CamelLogger}
+ * @param message log message
+ * @return log message, possibly enriched by the listener
+ */
+ String onLog(Exchange exchange, CamelLogger camelLogger, String message);
+
+}
+```
+
+We can include a couple of more methods such as `beforeLog()` (which it seems
to be identical with the `onLog()` purpose) and `afterLog()` whose goal is to
set and clear the MDC context accordingly, therefore making sure that the
information is stored in the same thread that will write any log trace and
cleared after its consumption.
+
+Of course, we should include the call of these new methods from the core,
likely wrapping the existing `onLog()` execution. The presence of these new
methods may open potential future features as it would make much more flexible
the control of the logging lifecycle for any other `LogListener`s
implementation (for example, the telemetry ones).
+
+=== Higher cohesion and abstraction
+
+The new component would take over entirely the logic around MDC. It could be
more easily tested and maintained and it should be applicable to the regular
lifecycle of Camel application, either it runs synchronously or asynchronously.
With this mechanism we don't require any longer a context propagation, as, each
execution of logging would be in charge to do the work on the same thread where
it is executed.
+
+=== Include exchange values
+
+With this mechanism in place we should be also able to drive the setting of
properties to include in the MDC with Camel Exchange headers or properties,
instead of letting the user to handle that via lower level API. We should have
a component parameter that will let the user choose which are the variables
included in the MDC context.
+
+=== Available for any DSL
+
+With this higher level of abstraction, the usage of MDC would be possible
through any DSL. Moreover it would be much more consistent and less error
prone, as it's the same user the one that can set the required headers (with
the canonical Camel way of setting headers), configure the MDC components with
the headers to use and configure the logging system with the variables he wants
to trace.
+
+=== Long term maintenance
+
+If the above is proven to work effectively, then, in any future major version
we can remove all the existing parts in the core components which are related
to the MDC, simplifying a lot the long term maintenance of the project.
+
+== Development
+
+This design proposals should not introduce any breaking compatibility changes.
The old and new MDC mechanism can coexist, although it will be recommendable to
deprecate the old one once the new one proves to work correctly.
