This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
The following commit(s) were added to refs/heads/master by this push:
new d646821089 JAMES-4083 removed logback jackson dependency for james and
added it for crowdsec at runtime
d646821089 is described below
commit d646821089a95befde339bc65e07b0eb628190b3
Author: Florent Azavant <fazav...@linagora.com>
AuthorDate: Thu Oct 31 16:29:49 2024 +0100
JAMES-4083 removed logback jackson dependency for james and added it for
crowdsec at runtime
---
docs/modules/servers/partials/operate/logging.adoc | 148 +--------------------
pom.xml | 5 -
server/apps/cassandra-app/pom.xml | 4 -
.../helm-chart/james/configs/logback.xml | 52 ++++----
server/apps/distributed-app/pom.xml | 4 -
server/apps/distributed-pop3-app/pom.xml | 4 -
server/apps/jpa-app/pom.xml | 4 -
server/apps/jpa-smtp-app/pom.xml | 4 -
server/apps/memory-app/pom.xml | 4 -
server/apps/scaling-pulsar-smtp/pom.xml | 4 -
third-party/crowdsec/Dockerfile | 3 +
third-party/crowdsec/README.md | 22 ++-
third-party/crowdsec/docker-compose.yml | 4 +-
third-party/crowdsec/pom.xml | 6 +
.../crowdsec/sample-configuration/logback.xml | 83 ++++++------
15 files changed, 104 insertions(+), 247 deletions(-)
diff --git a/docs/modules/servers/partials/operate/logging.adoc
b/docs/modules/servers/partials/operate/logging.adoc
index f48f35d92a..e8da4c3a4e 100644
--- a/docs/modules/servers/partials/operate/logging.adoc
+++ b/docs/modules/servers/partials/operate/logging.adoc
@@ -25,149 +25,7 @@ and link:https://docs.fluentbit.io/[FluentBit] as
centralize logging.
Information about logback configuration can be found
link:http://logback.qos.ch/manual/configuration.html[here].
-== Structured logging
+Note that dedicated formatters can be added to your docker containers :
-=== Using FluentBit as a log forwarder
-
-==== Using Docker
-
-{server-name} leverages the use of MDC in order to achieve structured logging,
and better add context to the logged information. We furthermore ship json logs
to file with RollingFileAppender on the classpath to easily allow FluentBit to
directly tail the log file.
-Here is a sample conf/logback.xml configuration file for logback with the
following pre-requisites:
-
-Logging in a structured json fashion and write to file for centralizing
logging.
-Centralize logging third party like FluentBit can tail from logging’s file
then filter/process and put in to OpenSearch
-
-....
-<?xml version="1.0" encoding="UTF-8"?>
-<configuration>
-
- <contextListener
class="ch.qos.logback.classic.jul.LevelChangePropagator">
- <resetJUL>true</resetJUL>
- </contextListener>
-
- <appender name="LOG_FILE"
class="ch.qos.logback.core.rolling.RollingFileAppender">
- <rollingPolicy
class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
-
<fileNamePattern>logs/james.%d{yyyy-MM-dd}.%i.log</fileNamePattern>
- <maxHistory>1</maxHistory>
- <totalSizeCap>200MB</totalSizeCap>
- <maxFileSize>100MB</maxFileSize>
- </rollingPolicy>
-
- <encoder
class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
- <layout
class="ch.qos.logback.contrib.json.classic.JsonLayout">
-
<timestampFormat>yyyy-MM-dd'T'HH:mm:ss.SSSX</timestampFormat>
-
<timestampFormatTimezoneId>Etc/UTC</timestampFormatTimezoneId>
-
- <!-- Importance for handling multiple lines log -->
- <appendLineSeparator>true</appendLineSeparator>
-
- <jsonFormatter
class="ch.qos.logback.contrib.jackson.JacksonJsonFormatter">
- <prettyPrint>false</prettyPrint>
- </jsonFormatter>
- </layout>
- </encoder>
- </appender>
-
- <root level="INFO">
- <appender-ref ref="LOG_FILE" />
- </root>
-
-</configuration>
-....
-
-First you need to create a `logs` folder, then mount it to James container and
to FluentBit.
-
-docker-compose:
-
-include::{docker-compose-code-block-sample}[]
-
-FluentBit config as:
-the `Host opensearch` pointing to `opensearch` service in docker-compose file.
-....
-[SERVICE]
- Parsers_File /fluent-bit/etc/parsers.conf
-
-[INPUT]
- name tail
- path /fluent-bit/log/*.log
- Parser docker
- docker_mode on
- buffer_chunk_size 1MB
- buffer_max_size 1MB
- mem_buf_limit 64MB
- Refresh_Interval 30
-
-[OUTPUT]
- Name stdout
- Match *
-
-
-[OUTPUT]
- Name es
- Match *
- Host opensearch
- Port 9200
- Index fluentbit
- Logstash_Format On
- Logstash_Prefix fluentbit-james
- Type docker
-....
-
-FluentBit Parser config:
-....
-[PARSER]
- Name docker
- Format json
- Time_Key timestamp
- Time_Format %Y-%m-%dT%H:%M:%S.%LZ
- Time_Keep On
- Decode_Field_As escaped_utf8 log do_next
- Decode_Field_As escaped log do_next
- Decode_Field_As json log
-....
-
-==== Using Kubernetes
-
-If using James in a Kubernetes environment, you can just append the logs to
the console in a JSON formatted way
-using Jackson to easily allow FluentBit to directly tail them.
-
-Here is a sample conf/logback.xml configuration file for achieving this:
-
-....
-<?xml version="1.0" encoding="UTF-8"?>
-<configuration>
-
- <contextListener
class="ch.qos.logback.classic.jul.LevelChangePropagator">
- <resetJUL>true</resetJUL>
- </contextListener>
-
- <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
- <encoder
class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
- <layout
class="ch.qos.logback.contrib.json.classic.JsonLayout">
-
<timestampFormat>yyyy-MM-dd'T'HH:mm:ss.SSSX</timestampFormat>
-
<timestampFormatTimezoneId>Etc/UTC</timestampFormatTimezoneId>
-
- <!-- Importance for handling multiple lines log -->
- <appendLineSeparator>true</appendLineSeparator>
-
- <jsonFormatter
class="ch.qos.logback.contrib.jackson.JacksonJsonFormatter">
- <prettyPrint>false</prettyPrint>
- </jsonFormatter>
- </layout>
- </encoder>
- </appender>
-
- <root level="INFO">
- <appender-ref ref="CONSOLE" />
- </root>
-
-</configuration>
-....
-
-Regarding FluentBit on Kubernetes, you need to install it as a DaemonSet. Some
official template exist
-with FluentBit outputting logs to OpenSearch. For more information on how to
install it,
-with your cluster, you can look at this
https://docs.fluentbit.io/manual/installation/kubernetes[documentation].
-
-As stated by the
https://docs.fluentbit.io/manual/installation/kubernetes#details[detail] of the
-official documentation, FluentBit is configured to consume out of the box logs
from containers
-on the same running node. So it should scrap your James logs without extra
configuration.
+ - Add the corresponding JAR onto the `/root/libs` folder of docker container
+ - Add the JAR path onto `/root/jib-classpath-file` for it to be loaded on the
classpath by the JAVA runtime.
diff --git a/pom.xml b/pom.xml
index 949aee9571..12a9e45924 100644
--- a/pom.xml
+++ b/pom.xml
@@ -2076,11 +2076,6 @@
<artifactId>logback-classic</artifactId>
<version>${logback.version}</version>
</dependency>
- <dependency>
- <groupId>ch.qos.logback.contrib</groupId>
- <artifactId>logback-jackson</artifactId>
- <version>0.1.5</version>
- </dependency>
<dependency>
<groupId>ch.qos.logback.contrib</groupId>
<artifactId>logback-json-classic</artifactId>
diff --git a/server/apps/cassandra-app/pom.xml
b/server/apps/cassandra-app/pom.xml
index 52e2d44fc2..a78b343daf 100644
--- a/server/apps/cassandra-app/pom.xml
+++ b/server/apps/cassandra-app/pom.xml
@@ -247,10 +247,6 @@
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
</dependency>
- <dependency>
- <groupId>ch.qos.logback.contrib</groupId>
- <artifactId>logback-jackson</artifactId>
- </dependency>
<dependency>
<groupId>ch.qos.logback.contrib</groupId>
<artifactId>logback-json-classic</artifactId>
diff --git a/server/apps/distributed-app/helm-chart/james/configs/logback.xml
b/server/apps/distributed-app/helm-chart/james/configs/logback.xml
index dd280c0d5c..981c842472 100644
--- a/server/apps/distributed-app/helm-chart/james/configs/logback.xml
+++ b/server/apps/distributed-app/helm-chart/james/configs/logback.xml
@@ -1,40 +1,40 @@
<?xml version="1.0" encoding="UTF-8"?>
-<configuration scan="true" scanPeriod="30 seconds">
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+ -->
+
+<configuration>
<contextListener
class="ch.qos.logback.classic.jul.LevelChangePropagator">
<resetJUL>true</resetJUL>
</contextListener>
<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
- <encoder
class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
- <layout
class="ch.qos.logback.contrib.json.classic.JsonLayout">
-
<timestampFormat>yyyy-MM-dd'T'HH:mm:ss.SSSX</timestampFormat>
-
<timestampFormatTimezoneId>Etc/UTC</timestampFormatTimezoneId>
-
- <!-- Importance for handling multiple lines log -->
- <appendLineSeparator>true</appendLineSeparator>
-
- <jsonFormatter
class="ch.qos.logback.contrib.jackson.JacksonJsonFormatter">
- <prettyPrint>false</prettyPrint>
- </jsonFormatter>
- </layout>
+ <encoder>
+ <pattern>%d{HH:mm:ss.SSS} %highlight([%-5level])
%logger{15} - %msg%n%rEx</pattern>
+ <immediateFlush>false</immediateFlush>
</encoder>
</appender>
- <root level="INFO">
+ <root level="WARN">
<appender-ref ref="CONSOLE" />
+ <appender-ref ref="LOG_FILE" />
</root>
- <logger name="http.jmap" level="INFO" />
- <logger name="org.apache.james.smtpserver" level="DEBUG" />
- <logger name="org.apache.mailet.base.MailetPipelineLogging"
level="DEBUG" />
- <logger name="org.apache.james.mailetcontainer" level="DEBUG" />
- <logger name="com.datastax.driver.core.QueryLogger.SLOW" level="DEBUG"
/>
- <logger name="org.apache.james.mailetcontainer" level="DEBUG" />
- <logger name="org.apache.james" level="DEBUG" />
- <logger name="org.apache.james.backends.es" level="INFO" />
- <logger name="com.datastax.driver.core.QueryLogger.SLOW" level="DEBUG"
/>
- <logger name="com.datastax.driver.core.QueryLogger.ERROR"
level="TRACE" />
- <logger name="http.jmap" level="DEBUG" />
-
+ <logger name="org.apache.james" level="INFO" />
</configuration>
diff --git a/server/apps/distributed-app/pom.xml
b/server/apps/distributed-app/pom.xml
index 5160986f7d..95972160e1 100644
--- a/server/apps/distributed-app/pom.xml
+++ b/server/apps/distributed-app/pom.xml
@@ -308,10 +308,6 @@
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
</dependency>
- <dependency>
- <groupId>ch.qos.logback.contrib</groupId>
- <artifactId>logback-jackson</artifactId>
- </dependency>
<dependency>
<groupId>ch.qos.logback.contrib</groupId>
<artifactId>logback-json-classic</artifactId>
diff --git a/server/apps/distributed-pop3-app/pom.xml
b/server/apps/distributed-pop3-app/pom.xml
index 1a588701fd..3cff0f9130 100644
--- a/server/apps/distributed-pop3-app/pom.xml
+++ b/server/apps/distributed-pop3-app/pom.xml
@@ -299,10 +299,6 @@
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
</dependency>
- <dependency>
- <groupId>ch.qos.logback.contrib</groupId>
- <artifactId>logback-jackson</artifactId>
- </dependency>
<dependency>
<groupId>ch.qos.logback.contrib</groupId>
<artifactId>logback-json-classic</artifactId>
diff --git a/server/apps/jpa-app/pom.xml b/server/apps/jpa-app/pom.xml
index ef8b3025a9..52d9a102aa 100644
--- a/server/apps/jpa-app/pom.xml
+++ b/server/apps/jpa-app/pom.xml
@@ -200,10 +200,6 @@
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
</dependency>
- <dependency>
- <groupId>ch.qos.logback.contrib</groupId>
- <artifactId>logback-jackson</artifactId>
- </dependency>
<dependency>
<groupId>ch.qos.logback.contrib</groupId>
<artifactId>logback-json-classic</artifactId>
diff --git a/server/apps/jpa-smtp-app/pom.xml b/server/apps/jpa-smtp-app/pom.xml
index 5efa240e03..10998e1d54 100644
--- a/server/apps/jpa-smtp-app/pom.xml
+++ b/server/apps/jpa-smtp-app/pom.xml
@@ -148,10 +148,6 @@
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
</dependency>
- <dependency>
- <groupId>ch.qos.logback.contrib</groupId>
- <artifactId>logback-jackson</artifactId>
- </dependency>
<dependency>
<groupId>ch.qos.logback.contrib</groupId>
<artifactId>logback-json-classic</artifactId>
diff --git a/server/apps/memory-app/pom.xml b/server/apps/memory-app/pom.xml
index bb0fd1375a..1ed803261c 100644
--- a/server/apps/memory-app/pom.xml
+++ b/server/apps/memory-app/pom.xml
@@ -218,10 +218,6 @@
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
</dependency>
- <dependency>
- <groupId>ch.qos.logback.contrib</groupId>
- <artifactId>logback-jackson</artifactId>
- </dependency>
<dependency>
<groupId>ch.qos.logback.contrib</groupId>
<artifactId>logback-json-classic</artifactId>
diff --git a/server/apps/scaling-pulsar-smtp/pom.xml
b/server/apps/scaling-pulsar-smtp/pom.xml
index 9e2cdd6375..77e3559318 100644
--- a/server/apps/scaling-pulsar-smtp/pom.xml
+++ b/server/apps/scaling-pulsar-smtp/pom.xml
@@ -158,10 +158,6 @@
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
</dependency>
- <dependency>
- <groupId>ch.qos.logback.contrib</groupId>
- <artifactId>logback-jackson</artifactId>
- </dependency>
<dependency>
<groupId>ch.qos.logback.contrib</groupId>
<artifactId>logback-json-classic</artifactId>
diff --git a/third-party/crowdsec/Dockerfile b/third-party/crowdsec/Dockerfile
new file mode 100644
index 0000000000..dd1da99275
--- /dev/null
+++ b/third-party/crowdsec/Dockerfile
@@ -0,0 +1,3 @@
+FROM apache/james:memory-latest
+
+RUN echo
":/root/libs/logback-jackson-0.1.5.jar:/root/libs/logback-json-core-0.1.5.jar"
>> /root/jib-classpath-file
\ No newline at end of file
diff --git a/third-party/crowdsec/README.md b/third-party/crowdsec/README.md
index db23dc62c9..e43ebabff8 100644
--- a/third-party/crowdsec/README.md
+++ b/third-party/crowdsec/README.md
@@ -17,6 +17,12 @@ This module is for developing and delivering extensions to
James for the [Crowds
guice.extension.module=org.apache.james.crowdsec.module.CrowdsecModule
```
+- Mount the configuration file onto the classpath
+
+You can find a `logback.xml` configuration file in the `sample-configuration`
directory. \
+To use this configuration at runtime, ensure the file is available in the
application's classpath by placing it in a directory accessible by the
application.
+Use a `-Dlogback.configurationFile` JVM argument to specify the file's
location when starting the application.
+
### CrowdSec support for SMTP
- Declare the Crowdsec EhloHook in `smtpserver.xml`. Eg:
@@ -67,7 +73,21 @@ The EHLO hook will block banned clients with `554 Email
rejected` whereas the co
```
mvn clean install -DskipTests
```
-then run it: `docker-compose up`
+
+Customise the crowdsec image:
+
+```
+docker build -t apache/james:memory-crowdsec .
+```
+
+Download extra dependencies:
+
+```
+$ wget
https://repo1.maven.org/maven2/ch/qos/logback/contrib/logback-jackson/0.1.5/logback-jackson-0.1.5.jar
+$ wget
https://repo1.maven.org/maven2/ch/qos/logback/contrib/logback-json-core/0.1.5/logback-json-core-0.1.5.jar
+```
+
+Then run it: `docker-compose up`
## Crowdsec endpoints
diff --git a/third-party/crowdsec/docker-compose.yml
b/third-party/crowdsec/docker-compose.yml
index 0f95146a48..47c064196d 100644
--- a/third-party/crowdsec/docker-compose.yml
+++ b/third-party/crowdsec/docker-compose.yml
@@ -3,12 +3,14 @@ version: '3'
services:
james:
- image: apache/james:memory-latest
+ image: apache/james:memory-crowdsec
container_name: james
hostname: james.local
command:
- --generate-keystore
volumes:
+ - ./logback-jackson-0.1.5.jar:/root/libs/logback-jackson-0.1.5.jar
+ - ./logback-json-core-0.1.5.jar:/root/libs/logback-json-core-0.1.5.jar
- ./sample-configuration/logback.xml:/root/conf/logback.xml
-
./target/apache-james-crowdsec-jar-with-dependencies.jar:/root/extensions-jars/james-server-crowdsec.jar
-
./sample-configuration/extensions.properties:/root/conf/extensions.properties
diff --git a/third-party/crowdsec/pom.xml b/third-party/crowdsec/pom.xml
index bcd171ea13..f70c91b29c 100644
--- a/third-party/crowdsec/pom.xml
+++ b/third-party/crowdsec/pom.xml
@@ -74,6 +74,12 @@
<type>test-jar</type>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>ch.qos.logback.contrib</groupId>
+ <artifactId>logback-jackson</artifactId>
+ <version>0.1.5</version>
+ <scope>provided</scope>
+ </dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
diff --git a/third-party/crowdsec/sample-configuration/logback.xml
b/third-party/crowdsec/sample-configuration/logback.xml
index 3d56ea2e15..af4b9feb1d 100644
--- a/third-party/crowdsec/sample-configuration/logback.xml
+++ b/third-party/crowdsec/sample-configuration/logback.xml
@@ -1,48 +1,49 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
-->
<configuration>
- <contextListener class="ch.qos.logback.classic.jul.LevelChangePropagator">
- <resetJUL>true</resetJUL>
- </contextListener>
-
- <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
- <encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
- <layout class="ch.qos.logback.contrib.json.classic.JsonLayout">
- <timestampFormat>yyyy-MM-dd'T'HH:mm:ss.SSSX</timestampFormat>
- <timestampFormatTimezoneId>Etc/UTC</timestampFormatTimezoneId>
-
- <!-- Importance for handling multiple lines log -->
- <appendLineSeparator>true</appendLineSeparator>
-
- <jsonFormatter
class="ch.qos.logback.contrib.jackson.JacksonJsonFormatter">
- <prettyPrint>false</prettyPrint>
- </jsonFormatter>
- </layout>
- </encoder>
- <immediateFlush>false</immediateFlush>
- </appender>
- <root level="WARN">
- <appender-ref ref="CONSOLE" />
- </root>
- <logger name="org.apache.james" level="INFO" />
-</configuration>
\ No newline at end of file
+ <contextListener
class="ch.qos.logback.classic.jul.LevelChangePropagator">
+ <resetJUL>true</resetJUL>
+ </contextListener>
+
+ <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
+ <encoder
class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
+ <layout
class="ch.qos.logback.contrib.json.classic.JsonLayout">
+
<timestampFormat>yyyy-MM-dd'T'HH:mm:ss.SSSX</timestampFormat>
+
<timestampFormatTimezoneId>Etc/UTC</timestampFormatTimezoneId>
+
+ <!-- Importance for handling multiple lines
log -->
+ <appendLineSeparator>true</appendLineSeparator>
+
+ <jsonFormatter
class="ch.qos.logback.contrib.jackson.JacksonJsonFormatter">
+ <prettyPrint>false</prettyPrint>
+ </jsonFormatter>
+ </layout>
+ </encoder>
+ <immediateFlush>false</immediateFlush>
+ </appender>
+ <root level="WARN">
+ <appender-ref ref="CONSOLE" />
+ </root>
+
+ <logger name="org.apache.james" level="INFO" />
+</configuration>
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org
For additional commands, e-mail: notifications-h...@james.apache.org
