gagraler opened a new issue, #1306:
URL: https://github.com/apache/rocketmq-clients/issues/1306
### Before Creating the Enhancement Request
- [x] I have confirmed that this should be classified as an enhancement
rather than a bug/feature.
### Programming Language of the Client
Go
### Summary
The Go client currently initializes a package-level logger during `init()`
and writes logs to a rotating file under:
```text
$HOME/logs/rocketmq/rocketmq_client_go.log
```
It also does not support custom loggers or JSON format output.
### Motivation
The Go client currently initializes a package-level logger during `init()`
and writes logs to a rotating file under:
```text
$HOME/logs/rocketmq/rocketmq_client_go.log
```
The output destination can be changed to stdout with:
```text
mq.consoleAppender.enabled=true
```
However, the main client logger always uses `zapcore.NewConsoleEncoder`, so
its output remains plain text. There is currently no public API for replacing
the package-level logger or configuring its encoder.
This causes several problems in containerized environments:
1. Containers commonly use a read-only root filesystem. When `$HOME` is
`/root`, the client repeatedly reports errors such as:
```text
write error: can't make directories for new logfile:
mkdir /root/logs: read-only file system
```
2. Kubernetes applications generally write logs to stdout/stderr and
delegate collection, rotation, and retention to the platform.
3. Applications using structured JSON logging cannot make RocketMQ client
logs conform to the same schema.
4. `WithZapLogger` only configures the gRPC connection/interceptor logger.
It does not replace the package-level `sugarBaseLogger` used by the main client
implementation.
As a result, applications must either accept mixed log formats, mount a
writable log directory, suppress RocketMQ logs, or maintain a patched fork.
## Proposed solution
Provide a supported API for injecting a custom `*zap.Logger` into the
RocketMQ client.
### Describe the Solution You'd Like
### Option 1: Accept a custom Zap logger
Provide a public API for injecting a custom `*zap.Logger` into the RocketMQ
client.
Example:
```go
logger := zap.New(
zapcore.NewCore(
zapcore.NewJSONEncoder(encoderConfig),
zapcore.AddSync(os.Stdout),
zap.InfoLevel,
),
)
rocketmq.SetLogger(logger)
```
The supplied logger should be used by the package-level client logger and
inherited by newly created producers and consumers.
A minimal backward-compatible API could look like:
```go
// SetLogger replaces the package-level logger used by RocketMQ clients.
//
// It must be called before creating any producer or consumer.
func SetLogger(logger *zap.Logger) {
sugarBaseLogger = logger.Sugar()
}
```
The existing file-based logger and environment variables can remain the
default, preserving current behavior for existing users.
If runtime replacement is not intended to be supported, the documentation
should explicitly require `SetLogger` to be called before creating any client.
This avoids synchronization and data-race concerns during normal operation.
A client-level option could also be provided:
```go
producer, err := rocketmq.NewProducer(
config,
rocketmq.WithTopics(topic),
rocketmq.WithLogger(logger),
)
```
A client-level option would avoid global state and allow different clients
to use different loggers. Ideally, the supplied logger should also be
propagated to the gRPC connection layer so users do not need to configure two
separate logging systems.
#### Advantages
- Small implementation change.
- Preserves compatibility with the SDK's existing Zap implementation.
- Allows applications to configure JSON encoding, stdout output, log levels,
sampling, and additional fields.
- Easier to introduce without refactoring every logging call site.
#### Disadvantages
- Exposes Zap as part of the public API.
- Applications using `slog`, Zerolog, Logrus, or an internal logger must
create a Zap adapter.
- This will make it even more difficult to replace Zap within the SDK in the
future.
### Option 2: Define a logger interface
Instead of accepting a concrete `*zap.Logger`, define a small logging
interface. Applications could then integrate the RocketMQ client with their
existing logging framework without depending on Zap-specific types.
For example:
```go
type Logger interface {
Debug(msg string, keyValues ...any)
Info(msg string, keyValues ...any)
Warn(msg string, keyValues ...any)
Error(msg string, keyValues ...any)
With(keyValues ...any) Logger
}
```
The logger could be supplied globally:
```go
rocketmq.SetLogger(newApplicationLogger())
```
or per client:
```go
producer, err := rocketmq.NewProducer(
config,
rocketmq.WithTopics(topic),
rocketmq.WithLogger(newApplicationLogger()),
)
```
The SDK could provide a default Zap-backed implementation to preserve the
current behavior:
```go
type zapLogger struct {
logger *zap.SugaredLogger
}
func (l *zapLogger) Debug(msg string, keyValues ...any) {
l.logger.Debugw(msg, keyValues...)
}
func (l *zapLogger) Info(msg string, keyValues ...any) {
l.logger.Infow(msg, keyValues...)
}
func (l *zapLogger) Warn(msg string, keyValues ...any) {
l.logger.Warnw(msg, keyValues...)
}
func (l *zapLogger) Error(msg string, keyValues ...any) {
l.logger.Errorw(msg, keyValues...)
}
func (l *zapLogger) With(keyValues ...any) Logger {
return &zapLogger{
logger: l.logger.With(keyValues...),
}
}
```
Applications could implement the interface using `log/slog`:
```go
type applicationL
### Describe Alternatives You've Considered
### Enable the console appender
```text
mq.consoleAppender.enabled=true
```
This solves the read-only filesystem error by writing to stdout, but the
main SDK logger still uses console encoding rather than JSON. Applications
therefore produce mixed structured and unstructured logs.
### Change `rocketmq.client.logRoot`
Pointing the SDK at `/tmp` or another writable volume avoids the immediate
error, but it still requires file lifecycle management inside the container.
It also conflicts with the common Kubernetes logging model, where
applications write to stdout/stderr and the platform manages collection and
rotation.
### Mount a writable volume at `$HOME/logs`
This works operationally but adds deployment complexity solely to satisfy an
SDK logging implementation detail. It also requires separate collection and
retention configuration for the RocketMQ log files.
### Use `WithZapLogger`
The existing `WithZapLogger` option configures the gRPC
connection/interceptor logger only. It does not affect the package-level logger
used by most RocketMQ client code, so it cannot provide consistent JSON output.
### Set the RocketMQ log level to `error`
Reducing the SDK log level limits the amount of unstructured output, but it
does not solve the formatting issue and removes useful diagnostic information.
### Parse and re-encode console logs in the log collector
A log collector could parse the console-formatted SDK logs and wrap them in
JSON. This is fragile because it depends on the SDK's text format and cannot
reliably preserve
### Additional Context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
