ziyanTOP opened a new issue, #662:
URL: https://github.com/apache/doris-flink-connector/issues/662
## Problem Summary
When using `mongodb-sync-database` to sync MongoDB data to Doris, BSON Date
fields are always written in UTC timezone, regardless of user expectations.
There is currently no way to configure the target timezone for datetime
conversion.
Additionally, even if a `date.timezone` parameter were added, the timezone
configuration would not take effect on TaskManagers due to the use of a static
variable (`MongoDateConverter.timeZone`) that resets to \"UTC\" after
serializer deserialization.
## Root Causes
1. **Missing timezone configuration**: `mongodb-sync-database` does not
expose a `date.timezone` parameter for users to specify the target timezone.
2. **Debezium serializes BSON Date as ISO 8601 strings**: The Change Stream
JSON represents dates as strings like `\"2024-06-04T08:03:37Z\"` instead of
`{\"$date\": ...}` objects. The existing `formatSpecialFieldData()` only
handles the object form, so these strings bypass timezone conversion entirely.
3. **Static variable not serializable to TaskManager**:
`MongoDateConverter.timeZone` is a static variable set on the Flink client
side. After the serializer is serialized and distributed to TaskManagers, the
static variable resets to its default value (`\"UTC\"`), making the
configuration ineffective at runtime.
## Expected Behavior
- Users should be able to configure `--mongodb-conf
date.timezone=Asia/Shanghai` (or any valid IANA timezone).
- MongoDB datetime fields should be converted to the configured timezone
before being written to Doris.
- The timezone configuration should survive serialization and take effect on
TaskManagers.
## Proposed Changes
- Add `date.timezone` ConfigOption to `MongoDBDatabaseSync` with default
value `\"UTC\"`.
- Add ISO 8601 string parsing support in
`MongoJsonDebeziumSchemaChange.formatSpecialFieldData()`.
- Pass timezone through `JsonDebeziumChangeContext` (which is
`Serializable`) to ensure it reaches TaskManagers.
- Add `convertTimestampToString(long, String)` overload to
`MongoDateConverter` to avoid relying on static state.
- Also fixes several related issues discovered during implementation:
- Filter `system.*` collections to avoid Unauthorized errors
- Support both `connection-options` and `connection.options` keys
- Expose `split.meta.group.size` and `close.idle.readers` options
- Fix `scheme` and `connectionOptions` propagation to
`MongoDBSourceBuilder`
- Fix `fullDocument` TextNode parsing error in
`MongoJsonDebeziumDataChange`
## Checklist
1. Does it affect the original behavior: Yes (adds timezone conversion)
2. Has unit tests been added: No (existing tests cover the formatter)
3. Has document been added or modified: No Need (self-documenting via
ConfigOption description)
4. Does it need to update dependencies: No
5. Are there any changes that cannot be rolled back: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
