zhengxiang378928908-code opened a new pull request, #12945:
URL: https://github.com/apache/seatunnel/pull/12945
### Purpose of this pull request
Follow-up to #12884, as agreed with @SEZ9 there.
Since #10724, MySQL `TIMESTAMP` reaches the Doris sink as `TIMESTAMP_TZ`
instead of `TIMESTAMP`. Doris has no timezone-aware datetime type, so since
#11524 the sink writes `TIMESTAMP_TZ` values as wall-clock time in
`sink.datetime-timezone`, or in the JVM default zone when that option is unset.
A job upgraded from 2.3.x with the option unset can therefore store different
wall-clock values when the JVM zone is not the zone Doris expects, and the
Doris sink currently logs nothing about this fallback.
This PR logs one warning when the option is unset and the table has
`TIMESTAMP_TZ` values, either at the top level or nested in `ROW`, `ARRAY` or
`MAP` (both serializers convert nested values the same way):
```
Doris sink table db.orders has TIMESTAMP_TZ columns [created_at,
payload.created_at, events[]] and 'sink.datetime-timezone' is not set. These
values are written as wall-clock time in the JVM default zone UTC. If Doris
expects another zone, set 'sink.datetime-timezone'.
```
The check runs when the writer creates its serializer, so at startup and
after a schema change. Each writer instance logs it at most once; a writer
created again (for example after a restore) can log it again. Serialization
behavior is unchanged. Deriving the default zone from Doris itself is left for
a separate discussion, as suggested in #12884.
The `sink.datetime-timezone` row in the Doris sink docs (EN/ZH) now mentions
the warning.
### Does this PR introduce _any_ user-facing change?
Only a new `WARN` log line in the case above, plus the doc note. No change
to the written data or to options.
### How was this patch tested?
Unit tests in `DorisSinkWriterTest`:
- `testTimestampTzZoneWarningNamesColumnsAndJvmZoneWhenZoneUnset`: the
message names the table, only the `TIMESTAMP_TZ` columns, and the JVM zone.
- `testTimestampTzZoneWarningFindsNestedColumns`: `ROW` field, `ARRAY`
element and `MAP` value paths are listed.
- `testNoTimestampTzZoneWarningWhenZoneSetOrNoTimestampTzColumn`: no warning
when the option is set or there is no `TIMESTAMP_TZ` value.
-
`testTimestampTzZoneWarningLoggedOnceWhenWriterCreatedWithTimestampTzColumn`
and
`testTimestampTzZoneWarningLoggedOnceWhenSchemaChangeAddsTimestampTzColumn`:
capture the writer's log4j output and check that exactly one `WARN` is logged,
at creation or on the schema change that first adds such a column, and that
later schema changes do not log it again.
- `testNoTimestampTzZoneWarningLoggedWhenZoneConfigured`: no `WARN` when the
option is set.
Whole `connector-doris` module: 158 tests pass locally (JDK 23 with
`-Dnet.bytebuddy.experimental=true` for Mockito; CI covers JDK 11 and 17).
Removing the `log.warn` call, or the once-per-writer check, makes the two
writer tests fail.
No E2E change: the change only adds a log line, and the existing Doris E2E
tests do not assert on log output.
### Check list
* [ ] If any new Jar binary package adding in your PR, please add License
Notice according
[New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
* [ ] If necessary, please update the documentation to describe the new
feature. https://github.com/apache/seatunnel/tree/dev/docs
* [ ] If necessary, please update `incompatible-changes.md` to describe the
incompatibility caused by this PR.
* [ ] If you are contributing the connector code, please check that the
following files are updated:
1. Update
[plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties)
and add new connector information in it
2. Update the pom file of
[seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
3. Add ci label in
[label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
4. Add e2e testcase in
[seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/)
5. Update connector
[plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
