Hi all,
I would like to get your opinion on a small change to the JSON output of
the new VARIANT type.
Variant.toJson() renders TIMESTAMP_LTZ and TIMESTAMP_LTZ_NS values as an
ISO 8601 timestamp with a numeric UTC offset. For a value at UTC, the
offset is printed as +00:00.
Current output:
"1970-01-01T00:00:00+00:00"
"1970-01-01T00:00:00.123456789+00:00"
The relevant formatter is in BinaryVariantUtil [1]:
public static final DateTimeFormatter TIMESTAMP_LTZ_FORMATTER =
new DateTimeFormatterBuilder()
.append(TIMESTAMP_FORMATTER)
.appendOffset("+HH:MM", "+00:00")
.toFormatter(Locale.US);
I propose emitting Z for the zero offset instead. The change is one line.
We set the "no offset" text to Z:
.appendOffset("+HH:MM", "Z")
Proposed output:
"1970-01-01T00:00:00Z"
"1970-01-01T00:00:00.123456789Z"
Only the zero offset changes from +00:00 to Z.
Why I think this is worth doing:
1. It is the canonical form for UTC. RFC 3339 [2] and ISO 8601 [3] both
define Z as the marker for UTC. It is the form most readers expect for a
"Zulu time" timestamp.
2. It matches standard tooling. java.time.Instant.toString(),
DateTimeFormatter.ISO_OFFSET_DATE_TIME, Jackson, and most JSON libraries
emit Z for UTC. Consumers that parse our output with a standard library
round-trip more cleanly.
3. It is more compact. Z is one character versus six.
4. The blast radius is small right now. VARIANT is new in Flink 2.x.
Aligning the format before it is widely depended upon is cheaper than
changing it later.
This is a behavior change to a public string output. Anyone doing string
equality on toJson() for a UTC TIMESTAMP_LTZ value will see a diff. I would
treat it as a documented change with a release note.
If there are no objections, I will file a JIRA and open a PR with the one
line change and updated tests. Feedback welcome, especially from anyone
consuming Variant.toJson() downstream.
Thanks,
Ramin
[1]
https://github.com/apache/flink/blob/4a23ab12063d4d1354fae905f01e47b9777c402e/flink-core/src/main/java/org/apache/flink/types/variant/BinaryVariantUtil.java#L238
[2] https://www.rfc-editor.org/info/rfc3339/
[3] https://en.wikipedia.org/wiki/ISO_8601
