Grzegorz Liter created AVRO-4334:
------------------------------------
Summary: Utf8.hashCode() changed silently in 1.12.1
(AVRO-4060/AVRO-4061) — undocumented breaking change for any hash-based consumer
Key: AVRO-4334
URL: https://issues.apache.org/jira/browse/AVRO-4334
Project: Apache Avro
Issue Type: Bug
Affects Versions: 1.12.1, 1.12.2
Reporter: Grzegorz Liter
Utf8.hashCode() (and therefore GenericData.hashCode() / every Avro-generated
SpecificRecord's hashCode() for any string field) changed behavior between
1.12.0 and 1.12.1, introduced by:
- AVRO-4061 "Use Default Value of 1 For UTF8 Hash" — commit
31221f0f9acbb43a947bd04d239fa4317bbf0119
- AVRO-4060 "Use JDK to Hash Byte Array in UTF8" — commit
057053f86074ec854f132b491ef0429a0eb1f977
Before (1.12.0 and every earlier release, confirmed identical back through at
least 1.11.3):
{quote}{{public int hashCode() {}}
{{ int h = this.hash; // defaults to 0}}
{{ if (h == 0) {}}
{{ for (int i = 0; i < length; i++) {}}
{{ h = h * 31 + bytes[i]; // accumulator starts at 0}}
{{ }}}
{{ this.hash = h;}}
{{ }}}
{{ return h;}}
{{}}}{quote}
After (1.12.1+):
{{public Utf8() {}}
{{ this.bytes = EMPTY;}}
{{ this.hash = 1; // seeded to 1}}
{{}}}
{{...}}
{{public int hashCode() {}}
{{ int h = this.hash;}}
{{ if (h == 0) {}}
{{ if (length > 7 && bytes.length == length) {}}
{{ h = Arrays.hashCode(bytes); // JDK hasher, also starts at 1}}
{{ } else {}}
{{ h = 1; // accumulator now starts at 1, not 0}}
{{ for (int i = 0; i < length; i++) {}}
{{ h = h * 31 + bytes[i];}}
{{ }}}
{{ }}}
{{ this.hash = h;}}
{{ }}}
{{ return h;}}
{{}}}
This produces a completely different hashCode() for every non-empty string
compared to 1.12.0 and earlier. Both tickets were filed and reviewed purely as
performance/caching fixes (AVRO-4061's stated motivation was that empty strings
hashed to 0 and were therefore never cached; AVRO-4060 added a JDK-vectorized
fast path). Neither ticket, their linked PRs, nor the 1.12.1 release notes
mention that this changes hashCode() output for existing data, or call out any
compatibility impact — AVRO-4060/AVRO-4061 aren't even named in the release
notes; they're folded into the generic "Java: minor bugfixes" line.
Impact:
Avro-generated SpecificRecord classes delegate hashCode() to
GenericData.hashCode() at runtime rather than baking it into generated code, so
any consumer relying on hashCode stability across an upgrade is silently
affected — no compile error, no exception, no warning.
We hit this in production: an Apache Flink job uses an Avro type as a .keyBy()
partitioning key. Flink's KeyGroupRangeAssignment.assignToKeyGroup() calls
key.hashCode() to decide both routing and which key-group a piece of
already-written state lives under. Crossing 1.12.0→1.12.1 with no other change,
every previously-written entry's recomputed key-group silently disagreed with
where it was stored — either a hard IllegalArgumentException: Key group X is
not in KeyGroupRange\{...} (State Processor API, which revalidates hashCode
against stored location) or, worse, silent state loss/duplication in normal
streaming (which never cross-validates against history). This corrupted the
majority of one operator's keyed state before being root-caused.
Ask:
at minimum, document in the release notes/upgrade guide that hashCode() for
Avro-generated types is not a stable contract across versions and changed in
1.12.1, so downstream consumers doing hash-based routing/partitioning are aware
to pin an explicit hashCode() override (as we now do) rather than discovering
this via silent data corruption. Longer-term, consider whether
Utf8/GenericData.hashCode()'s stability should itself be documented as
guaranteed-or-not, so this doesn't recur on a future "trivial" performance
tweak.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
