iemejia opened a new pull request, #3963:
URL: https://github.com/apache/avro/pull/3963
## What changes were proposed in this pull request?
`FastReaderBuilder`'s `RecordReader` cache was a weak-identity map keyed on
the reader `Schema`:
```java
private final Map<Schema, Map<Schema, RecordReader>> readerCache =
Collections.synchronizedMap(new WeakIdentityHashMap<>());
```
However, each cached `RecordReader` holds a **strong** reference to that
same reader `Schema` (`RecordReader.schema`, needed by its `InstanceSupplier`
at read time — `getNewRecordSupplier` returns `this::newRecord`, which takes
the schema as a parameter). Because the cached **value** strongly references
its own **weak key**, the weak entries can never be reclaimed. The cache
therefore grows without bound whenever many distinct `Schema` instances are
used — e.g. when a schema is re-parsed for every file or message in a
long-running process — leading to unbounded memory growth (reports of caches
reaching 100GB).
This is the memory leak tracked by AVRO-4253 (and its duplicate AVRO-3524).
Note the earlier commit referencing AVRO-4253 (#3764) actually addressed an
unrelated logging concern; the leak itself was still present.
## How was this patch fixed?
Replace the weak two-level map with a **bounded LRU cache** keyed on an
identity-based `(reader, writer)` schema pair:
- Bounds memory unconditionally, independent of the value-references-key
cycle.
- Identity-based key preserves the original cache semantics and avoids the
cost of `Schema.equals`/`hashCode` on large schemas.
- **Recursion-safe:** entries that are still `INITIALIZING` are never
evicted, so recursive schema resolution still terminates by resolving back to
the same in-flight instance instead of rebuilding endlessly.
- Evicting a cached reader is safe for in-use readers: the caller holds the
returned `DatumReader` directly, so eviction only means a future lookup
rebuilds it.
- The bound defaults to `2048` and is configurable via the
`org.apache.avro.fastreader.recordReaderCacheSize` system property.
## How was this patch tested?
New `TestFastReaderBuilderCacheBounded`:
- `cacheStaysBoundedAcrossDistinctSchemaInstances` — 3048 freshly-parsed
(distinct-identity) schemas leave the cache bounded at `<= 2048` (fails on the
old unbounded code).
- `reusedSchemaInstanceHitsCache` — a reused schema instance keeps a single
cache entry (cache still effective).
- `recursiveSchemaReadsCorrectly` — a self-referential (linked-list) schema
round-trips correctly.
Also verified no regressions across the read path: `TestResolvingIO` (816),
`TestResolvingIOResolving` (192), `TestGenericDatumReader`, `TestGenericData`,
`TestDataFile`, `TestResolver`, and the FastReaderBuilder tests all pass under
the default, custom-coders, and without-fast-reader surefire profiles.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
