ldwnt opened a new issue, #5652:
URL: https://github.com/apache/iceberg/issues/5652
### Apache Iceberg version
0.13.1
### Query engine
Flink
### Please describe the bug 🐞
I have a flink job writing to iceberg tables. When the job runs for several
days, an OOM occurs. The reason is described below:
The DecoderResolver holds a ThreadLocal variable of a two-layer map:
```
private static final ThreadLocal<Map<Schema, Map<Schema,
ResolvingDecoder>>> DECODER_CACHES = ThreadLocal.withInitial(() -> {
return (new MapMaker()).weakKeys().makeMap();
});
private static ResolvingDecoder resolve(Decoder decoder, Schema
readSchema, Schema fileSchema) throws IOException {
Map<Schema, Map<Schema, ResolvingDecoder>> cache =
(Map)DECODER_CACHES.get();
Map<Schema, ResolvingDecoder> fileSchemaToResolver =
(Map)cache.computeIfAbsent(readSchema, (k) -> {
return Maps.newHashMap();
});
...
}
```
The outer map has a weak key while the inner map has a strong one. As the
inner map holds a reference to a Schema object, the outer map holding the same
weak reference to the Schema object will not release the weak key. That leads
to the OOM.
What I suggest is to change the inner map to one with weak key, too:
```
Map<Schema, ResolvingDecoder> fileSchemaToResolver =
(Map)cache.computeIfAbsent(readSchema, (k) -> {
// return Maps.newHashMap();
return new WeakHashMap<>();
});
```
So far it seems working with my jobs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
