thomas-pfeiffer commented on issue #2325: URL: https://github.com/apache/iceberg-python/issues/2325#issuecomment-3606584943
We ran into this issue (like @andormarkus' case) on AWS Lambda as well. We tried the above mentioned option with regularly clearing the cache but the results were similar with the test script and the memory usage in the real Lambda was still too high for our scenario, hence we opted to deactivated the cache completely: ```py from pyiceberg.manifest import _manifests # unwrap the cached function to use the function directly _manifests = _manifests.__wrapped__ # type: ignore[assignment] ``` (reg. `__wrapped__` see https://cachetools.readthedocs.io/en/latest/#cachetools.cached) So it's only a workaround and probably we leave some performance on the table, but we valued the stability and less retried Lambda executions above pure performance. Your milage may vary. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
