alamb opened a new issue, #19052: URL: https://github.com/apache/datafusion/issues/19052
Also `listing file statistics cache` seems to not have any memory limit unlike `metadata cache` for example. Is that by design , do you think we need to add similar limit for this cache too ? _Originally posted by @bharath-techie in https://github.com/apache/datafusion/issues/18971#issuecomment-3600061959_ Basically the cache used in ListingTable comes from here: https://github.com/apache/datafusion/blob/81512da2b0aaa474f6c4ba205b05eea7b3095176/datafusion/core/src/datasource/listing_table_factory.rs#L188-L187 Which somewhat unobviously sets a `DefaultFileStatisticsCache` here https://github.com/apache/datafusion/blob/9f725d9c7064813cda0de0f87d115354b68d76e6/datafusion/catalog-listing/src/table.rs#L260-L259 The `DefaultFileStatisticsCache` has no limit: https://github.com/apache/datafusion/blob/7d8b8602ad1be2f61f6a8ebb253ace9d85304ea7/datafusion/execution/src/cache/cache_unit.rs#L41-L40 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
