steveloughran commented on issue #16640: URL: https://github.com/apache/iceberg/issues/16640#issuecomment-4601374748
@palladium-coder I'd be -1 to any change to iceberg here - upgrade your hadoop libraries. If for some reason you can't go to hadoop 4.x, check out and do your own release of hadoop branch-3.3 which has the patch too, use it. - renable caching. You must be taking a performance hit by not doing this, as there's a thread pool, and shared buffer pool for prefetching. If you've got more than one writer then duplicate fs instances will be created, so lots more local resources will be used. if you really can't cache filesystems, I'd be curious why and whether it's a bug in abfs you are finding. In which case, I'd encourage a move to Hadoop 3.4.3 and enabling of the httpclient connection pool (over the java.net.URL code in 3.3.x) to see if that helps. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
