vinothchandar commented on issue #620: Fix HUDI-27 : Support num_cores > 1 for writing through spark URL: https://github.com/apache/incubator-hudi/pull/620#issuecomment-478077857 @leilinen did you have the same problem with this patch applied as well? Some background. FileSystem class caches and hands out references of FileSystem object across a jvm (i.e driver or . single executor) and also registers hooks to close the cached instances when shutting down.. So that will take care of reclaiming resources.. @n3nash Not following. General practice for spark apps is to not call fs.close() at all and Hudi should be no different.. However as @bvaradar pointed out to me, may be its worth calling `super.close()` just in case.. I ll send in a PR for that.. If you still see follow up to be done, please open a JIRA and we can discuss and prioritize there
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
