xborder commented on code in PR #1141:
URL: https://github.com/apache/arrow-java/pull/1141#discussion_r3938572193
##########
dataset/src/main/java/org/apache/arrow/dataset/file/FileSystemDatasetFactory.java:
##########
@@ -51,6 +60,65 @@ public FileSystemDatasetFactory(
String[] uris,
Optional<FragmentScanOptions> fragmentScanOptions) {
super(allocator, memoryPool, createNative(format, uris,
fragmentScanOptions));
+ this.hdfsFileSystems = toHdfsFileSystems(uris);
+ }
+
+ /**
+ * Close this factory and release the native instance. For HDFS URIs, also
closes the cached
+ * Hadoop FileSystem to release non-daemon threads that would otherwise
prevent JVM exit. See <a
+ * href="https://github.com/apache/arrow-java/issues/1067";>#1067</a>.
+ */
+ @Override
+ public synchronized void close() {
+ try {
+ super.close();
+ } finally {
+ hdfsFileSystems.forEach(FileSystemDatasetFactory::closeHadoopFileSystem);
+ }
Review Comment:
I think we could use AutoCloseables by casting the `Object` obtained through
reflection, but it would not simplify or improve the cleanup logic.
Is there any benefit in changing it that I'm not seeing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
