lidavidm commented on code in PR #1141:
URL: https://github.com/apache/arrow-java/pull/1141#discussion_r3975574283
##########
dataset/src/main/java/org/apache/arrow/dataset/file/FileSystemDatasetFactory.java:
##########
@@ -51,6 +60,65 @@ public FileSystemDatasetFactory(
String[] uris,
Optional<FragmentScanOptions> fragmentScanOptions) {
super(allocator, memoryPool, createNative(format, uris,
fragmentScanOptions));
+ this.hdfsFileSystems = toHdfsFileSystems(uris);
+ }
+
+ /**
+ * Close this factory and release the native instance. For HDFS URIs, also
closes the cached
+ * Hadoop FileSystem to release non-daemon threads that would otherwise
prevent JVM exit. See <a
+ * href="https://github.com/apache/arrow-java/issues/1067";>#1067</a>.
+ */
+ @Override
+ public synchronized void close() {
+ try {
+ super.close();
+ } finally {
+ hdfsFileSystems.forEach(FileSystemDatasetFactory::closeHadoopFileSystem);
+ }
Review Comment:
Well, you would assemble a list of lambdas essentially and close it all at
once; that way a failure to close one won't affect the others. But this is fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
