siddharthteotia commented on a change in pull request #4418: Fix potential resource leak in the way we close a collection of closeables URL: https://github.com/apache/incubator-pinot/pull/4418#discussion_r302683406
########## File path: pinot-common/src/main/java/org/apache/pinot/common/Closeables.java ########## @@ -0,0 +1,61 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pinot.common; + +import java.io.Closeable; +import java.io.IOException; +import java.util.Arrays; + + +public class Closeables { + + public static void close(Iterable<? extends Closeable> ac) + throws IOException { + + if (ac == null) { + return; + } else if (ac instanceof Closeable) { + ((Closeable) ac).close(); + return; + } + + IOException topLevelException = null; + + for (Closeable closeable : ac) { + try { + if (closeable != null) { + closeable.close(); + } + } catch (IOException e) { + if (topLevelException == null) { + topLevelException = e; + } else if (e != topLevelException) { + topLevelException.addSuppressed(e); + } + } + } + + if (topLevelException != null) { + throw topLevelException; Review comment: @mcvsubbu , to see the the extent of logging, I created a simple test: ``` public static void main(String[] args) throws Exception { Closeables.close(new Closeable1(), new Closeable2(), new Closeable3()); } private static class Closeable1 implements Closeable { @Override public void close() throws IOException { throw new IOException("io exception 1"); } } private static class Closeable2 implements Closeable { @Override public void close() throws IOException { throw new FileNotFoundException("io exception 2"); } } private static class Closeable3 implements Closeable { @Override public void close() throws IOException { throw new IOException("io exception 3"); } } ``` **10:15:06.577 [main] ERROR org.apache.pinot.common.Closeables - Failed to close resource: class org.apache.pinot.core.segment.creator.impl.SegmentColumnarIndexCreator$Closeable1 java.io.IOException: io exception 1** at SegmentColumnarIndexCreator$Closeable1.close(SegmentColumnarIndexCreator.java:548) at org.apache.pinot.common.Closeables.close(Closeables.java:66) [classes/:?] at org.apache.pinot.common.Closeables.close(Closeables.java:94) [classes/:?] at SegmentColumnarIndexCreator.main(SegmentColumnarIndexCreator.java:542) **10:15:06.583 [main] ERROR org.apache.pinot.common.Closeables - Failed to close resource: class org.apache.pinot.core.segment.creator.impl.SegmentColumnarIndexCreator$Closeable2 java.io.FileNotFoundException: io exception 2** at SegmentColumnarIndexCreator$Closeable2.close(SegmentColumnarIndexCreator.java:555) at org.apache.pinot.common.Closeables.close(Closeables.java:66) [classes/:?] at org.apache.pinot.common.Closeables.close(Closeables.java:94) [classes/:?] at SegmentColumnarIndexCreator.main(SegmentColumnarIndexCreator.java:542) [classes/:?] **10:15:06.584 [main] ERROR org.apache.pinot.common.Closeables - Failed to close resource: class org.apache.pinot.core.segment.creator.impl.SegmentColumnarIndexCreator$Closeable3 java.io.IOException: io exception 3** at SegmentColumnarIndexCreator$Closeable3.close(SegmentColumnarIndexCreator.java:562) at org.apache.pinot.common.Closeables.close(Closeables.java:66) [classes/:?] at org.apache.pinot.common.Closeables.close(Closeables.java:94) [classes/:?] at SegmentColumnarIndexCreator.main(SegmentColumnarIndexCreator.java:542) [classes/:?] Above shows how each exception on close() is logged along with the exact resource type The following shows the final log message with all the above exceptions (last 2 being suppressed) **11:02:02.846 [main] ERROR org.apache.pinot.common.Closeables - Failed to close one or more resources java.io.IOException: io exception 1** at SegmentColumnarIndexCreator$Closeable1.close(SegmentColumnarIndexCreator.java:548) at org.apache.pinot.common.Closeables.close(Closeables.java:65) [classes/:?] at org.apache.pinot.common.Closeables.close(Closeables.java:93) [classes/:?] at SegmentColumnarIndexCreator.main(SegmentColumnarIndexCreator.java:542) [classes/:?] **Suppressed: java.io.FileNotFoundException: io exception 2** at SegmentColumnarIndexCreator$Closeable2.close(SegmentColumnarIndexCreator.java:555) at org.apache.pinot.common.Closeables.close(Closeables.java:65) [classes/:?] at org.apache.pinot.common.Closeables.close(Closeables.java:93) [classes/:?] at SegmentColumnarIndexCreator.main(SegmentColumnarIndexCreator.java:542) **Suppressed: java.io.IOException: io exception 3** at SegmentColumnarIndexCreator$Closeable3.close(SegmentColumnarIndexCreator.java:562) at org.apache.pinot.common.Closeables.close(Closeables.java:65) [classes/:?] at org.apache.pinot.common.Closeables.close(Closeables.java:93) [classes/:?] at .SegmentColumnarIndexCreator.main(SegmentColumnarIndexCreator.java:542) Finally this is what the end user exception looks like when we throw: **Exception in thread "main" java.io.IOException: io exception 1** at SegmentColumnarIndexCreator$Closeable1.close(SegmentColumnarIndexCreator.java:548) at org.apache.pinot.common.Closeables.close(Closeables.java:65) at org.apache.pinot.common.Closeables.close(Closeables.java:93) at .SegmentColumnarIndexCreator.main(SegmentColumnarIndexCreator.java:542) **Suppressed: java.io.FileNotFoundException: io exception 2** at SegmentColumnarIndexCreator$Closeable2.close(SegmentColumnarIndexCreator.java:555) ... 3 more Suppressed: java.io.IOException: io exception 3 at SegmentColumnarIndexCreator$Closeable3.close(SegmentColumnarIndexCreator.java:562) ... 3 more ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
