Is the excetion about a leak supposed to happen ?
Sent from my android device. -----Original Message----- From: Steven Phillips <[email protected]> To: [email protected] Sent: Tue, 27 Aug 2013 10:48 Subject: Re: Error with ParquetRecordReaderTest Maybe I'm just not looking closely enough, but I don't see the problem. It looks like the test passed without issue. On Mon, Aug 26, 2013 at 9:06 PM, Tanujit Ghosh <[email protected]>wrote: > Hi, > > I'm facing this issue when i try to run the test in > > org.apache.drill.exec.store.parquet.ParquetRecordReaderTest > > [tanujit@legolas java-exec]$ mvn > -Dtest=ParquetRecordReaderTest#testMultipleRowGroupsAndReads test > [INFO] Scanning for projects... > [WARNING] > [WARNING] Some problems were encountered while building the effective model > for org.apache.drill.exec:java-exec:jar:1.0-SNAPSHOT > [WARNING] Detected profiles.xml alongside > org.apache.drill.exec:java-exec:1.0-SNAPSHOT, this file is no longer > supported and was ignored, please use the settings.xml instead > [WARNING] > [WARNING] It is highly recommended to fix these problems because they > threaten the stability of your build. > [WARNING] > [WARNING] For this reason, future Maven versions might no longer support > building such malformed projects. > [WARNING] > [INFO] > > [INFO] > ------------------------------------------------------------------------ > [INFO] Building java-exec 1.0-SNAPSHOT > [INFO] > ------------------------------------------------------------------------ > Downloading: > > http://repository.apache.org/snapshots/org/apache/drill/common/1.0-SNAPSHOT/maven-metadata.xml > Downloading: > > http://conjars.org/repo/org/apache/drill/common/1.0-SNAPSHOT/maven-metadata.xml > Downloading: > > https://oss.sonatype.org/content/repositories/snapshots/org/apache/drill/common/1.0-SNAPSHOT/maven-metadata.xml > Downloading: > > http://repository.apache.org/snapshots/org/apache/drill/prototype-parent/1.0-SNAPSHOT/maven-metadata.xml > Downloading: > > https://oss.sonatype.org/content/repositories/snapshots/org/apache/drill/prototype-parent/1.0-SNAPSHOT/maven-metadata.xml > Downloading: > > http://conjars.org/repo/org/apache/drill/prototype-parent/1.0-SNAPSHOT/maven-metadata.xml > Downloading: > > http://repository.apache.org/snapshots/org/apache/drill/exec/exec-parent/1.0-SNAPSHOT/maven-metadata.xml > Downloading: > > http://conjars.org/repo/org/apache/drill/exec/exec-parent/1.0-SNAPSHOT/maven-metadata.xml > Downloading: > > https://oss.sonatype.org/content/repositories/snapshots/org/apache/drill/exec/exec-parent/1.0-SNAPSHOT/maven-metadata.xml > [INFO] > [INFO] --- maven-resources-plugin:2.6:copy-resources (copy-resources) @ > java-exec --- > [INFO] Using 'UTF-8' encoding to copy filtered resources. > [INFO] Copying 370 resources > [INFO] > [INFO] --- maven-enforcer-plugin:1.2:enforce (no_commons_logging) @ > java-exec --- > [INFO] > [INFO] --- maven-antrun-plugin:1.6:run (generate-sources) @ java-exec --- > [WARNING] Parameter tasks is deprecated, use target instead > [INFO] Executing tasks > > main: > [INFO] Executed tasks > [INFO] Registering compile source root > > /backup/java/wip/incubator-drill/sandbox/prototype/exec/java-exec/target/generated-sources > [INFO] > [INFO] --- fmpp-maven-plugin:1.0:generate (generate-sources) @ java-exec > --- > Downloading: > > https://oss.sonatype.org/content/repositories/snapshots/org/freemarker/freemarker/maven-metadata.xml > Downloading: > > https://oss.sonatype.org/content/repositories/snapshots/oro/oro/maven-metadata.xml > Downloading: > > https://oss.sonatype.org/content/repositories/snapshots/org/beanshell/bsh/maven-metadata.xml > Downloading: > > https://oss.sonatype.org/content/repositories/snapshots/xml-resolver/xml-resolver/maven-metadata.xml > - Executing: ValueHolders.java > log4j:WARN No appenders could be found for logger (freemarker.cache). > log4j:WARN Please initialize the log4j system properly. > - Executing: VariableLengthVectors.java > - Executing: FixedValueVectors.java > - Executing: TypeHelper.java > - Executing: NullableValueVectors.java > - Executing: RepeatedValueVectors.java > [INFO] Done > [INFO] > [INFO] --- maven-remote-resources-plugin:1.1:process (default) @ java-exec > --- > [INFO] Setting property: classpath.resource.loader.class => > 'org.codehaus.plexus.velocity.ContextClassLoaderResourceLoader'. > [INFO] Setting property: velocimacro.messages.on => 'false'. > [INFO] Setting property: resource.loader => 'classpath'. > [INFO] Setting property: resource.manager.logwhenfound => 'false'. > [INFO] > [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ > java-exec --- > [INFO] Using 'UTF-8' encoding to copy filtered resources. > [INFO] Copying 1 resource > [INFO] Copying 3 resources > [INFO] > [INFO] --- maven-compiler-plugin:3.0:compile (default-compile) @ java-exec > --- > [INFO] Changes detected - recompiling the module! > [INFO] Compiling 460 source files to > > /backup/java/wip/incubator-drill/sandbox/prototype/exec/java-exec/target/classes > [WARNING] > > /backup/java/wip/incubator-drill/sandbox/prototype/exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetStorageEngine.java:[20,21] > com.sun.corba.se.impl.interceptors.CodecFactoryImpl is internal proprietary > API and may be removed in a future release > [WARNING] > > /backup/java/wip/incubator-drill/sandbox/prototype/exec/java-exec/src/main/java/org/apache/drill/exec/rpc/bit/BitConnection.java: > Some input files use or override a deprecated API. > [WARNING] > > /backup/java/wip/incubator-drill/sandbox/prototype/exec/java-exec/src/main/java/org/apache/drill/exec/rpc/bit/BitConnection.java: > Recompile with -Xlint:deprecation for details. > [WARNING] > > /backup/java/wip/incubator-drill/sandbox/prototype/exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetGroupScan.java: > Some input files use unchecked or unsafe operations. > [WARNING] > > /backup/java/wip/incubator-drill/sandbox/prototype/exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetGroupScan.java: > Recompile with -Xlint:unchecked for details. > [INFO] > [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) > @ java-exec --- > [INFO] Using 'UTF-8' encoding to copy filtered resources. > [INFO] Copying 37 resources > [INFO] Copying 3 resources > [INFO] > [INFO] --- maven-compiler-plugin:3.0:testCompile (default-testCompile) @ > java-exec --- > [INFO] Changes detected - recompiling the module! > [INFO] Compiling 127 source files to > > /backup/java/wip/incubator-drill/sandbox/prototype/exec/java-exec/target/test-classes > [WARNING] > > /backup/java/wip/incubator-drill/sandbox/prototype/exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TestExecutionAbstractions.java:[25,16] > sun.misc.Unsafe is internal proprietary API and may be removed in a future > release > [WARNING] > > /backup/java/wip/incubator-drill/sandbox/prototype/exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TestExecutionAbstractions.java:[53,17] > sun.misc.Unsafe is internal proprietary API and may be removed in a future > release > [WARNING] > > /backup/java/wip/incubator-drill/sandbox/prototype/exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TestExecutionAbstractions.java:[94,12] > sun.misc.Unsafe is internal proprietary API and may be removed in a future > release > [WARNING] > > /backup/java/wip/incubator-drill/sandbox/prototype/exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TestExecutionAbstractions.java:[95,14] > sun.misc.Unsafe is internal proprietary API and may be removed in a future > release > [WARNING] > > /backup/java/wip/incubator-drill/sandbox/prototype/exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TestExecutionAbstractions.java:[98,26] > sun.misc.Unsafe is internal proprietary API and may be removed in a future > release > [WARNING] > > /backup/java/wip/incubator-drill/sandbox/prototype/exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TestExecutionAbstractions.java:[100,27] > sun.misc.Unsafe is internal proprietary API and may be removed in a future > release > [WARNING] > > /backup/java/wip/incubator-drill/sandbox/prototype/exec/java-exec/src/test/java/org/apache/drill/exec/store/TestParquetPhysicalPlan.java: > Some input files use or override a deprecated API. > [WARNING] > > /backup/java/wip/incubator-drill/sandbox/prototype/exec/java-exec/src/test/java/org/apache/drill/exec/store/TestParquetPhysicalPlan.java: > Recompile with -Xlint:deprecation for details. > [WARNING] > > /backup/java/wip/incubator-drill/sandbox/prototype/exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet/TestFileGenerator.java: > Some input files use unchecked or unsafe operations. > [WARNING] > > /backup/java/wip/incubator-drill/sandbox/prototype/exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet/TestFileGenerator.java: > Recompile with -Xlint:unchecked for details. > [INFO] > [INFO] --- maven-surefire-plugin:2.15:test (default-test) @ java-exec --- > [INFO] Surefire report directory: > > /backup/java/wip/incubator-drill/sandbox/prototype/exec/java-exec/target/surefire-reports > > ------------------------------------------------------- > T E S T S > ------------------------------------------------------- > Running org.apache.drill.exec.store.parquet.ParquetRecordReaderTest > 09:23:52.569 [main] DEBUG org.apache.hadoop.security.Groups - Creating new > Groups object > 09:23:52.662 [main] DEBUG org.apache.hadoop.security.Groups - Group mapping > impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; > cacheTimeout=300000 > 09:23:52.841 [main] DEBUG o.a.h.security.UserGroupInformation - hadoop > login > 09:23:52.842 [main] DEBUG o.a.h.security.UserGroupInformation - hadoop > login commit > 09:23:52.863 [main] DEBUG o.a.h.security.UserGroupInformation - using local > user:UnixPrincipal: tanujit > 09:23:52.867 [main] DEBUG o.a.h.security.UserGroupInformation - UGI > loginUser:tanujit > 09:23:52.921 [main] DEBUG org.apache.hadoop.fs.FileSystem - Creating > filesystem for file:/// > 09:23:53.011 [main] DEBUG o.a.hadoop.util.NativeCodeLoader - Trying to load > the custom-built native-hadoop library... > 09:23:53.033 [main] DEBUG o.a.hadoop.util.NativeCodeLoader - Failed to load > native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in > java.library.path > 09:23:53.034 [main] DEBUG o.a.hadoop.util.NativeCodeLoader - > > java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib > 09:23:53.034 [main] WARN o.a.hadoop.util.NativeCodeLoader - Unable to load > native-hadoop library for your platform... using builtin-java classes where > applicable > SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". > SLF4J: Defaulting to no-operation (NOP) logger implementation > SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further > details. > 09:23:55.443 [main] DEBUG org.reflections.Reflections - going to scan these > urls: > > jar:file:/home/tanujit/.m2/repository/org/apache/drill/common/1.0-SNAPSHOT/common-1.0-SNAPSHOT.jar!/ > > file:/backup/java/wip/incubator-drill/sandbox/prototype/exec/java-exec/target/test-classes/ > > file:/backup/java/wip/incubator-drill/sandbox/prototype/exec/java-exec/target/classes/ > > jar:file:/home/tanujit/.m2/repository/org/apache/drill/common/1.0-SNAPSHOT/common-1.0-SNAPSHOT-tests.jar!/ > > 09:23:56.000 [main] INFO org.reflections.Reflections - Reflections took > 556 ms to scan 4 urls, producing 669 keys and 1634 values > 09:23:56.028 [main] DEBUG i.n.u.i.l.InternalLoggerFactory - Using SLF4J as > the default logging framework > 09:23:56.037 [main] DEBUG i.n.c.MultithreadEventLoopGroup - > -Dio.netty.eventLoopThreads: 8 > 09:23:56.074 [main] DEBUG i.n.util.internal.PlatformDependent - UID: 1000 > 09:23:56.075 [main] DEBUG i.n.util.internal.PlatformDependent - Java > version: 7 > 09:23:56.076 [main] DEBUG i.n.util.internal.PlatformDependent - > -Dio.netty.noUnsafe: false > 09:23:56.079 [main] DEBUG i.n.util.internal.PlatformDependent0 - > java.nio.ByteBuffer.cleaner: available > 09:23:56.079 [main] DEBUG i.n.util.internal.PlatformDependent0 - > java.nio.Buffer.address: available > 09:23:56.080 [main] DEBUG i.n.util.internal.PlatformDependent0 - > sun.misc.Unsafe.theUnsafe: available > 09:23:56.081 [main] DEBUG i.n.util.internal.PlatformDependent0 - > sun.misc.Unsafe.copyMemory: available > 09:23:56.081 [main] DEBUG i.n.util.internal.PlatformDependent0 - > java.nio.Bits.unaligned: true > 09:23:56.081 [main] DEBUG i.n.util.internal.PlatformDependent - > sun.misc.Unsafe: available > 09:23:56.082 [main] DEBUG i.n.util.internal.PlatformDependent - > -Dio.netty.noJavassist: false > 09:23:56.145 [main] DEBUG i.n.util.internal.PlatformDependent - Javassist: > available > 09:23:56.145 [main] DEBUG i.n.util.internal.PlatformDependent - > -Dio.netty.noPreferDirect: false > 09:23:56.164 [main] DEBUG io.netty.channel.nio.NioEventLoop - > -Dio.netty.noKeySetOptimization: false > 09:23:56.164 [main] DEBUG io.netty.channel.nio.NioEventLoop - > -Dio.netty.selectorAutoRebuildThreshold: 512 > 09:23:56.200 [main] DEBUG i.n.buffer.PooledByteBufAllocatorL - > -Dio.netty.allocator.numHeapArenas: 4 > 09:23:56.200 [main] DEBUG i.n.buffer.PooledByteBufAllocatorL - > -Dio.netty.allocator.numDirectArenas: 4 > 09:23:56.200 [main] DEBUG i.n.buffer.PooledByteBufAllocatorL - > -Dio.netty.allocator.pageSize: 8192 > 09:23:56.200 [main] DEBUG i.n.buffer.PooledByteBufAllocatorL - > -Dio.netty.allocator.maxOrder: 11 > 09:23:56.200 [main] DEBUG i.n.buffer.PooledByteBufAllocatorL - > -Dio.netty.allocator.chunkSize: 16777216 > running bit > 09:23:56.277 [main] DEBUG io.netty.util.NetUtil - Loopback interface: lo > 09:23:56.277 [main] DEBUG io.netty.util.NetUtil - Loopback address: > /0:0:0:0:0:0:0:1%1 (primary) > 09:23:56.277 [main] DEBUG io.netty.util.NetUtil - Loopback address: / > 127.0.0.1 > 09:23:56.277 [main] DEBUG io.netty.util.NetUtil - > /proc/sys/net/core/somaxconn: 128 > 09:23:56.288 [UserServer-1] INFO i.n.handler.logging.LoggingHandler - [id: > 0x036be036] REGISTERED > 09:23:56.290 [UserServer-1] INFO i.n.handler.logging.LoggingHandler - [id: > 0x036be036] BIND(0.0.0.0/0.0.0.0:31010) > 09:23:56.294 [UserServer-1] INFO i.n.handler.logging.LoggingHandler - [id: > 0x036be036, /0:0:0:0:0:0:0:0:31010] ACTIVE > 09:23:56.310 [BitServer-1] INFO i.n.handler.logging.LoggingHandler - [id: > 0xb8299b35] REGISTERED > 09:23:56.311 [BitServer-1] INFO i.n.handler.logging.LoggingHandler - [id: > 0xb8299b35] BIND(0.0.0.0/0.0.0.0:31011) > 09:23:56.311 [BitServer-1] INFO i.n.handler.logging.LoggingHandler - [id: > 0xb8299b35, /0:0:0:0:0:0:0:0:31011] ACTIVE > running client > connecting - legolas -- 31010 > 09:23:56.545 [Client-1] DEBUG > i.n.u.i.JavassistTypeParameterMatcherGenerator - Generated: > io.netty.util.internal.__matchers__.io.netty.buffer.ByteBufMatcher > 09:23:56.549 [Client-1] DEBUG > i.n.u.i.JavassistTypeParameterMatcherGenerator - Generated: > > io.netty.util.internal.__matchers__.org.apache.drill.exec.rpc.OutboundRpcMessageMatcher > 09:23:56.550 [Client-1] DEBUG > i.n.u.i.JavassistTypeParameterMatcherGenerator - Generated: > > io.netty.util.internal.__matchers__.org.apache.drill.exec.rpc.InboundRpcMessageMatcher > 09:23:56.554 [UserServer-1] INFO i.n.handler.logging.LoggingHandler - [id: > 0x036be036, /0:0:0:0:0:0:0:0:31010] RECEIVED: [id: 0xf046bccd, / > 192.168.0.103:32952 => /192.168.0.103:31010] > 09:23:56.568 [Client-1] DEBUG io.netty.util.ResourceLeakDetector - > -Dio.netty.noResourceLeakDetection: false > running the query > Screen [endpoint=address: "legolas" > user_port: 31010 > bit_port: 31011 > , getChild()=org.apache.drill.exec.store.parquet.ParquetGroupScan@59c81460 > ] > 09:24:00.420 [WorkManager-2] WARN io.netty.util.ResourceLeakDetector - > LEAK: ByteBuf was GC'd before being released correctly. The following > stack trace shows where the leaked object was created, rather than where > you failed to release it. > io.netty.util.ResourceLeakException: > io.netty.buffer.PooledUnsafeDirectByteBufL@659974ba > at > > io.netty.util.ResourceLeakDetector$DefaultResourceLeak.<init>(ResourceLeakDetector.java:174) > ~[netty-common-4.0.7.Final.jar:na] > at io.netty.util.ResourceLeakDetector.open(ResourceLeakDetector.java:116) > ~[netty-common-4.0.7.Final.jar:na] > at io.netty.buffer.PooledByteBufL.<init>(PooledByteBufL.java:45) > ~[netty-bufferl-4.0.7.Final.jar:na] > at > > io.netty.buffer.PooledUnsafeDirectByteBufL.<init>(PooledUnsafeDirectByteBufL.java:53) > ~[netty-bufferl-4.0.7.Final.jar:na] > at > > io.netty.buffer.PooledUnsafeDirectByteBufL.<init>(PooledUnsafeDirectByteBufL.java:32) > ~[netty-bufferl-4.0.7.Final.jar:na] > at > > io.netty.buffer.PooledUnsafeDirectByteBufL$1.newObject(PooledUnsafeDirectByteBufL.java:39) > ~[netty-bufferl-4.0.7.Final.jar:na] > at > > io.netty.buffer.PooledUnsafeDirectByteBufL$1.newObject(PooledUnsafeDirectByteBufL.java:36) > ~[netty-bufferl-4.0.7.Final.jar:na] > at io.netty.util.Recycler.get(Recycler.java:40) > ~[netty-common-4.0.7.Final.jar:na] > at > > io.netty.buffer.PooledUnsafeDirectByteBufL.newInstance(PooledUnsafeDirectByteBufL.java:44) > ~[netty-bufferl-4.0.7.Final.jar:na] > at io.netty.buffer.PoolArenaL$DirectArena.newByteBuf(PoolArenaL.java:397) > ~[netty-bufferl-4.0.7.Final.jar:na] > at io.netty.buffer.PoolArenaL.allocate(PoolArenaL.java:93) > ~[netty-bufferl-4.0.7.Final.jar:na] > at > > io.netty.buffer.PooledByteBufAllocatorL.newDirectBuffer(PooledByteBufAllocatorL.java:235) > ~[netty-bufferl-4.0.7.Final.jar:na] > at > > io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:132) > ~[netty-buffer-4.0.7.Final.jar:na] > at > > io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:123) > ~[netty-buffer-4.0.7.Final.jar:na] > at > > org.apache.drill.exec.memory.DirectBufferAllocator.buffer(DirectBufferAllocator.java:35) > ~[classes/:na] > at org.apache.drill.exec.vector.BitVector.allocateNew(BitVector.java:54) > ~[classes/:na] > at > > org.apache.drill.exec.vector.AllocationHelper.allocate(AllocationHelper.java:12) > ~[classes/:na] > at > org.apache.drill.exec.store.VectorHolder.allocateNew(VectorHolder.java:87) > ~[classes/:na] > at org.apache.drill.exec.store.VectorHolder.reset(VectorHolder.java:74) > ~[classes/:na] > at > > org.apache.drill.exec.store.parquet.ParquetRecordReader.resetBatch(ParquetRecordReader.java:276) > ~[classes/:na] > at > > org.apache.drill.exec.store.parquet.ParquetRecordReader.next(ParquetRecordReader.java:318) > ~[classes/:na] > at org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:99) > ~[classes/:na] > at > > org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.next(ScreenCreator.java:77) > ~[classes/:na] > at org.apache.drill.exec.work.FragmentRunner.run(FragmentRunner.java:79) > ~[classes/:na] > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > ~[na:1.7.0_25] > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > ~[na:1.7.0_25] > at java.lang.Thread.run(Thread.java:724) ~[na:1.7.0_25] > Took 10003 ms to run query > 09:24:07.614 [BitServer-1] INFO i.n.handler.logging.LoggingHandler - [id: > 0xb8299b35, /0:0:0:0:0:0:0:0:31011] UNREGISTERED > 09:24:07.614 [UserServer-1] INFO i.n.handler.logging.LoggingHandler - [id: > 0x036be036, /0:0:0:0:0:0:0:0:31010] UNREGISTERED > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.329 sec > - in org.apache.drill.exec.store.parquet.ParquetRecordReaderTest > 09:24:07.621 [Thread-1] DEBUG org.apache.hadoop.fs.FileSystem - Starting > clear of FileSystem cache with 1 elements. > 09:24:07.623 [Thread-1] DEBUG org.apache.hadoop.fs.FileSystem - Removing > filesystem for file:/// > 09:24:07.623 [Thread-1] DEBUG org.apache.hadoop.fs.FileSystem - Removing > filesystem for file:/// > 09:24:07.623 [Thread-1] DEBUG org.apache.hadoop.fs.FileSystem - Done > clearing cache > > Results : > > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 > > [INFO] > ------------------------------------------------------------------------ > [INFO] BUILD SUCCESS > [INFO] > ------------------------------------------------------------------------ > [INFO] Total time: 55.501s > [INFO] Finished at: Tue Aug 27 09:24:07 IST 2013 > [INFO] Final Memory: 36M/429M > [INFO] > ------------------------------------------------------------------------ > > > -- > Regards, > Tanujit >
