Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/5718/ Java: 64bit/jdk-13.0.2 -XX:-UseCompressedOops -XX:+UseG1GC
1 tests failed. FAILED: junit.framework.TestSuite.org.apache.solr.cloud.hdfs.HdfsRecoverLeaseTest Error Message: 1 thread leaked from SUITE scope at org.apache.solr.cloud.hdfs.HdfsRecoverLeaseTest: 1) Thread[id=11901, name=globalEventExecutor-1-4, state=TIMED_WAITING, group=TGRP-HdfsRecoverLeaseTest] at [email protected]/jdk.internal.misc.Unsafe.park(Native Method) at [email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:235) at [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at [email protected]/java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:458) at app//io.netty.util.concurrent.GlobalEventExecutor.takeTask(GlobalEventExecutor.java:92) at app//io.netty.util.concurrent.GlobalEventExecutor$TaskRunner.run(GlobalEventExecutor.java:245) at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at [email protected]/java.lang.Thread.run(Thread.java:830) Stack Trace: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.hdfs.HdfsRecoverLeaseTest: 1) Thread[id=11901, name=globalEventExecutor-1-4, state=TIMED_WAITING, group=TGRP-HdfsRecoverLeaseTest] at [email protected]/jdk.internal.misc.Unsafe.park(Native Method) at [email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:235) at [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at [email protected]/java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:458) at app//io.netty.util.concurrent.GlobalEventExecutor.takeTask(GlobalEventExecutor.java:92) at app//io.netty.util.concurrent.GlobalEventExecutor$TaskRunner.run(GlobalEventExecutor.java:245) at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at [email protected]/java.lang.Thread.run(Thread.java:830) at __randomizedtesting.SeedInfo.seed([AAC78AF0DA00FD38]:0) Build Log: [...truncated 14005 lines...] [junit4] Suite: org.apache.solr.cloud.hdfs.HdfsRecoverLeaseTest [junit4] 2> 948778 INFO (SUITE-HdfsRecoverLeaseTest-seed#[AAC78AF0DA00FD38]-worker) [ ] o.a.s.SolrTestCase Setting 'solr.default.confdir' system property to test-framework derived value of '/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/server/solr/configsets/_default/conf' [junit4] 2> 948778 INFO (SUITE-HdfsRecoverLeaseTest-seed#[AAC78AF0DA00FD38]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom [junit4] 2> 948781 INFO (SUITE-HdfsRecoverLeaseTest-seed#[AAC78AF0DA00FD38]-worker) [ ] o.a.s.SolrTestCaseJ4 Created dataDir: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsRecoverLeaseTest_AAC78AF0DA00FD38-001/data-dir-67-001 [junit4] 2> 948781 WARN (SUITE-HdfsRecoverLeaseTest-seed#[AAC78AF0DA00FD38]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=20 numCloses=20 [junit4] 2> 948781 INFO (SUITE-HdfsRecoverLeaseTest-seed#[AAC78AF0DA00FD38]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false [junit4] 2> 948782 INFO (SUITE-HdfsRecoverLeaseTest-seed#[AAC78AF0DA00FD38]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason="", value=0.0/0.0, ssl=0.0/0.0, clientAuth=0.0/0.0) w/ MAC_OS_X supressed clientAuth [junit4] 1> Formatting using clusterid: testClusterID [junit4] 2> 949326 WARN (SUITE-HdfsRecoverLeaseTest-seed#[AAC78AF0DA00FD38]-worker) [ ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j [junit4] 2> 949331 INFO (SUITE-HdfsRecoverLeaseTest-seed#[AAC78AF0DA00FD38]-worker) [ ] o.e.j.s.Server jetty-9.4.24.v20191120; built: 2019-11-20T21:37:49.771Z; git: 363d5f2df3a8a28de40604320230664b9c793c16; jvm 13.0.2+8 [junit4] 2> 949338 INFO (SUITE-HdfsRecoverLeaseTest-seed#[AAC78AF0DA00FD38]-worker) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0 [junit4] 2> 949338 INFO (SUITE-HdfsRecoverLeaseTest-seed#[AAC78AF0DA00FD38]-worker) [ ] o.e.j.s.session No SessionScavenger set, using defaults [junit4] 2> 949339 INFO (SUITE-HdfsRecoverLeaseTest-seed#[AAC78AF0DA00FD38]-worker) [ ] o.e.j.s.session node0 Scavenging every 660000ms [junit4] 2> 949340 INFO (SUITE-HdfsRecoverLeaseTest-seed#[AAC78AF0DA00FD38]-worker) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7a4e301a{static,/static,jar:file:/Users/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-3.2.0-tests.jar!/webapps/static,AVAILABLE} [junit4] 2> 949698 INFO (SUITE-HdfsRecoverLeaseTest-seed#[AAC78AF0DA00FD38]-worker) [ ] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@3af22af5{hdfs,/,file:///Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/jetty-localhost-56603-hadoop-hdfs-3_2_0-tests_jar-_-any-12835043176089337808.dir/webapp/,AVAILABLE}{jar:file:/Users/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-3.2.0-tests.jar!/webapps/hdfs} [junit4] 2> 949702 INFO (SUITE-HdfsRecoverLeaseTest-seed#[AAC78AF0DA00FD38]-worker) [ ] o.e.j.s.AbstractConnector Started ServerConnector@2ea56256{HTTP/1.1,[http/1.1]}{localhost:56603} [junit4] 2> 949703 INFO (SUITE-HdfsRecoverLeaseTest-seed#[AAC78AF0DA00FD38]-worker) [ ] o.e.j.s.Server Started @949835ms [junit4] 2> 950173 WARN (SUITE-HdfsRecoverLeaseTest-seed#[AAC78AF0DA00FD38]-worker) [ ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j [junit4] 2> 950176 INFO (SUITE-HdfsRecoverLeaseTest-seed#[AAC78AF0DA00FD38]-worker) [ ] o.e.j.s.Server jetty-9.4.24.v20191120; built: 2019-11-20T21:37:49.771Z; git: 363d5f2df3a8a28de40604320230664b9c793c16; jvm 13.0.2+8 [junit4] 2> 950178 INFO (SUITE-HdfsRecoverLeaseTest-seed#[AAC78AF0DA00FD38]-worker) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0 [junit4] 2> 950178 INFO (SUITE-HdfsRecoverLeaseTest-seed#[AAC78AF0DA00FD38]-worker) [ ] o.e.j.s.session No SessionScavenger set, using defaults [junit4] 2> 950178 INFO (SUITE-HdfsRecoverLeaseTest-seed#[AAC78AF0DA00FD38]-worker) [ ] o.e.j.s.session node0 Scavenging every 660000ms [junit4] 2> 950179 INFO (SUITE-HdfsRecoverLeaseTest-seed#[AAC78AF0DA00FD38]-worker) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3d76c192{static,/static,jar:file:/Users/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-3.2.0-tests.jar!/webapps/static,AVAILABLE} [junit4] 2> 950428 INFO (SUITE-HdfsRecoverLeaseTest-seed#[AAC78AF0DA00FD38]-worker) [ ] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@7ae6b091{datanode,/,file:///Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/jetty-localhost-56608-hadoop-hdfs-3_2_0-tests_jar-_-any-2258634406076819031.dir/webapp/,AVAILABLE}{jar:file:/Users/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-3.2.0-tests.jar!/webapps/datanode} [junit4] 2> 950430 INFO (SUITE-HdfsRecoverLeaseTest-seed#[AAC78AF0DA00FD38]-worker) [ ] o.e.j.s.AbstractConnector Started ServerConnector@604f0899{HTTP/1.1,[http/1.1]}{localhost:56608} [junit4] 2> 950430 INFO (SUITE-HdfsRecoverLeaseTest-seed#[AAC78AF0DA00FD38]-worker) [ ] o.e.j.s.Server Started @950562ms [junit4] 2> 950606 WARN (SUITE-HdfsRecoverLeaseTest-seed#[AAC78AF0DA00FD38]-worker) [ ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j [junit4] 2> 950609 INFO (SUITE-HdfsRecoverLeaseTest-seed#[AAC78AF0DA00FD38]-worker) [ ] o.e.j.s.Server jetty-9.4.24.v20191120; built: 2019-11-20T21:37:49.771Z; git: 363d5f2df3a8a28de40604320230664b9c793c16; jvm 13.0.2+8 [junit4] 2> 950629 INFO (SUITE-HdfsRecoverLeaseTest-seed#[AAC78AF0DA00FD38]-worker) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0 [junit4] 2> 950629 INFO (SUITE-HdfsRecoverLeaseTest-seed#[AAC78AF0DA00FD38]-worker) [ ] o.e.j.s.session No SessionScavenger set, using defaults [junit4] 2> 950629 INFO (SUITE-HdfsRecoverLeaseTest-seed#[AAC78AF0DA00FD38]-worker) [ ] o.e.j.s.session node0 Scavenging every 600000ms [junit4] 2> 950633 INFO (SUITE-HdfsRecoverLeaseTest-seed#[AAC78AF0DA00FD38]-worker) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7632140b{static,/static,jar:file:/Users/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-3.2.0-tests.jar!/webapps/static,AVAILABLE} [junit4] 2> 950901 INFO (Block report processor) [ ] BlockStateChange BLOCK* processReport 0xb739181445a3d8ec: Processing first storage report for DS-b9f4546b-8c08-41a6-8f97-124994bbc289 from datanode 6dd644e1-9e83-443e-b72c-3ae44e9a6135 [junit4] 2> 950901 INFO (Block report processor) [ ] BlockStateChange BLOCK* processReport 0xb739181445a3d8ec: from storage DS-b9f4546b-8c08-41a6-8f97-124994bbc289 node DatanodeRegistration(127.0.0.1:56607, datanodeUuid=6dd644e1-9e83-443e-b72c-3ae44e9a6135, infoPort=56609, infoSecurePort=0, ipcPort=56610, storageInfo=lv=-57;cid=testClusterID;nsid=1315950498;c=1582983875483), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 [junit4] 2> 950903 INFO (Block report processor) [ ] BlockStateChange BLOCK* processReport 0xb739181445a3d8ec: Processing first storage report for DS-a26f5786-ef77-4f03-84e5-2565d3307239 from datanode 6dd644e1-9e83-443e-b72c-3ae44e9a6135 [junit4] 2> 950904 INFO (Block report processor) [ ] BlockStateChange BLOCK* processReport 0xb739181445a3d8ec: from storage DS-a26f5786-ef77-4f03-84e5-2565d3307239 node DatanodeRegistration(127.0.0.1:56607, datanodeUuid=6dd644e1-9e83-443e-b72c-3ae44e9a6135, infoPort=56609, infoSecurePort=0, ipcPort=56610, storageInfo=lv=-57;cid=testClusterID;nsid=1315950498;c=1582983875483), blocks: 0, hasStaleStorage: false, processing time: 1 msecs, invalidatedBlocks: 0 [junit4] 2> 951084 INFO (SUITE-HdfsRecoverLeaseTest-seed#[AAC78AF0DA00FD38]-worker) [ ] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@33d70eaf{datanode,/,file:///Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/jetty-localhost-56613-hadoop-hdfs-3_2_0-tests_jar-_-any-5960977767298714556.dir/webapp/,AVAILABLE}{jar:file:/Users/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-3.2.0-tests.jar!/webapps/datanode} [junit4] 2> 951094 INFO (SUITE-HdfsRecoverLeaseTest-seed#[AAC78AF0DA00FD38]-worker) [ ] o.e.j.s.AbstractConnector Started ServerConnector@416d7fba{HTTP/1.1,[http/1.1]}{localhost:56613} [junit4] 2> 951094 INFO (SUITE-HdfsRecoverLeaseTest-seed#[AAC78AF0DA00FD38]-worker) [ ] o.e.j.s.Server Started @951226ms [junit4] 2> 951347 INFO (Block report processor) [ ] BlockStateChange BLOCK* processReport 0x44e6d6d7f23b44ac: Processing first storage report for DS-8e4cf61f-d69d-4963-a1e6-8daf7d3bf370 from datanode 746adaab-5d74-41bc-ba33-7960e5baff53 [junit4] 2> 951347 INFO (Block report processor) [ ] BlockStateChange BLOCK* processReport 0x44e6d6d7f23b44ac: from storage DS-8e4cf61f-d69d-4963-a1e6-8daf7d3bf370 node DatanodeRegistration(127.0.0.1:56612, datanodeUuid=746adaab-5d74-41bc-ba33-7960e5baff53, infoPort=56616, infoSecurePort=0, ipcPort=56617, storageInfo=lv=-57;cid=testClusterID;nsid=1315950498;c=1582983875483), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 [junit4] 2> 951347 INFO (Block report processor) [ ] BlockStateChange BLOCK* processReport 0x44e6d6d7f23b44ac: Processing first storage report for DS-7fc73cf9-874d-4955-9ed9-6c7b5c06270f from datanode 746adaab-5d74-41bc-ba33-7960e5baff53 [junit4] 2> 951347 INFO (Block report processor) [ ] BlockStateChange BLOCK* processReport 0x44e6d6d7f23b44ac: from storage DS-7fc73cf9-874d-4955-9ed9-6c7b5c06270f node DatanodeRegistration(127.0.0.1:56612, datanodeUuid=746adaab-5d74-41bc-ba33-7960e5baff53, infoPort=56616, infoSecurePort=0, ipcPort=56617, storageInfo=lv=-57;cid=testClusterID;nsid=1315950498;c=1582983875483), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 [junit4] 2> 952184 INFO (TEST-HdfsRecoverLeaseTest.testMultiThreaded-seed#[AAC78AF0DA00FD38]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testMultiThreaded [junit4] 2> 954484 INFO (TEST-HdfsRecoverLeaseTest.testMultiThreaded-seed#[AAC78AF0DA00FD38]) [ ] o.a.s.u.FSHDFSUtils Recovering lease on dfs file hdfs://localhost:56606/file-1 [junit4] 2> 954512 WARN (IPC Server handler 1 on 56606) [ ] o.a.h.h.StateChange DIR* NameSystem.internalReleaseLease: File /file-1 has not been closed. Lease recovery is in progress. RecoveryId = 1003 for block blk_1073741826_1002 [junit4] 2> 954534 INFO (TEST-HdfsRecoverLeaseTest.testMultiThreaded-seed#[AAC78AF0DA00FD38]) [ ] o.a.s.u.FSHDFSUtils recoverLease=false, attempt=0 on file=hdfs://localhost:56606/file-1 after 37ms [junit4] 2> 957378 ERROR (DataXceiver for client DFSClient_NONMAPREDUCE_2032772839_11687 at /127.0.0.1:56624 [Receiving block BP-1020542081-127.0.0.1-1582983875483:blk_1073741826_1002]) [ ] o.a.h.h.s.d.DataNode 127.0.0.1:56607:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:56624 dst: /127.0.0.1:56607 [junit4] 2> => java.io.InterruptedIOException: Interrupted while waiting for IO on channel java.nio.channels.SocketChannel[connected local=localhost/127.0.0.1:56607 remote=/127.0.0.1:56624]. 60000 millis timeout left. [junit4] 2> at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:342) [junit4] 2> java.io.InterruptedIOException: Interrupted while waiting for IO on channel java.nio.channels.SocketChannel[connected local=localhost/127.0.0.1:56607 remote=/127.0.0.1:56624]. 60000 millis timeout left. [junit4] 2> at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:342) ~[hadoop-common-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) ~[hadoop-common-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) ~[hadoop-common-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) ~[hadoop-common-3.2.0.jar:?] [junit4] 2> at java.io.BufferedInputStream.fill(BufferedInputStream.java:245) ~[?:?] [junit4] 2> at java.io.BufferedInputStream.read1(BufferedInputStream.java:285) ~[?:?] [junit4] 2> at java.io.BufferedInputStream.read(BufferedInputStream.java:344) ~[?:?] [junit4] 2> at java.io.DataInputStream.read(DataInputStream.java:149) ~[?:?] [junit4] 2> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:210) ~[hadoop-common-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) ~[hadoop-hdfs-client-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) ~[hadoop-hdfs-client-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) ~[hadoop-hdfs-client-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) ~[hadoop-hdfs-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:971) ~[hadoop-hdfs-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:891) ~[hadoop-hdfs-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:173) ~[hadoop-hdfs-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:107) ~[hadoop-hdfs-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) ~[hadoop-hdfs-3.2.0.jar:?] [junit4] 2> at java.lang.Thread.run(Thread.java:830) [?:?] [junit4] 2> 957381 ERROR (DataXceiver for client DFSClient_NONMAPREDUCE_2032772839_11687 at /127.0.0.1:56625 [Receiving block BP-1020542081-127.0.0.1-1582983875483:blk_1073741826_1002]) [ ] o.a.h.h.s.d.DataNode 127.0.0.1:56612:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:56625 dst: /127.0.0.1:56612 [junit4] 2> => java.io.IOException: Premature EOF from inputStream [junit4] 2> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:212) [junit4] 2> java.io.IOException: Premature EOF from inputStream [junit4] 2> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:212) ~[hadoop-common-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) ~[hadoop-hdfs-client-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) ~[hadoop-hdfs-client-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) ~[hadoop-hdfs-client-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) ~[hadoop-hdfs-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:971) ~[hadoop-hdfs-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:891) ~[hadoop-hdfs-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:173) ~[hadoop-hdfs-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:107) ~[hadoop-hdfs-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) ~[hadoop-hdfs-3.2.0.jar:?] [junit4] 2> at java.lang.Thread.run(Thread.java:830) [?:?] [junit4] 2> 957401 WARN (ResponseProcessor for block BP-1020542081-127.0.0.1-1582983875483:blk_1073741826_1002) [ ] o.a.h.h.DataStreamer Exception for BP-1020542081-127.0.0.1-1582983875483:blk_1073741826_1002 [junit4] 2> => java.io.EOFException: Unexpected EOF while trying to read response from server [junit4] 2> at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:549) [junit4] 2> java.io.EOFException: Unexpected EOF while trying to read response from server [junit4] 2> at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:549) ~[hadoop-hdfs-client-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213) ~[hadoop-hdfs-client-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1086) ~[hadoop-hdfs-client-3.2.0.jar:?] [junit4] 2> 957417 WARN (DataStreamer for file /file-1 block BP-1020542081-127.0.0.1-1582983875483:blk_1073741826_1002) [ ] o.a.h.h.DataStreamer Error Recovery for BP-1020542081-127.0.0.1-1582983875483:blk_1073741826_1002 in pipeline [DatanodeInfoWithStorage[127.0.0.1:56607,DS-b9f4546b-8c08-41a6-8f97-124994bbc289,DISK], DatanodeInfoWithStorage[127.0.0.1:56612,DS-8e4cf61f-d69d-4963-a1e6-8daf7d3bf370,DISK]]: datanode 0(DatanodeInfoWithStorage[127.0.0.1:56607,DS-b9f4546b-8c08-41a6-8f97-124994bbc289,DISK]) is bad. [junit4] 2> 957446 WARN (DataStreamer for file /file-1 block BP-1020542081-127.0.0.1-1582983875483:blk_1073741826_1002) [ ] o.a.h.h.DataStreamer DataStreamer Exception [junit4] 2> => org.apache.hadoop.ipc.RemoteException(java.io.IOException): Unexpected BlockUCState: BP-1020542081-127.0.0.1-1582983875483:blk_1073741826_1002 is UNDER_RECOVERY but not UNDER_CONSTRUCTION [junit4] 2> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkUCBlock(FSNamesystem.java:5312) [junit4] 2> org.apache.hadoop.ipc.RemoteException: Unexpected BlockUCState: BP-1020542081-127.0.0.1-1582983875483:blk_1073741826_1002 is UNDER_RECOVERY but not UNDER_CONSTRUCTION [junit4] 2> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkUCBlock(FSNamesystem.java:5312) [junit4] 2> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.bumpBlockGenerationStamp(FSNamesystem.java:5380) [junit4] 2> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.updateBlockForPipeline(NameNodeRpcServer.java:948) [junit4] 2> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolServerSideTranslatorPB.java:1102) [junit4] 2> at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) [junit4] 2> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524) [junit4] 2> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) [junit4] 2> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) [junit4] 2> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) [junit4] 2> at java.base/java.security.AccessController.doPrivileged(AccessController.java:691) [junit4] 2> at java.base/javax.security.auth.Subject.doAs(Subject.java:423) [junit4] 2> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) [junit4] 2> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) [junit4] 2> [junit4] 2> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1511) ~[hadoop-common-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.ipc.Client.call(Client.java:1457) ~[hadoop-common-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.ipc.Client.call(Client.java:1367) ~[hadoop-common-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) ~[hadoop-common-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) ~[hadoop-common-3.2.0.jar:?] [junit4] 2> at com.sun.proxy.$Proxy68.updateBlockForPipeline(Unknown Source) ~[?:?] [junit4] 2> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolTranslatorPB.java:1041) ~[hadoop-hdfs-client-3.2.0.jar:?] [junit4] 2> at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] [junit4] 2> at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] [junit4] 2> at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] [junit4] 2> at java.lang.reflect.Method.invoke(Method.java:567) ~[?:?] [junit4] 2> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) ~[hadoop-common-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) ~[hadoop-common-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) ~[hadoop-common-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) ~[hadoop-common-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) ~[hadoop-common-3.2.0.jar:?] [junit4] 2> at com.sun.proxy.$Proxy71.updateBlockForPipeline(Unknown Source) ~[?:?] [junit4] 2> at org.apache.hadoop.hdfs.DataStreamer.updateBlockForPipeline(DataStreamer.java:1625) ~[hadoop-hdfs-client-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.DataStreamer.setupPipelineInternal(DataStreamer.java:1502) ~[hadoop-hdfs-client-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1481) ~[hadoop-hdfs-client-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.DataStreamer.processDatanodeOrExternalError(DataStreamer.java:1256) ~[hadoop-hdfs-client-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:667) ~[hadoop-hdfs-client-3.2.0.jar:?] [junit4] 2> 958543 INFO (TEST-HdfsRecoverLeaseTest.testMultiThreaded-seed#[AAC78AF0DA00FD38]) [ ] o.a.s.u.FSHDFSUtils recoverLease=true, attempt=1 on file=hdfs://localhost:56606/file-1 after 4058ms [junit4] 2> 958544 INFO (TEST-HdfsRecoverLeaseTest.testMultiThreaded-seed#[AAC78AF0DA00FD38]) [ ] o.a.s.u.FSHDFSUtils Recovering lease on dfs file hdfs://localhost:56606/file-2 [junit4] 2> 958549 WARN (IPC Server handler 3 on 56606) [ ] o.a.h.h.StateChange BLOCK* internalReleaseLease: All existing blocks are COMPLETE, lease removed, file /file-2 closed. [junit4] 2> 958550 INFO (TEST-HdfsRecoverLeaseTest.testMultiThreaded-seed#[AAC78AF0DA00FD38]) [ ] o.a.s.u.FSHDFSUtils recoverLease=true, attempt=0 on file=hdfs://localhost:56606/file-2 after 5ms [junit4] 2> 958550 INFO (TEST-HdfsRecoverLeaseTest.testMultiThreaded-seed#[AAC78AF0DA00FD38]) [ ] o.a.s.u.FSHDFSUtils Recovering lease on dfs file hdfs://localhost:56606/file-0 [junit4] 2> 958559 WARN (IPC Server handler 4 on 56606) [ ] o.a.h.h.StateChange DIR* NameSystem.internalReleaseLease: File /file-0 has not been closed. Lease recovery is in progress. RecoveryId = 1004 for block blk_1073741825_1001 [junit4] 2> 958560 INFO (TEST-HdfsRecoverLeaseTest.testMultiThreaded-seed#[AAC78AF0DA00FD38]) [ ] o.a.s.u.FSHDFSUtils recoverLease=false, attempt=0 on file=hdfs://localhost:56606/file-0 after 9ms [junit4] 2> 961179 ERROR (DataXceiver for client DFSClient_NONMAPREDUCE_-1273894514_11687 at /127.0.0.1:56622 [Receiving block BP-1020542081-127.0.0.1-1582983875483:blk_1073741825_1001]) [ ] o.a.h.h.s.d.DataNode 127.0.0.1:56607:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:56622 dst: /127.0.0.1:56607 [junit4] 2> => java.io.InterruptedIOException: Interrupted while waiting for IO on channel java.nio.channels.SocketChannel[connected local=localhost/127.0.0.1:56607 remote=/127.0.0.1:56622]. 60000 millis timeout left. [junit4] 2> at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:342) [junit4] 2> java.io.InterruptedIOException: Interrupted while waiting for IO on channel java.nio.channels.SocketChannel[connected local=localhost/127.0.0.1:56607 remote=/127.0.0.1:56622]. 60000 millis timeout left. [junit4] 2> at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:342) ~[hadoop-common-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) ~[hadoop-common-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) ~[hadoop-common-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) ~[hadoop-common-3.2.0.jar:?] [junit4] 2> at java.io.BufferedInputStream.fill(BufferedInputStream.java:245) ~[?:?] [junit4] 2> at java.io.BufferedInputStream.read1(BufferedInputStream.java:285) ~[?:?] [junit4] 2> at java.io.BufferedInputStream.read(BufferedInputStream.java:344) ~[?:?] [junit4] 2> at java.io.DataInputStream.read(DataInputStream.java:149) ~[?:?] [junit4] 2> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:210) ~[hadoop-common-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) ~[hadoop-hdfs-client-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) ~[hadoop-hdfs-client-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) ~[hadoop-hdfs-client-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) ~[hadoop-hdfs-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:971) ~[hadoop-hdfs-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:891) ~[hadoop-hdfs-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:173) ~[hadoop-hdfs-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:107) ~[hadoop-hdfs-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) ~[hadoop-hdfs-3.2.0.jar:?] [junit4] 2> at java.lang.Thread.run(Thread.java:830) [?:?] [junit4] 2> 961184 WARN (ResponseProcessor for block BP-1020542081-127.0.0.1-1582983875483:blk_1073741825_1001) [ ] o.a.h.h.DataStreamer Exception for BP-1020542081-127.0.0.1-1582983875483:blk_1073741825_1001 [junit4] 2> => java.io.EOFException: Unexpected EOF while trying to read response from server [junit4] 2> at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:549) [junit4] 2> java.io.EOFException: Unexpected EOF while trying to read response from server [junit4] 2> at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:549) ~[hadoop-hdfs-client-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213) ~[hadoop-hdfs-client-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1086) ~[hadoop-hdfs-client-3.2.0.jar:?] [junit4] 2> 961184 ERROR (DataXceiver for client DFSClient_NONMAPREDUCE_-1273894514_11687 at /127.0.0.1:56623 [Receiving block BP-1020542081-127.0.0.1-1582983875483:blk_1073741825_1001]) [ ] o.a.h.h.s.d.DataNode 127.0.0.1:56612:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:56623 dst: /127.0.0.1:56612 [junit4] 2> => java.io.IOException: Premature EOF from inputStream [junit4] 2> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:212) [junit4] 2> java.io.IOException: Premature EOF from inputStream [junit4] 2> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:212) ~[hadoop-common-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) ~[hadoop-hdfs-client-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) ~[hadoop-hdfs-client-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) ~[hadoop-hdfs-client-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) ~[hadoop-hdfs-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:971) ~[hadoop-hdfs-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:891) ~[hadoop-hdfs-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:173) ~[hadoop-hdfs-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:107) ~[hadoop-hdfs-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) ~[hadoop-hdfs-3.2.0.jar:?] [junit4] 2> at java.lang.Thread.run(Thread.java:830) [?:?] [junit4] 2> 961185 WARN (DataStreamer for file /file-0 block BP-1020542081-127.0.0.1-1582983875483:blk_1073741825_1001) [ ] o.a.h.h.DataStreamer Error Recovery for BP-1020542081-127.0.0.1-1582983875483:blk_1073741825_1001 in pipeline [DatanodeInfoWithStorage[127.0.0.1:56607,DS-b9f4546b-8c08-41a6-8f97-124994bbc289,DISK], DatanodeInfoWithStorage[127.0.0.1:56612,DS-8e4cf61f-d69d-4963-a1e6-8daf7d3bf370,DISK]]: datanode 0(DatanodeInfoWithStorage[127.0.0.1:56607,DS-b9f4546b-8c08-41a6-8f97-124994bbc289,DISK]) is bad. [junit4] 2> 961190 WARN (DataStreamer for file /file-0 block BP-1020542081-127.0.0.1-1582983875483:blk_1073741825_1001) [ ] o.a.h.h.DataStreamer DataStreamer Exception [junit4] 2> => org.apache.hadoop.ipc.RemoteException(java.io.IOException): Unexpected BlockUCState: BP-1020542081-127.0.0.1-1582983875483:blk_1073741825_1001 is UNDER_RECOVERY but not UNDER_CONSTRUCTION [junit4] 2> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkUCBlock(FSNamesystem.java:5312) [junit4] 2> org.apache.hadoop.ipc.RemoteException: Unexpected BlockUCState: BP-1020542081-127.0.0.1-1582983875483:blk_1073741825_1001 is UNDER_RECOVERY but not UNDER_CONSTRUCTION [junit4] 2> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkUCBlock(FSNamesystem.java:5312) [junit4] 2> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.bumpBlockGenerationStamp(FSNamesystem.java:5380) [junit4] 2> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.updateBlockForPipeline(NameNodeRpcServer.java:948) [junit4] 2> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolServerSideTranslatorPB.java:1102) [junit4] 2> at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) [junit4] 2> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524) [junit4] 2> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) [junit4] 2> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) [junit4] 2> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) [junit4] 2> at java.base/java.security.AccessController.doPrivileged(AccessController.java:691) [junit4] 2> at java.base/javax.security.auth.Subject.doAs(Subject.java:423) [junit4] 2> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) [junit4] 2> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) [junit4] 2> [junit4] 2> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1511) ~[hadoop-common-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.ipc.Client.call(Client.java:1457) ~[hadoop-common-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.ipc.Client.call(Client.java:1367) ~[hadoop-common-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) ~[hadoop-common-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) ~[hadoop-common-3.2.0.jar:?] [junit4] 2> at com.sun.proxy.$Proxy68.updateBlockForPipeline(Unknown Source) ~[?:?] [junit4] 2> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolTranslatorPB.java:1041) ~[hadoop-hdfs-client-3.2.0.jar:?] [junit4] 2> at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] [junit4] 2> at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] [junit4] 2> at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] [junit4] 2> at java.lang.reflect.Method.invoke(Method.java:567) ~[?:?] [junit4] 2> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) ~[hadoop-common-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) ~[hadoop-common-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) ~[hadoop-common-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) ~[hadoop-common-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) ~[hadoop-common-3.2.0.jar:?] [junit4] 2> at com.sun.proxy.$Proxy71.updateBlockForPipeline(Unknown Source) ~[?:?] [junit4] 2> at org.apache.hadoop.hdfs.DataStreamer.updateBlockForPipeline(DataStreamer.java:1625) ~[hadoop-hdfs-client-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.DataStreamer.setupPipelineInternal(DataStreamer.java:1502) ~[hadoop-hdfs-client-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1481) ~[hadoop-hdfs-client-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.DataStreamer.processDatanodeOrExternalError(DataStreamer.java:1256) ~[hadoop-hdfs-client-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:667) ~[hadoop-hdfs-client-3.2.0.jar:?] [junit4] 2> 963970 INFO (TEST-HdfsRecoverLeaseTest.testMultiThreaded-seed#[AAC78AF0DA00FD38]) [ ] o.a.s.u.FSHDFSUtils recoverLease=true, attempt=1 on file=hdfs://localhost:56606/file-0 after 5419ms [junit4] 2> 963971 ERROR (TEST-HdfsRecoverLeaseTest.testMultiThreaded-seed#[AAC78AF0DA00FD38]) [ ] o.a.h.h.DFSClient Failed to close file: /file-1 with inode: 16393 [junit4] 2> => org.apache.hadoop.ipc.RemoteException(java.io.IOException): Unexpected BlockUCState: BP-1020542081-127.0.0.1-1582983875483:blk_1073741826_1002 is UNDER_RECOVERY but not UNDER_CONSTRUCTION [junit4] 2> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkUCBlock(FSNamesystem.java:5312) [junit4] 2> org.apache.hadoop.ipc.RemoteException: Unexpected BlockUCState: BP-1020542081-127.0.0.1-1582983875483:blk_1073741826_1002 is UNDER_RECOVERY but not UNDER_CONSTRUCTION [junit4] 2> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkUCBlock(FSNamesystem.java:5312) [junit4] 2> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.bumpBlockGenerationStamp(FSNamesystem.java:5380) [junit4] 2> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.updateBlockForPipeline(NameNodeRpcServer.java:948) [junit4] 2> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolServerSideTranslatorPB.java:1102) [junit4] 2> at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) [junit4] 2> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524) [junit4] 2> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) [junit4] 2> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) [junit4] 2> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) [junit4] 2> at java.base/java.security.AccessController.doPrivileged(AccessController.java:691) [junit4] 2> at java.base/javax.security.auth.Subject.doAs(Subject.java:423) [junit4] 2> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) [junit4] 2> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) [junit4] 2> [junit4] 2> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1511) ~[hadoop-common-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.ipc.Client.call(Client.java:1457) ~[hadoop-common-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.ipc.Client.call(Client.java:1367) ~[hadoop-common-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) ~[hadoop-common-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) ~[hadoop-common-3.2.0.jar:?] [junit4] 2> at com.sun.proxy.$Proxy68.updateBlockForPipeline(Unknown Source) ~[?:?] [junit4] 2> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolTranslatorPB.java:1041) ~[hadoop-hdfs-client-3.2.0.jar:?] [junit4] 2> at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] [junit4] 2> at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] [junit4] 2> at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] [junit4] 2> at java.lang.reflect.Method.invoke(Method.java:567) ~[?:?] [junit4] 2> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) ~[hadoop-common-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) ~[hadoop-common-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) ~[hadoop-common-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) ~[hadoop-common-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) ~[hadoop-common-3.2.0.jar:?] [junit4] 2> at com.sun.proxy.$Proxy71.updateBlockForPipeline(Unknown Source) ~[?:?] [junit4] 2> at org.apache.hadoop.hdfs.DataStreamer.updateBlockForPipeline(DataStreamer.java:1625) ~[hadoop-hdfs-client-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.DataStreamer.setupPipelineInternal(DataStreamer.java:1502) ~[hadoop-hdfs-client-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1481) ~[hadoop-hdfs-client-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.DataStreamer.processDatanodeOrExternalError(DataStreamer.java:1256) ~[hadoop-hdfs-client-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:667) ~[hadoop-hdfs-client-3.2.0.jar:?] [junit4] 2> 963976 ERROR (TEST-HdfsRecoverLeaseTest.testMultiThreaded-seed#[AAC78AF0DA00FD38]) [ ] o.a.h.h.DFSClient Failed to close file: /file-0 with inode: 16392 [junit4] 2> => org.apache.hadoop.ipc.RemoteException(java.io.IOException): Unexpected BlockUCState: BP-1020542081-127.0.0.1-1582983875483:blk_1073741825_1001 is UNDER_RECOVERY but not UNDER_CONSTRUCTION [junit4] 2> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkUCBlock(FSNamesystem.java:5312) [junit4] 2> org.apache.hadoop.ipc.RemoteException: Unexpected BlockUCState: BP-1020542081-127.0.0.1-1582983875483:blk_1073741825_1001 is UNDER_RECOVERY but not UNDER_CONSTRUCTION [junit4] 2> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkUCBlock(FSNamesystem.java:5312) [junit4] 2> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.bumpBlockGenerationStamp(FSNamesystem.java:5380) [junit4] 2> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.updateBlockForPipeline(NameNodeRpcServer.java:948) [junit4] 2> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolServerSideTranslatorPB.java:1102) [junit4] 2> at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) [junit4] 2> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524) [junit4] 2> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) [junit4] 2> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) [junit4] 2> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) [junit4] 2> at java.base/java.security.AccessController.doPrivileged(AccessController.java:691) [junit4] 2> at java.base/javax.security.auth.Subject.doAs(Subject.java:423) [junit4] 2> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) [junit4] 2> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) [junit4] 2> [junit4] 2> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1511) ~[hadoop-common-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.ipc.Client.call(Client.java:1457) ~[hadoop-common-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.ipc.Client.call(Client.java:1367) ~[hadoop-common-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) ~[hadoop-common-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) ~[hadoop-common-3.2.0.jar:?] [junit4] 2> at com.sun.proxy.$Proxy68.updateBlockForPipeline(Unknown Source) ~[?:?] [junit4] 2> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolTranslatorPB.java:1041) ~[hadoop-hdfs-client-3.2.0.jar:?] [junit4] 2> at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] [junit4] 2> at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] [junit4] 2> at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] [junit4] 2> at java.lang.reflect.Method.invoke(Method.java:567) ~[?:?] [junit4] 2> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) ~[hadoop-common-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) ~[hadoop-common-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) ~[hadoop-common-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) ~[hadoop-common-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) ~[hadoop-common-3.2.0.jar:?] [junit4] 2> at com.sun.proxy.$Proxy71.updateBlockForPipeline(Unknown Source) ~[?:?] [junit4] 2> at org.apache.hadoop.hdfs.DataStreamer.updateBlockForPipeline(DataStreamer.java:1625) ~[hadoop-hdfs-client-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.DataStreamer.setupPipelineInternal(DataStreamer.java:1502) ~[hadoop-hdfs-client-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1481) ~[hadoop-hdfs-client-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.DataStreamer.processDatanodeOrExternalError(DataStreamer.java:1256) ~[hadoop-hdfs-client-3.2.0.jar:?] [junit4] 2> at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:667) ~[hadoop-hdfs-client-3.2.0.jar:?] [junit4] 2> 963976 INFO (TEST-HdfsRecoverLeaseTest.testMultiThreaded-seed#[AAC78AF0DA00FD38]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testMultiThreaded [junit4] 2> 964120 INFO (TEST-HdfsRecoverLeaseTest.testBasic-seed#[AAC78AF0DA00FD38]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testBasic [junit4] 2> 964636 INFO (TEST-HdfsRecoverLeaseTest.testBasic-seed#[AAC78AF0DA00FD38]) [ ] o.a.s.u.FSHDFSUtils Recovering lease on dfs file hdfs://localhost:56606/testfile [junit4] 2> 964645 INFO (TEST-HdfsRecoverLeaseTest.testBasic-seed#[AAC78AF0DA00FD38]) [ ] o.a.s.u.FSHDFSUtils Recovering lease on dfs file hdfs://localhost:56606/testfile2 [junit4] 2> 964649 WARN (IPC Server handler 5 on 56606) [ ] o.a.h.h.StateChange BLOCK* internalReleaseLease: All existing blocks are COMPLETE, lease removed, file /testfile2 closed. [junit4] 2> 964650 INFO (TEST-HdfsRecoverLeaseTest.testBasic-seed#[AAC78AF0DA00FD38]) [ ] o.a.s.u.FSHDFSUtils recoverLease=true, attempt=0 on file=hdfs://localhost:56606/testfile2 after 4ms [junit4] 2> 964658 INFO (TEST-HdfsRecoverLeaseTest.testBasic-seed#[AAC78AF0DA00FD38]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testBasic [junit4] 2> 964669 WARN (SUITE-HdfsRecoverLeaseTest-seed#[AAC78AF0DA00FD38]-worker) [ ] o.a.h.h.s.d.DirectoryScanner DirectoryScanner: shutdown has been called [junit4] 2> 964690 INFO (SUITE-HdfsRecoverLeaseTest-seed#[AAC78AF0DA00FD38]-worker) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.w.WebAppContext@33d70eaf{datanode,/,null,UNAVAILABLE}{jar:file:/Users/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-3.2.0-tests.jar!/webapps/datanode} [junit4] 2> 964691 INFO (SUITE-HdfsRecoverLeaseTest-seed#[AAC78AF0DA00FD38]-worker) [ ] o.e.j.s.AbstractConnector Stopped ServerConnector@416d7fba{HTTP/1.1,[http/1.1]}{localhost:0} [junit4] 2> 964691 INFO (SUITE-HdfsRecoverLeaseTest-seed#[AAC78AF0DA00FD38]-worker) [ ] o.e.j.s.session node0 Stopped scavenging [junit4] 2> 964692 INFO (SUITE-HdfsRecoverLeaseTest-seed#[AAC78AF0DA00FD38]-worker) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@7632140b{static,/static,jar:file:/Users/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-3.2.0-tests.jar!/webapps/static,UNAVAILABLE} [junit4] 2> 964702 WARN (BP-1020542081-127.0.0.1-1582983875483 heartbeating to localhost/127.0.0.1:56606) [ ] o.a.h.h.s.d.IncrementalBlockReportManager IncrementalBlockReportManager interrupted [junit4] 2> 964702 WARN (BP-1020542081-127.0.0.1-1582983875483 heartbeating to localhost/127.0.0.1:56606) [ ] o.a.h.h.s.d.DataNode Ending block pool service for: Block pool BP-1020542081-127.0.0.1-1582983875483 (Datanode Uuid 746adaab-5d74-41bc-ba33-7960e5baff53) service to localhost/127.0.0.1:56606 [junit4] 2> 964739 WARN (SUITE-HdfsRecoverLeaseTest-seed#[AAC78AF0DA00FD38]-worker) [ ] o.a.h.h.s.d.DirectoryScanner DirectoryScanner: shutdown has been called [junit4] 2> 964756 INFO (SUITE-HdfsRecoverLeaseTest-seed#[AAC78AF0DA00FD38]-worker) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.w.WebAppContext@7ae6b091{datanode,/,null,UNAVAILABLE}{jar:file:/Users/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-3.2.0-tests.jar!/webapps/datanode} [junit4] 2> 964757 INFO (SUITE-HdfsRecoverLeaseTest-seed#[AAC78AF0DA00FD38]-worker) [ ] o.e.j.s.AbstractConnector Stopped ServerConnector@604f0899{HTTP/1.1,[http/1.1]}{localhost:0} [junit4] 2> 964757 INFO (SUITE-HdfsRecoverLeaseTest-seed#[AAC78AF0DA00FD38]-worker) [ ] o.e.j.s.session node0 Stopped scavenging [junit4] 2> 964757 INFO (SUITE-HdfsRecoverLeaseTest-seed#[AAC78AF0DA00FD38]-worker) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@3d76c192{static,/static,jar:file:/Users/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-3.2.0-tests.jar!/webapps/static,UNAVAILABLE} [junit4] 2> 964766 WARN (BP-1020542081-127.0.0.1-1582983875483 heartbeating to localhost/127.0.0.1:56606) [ ] o.a.h.h.s.d.IncrementalBlockReportManager IncrementalBlockReportManager interrupted [junit4] 2> 964766 WARN (BP-1020542081-127.0.0.1-1582983875483 heartbeating to localhost/127.0.0.1:56606) [ ] o.a.h.h.s.d.DataNode Ending block pool service for: Block pool BP-1020542081-127.0.0.1-1582983875483 (Datanode Uuid 6dd644e1-9e83-443e-b72c-3ae44e9a6135) service to localhost/127.0.0.1:56606 [junit4] 2> 964817 INFO (SUITE-HdfsRecoverLeaseTest-seed#[AAC78AF0DA00FD38]-worker) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.w.WebAppContext@3af22af5{hdfs,/,null,UNAVAILABLE}{jar:file:/Users/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-3.2.0-tests.jar!/webapps/hdfs} [junit4] 2> 964820 INFO (SUITE-HdfsRecoverLeaseTest-seed#[AAC78AF0DA00FD38]-worker) [ ] o.e.j.s.AbstractConnector Stopped ServerConnector@2ea56256{HTTP/1.1,[http/1.1]}{localhost:0} [junit4] 2> 964820 INFO (SUITE-HdfsRecoverLeaseTest-seed#[AAC78AF0DA00FD38]-worker) [ ] o.e.j.s.session node0 Stopped scavenging [junit4] 2> 964820 INFO (SUITE-HdfsRecoverLeaseTest-seed#[AAC78AF0DA00FD38]-worker) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@7a4e301a{static,/static,jar:file:/Users/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-3.2.0-tests.jar!/webapps/static,UNAVAILABLE} [junit4] 2> 964925 INFO (SUITE-HdfsRecoverLeaseTest-seed#[AAC78AF0DA00FD38]-worker) [ ] o.a.s.SolrTestCaseJ4 ------------------------------------------------------- Done waiting for tracked resources to be released [junit4] 2> Feb 29, 2020 1:44:51 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks [junit4] 2> WARNING: Will linger awaiting termination of 48 leaked thread(s). [junit4] 2> Feb 29, 2020 1:45:01 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks [junit4] 2> SEVERE: 1 thread leaked from SUITE scope at org.apache.solr.cloud.hdfs.HdfsRecoverLeaseTest: [junit4] 2> 1) Thread[id=11901, name=globalEventExecutor-1-4, state=TIMED_WAITING, group=TGRP-HdfsRecoverLeaseTest] [junit4] 2> at [email protected]/jdk.internal.misc.Unsafe.park(Native Method) [junit4] 2> at [email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:235) [junit4] 2> at [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) [junit4] 2> at [email protected]/java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:458) [junit4] 2> at app//io.netty.util.concurrent.GlobalEventExecutor.takeTask(GlobalEventExecutor.java:92) [junit4] 2> at app//io.netty.util.concurrent.GlobalEventExecutor$TaskRunner.run(GlobalEventExecutor.java:245) [junit4] 2> at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [junit4] 2> at [email protected]/java.lang.Thread.run(Thread.java:830) [junit4] 2> Feb 29, 2020 1:45:01 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll [junit4] 2> INFO: Starting to interrupt leaked threads: [junit4] 2> 1) Thread[id=11901, name=globalEventExecutor-1-4, state=TIMED_WAITING, group=TGRP-HdfsRecoverLeaseTest] [junit4] 2> Feb 29, 2020 1:45:01 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll [junit4] 2> INFO: All leaked threads terminated. [junit4] 2> NOTE: test params are: codec=Asserting(Lucene84): {}, docValues:{}, maxPointsInLeafNode=835, maxMBSortInHeap=5.814976738745123, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@4899ec24), locale=ff-Latn-SL, timezone=Iran [junit4] 2> NOTE: Mac OS X 10.14.6 x86_64/AdoptOpenJDK 13.0.2 (64-bit)/cpus=6,threads=6,free=165174080,total=536870912 [junit4] 2> NOTE: All tests run in this JVM: [MultiSolrCloudTestCaseTest, TestPayloadCheckQParserPlugin, TestDistributedStatsComponentCardinality, CollectionReloadTest, CloudReplicaSourceTest, DataDrivenBlockJoinTest, MigrateRouteKeyTest, TestAddFieldRealTimeGet, CachingDirectoryFactoryTest, TestExactStatsCache, TestFieldTypeCollectionResource, OverriddenZkACLAndCredentialsProvidersTest, UUIDFieldTest, TestMergePolicyConfig, SplitByPrefixTest, FullSolrCloudDistribCmdsTest, TestCloudNestedDocsSort, TestMinHashQParser, MetricsHistoryHandlerTest, TestShardHandlerFactory, TestJsonFacetsStatsParsing, TransactionLogTest, TestWaitForStateWithJettyShutdowns, ResolveAnalyzerByNameTest, TestStressThreadBackup, BasicFunctionalityTest, TestMacros, SuggesterTest, SpellingQueryConverterTest, TestSegmentSorting, ChaosMonkeySafeLeaderTest, TestExactStatsCacheCloud, InfixSuggestersTest, TestHdfsBackupRestoreCore, CdcrReplicationHandlerTest, IndexSchemaTest, ComputePlanActionTest, TestManagedSchemaThreadSafety, ResourceLoaderTest, TriggerSetPropertiesIntegrationTest, XsltUpdateRequestHandlerTest, TestSolrCLIRunExample, BasicDistributedZk2Test, TestClassicSimilarityFactory, SuggestComponentContextFilterQueryTest, TestPushWriter, SolrCloudReportersTest, TestUnifiedSolrHighlighterWithoutStoredId, ClassificationUpdateProcessorIntegrationTest, HighlighterMaxOffsetTest, SearchHandlerTest, TestChildDocTransformer, DistributedFacetPivotSmallTest, ClusterStateMockUtilTest, LeaderTragicEventTest, TestStressUserVersions, ScheduledMaintenanceTriggerTest, TestLeaderElectionZkExpiry, HdfsBasicDistributedZk2Test, SolrMetricsIntegrationTest, TestExportWriter, TestConfigSetProperties, TestSolrCachePerf, DistributedExpandComponentTest, TestCustomStream, HdfsRecoverLeaseTest] [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=HdfsRecoverLeaseTest -Dtests.seed=AAC78AF0DA00FD38 -Dtests.slow=true -Dtests.locale=ff-Latn-SL -Dtests.timezone=Iran -Dtests.asserts=true -Dtests.file.encoding=US-ASCII [junit4] ERROR 0.00s J1 | HdfsRecoverLeaseTest (suite) <<< [junit4] > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.hdfs.HdfsRecoverLeaseTest: [junit4] > 1) Thread[id=11901, name=globalEventExecutor-1-4, state=TIMED_WAITING, group=TGRP-HdfsRecoverLeaseTest] [junit4] > at [email protected]/jdk.internal.misc.Unsafe.park(Native Method) [junit4] > at [email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:235) [junit4] > at [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) [junit4] > at [email protected]/java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:458) [junit4] > at app//io.netty.util.concurrent.GlobalEventExecutor.takeTask(GlobalEventExecutor.java:92) [junit4] > at app//io.netty.util.concurrent.GlobalEventExecutor$TaskRunner.run(GlobalEventExecutor.java:245) [junit4] > at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [junit4] > at [email protected]/java.lang.Thread.run(Thread.java:830) [junit4] > at __randomizedtesting.SeedInfo.seed([AAC78AF0DA00FD38]:0) [junit4] Completed [359/905 (1!)] on J1 in 26.34s, 2 tests, 1 error <<< FAILURES! [...truncated 54434 lines...] [repro] Jenkins log URL: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/5718/consoleText [repro] Revision: 5f9bf6b707a398e5cb42bf313b3a444e169ae6fa [repro] Ant options: "-Dargs=-XX:-UseCompressedOops -XX:+UseG1GC" [repro] Repro line: ant test -Dtestcase=HdfsRecoverLeaseTest -Dtests.seed=AAC78AF0DA00FD38 -Dtests.slow=true -Dtests.locale=ff-Latn-SL -Dtests.timezone=Iran -Dtests.asserts=true -Dtests.file.encoding=US-ASCII [repro] JUnit rest result XML files will be moved to: ./repro-reports [repro] ant clean [...truncated 6 lines...] [repro] Test suites by module: [repro] solr/core [repro] HdfsRecoverLeaseTest [repro] ant compile-test [...truncated 3150 lines...] [repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=5 -Dtests.class="*.HdfsRecoverLeaseTest" -Dtests.showOutput=onerror "-Dargs=-XX:-UseCompressedOops -XX:+UseG1GC" -Dtests.seed=AAC78AF0DA00FD38 -Dtests.slow=true -Dtests.locale=ff-Latn-SL -Dtests.timezone=Iran -Dtests.asserts=true -Dtests.file.encoding=US-ASCII [...truncated 86 lines...] [repro] Failures w/original seeds: [repro] 0/5 failed: org.apache.solr.cloud.hdfs.HdfsRecoverLeaseTest [repro] Exiting with code 0 [...truncated 73 lines...]
--------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
