Hi Why does KNNVectorDict not use IndexInput with a IndexInput#randomAccessSlice()? Then problem is handled by mmapdirectory.
Lucene classes should never ever use direct access to file system and always go through the Directory. Uwe ----- Uwe Schindler Achterdiek 19, D-28357 Bremen https://www.thetaphi.de eMail: [email protected] > -----Original Message----- > From: Dawid Weiss <[email protected]> > Sent: Thursday, August 19, 2021 11:03 AM > To: Lucene Dev <[email protected]> > Cc: Uwe Schindler (SD DataSolutions GmbH) <[email protected]> > Subject: Re: [JENKINS] Lucene-main-Windows (64bit/jdk-15) - Build # 10089 - > Unstable! > > This is actually more complicated. KnnVectorDict uses a mapped byte > buffer - such buffers lock the source file on Windows and closing the > original channel doesn't really affect their state: > > private final MappedByteBuffer vbuffer; > > It'd need to use the unmapper hack currently present in mmapdirectory > to release this buffer. I CC Uwe - perhaps he has an idea how to solve > it cleanly. At the moment tests on Windows will fail because the > temporary file/ directory can't be unmapped. I think it's a legitimate > bug since we do want those resources released eagerly rather than rely > on the JVM garbage collector to unmap them. > > Dawid > > On Wed, Aug 18, 2021 at 9:25 PM Michael Sokolov <[email protected]> > wrote: > > > > Ah I'll check, thanks for the pointer > > > > On Wed, Aug 18, 2021 at 12:19 PM Dawid Weiss <[email protected]> > wrote: > > > > > > Looks like something (file handle) is not properly closed - this > > > causes temp. file removal to fail on Windows. > > > > > > D. > > > > > > On Wed, Aug 18, 2021 at 5:52 PM Policeman Jenkins Server > > > <[email protected]> wrote: > > > > > > > > Build: https://jenkins.thetaphi.de/job/Lucene-main-Windows/10089/ > > > > Java: 64bit/jdk-15 -XX:+UseCompressedOops -XX:+UseG1GC > > > > > > > > 3 tests failed. > > > > FAILED: org.apache.lucene.demo.TestDemo.classMethod > > > > > > > > Error Message: > > > > java.io.IOException: Could not remove the following files (in the order > > > > of > attempts): > > > > C:\Users\jenkins\workspace\Lucene-main- > Windows\lucene\demo\build\tmp\tests- > tmp\lucene.demo.TestDemo_94AFFE7CDE43C29F-001\ContribDemoTest- > 001\knn-dict.bin: java.nio.file.AccessDeniedException: > C:\Users\jenkins\workspace\Lucene-main- > Windows\lucene\demo\build\tmp\tests- > tmp\lucene.demo.TestDemo_94AFFE7CDE43C29F-001\ContribDemoTest- > 001\knn-dict.bin > > > > C:\Users\jenkins\workspace\Lucene-main- > Windows\lucene\demo\build\tmp\tests- > tmp\lucene.demo.TestDemo_94AFFE7CDE43C29F-001\ContribDemoTest-001: > java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene- > main-Windows\lucene\demo\build\tmp\tests- > tmp\lucene.demo.TestDemo_94AFFE7CDE43C29F-001\ContribDemoTest-001 > > > > C:\Users\jenkins\workspace\Lucene-main- > Windows\lucene\demo\build\tmp\tests- > tmp\lucene.demo.TestDemo_94AFFE7CDE43C29F-001: > java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene- > main-Windows\lucene\demo\build\tmp\tests- > tmp\lucene.demo.TestDemo_94AFFE7CDE43C29F-001 > > > > > > > > > > > > Stack Trace: > > > > java.io.IOException: Could not remove the following files (in the order > > > > of > attempts): > > > > C:\Users\jenkins\workspace\Lucene-main- > Windows\lucene\demo\build\tmp\tests- > tmp\lucene.demo.TestDemo_94AFFE7CDE43C29F-001\ContribDemoTest- > 001\knn-dict.bin: java.nio.file.AccessDeniedException: > C:\Users\jenkins\workspace\Lucene-main- > Windows\lucene\demo\build\tmp\tests- > tmp\lucene.demo.TestDemo_94AFFE7CDE43C29F-001\ContribDemoTest- > 001\knn-dict.bin > > > > C:\Users\jenkins\workspace\Lucene-main- > Windows\lucene\demo\build\tmp\tests- > tmp\lucene.demo.TestDemo_94AFFE7CDE43C29F-001\ContribDemoTest-001: > java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene- > main-Windows\lucene\demo\build\tmp\tests- > tmp\lucene.demo.TestDemo_94AFFE7CDE43C29F-001\ContribDemoTest-001 > > > > C:\Users\jenkins\workspace\Lucene-main- > Windows\lucene\demo\build\tmp\tests- > tmp\lucene.demo.TestDemo_94AFFE7CDE43C29F-001: > java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene- > main-Windows\lucene\demo\build\tmp\tests- > tmp\lucene.demo.TestDemo_94AFFE7CDE43C29F-001 > > > > > > > > at __randomizedtesting.SeedInfo.seed([94AFFE7CDE43C29F]:0) > > > > at org.apache.lucene.util.IOUtils.rm(IOUtils.java:315) > > > > at > org.apache.lucene.util.TestRuleTemporaryFilesCleanup.afterAlways(TestRuleTe > mporaryFilesCleanup.java:207) > > > > at > com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterAlways(Test > RuleAdapter.java:31) > > > > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme > ntAdapter.java:43) > > > > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme > ntAdapter.java:36) > > > > at > org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertio > nsRequired.java:53) > > > > at > org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfter > Rule.java:43) > > > > at > org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.ja > va:44) > > > > at > org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgn > oreAfterMaxFailures.java:60) > > > > at > org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTest > Suites.java:47) > > > > at org.junit.rules.RunRules.evaluate(RunRules.java:20) > > > > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme > ntAdapter.java:36) > > > > at > com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run( > ThreadLeakControl.java:370) > > > > at > com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutin > gTask$0(ThreadLeakControl.java:826) > > > > at java.base/java.lang.Thread.run(Thread.java:832) > > > > > > > > > > > > FAILED: org.apache.lucene.demo.knn.TestDemoEmbeddings.classMethod > > > > > > > > Error Message: > > > > java.io.IOException: Could not remove the following files (in the order > > > > of > attempts): > > > > C:\Users\jenkins\workspace\Lucene-main- > Windows\lucene\demo\build\tmp\tests- > tmp\lucene.demo.knn.TestDemoEmbeddings_94AFFE7CDE43C29F-001\knn- > demo-001\dict.bin: java.nio.file.AccessDeniedException: > C:\Users\jenkins\workspace\Lucene-main- > Windows\lucene\demo\build\tmp\tests- > tmp\lucene.demo.knn.TestDemoEmbeddings_94AFFE7CDE43C29F-001\knn- > demo-001\dict.bin > > > > C:\Users\jenkins\workspace\Lucene-main- > Windows\lucene\demo\build\tmp\tests- > tmp\lucene.demo.knn.TestDemoEmbeddings_94AFFE7CDE43C29F-001\knn- > demo-001: java.nio.file.DirectoryNotEmptyException: > C:\Users\jenkins\workspace\Lucene-main- > Windows\lucene\demo\build\tmp\tests- > tmp\lucene.demo.knn.TestDemoEmbeddings_94AFFE7CDE43C29F-001\knn- > demo-001 > > > > C:\Users\jenkins\workspace\Lucene-main- > Windows\lucene\demo\build\tmp\tests- > tmp\lucene.demo.knn.TestDemoEmbeddings_94AFFE7CDE43C29F-001: > java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene- > main-Windows\lucene\demo\build\tmp\tests- > tmp\lucene.demo.knn.TestDemoEmbeddings_94AFFE7CDE43C29F-001 > > > > > > > > > > > > Stack Trace: > > > > java.io.IOException: Could not remove the following files (in the order > > > > of > attempts): > > > > C:\Users\jenkins\workspace\Lucene-main- > Windows\lucene\demo\build\tmp\tests- > tmp\lucene.demo.knn.TestDemoEmbeddings_94AFFE7CDE43C29F-001\knn- > demo-001\dict.bin: java.nio.file.AccessDeniedException: > C:\Users\jenkins\workspace\Lucene-main- > Windows\lucene\demo\build\tmp\tests- > tmp\lucene.demo.knn.TestDemoEmbeddings_94AFFE7CDE43C29F-001\knn- > demo-001\dict.bin > > > > C:\Users\jenkins\workspace\Lucene-main- > Windows\lucene\demo\build\tmp\tests- > tmp\lucene.demo.knn.TestDemoEmbeddings_94AFFE7CDE43C29F-001\knn- > demo-001: java.nio.file.DirectoryNotEmptyException: > C:\Users\jenkins\workspace\Lucene-main- > Windows\lucene\demo\build\tmp\tests- > tmp\lucene.demo.knn.TestDemoEmbeddings_94AFFE7CDE43C29F-001\knn- > demo-001 > > > > C:\Users\jenkins\workspace\Lucene-main- > Windows\lucene\demo\build\tmp\tests- > tmp\lucene.demo.knn.TestDemoEmbeddings_94AFFE7CDE43C29F-001: > java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene- > main-Windows\lucene\demo\build\tmp\tests- > tmp\lucene.demo.knn.TestDemoEmbeddings_94AFFE7CDE43C29F-001 > > > > > > > > at __randomizedtesting.SeedInfo.seed([94AFFE7CDE43C29F]:0) > > > > at org.apache.lucene.util.IOUtils.rm(IOUtils.java:315) > > > > at > org.apache.lucene.util.TestRuleTemporaryFilesCleanup.afterAlways(TestRuleTe > mporaryFilesCleanup.java:207) > > > > at > com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterAlways(Test > RuleAdapter.java:31) > > > > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme > ntAdapter.java:43) > > > > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme > ntAdapter.java:36) > > > > at > org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertio > nsRequired.java:53) > > > > at > org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfter > Rule.java:43) > > > > at > org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.ja > va:44) > > > > at > org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgn > oreAfterMaxFailures.java:60) > > > > at > org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTest > Suites.java:47) > > > > at org.junit.rules.RunRules.evaluate(RunRules.java:20) > > > > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme > ntAdapter.java:36) > > > > at > com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run( > ThreadLeakControl.java:370) > > > > at > com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutin > gTask$0(ThreadLeakControl.java:826) > > > > at java.base/java.lang.Thread.run(Thread.java:832) > > > > > > > > > > > > FAILED: org.apache.lucene.demo.knn.TestKnnVectorDict.classMethod > > > > > > > > Error Message: > > > > java.io.IOException: Could not remove the following files (in the order > > > > of > attempts): > > > > C:\Users\jenkins\workspace\Lucene-main- > Windows\lucene\demo\build\tmp\tests- > tmp\lucene.demo.knn.TestKnnVectorDict_94AFFE7CDE43C29F-001\knn-demo- > 001\dict.bin: java.nio.file.AccessDeniedException: > C:\Users\jenkins\workspace\Lucene-main- > Windows\lucene\demo\build\tmp\tests- > tmp\lucene.demo.knn.TestKnnVectorDict_94AFFE7CDE43C29F-001\knn-demo- > 001\dict.bin > > > > C:\Users\jenkins\workspace\Lucene-main- > Windows\lucene\demo\build\tmp\tests- > tmp\lucene.demo.knn.TestKnnVectorDict_94AFFE7CDE43C29F-001\knn-demo- > 001: java.nio.file.DirectoryNotEmptyException: > C:\Users\jenkins\workspace\Lucene-main- > Windows\lucene\demo\build\tmp\tests- > tmp\lucene.demo.knn.TestKnnVectorDict_94AFFE7CDE43C29F-001\knn-demo- > 001 > > > > C:\Users\jenkins\workspace\Lucene-main- > Windows\lucene\demo\build\tmp\tests- > tmp\lucene.demo.knn.TestKnnVectorDict_94AFFE7CDE43C29F-001: > java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene- > main-Windows\lucene\demo\build\tmp\tests- > tmp\lucene.demo.knn.TestKnnVectorDict_94AFFE7CDE43C29F-001 > > > > > > > > > > > > Stack Trace: > > > > java.io.IOException: Could not remove the following files (in the order > > > > of > attempts): > > > > C:\Users\jenkins\workspace\Lucene-main- > Windows\lucene\demo\build\tmp\tests- > tmp\lucene.demo.knn.TestKnnVectorDict_94AFFE7CDE43C29F-001\knn-demo- > 001\dict.bin: java.nio.file.AccessDeniedException: > C:\Users\jenkins\workspace\Lucene-main- > Windows\lucene\demo\build\tmp\tests- > tmp\lucene.demo.knn.TestKnnVectorDict_94AFFE7CDE43C29F-001\knn-demo- > 001\dict.bin > > > > C:\Users\jenkins\workspace\Lucene-main- > Windows\lucene\demo\build\tmp\tests- > tmp\lucene.demo.knn.TestKnnVectorDict_94AFFE7CDE43C29F-001\knn-demo- > 001: java.nio.file.DirectoryNotEmptyException: > C:\Users\jenkins\workspace\Lucene-main- > Windows\lucene\demo\build\tmp\tests- > tmp\lucene.demo.knn.TestKnnVectorDict_94AFFE7CDE43C29F-001\knn-demo- > 001 > > > > C:\Users\jenkins\workspace\Lucene-main- > Windows\lucene\demo\build\tmp\tests- > tmp\lucene.demo.knn.TestKnnVectorDict_94AFFE7CDE43C29F-001: > java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene- > main-Windows\lucene\demo\build\tmp\tests- > tmp\lucene.demo.knn.TestKnnVectorDict_94AFFE7CDE43C29F-001 > > > > > > > > at __randomizedtesting.SeedInfo.seed([94AFFE7CDE43C29F]:0) > > > > at org.apache.lucene.util.IOUtils.rm(IOUtils.java:315) > > > > at > org.apache.lucene.util.TestRuleTemporaryFilesCleanup.afterAlways(TestRuleTe > mporaryFilesCleanup.java:207) > > > > at > com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterAlways(Test > RuleAdapter.java:31) > > > > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme > ntAdapter.java:43) > > > > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme > ntAdapter.java:36) > > > > at > org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertio > nsRequired.java:53) > > > > at > org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfter > Rule.java:43) > > > > at > org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.ja > va:44) > > > > at > org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgn > oreAfterMaxFailures.java:60) > > > > at > org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTest > Suites.java:47) > > > > at org.junit.rules.RunRules.evaluate(RunRules.java:20) > > > > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme > ntAdapter.java:36) > > > > at > com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run( > ThreadLeakControl.java:370) > > > > at > com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutin > gTask$0(ThreadLeakControl.java:826) > > > > at java.base/java.lang.Thread.run(Thread.java:832) > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: [email protected] > > > > For additional commands, e-mail: [email protected] > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: [email protected] > > > For additional commands, e-mail: [email protected] > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
