Hi, I am glad that this looks like demo code only. I was afraid that you used it inside codecs or in other public lucene classes! Standard Lucene code should only use Directory abstractions to access files!
To fix the demo issue: In KNNVectorDict demo application should use FSDirectory.open() instead of lowlevel code with FileChannel. Then open the file as IndexInput and call randomAccessSlice() on it (<https://lucene.apache.org/core/8_9_0/core/org/apache/lucene/store/IndexInput.html#randomAccessSlice-long-long->). Then you can read the values using RandomAccessInput#getXY: https://lucene.apache.org/core/8_9_0/core/org/apache/lucene/store/RandomAccessInput.html (which uses MMap behind scenes, if it is an MMapDirectory or on unsupported platform like 32 bit it will use the other directory implementations). Uwe ----- Uwe Schindler Achterdiek 19, D-28357 Bremen https://www.thetaphi.de eMail: [email protected] > -----Original Message----- > From: Uwe Schindler <[email protected]> > Sent: Thursday, August 19, 2021 11:17 AM > To: 'Dawid Weiss' <[email protected]> > Subject: RE: [JENKINS] Lucene-main-Windows (64bit/jdk-15) - Build # 10089 - > Unstable! > > Hi > > Why does KNNVectorDict not use IndexInput with a > IndexInput#randomAccessSlice()? Then problem is handled by mmapdirectory. > > Lucene classes should never ever use direct access to file system and always > go > through the Directory. > > Uwe > > ----- > Uwe Schindler > Achterdiek 19, D-28357 Bremen > https://www.thetaphi.de > eMail: [email protected] > > > -----Original Message----- > > From: Dawid Weiss <[email protected]> > > Sent: Thursday, August 19, 2021 11:03 AM > > To: Lucene Dev <[email protected]> > > Cc: Uwe Schindler (SD DataSolutions GmbH) <[email protected]> > > Subject: Re: [JENKINS] Lucene-main-Windows (64bit/jdk-15) - Build # 10089 - > > Unstable! > > > > This is actually more complicated. KnnVectorDict uses a mapped byte > > buffer - such buffers lock the source file on Windows and closing the > > original channel doesn't really affect their state: > > > > private final MappedByteBuffer vbuffer; > > > > It'd need to use the unmapper hack currently present in mmapdirectory > > to release this buffer. I CC Uwe - perhaps he has an idea how to solve > > it cleanly. At the moment tests on Windows will fail because the > > temporary file/ directory can't be unmapped. I think it's a legitimate > > bug since we do want those resources released eagerly rather than rely > > on the JVM garbage collector to unmap them. > > > > Dawid > > > > On Wed, Aug 18, 2021 at 9:25 PM Michael Sokolov <[email protected]> > > wrote: > > > > > > Ah I'll check, thanks for the pointer > > > > > > On Wed, Aug 18, 2021 at 12:19 PM Dawid Weiss > <[email protected]> > > wrote: > > > > > > > > Looks like something (file handle) is not properly closed - this > > > > causes temp. file removal to fail on Windows. > > > > > > > > D. > > > > > > > > On Wed, Aug 18, 2021 at 5:52 PM Policeman Jenkins Server > > > > <[email protected]> wrote: > > > > > > > > > > Build: https://jenkins.thetaphi.de/job/Lucene-main-Windows/10089/ > > > > > Java: 64bit/jdk-15 -XX:+UseCompressedOops -XX:+UseG1GC > > > > > > > > > > 3 tests failed. > > > > > FAILED: org.apache.lucene.demo.TestDemo.classMethod > > > > > > > > > > Error Message: > > > > > java.io.IOException: Could not remove the following files (in the > > > > > order of > > attempts): > > > > > C:\Users\jenkins\workspace\Lucene-main- > > Windows\lucene\demo\build\tmp\tests- > > tmp\lucene.demo.TestDemo_94AFFE7CDE43C29F-001\ContribDemoTest- > > 001\knn-dict.bin: java.nio.file.AccessDeniedException: > > C:\Users\jenkins\workspace\Lucene-main- > > Windows\lucene\demo\build\tmp\tests- > > tmp\lucene.demo.TestDemo_94AFFE7CDE43C29F-001\ContribDemoTest- > > 001\knn-dict.bin > > > > > C:\Users\jenkins\workspace\Lucene-main- > > Windows\lucene\demo\build\tmp\tests- > > tmp\lucene.demo.TestDemo_94AFFE7CDE43C29F-001\ContribDemoTest- > 001: > > java.nio.file.DirectoryNotEmptyException: > C:\Users\jenkins\workspace\Lucene- > > main-Windows\lucene\demo\build\tmp\tests- > > tmp\lucene.demo.TestDemo_94AFFE7CDE43C29F-001\ContribDemoTest-001 > > > > > C:\Users\jenkins\workspace\Lucene-main- > > Windows\lucene\demo\build\tmp\tests- > > tmp\lucene.demo.TestDemo_94AFFE7CDE43C29F-001: > > java.nio.file.DirectoryNotEmptyException: > C:\Users\jenkins\workspace\Lucene- > > main-Windows\lucene\demo\build\tmp\tests- > > tmp\lucene.demo.TestDemo_94AFFE7CDE43C29F-001 > > > > > > > > > > > > > > > Stack Trace: > > > > > java.io.IOException: Could not remove the following files (in the > > > > > order of > > attempts): > > > > > C:\Users\jenkins\workspace\Lucene-main- > > Windows\lucene\demo\build\tmp\tests- > > tmp\lucene.demo.TestDemo_94AFFE7CDE43C29F-001\ContribDemoTest- > > 001\knn-dict.bin: java.nio.file.AccessDeniedException: > > C:\Users\jenkins\workspace\Lucene-main- > > Windows\lucene\demo\build\tmp\tests- > > tmp\lucene.demo.TestDemo_94AFFE7CDE43C29F-001\ContribDemoTest- > > 001\knn-dict.bin > > > > > C:\Users\jenkins\workspace\Lucene-main- > > Windows\lucene\demo\build\tmp\tests- > > tmp\lucene.demo.TestDemo_94AFFE7CDE43C29F-001\ContribDemoTest- > 001: > > java.nio.file.DirectoryNotEmptyException: > C:\Users\jenkins\workspace\Lucene- > > main-Windows\lucene\demo\build\tmp\tests- > > tmp\lucene.demo.TestDemo_94AFFE7CDE43C29F-001\ContribDemoTest-001 > > > > > C:\Users\jenkins\workspace\Lucene-main- > > Windows\lucene\demo\build\tmp\tests- > > tmp\lucene.demo.TestDemo_94AFFE7CDE43C29F-001: > > java.nio.file.DirectoryNotEmptyException: > C:\Users\jenkins\workspace\Lucene- > > main-Windows\lucene\demo\build\tmp\tests- > > tmp\lucene.demo.TestDemo_94AFFE7CDE43C29F-001 > > > > > > > > > > at __randomizedtesting.SeedInfo.seed([94AFFE7CDE43C29F]:0) > > > > > at org.apache.lucene.util.IOUtils.rm(IOUtils.java:315) > > > > > at > > > org.apache.lucene.util.TestRuleTemporaryFilesCleanup.afterAlways(TestRuleTe > > mporaryFilesCleanup.java:207) > > > > > at > > > com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterAlways(Test > > RuleAdapter.java:31) > > > > > at > > > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme > > ntAdapter.java:43) > > > > > at > > > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme > > ntAdapter.java:36) > > > > > at > > > org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertio > > nsRequired.java:53) > > > > > at > > > org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfter > > Rule.java:43) > > > > > at > > > org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.ja > > va:44) > > > > > at > > > org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgn > > oreAfterMaxFailures.java:60) > > > > > at > > > org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTest > > Suites.java:47) > > > > > at org.junit.rules.RunRules.evaluate(RunRules.java:20) > > > > > at > > > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme > > ntAdapter.java:36) > > > > > at > > > com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run( > > ThreadLeakControl.java:370) > > > > > at > > > com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutin > > gTask$0(ThreadLeakControl.java:826) > > > > > at java.base/java.lang.Thread.run(Thread.java:832) > > > > > > > > > > > > > > > FAILED: > org.apache.lucene.demo.knn.TestDemoEmbeddings.classMethod > > > > > > > > > > Error Message: > > > > > java.io.IOException: Could not remove the following files (in the > > > > > order of > > attempts): > > > > > C:\Users\jenkins\workspace\Lucene-main- > > Windows\lucene\demo\build\tmp\tests- > > tmp\lucene.demo.knn.TestDemoEmbeddings_94AFFE7CDE43C29F-001\knn- > > demo-001\dict.bin: java.nio.file.AccessDeniedException: > > C:\Users\jenkins\workspace\Lucene-main- > > Windows\lucene\demo\build\tmp\tests- > > tmp\lucene.demo.knn.TestDemoEmbeddings_94AFFE7CDE43C29F-001\knn- > > demo-001\dict.bin > > > > > C:\Users\jenkins\workspace\Lucene-main- > > Windows\lucene\demo\build\tmp\tests- > > tmp\lucene.demo.knn.TestDemoEmbeddings_94AFFE7CDE43C29F-001\knn- > > demo-001: java.nio.file.DirectoryNotEmptyException: > > C:\Users\jenkins\workspace\Lucene-main- > > Windows\lucene\demo\build\tmp\tests- > > tmp\lucene.demo.knn.TestDemoEmbeddings_94AFFE7CDE43C29F-001\knn- > > demo-001 > > > > > C:\Users\jenkins\workspace\Lucene-main- > > Windows\lucene\demo\build\tmp\tests- > > tmp\lucene.demo.knn.TestDemoEmbeddings_94AFFE7CDE43C29F-001: > > java.nio.file.DirectoryNotEmptyException: > C:\Users\jenkins\workspace\Lucene- > > main-Windows\lucene\demo\build\tmp\tests- > > tmp\lucene.demo.knn.TestDemoEmbeddings_94AFFE7CDE43C29F-001 > > > > > > > > > > > > > > > Stack Trace: > > > > > java.io.IOException: Could not remove the following files (in the > > > > > order of > > attempts): > > > > > C:\Users\jenkins\workspace\Lucene-main- > > Windows\lucene\demo\build\tmp\tests- > > tmp\lucene.demo.knn.TestDemoEmbeddings_94AFFE7CDE43C29F-001\knn- > > demo-001\dict.bin: java.nio.file.AccessDeniedException: > > C:\Users\jenkins\workspace\Lucene-main- > > Windows\lucene\demo\build\tmp\tests- > > tmp\lucene.demo.knn.TestDemoEmbeddings_94AFFE7CDE43C29F-001\knn- > > demo-001\dict.bin > > > > > C:\Users\jenkins\workspace\Lucene-main- > > Windows\lucene\demo\build\tmp\tests- > > tmp\lucene.demo.knn.TestDemoEmbeddings_94AFFE7CDE43C29F-001\knn- > > demo-001: java.nio.file.DirectoryNotEmptyException: > > C:\Users\jenkins\workspace\Lucene-main- > > Windows\lucene\demo\build\tmp\tests- > > tmp\lucene.demo.knn.TestDemoEmbeddings_94AFFE7CDE43C29F-001\knn- > > demo-001 > > > > > C:\Users\jenkins\workspace\Lucene-main- > > Windows\lucene\demo\build\tmp\tests- > > tmp\lucene.demo.knn.TestDemoEmbeddings_94AFFE7CDE43C29F-001: > > java.nio.file.DirectoryNotEmptyException: > C:\Users\jenkins\workspace\Lucene- > > main-Windows\lucene\demo\build\tmp\tests- > > tmp\lucene.demo.knn.TestDemoEmbeddings_94AFFE7CDE43C29F-001 > > > > > > > > > > at __randomizedtesting.SeedInfo.seed([94AFFE7CDE43C29F]:0) > > > > > at org.apache.lucene.util.IOUtils.rm(IOUtils.java:315) > > > > > at > > > org.apache.lucene.util.TestRuleTemporaryFilesCleanup.afterAlways(TestRuleTe > > mporaryFilesCleanup.java:207) > > > > > at > > > com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterAlways(Test > > RuleAdapter.java:31) > > > > > at > > > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme > > ntAdapter.java:43) > > > > > at > > > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme > > ntAdapter.java:36) > > > > > at > > > org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertio > > nsRequired.java:53) > > > > > at > > > org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfter > > Rule.java:43) > > > > > at > > > org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.ja > > va:44) > > > > > at > > > org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgn > > oreAfterMaxFailures.java:60) > > > > > at > > > org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTest > > Suites.java:47) > > > > > at org.junit.rules.RunRules.evaluate(RunRules.java:20) > > > > > at > > > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme > > ntAdapter.java:36) > > > > > at > > > com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run( > > ThreadLeakControl.java:370) > > > > > at > > > com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutin > > gTask$0(ThreadLeakControl.java:826) > > > > > at java.base/java.lang.Thread.run(Thread.java:832) > > > > > > > > > > > > > > > FAILED: org.apache.lucene.demo.knn.TestKnnVectorDict.classMethod > > > > > > > > > > Error Message: > > > > > java.io.IOException: Could not remove the following files (in the > > > > > order of > > attempts): > > > > > C:\Users\jenkins\workspace\Lucene-main- > > Windows\lucene\demo\build\tmp\tests- > > tmp\lucene.demo.knn.TestKnnVectorDict_94AFFE7CDE43C29F-001\knn- > demo- > > 001\dict.bin: java.nio.file.AccessDeniedException: > > C:\Users\jenkins\workspace\Lucene-main- > > Windows\lucene\demo\build\tmp\tests- > > tmp\lucene.demo.knn.TestKnnVectorDict_94AFFE7CDE43C29F-001\knn- > demo- > > 001\dict.bin > > > > > C:\Users\jenkins\workspace\Lucene-main- > > Windows\lucene\demo\build\tmp\tests- > > tmp\lucene.demo.knn.TestKnnVectorDict_94AFFE7CDE43C29F-001\knn- > demo- > > 001: java.nio.file.DirectoryNotEmptyException: > > C:\Users\jenkins\workspace\Lucene-main- > > Windows\lucene\demo\build\tmp\tests- > > tmp\lucene.demo.knn.TestKnnVectorDict_94AFFE7CDE43C29F-001\knn- > demo- > > 001 > > > > > C:\Users\jenkins\workspace\Lucene-main- > > Windows\lucene\demo\build\tmp\tests- > > tmp\lucene.demo.knn.TestKnnVectorDict_94AFFE7CDE43C29F-001: > > java.nio.file.DirectoryNotEmptyException: > C:\Users\jenkins\workspace\Lucene- > > main-Windows\lucene\demo\build\tmp\tests- > > tmp\lucene.demo.knn.TestKnnVectorDict_94AFFE7CDE43C29F-001 > > > > > > > > > > > > > > > Stack Trace: > > > > > java.io.IOException: Could not remove the following files (in the > > > > > order of > > attempts): > > > > > C:\Users\jenkins\workspace\Lucene-main- > > Windows\lucene\demo\build\tmp\tests- > > tmp\lucene.demo.knn.TestKnnVectorDict_94AFFE7CDE43C29F-001\knn- > demo- > > 001\dict.bin: java.nio.file.AccessDeniedException: > > C:\Users\jenkins\workspace\Lucene-main- > > Windows\lucene\demo\build\tmp\tests- > > tmp\lucene.demo.knn.TestKnnVectorDict_94AFFE7CDE43C29F-001\knn- > demo- > > 001\dict.bin > > > > > C:\Users\jenkins\workspace\Lucene-main- > > Windows\lucene\demo\build\tmp\tests- > > tmp\lucene.demo.knn.TestKnnVectorDict_94AFFE7CDE43C29F-001\knn- > demo- > > 001: java.nio.file.DirectoryNotEmptyException: > > C:\Users\jenkins\workspace\Lucene-main- > > Windows\lucene\demo\build\tmp\tests- > > tmp\lucene.demo.knn.TestKnnVectorDict_94AFFE7CDE43C29F-001\knn- > demo- > > 001 > > > > > C:\Users\jenkins\workspace\Lucene-main- > > Windows\lucene\demo\build\tmp\tests- > > tmp\lucene.demo.knn.TestKnnVectorDict_94AFFE7CDE43C29F-001: > > java.nio.file.DirectoryNotEmptyException: > C:\Users\jenkins\workspace\Lucene- > > main-Windows\lucene\demo\build\tmp\tests- > > tmp\lucene.demo.knn.TestKnnVectorDict_94AFFE7CDE43C29F-001 > > > > > > > > > > at __randomizedtesting.SeedInfo.seed([94AFFE7CDE43C29F]:0) > > > > > at org.apache.lucene.util.IOUtils.rm(IOUtils.java:315) > > > > > at > > > org.apache.lucene.util.TestRuleTemporaryFilesCleanup.afterAlways(TestRuleTe > > mporaryFilesCleanup.java:207) > > > > > at > > > com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterAlways(Test > > RuleAdapter.java:31) > > > > > at > > > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme > > ntAdapter.java:43) > > > > > at > > > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme > > ntAdapter.java:36) > > > > > at > > > org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertio > > nsRequired.java:53) > > > > > at > > > org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfter > > Rule.java:43) > > > > > at > > > org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.ja > > va:44) > > > > > at > > > org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgn > > oreAfterMaxFailures.java:60) > > > > > at > > > org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTest > > Suites.java:47) > > > > > at org.junit.rules.RunRules.evaluate(RunRules.java:20) > > > > > at > > > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme > > ntAdapter.java:36) > > > > > at > > > com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run( > > ThreadLeakControl.java:370) > > > > > at > > > com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutin > > gTask$0(ThreadLeakControl.java:826) > > > > > at java.base/java.lang.Thread.run(Thread.java:832) > > > > > > > > > > --------------------------------------------------------------------- > > > > > To unsubscribe, e-mail: [email protected] > > > > > For additional commands, e-mail: [email protected] > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: [email protected] > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: [email protected] > > > For additional commands, e-mail: [email protected] > > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
