Thanks Uwe. I've filed an issue and may even look into it before Mike wakes up. https://issues.apache.org/jira/browse/LUCENE-10057
Dawid On Thu, Aug 19, 2021 at 11:27 AM Uwe Schindler <[email protected]> wrote: > > Hi, > > I am glad that this looks like demo code only. I was afraid that you used it > inside codecs or in other public lucene classes! Standard Lucene code should > only use Directory abstractions to access files! > > To fix the demo issue: In KNNVectorDict demo application should use > FSDirectory.open() instead of lowlevel code with FileChannel. Then open the > file as IndexInput and call randomAccessSlice() on it > (<https://lucene.apache.org/core/8_9_0/core/org/apache/lucene/store/IndexInput.html#randomAccessSlice-long-long->). > Then you can read the values using RandomAccessInput#getXY: > https://lucene.apache.org/core/8_9_0/core/org/apache/lucene/store/RandomAccessInput.html > (which uses MMap behind scenes, if it is an MMapDirectory or on unsupported > platform like 32 bit it will use the other directory implementations). > > Uwe > > ----- > Uwe Schindler > Achterdiek 19, D-28357 Bremen > https://www.thetaphi.de > eMail: [email protected] > > > -----Original Message----- > > From: Uwe Schindler <[email protected]> > > Sent: Thursday, August 19, 2021 11:17 AM > > To: 'Dawid Weiss' <[email protected]> > > Subject: RE: [JENKINS] Lucene-main-Windows (64bit/jdk-15) - Build # 10089 - > > Unstable! > > > > Hi > > > > Why does KNNVectorDict not use IndexInput with a > > IndexInput#randomAccessSlice()? Then problem is handled by mmapdirectory. > > > > Lucene classes should never ever use direct access to file system and > > always go > > through the Directory. > > > > Uwe > > > > ----- > > Uwe Schindler > > Achterdiek 19, D-28357 Bremen > > https://www.thetaphi.de > > eMail: [email protected] > > > > > -----Original Message----- > > > From: Dawid Weiss <[email protected]> > > > Sent: Thursday, August 19, 2021 11:03 AM > > > To: Lucene Dev <[email protected]> > > > Cc: Uwe Schindler (SD DataSolutions GmbH) <[email protected]> > > > Subject: Re: [JENKINS] Lucene-main-Windows (64bit/jdk-15) - Build # 10089 > > > - > > > Unstable! > > > > > > This is actually more complicated. KnnVectorDict uses a mapped byte > > > buffer - such buffers lock the source file on Windows and closing the > > > original channel doesn't really affect their state: > > > > > > private final MappedByteBuffer vbuffer; > > > > > > It'd need to use the unmapper hack currently present in mmapdirectory > > > to release this buffer. I CC Uwe - perhaps he has an idea how to solve > > > it cleanly. At the moment tests on Windows will fail because the > > > temporary file/ directory can't be unmapped. I think it's a legitimate > > > bug since we do want those resources released eagerly rather than rely > > > on the JVM garbage collector to unmap them. > > > > > > Dawid > > > > > > On Wed, Aug 18, 2021 at 9:25 PM Michael Sokolov <[email protected]> > > > wrote: > > > > > > > > Ah I'll check, thanks for the pointer > > > > > > > > On Wed, Aug 18, 2021 at 12:19 PM Dawid Weiss > > <[email protected]> > > > wrote: > > > > > > > > > > Looks like something (file handle) is not properly closed - this > > > > > causes temp. file removal to fail on Windows. > > > > > > > > > > D. > > > > > > > > > > On Wed, Aug 18, 2021 at 5:52 PM Policeman Jenkins Server > > > > > <[email protected]> wrote: > > > > > > > > > > > > Build: https://jenkins.thetaphi.de/job/Lucene-main-Windows/10089/ > > > > > > Java: 64bit/jdk-15 -XX:+UseCompressedOops -XX:+UseG1GC > > > > > > > > > > > > 3 tests failed. > > > > > > FAILED: org.apache.lucene.demo.TestDemo.classMethod > > > > > > > > > > > > Error Message: > > > > > > java.io.IOException: Could not remove the following files (in the > > > > > > order of > > > attempts): > > > > > > C:\Users\jenkins\workspace\Lucene-main- > > > Windows\lucene\demo\build\tmp\tests- > > > tmp\lucene.demo.TestDemo_94AFFE7CDE43C29F-001\ContribDemoTest- > > > 001\knn-dict.bin: java.nio.file.AccessDeniedException: > > > C:\Users\jenkins\workspace\Lucene-main- > > > Windows\lucene\demo\build\tmp\tests- > > > tmp\lucene.demo.TestDemo_94AFFE7CDE43C29F-001\ContribDemoTest- > > > 001\knn-dict.bin > > > > > > C:\Users\jenkins\workspace\Lucene-main- > > > Windows\lucene\demo\build\tmp\tests- > > > tmp\lucene.demo.TestDemo_94AFFE7CDE43C29F-001\ContribDemoTest- > > 001: > > > java.nio.file.DirectoryNotEmptyException: > > C:\Users\jenkins\workspace\Lucene- > > > main-Windows\lucene\demo\build\tmp\tests- > > > tmp\lucene.demo.TestDemo_94AFFE7CDE43C29F-001\ContribDemoTest-001 > > > > > > C:\Users\jenkins\workspace\Lucene-main- > > > Windows\lucene\demo\build\tmp\tests- > > > tmp\lucene.demo.TestDemo_94AFFE7CDE43C29F-001: > > > java.nio.file.DirectoryNotEmptyException: > > C:\Users\jenkins\workspace\Lucene- > > > main-Windows\lucene\demo\build\tmp\tests- > > > tmp\lucene.demo.TestDemo_94AFFE7CDE43C29F-001 > > > > > > > > > > > > > > > > > > Stack Trace: > > > > > > java.io.IOException: Could not remove the following files (in the > > > > > > order of > > > attempts): > > > > > > C:\Users\jenkins\workspace\Lucene-main- > > > Windows\lucene\demo\build\tmp\tests- > > > tmp\lucene.demo.TestDemo_94AFFE7CDE43C29F-001\ContribDemoTest- > > > 001\knn-dict.bin: java.nio.file.AccessDeniedException: > > > C:\Users\jenkins\workspace\Lucene-main- > > > Windows\lucene\demo\build\tmp\tests- > > > tmp\lucene.demo.TestDemo_94AFFE7CDE43C29F-001\ContribDemoTest- > > > 001\knn-dict.bin > > > > > > C:\Users\jenkins\workspace\Lucene-main- > > > Windows\lucene\demo\build\tmp\tests- > > > tmp\lucene.demo.TestDemo_94AFFE7CDE43C29F-001\ContribDemoTest- > > 001: > > > java.nio.file.DirectoryNotEmptyException: > > C:\Users\jenkins\workspace\Lucene- > > > main-Windows\lucene\demo\build\tmp\tests- > > > tmp\lucene.demo.TestDemo_94AFFE7CDE43C29F-001\ContribDemoTest-001 > > > > > > C:\Users\jenkins\workspace\Lucene-main- > > > Windows\lucene\demo\build\tmp\tests- > > > tmp\lucene.demo.TestDemo_94AFFE7CDE43C29F-001: > > > java.nio.file.DirectoryNotEmptyException: > > C:\Users\jenkins\workspace\Lucene- > > > main-Windows\lucene\demo\build\tmp\tests- > > > tmp\lucene.demo.TestDemo_94AFFE7CDE43C29F-001 > > > > > > > > > > > > at __randomizedtesting.SeedInfo.seed([94AFFE7CDE43C29F]:0) > > > > > > at org.apache.lucene.util.IOUtils.rm(IOUtils.java:315) > > > > > > at > > > > > org.apache.lucene.util.TestRuleTemporaryFilesCleanup.afterAlways(TestRuleTe > > > mporaryFilesCleanup.java:207) > > > > > > at > > > > > com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterAlways(Test > > > RuleAdapter.java:31) > > > > > > at > > > > > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme > > > ntAdapter.java:43) > > > > > > at > > > > > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme > > > ntAdapter.java:36) > > > > > > at > > > > > org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertio > > > nsRequired.java:53) > > > > > > at > > > > > org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfter > > > Rule.java:43) > > > > > > at > > > > > org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.ja > > > va:44) > > > > > > at > > > > > org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgn > > > oreAfterMaxFailures.java:60) > > > > > > at > > > > > org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTest > > > Suites.java:47) > > > > > > at org.junit.rules.RunRules.evaluate(RunRules.java:20) > > > > > > at > > > > > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme > > > ntAdapter.java:36) > > > > > > at > > > > > com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run( > > > ThreadLeakControl.java:370) > > > > > > at > > > > > com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutin > > > gTask$0(ThreadLeakControl.java:826) > > > > > > at java.base/java.lang.Thread.run(Thread.java:832) > > > > > > > > > > > > > > > > > > FAILED: > > org.apache.lucene.demo.knn.TestDemoEmbeddings.classMethod > > > > > > > > > > > > Error Message: > > > > > > java.io.IOException: Could not remove the following files (in the > > > > > > order of > > > attempts): > > > > > > C:\Users\jenkins\workspace\Lucene-main- > > > Windows\lucene\demo\build\tmp\tests- > > > tmp\lucene.demo.knn.TestDemoEmbeddings_94AFFE7CDE43C29F-001\knn- > > > demo-001\dict.bin: java.nio.file.AccessDeniedException: > > > C:\Users\jenkins\workspace\Lucene-main- > > > Windows\lucene\demo\build\tmp\tests- > > > tmp\lucene.demo.knn.TestDemoEmbeddings_94AFFE7CDE43C29F-001\knn- > > > demo-001\dict.bin > > > > > > C:\Users\jenkins\workspace\Lucene-main- > > > Windows\lucene\demo\build\tmp\tests- > > > tmp\lucene.demo.knn.TestDemoEmbeddings_94AFFE7CDE43C29F-001\knn- > > > demo-001: java.nio.file.DirectoryNotEmptyException: > > > C:\Users\jenkins\workspace\Lucene-main- > > > Windows\lucene\demo\build\tmp\tests- > > > tmp\lucene.demo.knn.TestDemoEmbeddings_94AFFE7CDE43C29F-001\knn- > > > demo-001 > > > > > > C:\Users\jenkins\workspace\Lucene-main- > > > Windows\lucene\demo\build\tmp\tests- > > > tmp\lucene.demo.knn.TestDemoEmbeddings_94AFFE7CDE43C29F-001: > > > java.nio.file.DirectoryNotEmptyException: > > C:\Users\jenkins\workspace\Lucene- > > > main-Windows\lucene\demo\build\tmp\tests- > > > tmp\lucene.demo.knn.TestDemoEmbeddings_94AFFE7CDE43C29F-001 > > > > > > > > > > > > > > > > > > Stack Trace: > > > > > > java.io.IOException: Could not remove the following files (in the > > > > > > order of > > > attempts): > > > > > > C:\Users\jenkins\workspace\Lucene-main- > > > Windows\lucene\demo\build\tmp\tests- > > > tmp\lucene.demo.knn.TestDemoEmbeddings_94AFFE7CDE43C29F-001\knn- > > > demo-001\dict.bin: java.nio.file.AccessDeniedException: > > > C:\Users\jenkins\workspace\Lucene-main- > > > Windows\lucene\demo\build\tmp\tests- > > > tmp\lucene.demo.knn.TestDemoEmbeddings_94AFFE7CDE43C29F-001\knn- > > > demo-001\dict.bin > > > > > > C:\Users\jenkins\workspace\Lucene-main- > > > Windows\lucene\demo\build\tmp\tests- > > > tmp\lucene.demo.knn.TestDemoEmbeddings_94AFFE7CDE43C29F-001\knn- > > > demo-001: java.nio.file.DirectoryNotEmptyException: > > > C:\Users\jenkins\workspace\Lucene-main- > > > Windows\lucene\demo\build\tmp\tests- > > > tmp\lucene.demo.knn.TestDemoEmbeddings_94AFFE7CDE43C29F-001\knn- > > > demo-001 > > > > > > C:\Users\jenkins\workspace\Lucene-main- > > > Windows\lucene\demo\build\tmp\tests- > > > tmp\lucene.demo.knn.TestDemoEmbeddings_94AFFE7CDE43C29F-001: > > > java.nio.file.DirectoryNotEmptyException: > > C:\Users\jenkins\workspace\Lucene- > > > main-Windows\lucene\demo\build\tmp\tests- > > > tmp\lucene.demo.knn.TestDemoEmbeddings_94AFFE7CDE43C29F-001 > > > > > > > > > > > > at __randomizedtesting.SeedInfo.seed([94AFFE7CDE43C29F]:0) > > > > > > at org.apache.lucene.util.IOUtils.rm(IOUtils.java:315) > > > > > > at > > > > > org.apache.lucene.util.TestRuleTemporaryFilesCleanup.afterAlways(TestRuleTe > > > mporaryFilesCleanup.java:207) > > > > > > at > > > > > com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterAlways(Test > > > RuleAdapter.java:31) > > > > > > at > > > > > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme > > > ntAdapter.java:43) > > > > > > at > > > > > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme > > > ntAdapter.java:36) > > > > > > at > > > > > org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertio > > > nsRequired.java:53) > > > > > > at > > > > > org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfter > > > Rule.java:43) > > > > > > at > > > > > org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.ja > > > va:44) > > > > > > at > > > > > org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgn > > > oreAfterMaxFailures.java:60) > > > > > > at > > > > > org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTest > > > Suites.java:47) > > > > > > at org.junit.rules.RunRules.evaluate(RunRules.java:20) > > > > > > at > > > > > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme > > > ntAdapter.java:36) > > > > > > at > > > > > com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run( > > > ThreadLeakControl.java:370) > > > > > > at > > > > > com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutin > > > gTask$0(ThreadLeakControl.java:826) > > > > > > at java.base/java.lang.Thread.run(Thread.java:832) > > > > > > > > > > > > > > > > > > FAILED: org.apache.lucene.demo.knn.TestKnnVectorDict.classMethod > > > > > > > > > > > > Error Message: > > > > > > java.io.IOException: Could not remove the following files (in the > > > > > > order of > > > attempts): > > > > > > C:\Users\jenkins\workspace\Lucene-main- > > > Windows\lucene\demo\build\tmp\tests- > > > tmp\lucene.demo.knn.TestKnnVectorDict_94AFFE7CDE43C29F-001\knn- > > demo- > > > 001\dict.bin: java.nio.file.AccessDeniedException: > > > C:\Users\jenkins\workspace\Lucene-main- > > > Windows\lucene\demo\build\tmp\tests- > > > tmp\lucene.demo.knn.TestKnnVectorDict_94AFFE7CDE43C29F-001\knn- > > demo- > > > 001\dict.bin > > > > > > C:\Users\jenkins\workspace\Lucene-main- > > > Windows\lucene\demo\build\tmp\tests- > > > tmp\lucene.demo.knn.TestKnnVectorDict_94AFFE7CDE43C29F-001\knn- > > demo- > > > 001: java.nio.file.DirectoryNotEmptyException: > > > C:\Users\jenkins\workspace\Lucene-main- > > > Windows\lucene\demo\build\tmp\tests- > > > tmp\lucene.demo.knn.TestKnnVectorDict_94AFFE7CDE43C29F-001\knn- > > demo- > > > 001 > > > > > > C:\Users\jenkins\workspace\Lucene-main- > > > Windows\lucene\demo\build\tmp\tests- > > > tmp\lucene.demo.knn.TestKnnVectorDict_94AFFE7CDE43C29F-001: > > > java.nio.file.DirectoryNotEmptyException: > > C:\Users\jenkins\workspace\Lucene- > > > main-Windows\lucene\demo\build\tmp\tests- > > > tmp\lucene.demo.knn.TestKnnVectorDict_94AFFE7CDE43C29F-001 > > > > > > > > > > > > > > > > > > Stack Trace: > > > > > > java.io.IOException: Could not remove the following files (in the > > > > > > order of > > > attempts): > > > > > > C:\Users\jenkins\workspace\Lucene-main- > > > Windows\lucene\demo\build\tmp\tests- > > > tmp\lucene.demo.knn.TestKnnVectorDict_94AFFE7CDE43C29F-001\knn- > > demo- > > > 001\dict.bin: java.nio.file.AccessDeniedException: > > > C:\Users\jenkins\workspace\Lucene-main- > > > Windows\lucene\demo\build\tmp\tests- > > > tmp\lucene.demo.knn.TestKnnVectorDict_94AFFE7CDE43C29F-001\knn- > > demo- > > > 001\dict.bin > > > > > > C:\Users\jenkins\workspace\Lucene-main- > > > Windows\lucene\demo\build\tmp\tests- > > > tmp\lucene.demo.knn.TestKnnVectorDict_94AFFE7CDE43C29F-001\knn- > > demo- > > > 001: java.nio.file.DirectoryNotEmptyException: > > > C:\Users\jenkins\workspace\Lucene-main- > > > Windows\lucene\demo\build\tmp\tests- > > > tmp\lucene.demo.knn.TestKnnVectorDict_94AFFE7CDE43C29F-001\knn- > > demo- > > > 001 > > > > > > C:\Users\jenkins\workspace\Lucene-main- > > > Windows\lucene\demo\build\tmp\tests- > > > tmp\lucene.demo.knn.TestKnnVectorDict_94AFFE7CDE43C29F-001: > > > java.nio.file.DirectoryNotEmptyException: > > C:\Users\jenkins\workspace\Lucene- > > > main-Windows\lucene\demo\build\tmp\tests- > > > tmp\lucene.demo.knn.TestKnnVectorDict_94AFFE7CDE43C29F-001 > > > > > > > > > > > > at __randomizedtesting.SeedInfo.seed([94AFFE7CDE43C29F]:0) > > > > > > at org.apache.lucene.util.IOUtils.rm(IOUtils.java:315) > > > > > > at > > > > > org.apache.lucene.util.TestRuleTemporaryFilesCleanup.afterAlways(TestRuleTe > > > mporaryFilesCleanup.java:207) > > > > > > at > > > > > com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterAlways(Test > > > RuleAdapter.java:31) > > > > > > at > > > > > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme > > > ntAdapter.java:43) > > > > > > at > > > > > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme > > > ntAdapter.java:36) > > > > > > at > > > > > org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertio > > > nsRequired.java:53) > > > > > > at > > > > > org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfter > > > Rule.java:43) > > > > > > at > > > > > org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.ja > > > va:44) > > > > > > at > > > > > org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgn > > > oreAfterMaxFailures.java:60) > > > > > > at > > > > > org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTest > > > Suites.java:47) > > > > > > at org.junit.rules.RunRules.evaluate(RunRules.java:20) > > > > > > at > > > > > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme > > > ntAdapter.java:36) > > > > > > at > > > > > com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run( > > > ThreadLeakControl.java:370) > > > > > > at > > > > > com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutin > > > gTask$0(ThreadLeakControl.java:826) > > > > > > at java.base/java.lang.Thread.run(Thread.java:832) > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > To unsubscribe, e-mail: [email protected] > > > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > --------------------------------------------------------------------- > > > > > To unsubscribe, e-mail: [email protected] > > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: [email protected] > > > > For additional commands, e-mail: [email protected] > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
