Hi Uwe, I will report the bug to ES, as you suggested.
Do you recon using Mmap would have an effect to the index corruption when using SMB? I have to report back to my manager in few days to decide wether to carry on with ACIs or find another hosting solution. It is unfortunate there seems to be problems with this solution. Microsoft seems not interested in extending the volume mapping options for ACIs and K8 is overkilling for our use case. Thank you for your help so far, you have been very kind :) Cheers, Seb > On 2 Jan 2023, at 19:09, Uwe Schindler <u...@thetaphi.de> wrote: > > Hi, > > Please open a bug report at ES. The setting vm.max_map_count is not needed > and should not be changed unless really needed, because it uses kernel > resources. > > This has to do with their support (they try to tell people to overshard and > to prevent support requests they ask to raise this seeting). The default > value on Linux is 65530. This would allow you to memory map 65530 chunks of 1 > GiB (thats the limitation without preview enabled in the Java 19 JVM). A > shard in Lucene has about 70 files, most of them with filesizes < 1 Gib and a > few >1 GiB (but < 5 GiB) chunks (so lets assume 90 mappings needed for a > shard), so you could have approx 728 shards per node. Sorry raising this > setting is not needed! > > Uwe > > Am 02.01.2023 um 18:24 schrieb S S: >> I also tried enabling preview but no joy, same error :( >> >> It looks like it is not possible to start a multinode ES cluster without >> setting vm.max_map_count. I also googled it and this check cannot be >> disabled. >> >> I guess MMapDirectory is not an option for ES on ACIs, unless you have >> something else I can try? >> >> Many thanks, >> >> Seb >> >>> On 2 Jan 2023, at 17:55, S S <sebastiano1...@yahoo.it.INVALID> wrote: >>> >>> Thank you Uwe, this is great! I am rebuilding the cluster using >>> MMapDirectory and no enable-preview, as you suggested. Let’s see what >>> happens. >>> >>> Cheers, >>> >>> Seb >>> >>>> On 2 Jan 2023, at 17:51, Uwe Schindler <u...@thetaphi.de> wrote: >>>> >>>> Hi, >>>> >>>> in recent versions it works like that: >>>> >>>> https://www.elastic.co/guide/en/elasticsearch/reference/current/advanced-configuration.html#set-jvm-options >>>> >>>> So in folder jvm.options.d/ add a new file (like "preview.conf") and put >>>> "19:--enable-preview" into it. It is basically the same like modifying >>>> heap size. >>>> >>>> But in general, you can simply use MMapDirectory, the max-map-count >>>> setting is only relevant in *huge* (huge means hundreds of huge indexes >>>> per node). In that case Java 19's preview features would be recommended. >>>> >>>> Uwe >>>> >>>> Am 02.01.2023 um 17:41 schrieb S S: >>>>> Hi Uwe, >>>>> >>>>> Sorry for the late reply but upgrading the docker image to use OpenJDK >>>>> was easier said that done. >>>>> >>>>> I am not a Java developer/expert so, sorry for the stupid question but, >>>>> how do I specify the --enable-preview flag? ES has got a quite complex >>>>> way to start so I cannot specify the flag on the command line. You >>>>> suggested to use a jvm.properties file but I cannot find anything useful >>>>> about it when google-ing. Where should it be placed? And what should I >>>>> write in it? >>>>> >>>>> I can see ES recognising OpenJDK 19 while bootstrapping and suggesting to >>>>> enable preview, but it does not suggest how, and I cannot find anything >>>>> on the ES website. >>>>> >>>>> Many thanks. >>>>> >>>>> Seb >>>>> >>>>>> On 2 Jan 2023, at 11:48, Uwe Schindler <u...@thetaphi.de> wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> in general you can still use MMapDirectory. There is no requirement to >>>>>> set vm.max_map_count for smaller clusters. The information in Elastics >>>>>> documentation is not mandatory and misleading. >>>>>> >>>>>> If you use newest version of Elasticsearch with Java 19 and you use >>>>>> `--enable-preview` in you jvm.properties file, you don't even need to >>>>>> change that setting even with larger clusters. >>>>>> >>>>>> Uwe >>>>>> >>>>>> Am 02.01.2023 um 11:18 schrieb S S: >>>>>>> We are experimenting with Elastic Search deployed in Azure Container >>>>>>> Instances (Debian + OpenJDK). The ES indexes are stored into an Azure >>>>>>> file share mounted via SMB (3.0). The Elastic Search cluster is made up >>>>>>> of 4 nodes, each one have a separate file share to store the indices. >>>>>>> >>>>>>> This configuration has been influenced by some ACIs limitations, >>>>>>> specifically: >>>>>>> >>>>>>> we cannot set the max_map_count value as we do not have access to the >>>>>>> underlying host >>>>>>> (https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html). >>>>>>> Unfortunately, this is required to run an ES cluster, therefore we >>>>>>> were forced to use NIOF >>>>>>> ACI’s storage is ephemera, therefore we had to map volumes to persist >>>>>>> the indexes. ACIs only allow volume mappings using Azure File Shares, >>>>>>> which only works with NFS or SMB. >>>>>>> >>>>>>> We are experiencing recurring index corruption, specifically a "read >>>>>>> past EOF" exception. I asked on the Elastic Search forum but the answer >>>>>>> I got was a bit generic and not really helpful other than confirming >>>>>>> that, from ES point of view, ES should work on an SMB share as long as >>>>>>> it behaves as a local drive. As the underlying exception relates to an >>>>>>> issue with a Lucene index, I was wondering if you could help out? >>>>>>> Specifically, can Lucene work on SMB? I can only find sparse >>>>>>> information on this configuration and, while NFS seems a no-no, for SMB >>>>>>> is not that clear. Below is the exception we are getting. >>>>>>> >>>>>>> java.io.IOException: read past EOF: >>>>>>> NIOFSIndexInput(path="/bitnami/elasticsearch/data/indices/mS2bUbLtSeG0FSAMuKX7JQ/0/index/_ldsn_1.fnm") >>>>>>> buffer: java.nio.HeapByteBuffer[pos=0 lim=1024 cap=1024] chunkLen: >>>>>>> 1024 end: 2331: >>>>>>> NIOFSIndexInput(path="/bitnami/elasticsearch/data/indices/mS2bUbLtSeG0FSAMuKX7JQ/0/index/_ldsn_1.fnm") >>>>>>> at >>>>>>> org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:200) >>>>>>> ~[lucene-core-9.3.0.jar:?] >>>>>>> at >>>>>>> org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:291) >>>>>>> ~[lucene-core-9.3.0.jar:?] >>>>>>> at >>>>>>> org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:55) >>>>>>> ~[lucene-core-9.3.0.jar:?] >>>>>>> at >>>>>>> org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:39) >>>>>>> ~[lucene-core-9.3.0.jar:?] >>>>>>> at >>>>>>> org.apache.lucene.codecs.CodecUtil.readBEInt(CodecUtil.java:667) >>>>>>> ~[lucene-core-9.3.0.jar:?] >>>>>>> at >>>>>>> org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:184) >>>>>>> ~[lucene-core-9.3.0.jar:?] >>>>>>> at >>>>>>> org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:253) >>>>>>> ~[lucene-core-9.3.0.jar:?] >>>>>>> at >>>>>>> org.apache.lucene.codecs.lucene90.Lucene90FieldInfosFormat.read(Lucene90FieldInfosFormat.java:128) >>>>>>> ~[lucene-core-9.3.0.jar:?] >>>>>>> at >>>>>>> org.apache.lucene.index.SegmentReader.initFieldInfos(SegmentReader.java:205) >>>>>>> ~[lucene-core-9.3.0.jar:?] >>>>>>> at >>>>>>> org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:156) >>>>>>> ~[lucene-core-9.3.0.jar:?] >>>>>>> at >>>>>>> org.apache.lucene.index.ReadersAndUpdates.createNewReaderWithLatestLiveDocs(ReadersAndUpdates.java:738) >>>>>>> ~[lucene-core-9.3.0.jar:?] >>>>>>> at >>>>>>> org.apache.lucene.index.ReadersAndUpdates.swapNewReaderWithLatestLiveDocs(ReadersAndUpdates.java:754) >>>>>>> ~[lucene-core-9.3.0.jar:?] >>>>>>> at >>>>>>> org.apache.lucene.index.ReadersAndUpdates.writeFieldUpdates(ReadersAndUpdates.java:678) >>>>>>> ~[lucene-core-9.3.0.jar:?] >>>>>>> at >>>>>>> org.apache.lucene.index.ReaderPool.writeAllDocValuesUpdates(ReaderPool.java:251) >>>>>>> ~[lucene-core-9.3.0.jar:?] >>>>>>> at >>>>>>> org.apache.lucene.index.IndexWriter.writeReaderPool(IndexWriter.java:3743) >>>>>>> ~[lucene-core-9.3.0.jar:?] >>>>>>> at >>>>>>> org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:591) >>>>>>> ~[lucene-core-9.3.0.jar:?] >>>>>>> at >>>>>>> org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:381) >>>>>>> ~[lucene-core-9.3.0.jar:?] >>>>>>> at >>>>>>> org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:355) >>>>>>> ~[lucene-core-9.3.0.jar:?] >>>>>>> at >>>>>>> org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:345) >>>>>>> ~[lucene-core-9.3.0.jar:?] >>>>>>> at >>>>>>> org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112) >>>>>>> ~[lucene-core-9.3.0.jar:?] >>>>>>> at >>>>>>> org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170) >>>>>>> ~[lucene-core-9.3.0.jar:?] >>>>>>> at >>>>>>> org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:48) >>>>>>> ~[elasticsearch-8.4.1.jar:?] >>>>>>> at >>>>>>> org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:27) >>>>>>> ~[elasticsearch-8.4.1.jar:?] >>>>>>> at >>>>>>> org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) >>>>>>> ~[lucene-core-9.3.0.jar:?] >>>>>>> at >>>>>>> org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240) >>>>>>> ~[lucene-core-9.3.0.jar:?] >>>>>>> at >>>>>>> org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:355) >>>>>>> ~[elasticsearch-8.4.1.jar:?] >>>>>>> at >>>>>>> org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:335) >>>>>>> ~[elasticsearch-8.4.1.jar:?] >>>>>>> at >>>>>>> org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) >>>>>>> ~[lucene-core-9.3.0.jar:?] >>>>>>> Many thanks. >>>>>>> >>>>>>> Seb >>>>>> -- >>>>>> Uwe Schindler >>>>>> Achterdiek 19, D-28357 Bremen >>>>>> https://www.thetaphi.de >>>>>> eMail: u...@thetaphi.de >>>>>> >>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >>>>>> For additional commands, e-mail: java-user-h...@lucene.apache.org >>>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >>>>> For additional commands, e-mail: java-user-h...@lucene.apache.org >>>>> >>>> -- >>>> Uwe Schindler >>>> Achterdiek 19, D-28357 Bremen >>>> https://www.thetaphi.de >>>> eMail: u...@thetaphi.de >>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >>>> For additional commands, e-mail: java-user-h...@lucene.apache.org >>>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >>> For additional commands, e-mail: java-user-h...@lucene.apache.org >>> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >> For additional commands, e-mail: java-user-h...@lucene.apache.org >> > -- > Uwe Schindler > Achterdiek 19, D-28357 Bremen > https://www.thetaphi.de > eMail: u...@thetaphi.de > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
