I also tried enabling preview but no joy, same error :( It looks like it is not possible to start a multinode ES cluster without setting vm.max_map_count. I also googled it and this check cannot be disabled.
I guess MMapDirectory is not an option for ES on ACIs, unless you have something else I can try? Many thanks, Seb > On 2 Jan 2023, at 17:55, S S <sebastiano1...@yahoo.it.INVALID> wrote: > > Thank you Uwe, this is great! I am rebuilding the cluster using MMapDirectory > and no enable-preview, as you suggested. Let’s see what happens. > > Cheers, > > Seb > >> On 2 Jan 2023, at 17:51, Uwe Schindler <u...@thetaphi.de> wrote: >> >> Hi, >> >> in recent versions it works like that: >> >> https://www.elastic.co/guide/en/elasticsearch/reference/current/advanced-configuration.html#set-jvm-options >> >> So in folder jvm.options.d/ add a new file (like "preview.conf") and put >> "19:--enable-preview" into it. It is basically the same like modifying heap >> size. >> >> But in general, you can simply use MMapDirectory, the max-map-count setting >> is only relevant in *huge* (huge means hundreds of huge indexes per node). >> In that case Java 19's preview features would be recommended. >> >> Uwe >> >> Am 02.01.2023 um 17:41 schrieb S S: >>> Hi Uwe, >>> >>> Sorry for the late reply but upgrading the docker image to use OpenJDK was >>> easier said that done. >>> >>> I am not a Java developer/expert so, sorry for the stupid question but, how >>> do I specify the --enable-preview flag? ES has got a quite complex way to >>> start so I cannot specify the flag on the command line. You suggested to >>> use a jvm.properties file but I cannot find anything useful about it when >>> google-ing. Where should it be placed? And what should I write in it? >>> >>> I can see ES recognising OpenJDK 19 while bootstrapping and suggesting to >>> enable preview, but it does not suggest how, and I cannot find anything on >>> the ES website. >>> >>> Many thanks. >>> >>> Seb >>> >>>> On 2 Jan 2023, at 11:48, Uwe Schindler <u...@thetaphi.de> wrote: >>>> >>>> Hi, >>>> >>>> in general you can still use MMapDirectory. There is no requirement to set >>>> vm.max_map_count for smaller clusters. The information in Elastics >>>> documentation is not mandatory and misleading. >>>> >>>> If you use newest version of Elasticsearch with Java 19 and you use >>>> `--enable-preview` in you jvm.properties file, you don't even need to >>>> change that setting even with larger clusters. >>>> >>>> Uwe >>>> >>>> Am 02.01.2023 um 11:18 schrieb S S: >>>>> We are experimenting with Elastic Search deployed in Azure Container >>>>> Instances (Debian + OpenJDK). The ES indexes are stored into an Azure >>>>> file share mounted via SMB (3.0). The Elastic Search cluster is made up >>>>> of 4 nodes, each one have a separate file share to store the indices. >>>>> >>>>> This configuration has been influenced by some ACIs limitations, >>>>> specifically: >>>>> >>>>> we cannot set the max_map_count value as we do not have access to the >>>>> underlying host >>>>> (https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html). >>>>> Unfortunately, this is required to run an ES cluster, therefore we were >>>>> forced to use NIOF >>>>> ACI’s storage is ephemera, therefore we had to map volumes to persist the >>>>> indexes. ACIs only allow volume mappings using Azure File Shares, which >>>>> only works with NFS or SMB. >>>>> >>>>> We are experiencing recurring index corruption, specifically a "read past >>>>> EOF" exception. I asked on the Elastic Search forum but the answer I got >>>>> was a bit generic and not really helpful other than confirming that, from >>>>> ES point of view, ES should work on an SMB share as long as it behaves as >>>>> a local drive. As the underlying exception relates to an issue with a >>>>> Lucene index, I was wondering if you could help out? Specifically, can >>>>> Lucene work on SMB? I can only find sparse information on this >>>>> configuration and, while NFS seems a no-no, for SMB is not that clear. >>>>> Below is the exception we are getting. >>>>> >>>>> java.io.IOException: read past EOF: >>>>> NIOFSIndexInput(path="/bitnami/elasticsearch/data/indices/mS2bUbLtSeG0FSAMuKX7JQ/0/index/_ldsn_1.fnm") >>>>> buffer: java.nio.HeapByteBuffer[pos=0 lim=1024 cap=1024] chunkLen: 1024 >>>>> end: 2331: >>>>> NIOFSIndexInput(path="/bitnami/elasticsearch/data/indices/mS2bUbLtSeG0FSAMuKX7JQ/0/index/_ldsn_1.fnm") >>>>> at >>>>> org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:200) >>>>> ~[lucene-core-9.3.0.jar:?] >>>>> at >>>>> org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:291) >>>>> ~[lucene-core-9.3.0.jar:?] >>>>> at >>>>> org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:55) >>>>> ~[lucene-core-9.3.0.jar:?] >>>>> at >>>>> org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:39) >>>>> ~[lucene-core-9.3.0.jar:?] >>>>> at >>>>> org.apache.lucene.codecs.CodecUtil.readBEInt(CodecUtil.java:667) >>>>> ~[lucene-core-9.3.0.jar:?] >>>>> at >>>>> org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:184) >>>>> ~[lucene-core-9.3.0.jar:?] >>>>> at >>>>> org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:253) >>>>> ~[lucene-core-9.3.0.jar:?] >>>>> at >>>>> org.apache.lucene.codecs.lucene90.Lucene90FieldInfosFormat.read(Lucene90FieldInfosFormat.java:128) >>>>> ~[lucene-core-9.3.0.jar:?] >>>>> at >>>>> org.apache.lucene.index.SegmentReader.initFieldInfos(SegmentReader.java:205) >>>>> ~[lucene-core-9.3.0.jar:?] >>>>> at >>>>> org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:156) >>>>> ~[lucene-core-9.3.0.jar:?] >>>>> at >>>>> org.apache.lucene.index.ReadersAndUpdates.createNewReaderWithLatestLiveDocs(ReadersAndUpdates.java:738) >>>>> ~[lucene-core-9.3.0.jar:?] >>>>> at >>>>> org.apache.lucene.index.ReadersAndUpdates.swapNewReaderWithLatestLiveDocs(ReadersAndUpdates.java:754) >>>>> ~[lucene-core-9.3.0.jar:?] >>>>> at >>>>> org.apache.lucene.index.ReadersAndUpdates.writeFieldUpdates(ReadersAndUpdates.java:678) >>>>> ~[lucene-core-9.3.0.jar:?] >>>>> at >>>>> org.apache.lucene.index.ReaderPool.writeAllDocValuesUpdates(ReaderPool.java:251) >>>>> ~[lucene-core-9.3.0.jar:?] >>>>> at >>>>> org.apache.lucene.index.IndexWriter.writeReaderPool(IndexWriter.java:3743) >>>>> ~[lucene-core-9.3.0.jar:?] >>>>> at >>>>> org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:591) >>>>> ~[lucene-core-9.3.0.jar:?] >>>>> at >>>>> org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:381) >>>>> ~[lucene-core-9.3.0.jar:?] >>>>> at >>>>> org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:355) >>>>> ~[lucene-core-9.3.0.jar:?] >>>>> at >>>>> org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:345) >>>>> ~[lucene-core-9.3.0.jar:?] >>>>> at >>>>> org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112) >>>>> ~[lucene-core-9.3.0.jar:?] >>>>> at >>>>> org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170) >>>>> ~[lucene-core-9.3.0.jar:?] >>>>> at >>>>> org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:48) >>>>> ~[elasticsearch-8.4.1.jar:?] >>>>> at >>>>> org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:27) >>>>> ~[elasticsearch-8.4.1.jar:?] >>>>> at >>>>> org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) >>>>> ~[lucene-core-9.3.0.jar:?] >>>>> at >>>>> org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240) >>>>> ~[lucene-core-9.3.0.jar:?] >>>>> at >>>>> org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:355) >>>>> ~[elasticsearch-8.4.1.jar:?] >>>>> at >>>>> org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:335) >>>>> ~[elasticsearch-8.4.1.jar:?] >>>>> at >>>>> org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) >>>>> ~[lucene-core-9.3.0.jar:?] >>>>> Many thanks. >>>>> >>>>> Seb >>>> -- >>>> Uwe Schindler >>>> Achterdiek 19, D-28357 Bremen >>>> https://www.thetaphi.de >>>> eMail: u...@thetaphi.de >>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >>>> For additional commands, e-mail: java-user-h...@lucene.apache.org >>>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >>> For additional commands, e-mail: java-user-h...@lucene.apache.org >>> >> -- >> Uwe Schindler >> Achterdiek 19, D-28357 Bremen >> https://www.thetaphi.de >> eMail: u...@thetaphi.de >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >> For additional commands, e-mail: java-user-h...@lucene.apache.org >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
