Thank you Uwe, this is great! I am rebuilding the cluster using MMapDirectory and no enable-preview, as you suggested. Let’s see what happens.
Cheers, Seb > On 2 Jan 2023, at 17:51, Uwe Schindler <u...@thetaphi.de> wrote: > > Hi, > > in recent versions it works like that: > > https://www.elastic.co/guide/en/elasticsearch/reference/current/advanced-configuration.html#set-jvm-options > > So in folder jvm.options.d/ add a new file (like "preview.conf") and put > "19:--enable-preview" into it. It is basically the same like modifying heap > size. > > But in general, you can simply use MMapDirectory, the max-map-count setting > is only relevant in *huge* (huge means hundreds of huge indexes per node). In > that case Java 19's preview features would be recommended. > > Uwe > > Am 02.01.2023 um 17:41 schrieb S S: >> Hi Uwe, >> >> Sorry for the late reply but upgrading the docker image to use OpenJDK was >> easier said that done. >> >> I am not a Java developer/expert so, sorry for the stupid question but, how >> do I specify the --enable-preview flag? ES has got a quite complex way to >> start so I cannot specify the flag on the command line. You suggested to use >> a jvm.properties file but I cannot find anything useful about it when >> google-ing. Where should it be placed? And what should I write in it? >> >> I can see ES recognising OpenJDK 19 while bootstrapping and suggesting to >> enable preview, but it does not suggest how, and I cannot find anything on >> the ES website. >> >> Many thanks. >> >> Seb >> >>> On 2 Jan 2023, at 11:48, Uwe Schindler <u...@thetaphi.de> wrote: >>> >>> Hi, >>> >>> in general you can still use MMapDirectory. There is no requirement to set >>> vm.max_map_count for smaller clusters. The information in Elastics >>> documentation is not mandatory and misleading. >>> >>> If you use newest version of Elasticsearch with Java 19 and you use >>> `--enable-preview` in you jvm.properties file, you don't even need to >>> change that setting even with larger clusters. >>> >>> Uwe >>> >>> Am 02.01.2023 um 11:18 schrieb S S: >>>> We are experimenting with Elastic Search deployed in Azure Container >>>> Instances (Debian + OpenJDK). The ES indexes are stored into an Azure file >>>> share mounted via SMB (3.0). The Elastic Search cluster is made up of 4 >>>> nodes, each one have a separate file share to store the indices. >>>> >>>> This configuration has been influenced by some ACIs limitations, >>>> specifically: >>>> >>>> we cannot set the max_map_count value as we do not have access to the >>>> underlying host >>>> (https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html). >>>> Unfortunately, this is required to run an ES cluster, therefore we were >>>> forced to use NIOF >>>> ACI’s storage is ephemera, therefore we had to map volumes to persist the >>>> indexes. ACIs only allow volume mappings using Azure File Shares, which >>>> only works with NFS or SMB. >>>> >>>> We are experiencing recurring index corruption, specifically a "read past >>>> EOF" exception. I asked on the Elastic Search forum but the answer I got >>>> was a bit generic and not really helpful other than confirming that, from >>>> ES point of view, ES should work on an SMB share as long as it behaves as >>>> a local drive. As the underlying exception relates to an issue with a >>>> Lucene index, I was wondering if you could help out? Specifically, can >>>> Lucene work on SMB? I can only find sparse information on this >>>> configuration and, while NFS seems a no-no, for SMB is not that clear. >>>> Below is the exception we are getting. >>>> >>>> java.io.IOException: read past EOF: >>>> NIOFSIndexInput(path="/bitnami/elasticsearch/data/indices/mS2bUbLtSeG0FSAMuKX7JQ/0/index/_ldsn_1.fnm") >>>> buffer: java.nio.HeapByteBuffer[pos=0 lim=1024 cap=1024] chunkLen: 1024 >>>> end: 2331: >>>> NIOFSIndexInput(path="/bitnami/elasticsearch/data/indices/mS2bUbLtSeG0FSAMuKX7JQ/0/index/_ldsn_1.fnm") >>>> at >>>> org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:200) >>>> ~[lucene-core-9.3.0.jar:?] >>>> at >>>> org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:291) >>>> ~[lucene-core-9.3.0.jar:?] >>>> at >>>> org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:55) >>>> ~[lucene-core-9.3.0.jar:?] >>>> at >>>> org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:39) >>>> ~[lucene-core-9.3.0.jar:?] >>>> at >>>> org.apache.lucene.codecs.CodecUtil.readBEInt(CodecUtil.java:667) >>>> ~[lucene-core-9.3.0.jar:?] >>>> at >>>> org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:184) >>>> ~[lucene-core-9.3.0.jar:?] >>>> at >>>> org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:253) >>>> ~[lucene-core-9.3.0.jar:?] >>>> at >>>> org.apache.lucene.codecs.lucene90.Lucene90FieldInfosFormat.read(Lucene90FieldInfosFormat.java:128) >>>> ~[lucene-core-9.3.0.jar:?] >>>> at >>>> org.apache.lucene.index.SegmentReader.initFieldInfos(SegmentReader.java:205) >>>> ~[lucene-core-9.3.0.jar:?] >>>> at >>>> org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:156) >>>> ~[lucene-core-9.3.0.jar:?] >>>> at >>>> org.apache.lucene.index.ReadersAndUpdates.createNewReaderWithLatestLiveDocs(ReadersAndUpdates.java:738) >>>> ~[lucene-core-9.3.0.jar:?] >>>> at >>>> org.apache.lucene.index.ReadersAndUpdates.swapNewReaderWithLatestLiveDocs(ReadersAndUpdates.java:754) >>>> ~[lucene-core-9.3.0.jar:?] >>>> at >>>> org.apache.lucene.index.ReadersAndUpdates.writeFieldUpdates(ReadersAndUpdates.java:678) >>>> ~[lucene-core-9.3.0.jar:?] >>>> at >>>> org.apache.lucene.index.ReaderPool.writeAllDocValuesUpdates(ReaderPool.java:251) >>>> ~[lucene-core-9.3.0.jar:?] >>>> at >>>> org.apache.lucene.index.IndexWriter.writeReaderPool(IndexWriter.java:3743) >>>> ~[lucene-core-9.3.0.jar:?] >>>> at >>>> org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:591) >>>> ~[lucene-core-9.3.0.jar:?] >>>> at >>>> org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:381) >>>> ~[lucene-core-9.3.0.jar:?] >>>> at >>>> org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:355) >>>> ~[lucene-core-9.3.0.jar:?] >>>> at >>>> org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:345) >>>> ~[lucene-core-9.3.0.jar:?] >>>> at >>>> org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112) >>>> ~[lucene-core-9.3.0.jar:?] >>>> at >>>> org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170) >>>> ~[lucene-core-9.3.0.jar:?] >>>> at >>>> org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:48) >>>> ~[elasticsearch-8.4.1.jar:?] >>>> at >>>> org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:27) >>>> ~[elasticsearch-8.4.1.jar:?] >>>> at >>>> org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) >>>> ~[lucene-core-9.3.0.jar:?] >>>> at >>>> org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240) >>>> ~[lucene-core-9.3.0.jar:?] >>>> at >>>> org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:355) >>>> ~[elasticsearch-8.4.1.jar:?] >>>> at >>>> org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:335) >>>> ~[elasticsearch-8.4.1.jar:?] >>>> at >>>> org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) >>>> ~[lucene-core-9.3.0.jar:?] >>>> Many thanks. >>>> >>>> Seb >>> -- >>> Uwe Schindler >>> Achterdiek 19, D-28357 Bremen >>> https://www.thetaphi.de >>> eMail: u...@thetaphi.de >>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >>> For additional commands, e-mail: java-user-h...@lucene.apache.org >>> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >> For additional commands, e-mail: java-user-h...@lucene.apache.org >> > -- > Uwe Schindler > Achterdiek 19, D-28357 Bremen > https://www.thetaphi.de > eMail: u...@thetaphi.de > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
