Hi,
Yeah, sorry for leading the issue into the wrong direction. I was just
stumbling on the exception message and because we do not spend much time
in improving/supporting the use of NIOFSDirectory, I may have moved this
mailing list thread into the wrong direction.
I don't think the directory implementation will change much when using
Samba. Recent versions of the CIFS specification (the spec behind Samba)
and the implementation in Windows Server are fine. But I doubt that
recent implementations in Linux kernel are correct and fully support the
spec.
The corrumption issue you see is just detected by NIOFSDirectory (which
I stumbled upon), but with MMapDirectory it will happen in the same way
as the problem is caused by fsync not working correctly and files
appearing in wrong order on network devices (this is very important for
Lucene). When writing and fsyncing index files, Lucene uses the same
code (sequential writes to segment files, then writing commit file and
finally atomic renaming it to its final location - a 2 phase commit).
This must be fully supported and implemented by the file system also
with making changes visible in the correct order, which is not the case
for NFS (due to stale caches), but *should* work with the recent CIFS
spec, but Samba may not have implemented it correctly in Linux kernel.
This is the reason for the recommendation to not use network file
systems. You're example just confirmed this recommendation.
Uwe
Am 02.01.2023 um 19:24 schrieb S S:
Hi Uwe,
I will report the bug to ES, as you suggested.
Do you recon using Mmap would have an effect to the index corruption when using
SMB? I have to report back to my manager in few days to decide wether to carry
on with ACIs or find another hosting solution. It is unfortunate there seems to
be problems with this solution. Microsoft seems not interested in extending the
volume mapping options for ACIs and K8 is overkilling for our use case.
Thank you for your help so far, you have been very kind :)
Cheers,
Seb
On 2 Jan 2023, at 19:09, Uwe Schindler <u...@thetaphi.de> wrote:
Hi,
Please open a bug report at ES. The setting vm.max_map_count is not needed and
should not be changed unless really needed, because it uses kernel resources.
This has to do with their support (they try to tell people to overshard and to prevent
support requests they ask to raise this seeting). The default value on Linux is 65530.
This would allow you to memory map 65530 chunks of 1 GiB (thats the limitation without
preview enabled in the Java 19 JVM). A shard in Lucene has about 70 files, most of them
with filesizes < 1 Gib and a few >1 GiB (but < 5 GiB) chunks (so lets assume 90
mappings needed for a shard), so you could have approx 728 shards per node. Sorry
raising this setting is not needed!
Uwe
Am 02.01.2023 um 18:24 schrieb S S:
I also tried enabling preview but no joy, same error :(
It looks like it is not possible to start a multinode ES cluster without
setting vm.max_map_count. I also googled it and this check cannot be disabled.
I guess MMapDirectory is not an option for ES on ACIs, unless you have
something else I can try?
Many thanks,
Seb
On 2 Jan 2023, at 17:55, S S <sebastiano1...@yahoo.it.INVALID> wrote:
Thank you Uwe, this is great! I am rebuilding the cluster using MMapDirectory
and no enable-preview, as you suggested. Let’s see what happens.
Cheers,
Seb
On 2 Jan 2023, at 17:51, Uwe Schindler <u...@thetaphi.de> wrote:
Hi,
in recent versions it works like that:
https://www.elastic.co/guide/en/elasticsearch/reference/current/advanced-configuration.html#set-jvm-options
So in folder jvm.options.d/ add a new file (like "preview.conf") and put
"19:--enable-preview" into it. It is basically the same like modifying heap size.
But in general, you can simply use MMapDirectory, the max-map-count setting is
only relevant in *huge* (huge means hundreds of huge indexes per node). In that
case Java 19's preview features would be recommended.
Uwe
Am 02.01.2023 um 17:41 schrieb S S:
Hi Uwe,
Sorry for the late reply but upgrading the docker image to use OpenJDK was
easier said that done.
I am not a Java developer/expert so, sorry for the stupid question but, how do
I specify the --enable-preview flag? ES has got a quite complex way to start so
I cannot specify the flag on the command line. You suggested to use a
jvm.properties file but I cannot find anything useful about it when google-ing.
Where should it be placed? And what should I write in it?
I can see ES recognising OpenJDK 19 while bootstrapping and suggesting to
enable preview, but it does not suggest how, and I cannot find anything on the
ES website.
Many thanks.
Seb
On 2 Jan 2023, at 11:48, Uwe Schindler <u...@thetaphi.de> wrote:
Hi,
in general you can still use MMapDirectory. There is no requirement to set
vm.max_map_count for smaller clusters. The information in Elastics
documentation is not mandatory and misleading.
If you use newest version of Elasticsearch with Java 19 and you use
`--enable-preview` in you jvm.properties file, you don't even need to change
that setting even with larger clusters.
Uwe
Am 02.01.2023 um 11:18 schrieb S S:
We are experimenting with Elastic Search deployed in Azure Container Instances
(Debian + OpenJDK). The ES indexes are stored into an Azure file share mounted
via SMB (3.0). The Elastic Search cluster is made up of 4 nodes, each one have
a separate file share to store the indices.
This configuration has been influenced by some ACIs limitations, specifically:
we cannot set the max_map_count value as we do not have access to the
underlying host
(https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html).
Unfortunately, this is required to run an ES cluster, therefore we were forced
to use NIOF
ACI’s storage is ephemera, therefore we had to map volumes to persist the
indexes. ACIs only allow volume mappings using Azure File Shares, which only
works with NFS or SMB.
We are experiencing recurring index corruption, specifically a "read past EOF"
exception. I asked on the Elastic Search forum but the answer I got was a bit generic and
not really helpful other than confirming that, from ES point of view, ES should work on
an SMB share as long as it behaves as a local drive. As the underlying exception relates
to an issue with a Lucene index, I was wondering if you could help out? Specifically, can
Lucene work on SMB? I can only find sparse information on this configuration and, while
NFS seems a no-no, for SMB is not that clear. Below is the exception we are getting.
java.io.IOException: read past EOF:
NIOFSIndexInput(path="/bitnami/elasticsearch/data/indices/mS2bUbLtSeG0FSAMuKX7JQ/0/index/_ldsn_1.fnm")
buffer: java.nio.HeapByteBuffer[pos=0 lim=1024 cap=1024] chunkLen: 1024 end: 2331:
NIOFSIndexInput(path="/bitnami/elasticsearch/data/indices/mS2bUbLtSeG0FSAMuKX7JQ/0/index/_ldsn_1.fnm")
at
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:200)
~[lucene-core-9.3.0.jar:?]
at
org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:291)
~[lucene-core-9.3.0.jar:?]
at
org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:55)
~[lucene-core-9.3.0.jar:?]
at
org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:39)
~[lucene-core-9.3.0.jar:?]
at
org.apache.lucene.codecs.CodecUtil.readBEInt(CodecUtil.java:667)
~[lucene-core-9.3.0.jar:?]
at
org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:184)
~[lucene-core-9.3.0.jar:?]
at
org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:253)
~[lucene-core-9.3.0.jar:?]
at
org.apache.lucene.codecs.lucene90.Lucene90FieldInfosFormat.read(Lucene90FieldInfosFormat.java:128)
~[lucene-core-9.3.0.jar:?]
at
org.apache.lucene.index.SegmentReader.initFieldInfos(SegmentReader.java:205)
~[lucene-core-9.3.0.jar:?]
at
org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:156)
~[lucene-core-9.3.0.jar:?]
at
org.apache.lucene.index.ReadersAndUpdates.createNewReaderWithLatestLiveDocs(ReadersAndUpdates.java:738)
~[lucene-core-9.3.0.jar:?]
at
org.apache.lucene.index.ReadersAndUpdates.swapNewReaderWithLatestLiveDocs(ReadersAndUpdates.java:754)
~[lucene-core-9.3.0.jar:?]
at
org.apache.lucene.index.ReadersAndUpdates.writeFieldUpdates(ReadersAndUpdates.java:678)
~[lucene-core-9.3.0.jar:?]
at
org.apache.lucene.index.ReaderPool.writeAllDocValuesUpdates(ReaderPool.java:251)
~[lucene-core-9.3.0.jar:?]
at
org.apache.lucene.index.IndexWriter.writeReaderPool(IndexWriter.java:3743)
~[lucene-core-9.3.0.jar:?]
at
org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:591)
~[lucene-core-9.3.0.jar:?]
at
org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:381)
~[lucene-core-9.3.0.jar:?]
at
org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:355)
~[lucene-core-9.3.0.jar:?]
at
org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:345)
~[lucene-core-9.3.0.jar:?]
at
org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112)
~[lucene-core-9.3.0.jar:?]
at
org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170)
~[lucene-core-9.3.0.jar:?]
at
org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:48)
~[elasticsearch-8.4.1.jar:?]
at
org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:27)
~[elasticsearch-8.4.1.jar:?]
at
org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167)
~[lucene-core-9.3.0.jar:?]
at
org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240)
~[lucene-core-9.3.0.jar:?]
at
org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:355)
~[elasticsearch-8.4.1.jar:?]
at
org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:335)
~[elasticsearch-8.4.1.jar:?]
at
org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167)
~[lucene-core-9.3.0.jar:?]
Many thanks.
Seb
--
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail: u...@thetaphi.de
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
--
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail: u...@thetaphi.de
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
--
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail: u...@thetaphi.de
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
--
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail: u...@thetaphi.de
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
