I'm not on top of performance degradation issues with the NFS storage type. But by reading more on this topic, I understand the reasoning behind your suggestion.
NFS version 4 improved file locking. MySQL documentation says "If you choose to use NFS, NFS Version 4 or later is recommended, as is testing your NFS setup thoroughly before deploying into a production environment". But Prometheus documentation and also the source code warn users to stay away from it. https://github.com/prometheus/prometheus/blob/main/cmd/prometheus/main.go#L990 case "NFS_SUPER_MAGIC": level.Warn(logger).Log("fs_type", fsType, "msg", "This filesystem is not supported and may lead to data corruption and data loss. Please carefully read https://prometheus.io/docs/prometheus/latest/storage/ to learn more about supported filesystems.") What do you recommend if certain users have access to ONLY NFS storage type? Do you suggest sending/streaming Prometheus data to someother long term data store like Victoria Metrics or Cortext? I still wonder about storage type for fsyncs and WAL file handling. On K8S, if ephemeral storage is used to store Prometheus data, data is not persistent and highly available. Thanks! On Tuesday, May 31, 2022 at 9:37:12 PM UTC+2 [email protected] wrote: > Last I remember, most of the issues are around file locking and atomic > operations. > > Agent mode with WAL-only should be less of an issue, except possibly > fsyncs. > > Either way, depending on NFS is a bad idea from a reliability perspective, > as your monitoring system now depends on the network and storage server to > properly alert. > > Like Stuart says, would you run your SQL servers over NFS? > > On Tue, May 31, 2022 at 7:38 PM tejaswini vadlamudi <[email protected]> > wrote: > >> What do you mean by not meeting desired performance characteristics? Is >> it the write conflicts, file/block size limiations, etc.. eventually over >> time, or is it with a high ingestion load? >> Do we have similar problems if we deploy Prometheus in Agent mode (data >> streamed to a remote location) but still need WAL storage to handle >> receiver connection failures? >> >> Thanks, Teja >> On Tuesday, May 31, 2022 at 5:41:11 PM UTC+2 Stuart Clark wrote: >> >>> On 2022-05-31 15:45, tejaswini vadlamudi wrote: >>> > Hi Stuart, >>> > >>> > I forgot to ask the most important question on this topic :-) >>> > Could you explain the reason for not supporting NFS based storage in >>> > Prometheus? >>> > >>> >>> At its core Prometheus contains a high performance time-series database. >>> Network filesystems just don't have the same performance characteristics >>> or features as a direct local disk. It is similar to not expecting >>> support for storing a MySQL/Oracle/MSSQL/etc. database on a network >>> storage system - it might technically work, but you are likely to >>> encounter problems at some point, and you are unlikely to recenive any >>> free/paid support in that architecture. >>> >>> -- >>> Stuart Clark >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/993b1dc3-120d-4ec7-84d1-e99387e30e15n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/prometheus-users/993b1dc3-120d-4ec7-84d1-e99387e30e15n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/43835db9-7c20-49eb-8890-a54a13dfc033n%40googlegroups.com.
