Ben and Stuart, any comments? On Thursday, June 2, 2022 at 1:53:49 PM UTC+2 tejaswini vadlamudi wrote:
> I'm not on top of performance degradation issues with the NFS storage > type. But by reading more on this topic, I understand the reasoning behind > your suggestion. > > NFS version 4 improved file locking. > > MySQL documentation says "If you choose to use NFS, NFS Version 4 or later > is recommended, as is testing your NFS setup thoroughly before deploying > into a production environment". > > But Prometheus documentation and also the source code warn users to stay > away from it. > > > https://github.com/prometheus/prometheus/blob/main/cmd/prometheus/main.go#L990 > > case "NFS_SUPER_MAGIC": > > level.Warn(logger).Log("fs_type", fsType, "msg", "This > filesystem is not supported and may lead to data corruption and data loss. > Please carefully read > https://prometheus.io/docs/prometheus/latest/storage/ to learn more about > supported filesystems.") > > What do you recommend if certain users have access to ONLY NFS storage > type? Do you suggest sending/streaming Prometheus data to someother long > term data store like Victoria Metrics or Cortext? I still wonder about > storage type for fsyncs and WAL file handling. > On K8S, if ephemeral storage is used to store Prometheus data, data is > not persistent and highly available. > > Thanks! > On Tuesday, May 31, 2022 at 9:37:12 PM UTC+2 [email protected] wrote: > >> Last I remember, most of the issues are around file locking and atomic >> operations. >> >> Agent mode with WAL-only should be less of an issue, except possibly >> fsyncs. >> >> Either way, depending on NFS is a bad idea from a reliability >> perspective, as your monitoring system now depends on the network and >> storage server to properly alert. >> >> Like Stuart says, would you run your SQL servers over NFS? >> >> On Tue, May 31, 2022 at 7:38 PM tejaswini vadlamudi <[email protected]> >> wrote: >> >>> What do you mean by not meeting desired performance characteristics? Is >>> it the write conflicts, file/block size limiations, etc.. eventually over >>> time, or is it with a high ingestion load? >>> Do we have similar problems if we deploy Prometheus in Agent mode (data >>> streamed to a remote location) but still need WAL storage to handle >>> receiver connection failures? >>> >>> Thanks, Teja >>> On Tuesday, May 31, 2022 at 5:41:11 PM UTC+2 Stuart Clark wrote: >>> >>>> On 2022-05-31 15:45, tejaswini vadlamudi wrote: >>>> > Hi Stuart, >>>> > >>>> > I forgot to ask the most important question on this topic :-) >>>> > Could you explain the reason for not supporting NFS based storage in >>>> > Prometheus? >>>> > >>>> >>>> At its core Prometheus contains a high performance time-series >>>> database. >>>> Network filesystems just don't have the same performance >>>> characteristics >>>> or features as a direct local disk. It is similar to not expecting >>>> support for storing a MySQL/Oracle/MSSQL/etc. database on a network >>>> storage system - it might technically work, but you are likely to >>>> encounter problems at some point, and you are unlikely to recenive any >>>> free/paid support in that architecture. >>>> >>>> -- >>>> Stuart Clark >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Prometheus Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/prometheus-users/993b1dc3-120d-4ec7-84d1-e99387e30e15n%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/prometheus-users/993b1dc3-120d-4ec7-84d1-e99387e30e15n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/7cfd5704-a10f-4007-92e2-7f2ffc21429dn%40googlegroups.com.
