Hi Christian, This does appear to be the issue.
In the Azure ecosystem Azure Disks should be posix compliant, would switching to those resolve the issue or would we still have issues with NFS? Reference at the very bottom of this link: https://social.msdn.microsoft.com/Forums/en-US/0e19f5f8-1767-41c9-b3cf-b6aaa770433d/azure-files-storage-posix-compliant And if that also would be an issue, do you know if we would see similar problems trying to use Thanos to manage storage on Azure instead of PVCs? On Friday, April 3, 2020 at 2:56:22 PM UTC-4, Christian Hoffmann wrote: > > Hi Robin, > > On 4/3/20 8:36 PM, Robin Pharaoh wrote: > > Was this ever resolved? We are hitting a the exact same issue right now. > > > > We have a single instance of prometheus > > We are using Azure File Share with a volume claim > I don't have any Azure knowledge, but I assume this is a SMB-based mount? > > Then I guess this is something which is not supported as I don't think > SMB matches all POSIX requirements which Prometheus relies on? > > Citing the docs [1]: > "Non POSIX compliant filesystems are not supported by Prometheus's local > storage, corruptions may happen, without possibility to recover. NFS is > only potentially POSIX, most implementations are not." > > I have never seen such corruption issues when running on xfs. I guess > your best bet would be block-based storage and a standard *nix > filesystem such as xfs or ext4. > > Kind regards, > Christian > > [1] https://prometheus.io/docs/prometheus/latest/storage/ > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/5a458082-aa19-423f-a6d2-bbf215c64c07%40googlegroups.com.
