Re: fsync()-in webdav PUT

This needs to stop On Monday, March 5, 2018, Nagy, Attila wrote: > On 03/05/2018 12:54 PM, Reinis Rozitis wrote: > >> Jesus, why? You start the fsync in a thread and you wait for it to be >>> completed >>> with the HTTP response. Until this happens, the main thread can service >>>

Re: fsync()-in webdav PUT

On 03/05/2018 12:54 PM, Reinis Rozitis wrote: Jesus, why? You start the fsync in a thread and you wait for it to be completed with the HTTP response. Until this happens, the main thread can service other requests. Have you ever seen an async program which uses threads to run blocking operations?

Re: fsync()-in webdav PUT

On 05-03-18 12:54, Reinis Rozitis wrote: Have you ever seen an async program which uses threads to run blocking operations? The point was that it's odd that you are going to "trust" the userland daemon to finish the sync operation (which obviously has the possibility to fail) in some

RE: fsync()-in webdav PUT

> Jesus, why? You start the fsync in a thread and you wait for it to be > completed > with the HTTP response. Until this happens, the main thread can service other > requests. > Have you ever seen an async program which uses threads to run blocking > operations? The point was that it's odd that

Re: fsync()-in webdav PUT

On 03/04/2018 12:45 PM, Reinis Rozitis wrote: That's what really kills performance, because of the async nature of nginx. That's why I'm proposing an option to do the fsync at the end of the PUT (or maybe even the whole operation) in a thread(pool). Then again this way you make it again

Re: fsync()-in webdav PUT

Hello! On Fri, Mar 02, 2018 at 08:47:17PM +0100, Valery Kholodkov wrote: > On 02-03-18 17:06, Maxim Dounin wrote: > >>> The question here is - why you want the file to be on disk, and > >>> not just in a buffer? Because you expect the server to die in a > >>> few seconds without flushing the

Re: fsync()-in webdav PUT

On 02-03-18 17:06, Maxim Dounin wrote: The question here is - why you want the file to be on disk, and not just in a buffer? Because you expect the server to die in a few seconds without flushing the file to disk? How probable it is, compared to the probability of the disk to die? A more

Re: fsync()-in webdav PUT

On 03/02/2018 11:42 AM, Aziz Rozyev wrote: man page quote is related to the Valery’s argument that fsync wont affect performance, forget it. Of course it affects performance. But as for how much: it depends on many factors. It's possible to build servers where the overall effect will be

Re: fsync()-in webdav PUT

Atilla, man page quote is related to the Valery’s argument that fsync wont affect performance, forget it. It’s nonsense because you’re trying to solve the reliability problem at the different level, it has been multiple times suggested here already by maxim and Paul, that it’s better to

Re: fsync()-in webdav PUT

On 02/28/2018 10:41 PM, Aziz Rozyev wrote: Without fsyncing file's data and metadata a client will receive a positive reply before data has reached the storage, thus leaving non-zero probability that states of two systems involved into a web transaction end up inconsistent. I understand why

Re: fsync()-in webdav PUT

On 02/28/2018 03:08 PM, Maxim Dounin wrote: The question here is - why you want the file to be on disk, and not just in a buffer? Because you expect the server to die in a few seconds without flushing the file to disk? How probable it is, compared to the probability of the disk to die? A more

Re: fsync()-in webdav PUT

On 02/28/2018 11:33 PM, Peter Booth wrote: This discussion is interesting, educational, and thought provoking.  Web architects only learn “the right way” by first doing things “the wrong way” and seeing what happens. Attila and Valery asked questions that sound logical, and I think there's

Re: fsync()-in webdav PUT

On 02/28/2018 11:04 AM, Aziz Rozyev wrote: While it’s not clear why one may need to flush the data on each http operation, I can imagine to what performance degradation that may lead of. I store data on HTTP servers in a distributed manner and have a catalog of where each file is. If I get back

Re: fsync()-in webdav PUT

You can also apply online: https://angel.co/qubiq-digital-b-v/jobs That's more 2018-nish. On 01-03-18 13:24, Valery Kholodkov wrote: I admire your wise approach to this discussion, as well your technical expertise! I see the value in people who know the right way, but I see the value in

Re: fsync()-in webdav PUT

I admire your wise approach to this discussion, as well your technical expertise! I see the value in people who know the right way, but I see the value in people who dare to explore and want learning the right way. Coincidently, at Qubiq Labs we're looking for that kind of kick-ass Systems and

Re: fsync()-in webdav PUT

This discussion is interesting, educational, and thought provoking. Web architects only learn “the right way” by first doing things “the wrong way” and seeing what happens. Attila and Valery asked questions that sound logical, and I think there's value in exploring what would happen if their

Re: fsync()-in webdav PUT

here is a synthetic test on vm, not perfect, but representative: [root@nginx-single ~]# dd if=/dev/zero of=/writetest bs=8k count=30960 30960+0 records in 30960+0 records out 253624320 bytes (254 MB) copied, 0.834861 s, 304 MB/s [root@nginx-single ~]# dd if=/dev/zero of=/writetest bs=8k

Re: fsync()-in webdav PUT

Valery, may you please suggest how you came to the conclusion that “fsync simply instructs OS to ensure consistency of a file"? As far as understand simply instructing OS staff come at no cost, right? > Without fsyncing file's data and metadata a client will receive a positive > reply

Re: fsync()-in webdav PUT

Not waiting for fsync to complete makes calling fsync pointless, waiting for fsync is blocking, thread based or otherwise. The only midway solution is to implement fsync as a cgi, ea. a none-blocking (background)fc call in combination with an OS resource lock. Posted at Nginx Forum:

Re: fsync()-in webdav PUT

On 28-02-18 15:08, Maxim Dounin wrote: What do you mean by a reliable server? I want to make sure when the HTTP operation returns, the file is on the disk, not just in a buffer waiting for an indefinite amount of time to be flushed. This is what fsync is for. The question here is - why you

Re: fsync()-in webdav PUT

It's completely clear why someone would need to flush file's data and metadata upon a WebDAV PUT operation. That is because many architectures expect a PUT operation to be completely settled before a reply is returned. Without fsyncing file's data and metadata a client will receive a positive

Re: fsync()-in webdav PUT

Hello! On Wed, Feb 28, 2018 at 10:30:08AM +0100, Nagy, Attila wrote: > On 02/27/2018 02:24 PM, Maxim Dounin wrote: > > > >> Now, that nginx supports running threads, are there plans to convert at > >> least DAV PUTs into it's own thread(pool), so make it possible to do > >> non-blocking (from

Re: fsync()-in webdav PUT

While it’s not clear why one may need to flush the data on each http operation, I can imagine to what performance degradation that may lead of. if it’s not a some kind of funny clustering among nodes, I wouldn't care much where actual data is, RAM still should be much faster, than disk I/O.

Re: fsync()-in webdav PUT

Hello! On Tue, Feb 27, 2018 at 11:21:23AM +0100, Nagy, Attila wrote: > I would like to make sure when a WebDAV (with ngx_http_dav) PUT returns > the file is reliably on the storage. This needs an fsync() on the file. > It would be easy to put that into the module, but it would block the >

fsync()-in webdav PUT

Hi, I would like to make sure when a WebDAV (with ngx_http_dav) PUT returns the file is reliably on the storage. This needs an fsync() on the file. It would be easy to put that into the module, but it would block the whole nginx process. Now, that nginx supports running threads, are there