This needs to stop
On Monday, March 5, 2018, Nagy, Attila wrote:
> On 03/05/2018 12:54 PM, Reinis Rozitis wrote:
>
>> Jesus, why? You start the fsync in a thread and you wait for it to be
>>> completed
>>> with the HTTP response. Until this happens, the main thread can service
>>> other
>>> requ
On 03/05/2018 12:54 PM, Reinis Rozitis wrote:
Jesus, why? You start the fsync in a thread and you wait for it to be completed
with the HTTP response. Until this happens, the main thread can service other
requests.
Have you ever seen an async program which uses threads to run blocking
operations?
On 05-03-18 12:54, Reinis Rozitis wrote:
Have you ever seen an async program which uses threads to run blocking
operations?
The point was that it's odd that you are going to "trust" the userland daemon to finish the sync
operation (which obviously has the possibility to fail) in some backgroun
> Jesus, why? You start the fsync in a thread and you wait for it to be
> completed
> with the HTTP response. Until this happens, the main thread can service other
> requests.
> Have you ever seen an async program which uses threads to run blocking
> operations?
The point was that it's odd that y
On 03/04/2018 12:45 PM, Reinis Rozitis wrote:
That's what really kills performance, because of the async nature of
nginx. That's why I'm proposing an option to do the fsync at the end of
the PUT (or maybe even the whole operation) in a thread(pool).
Then again this way you make it again "asynchr
> That's what really kills performance, because of the async nature of
> nginx. That's why I'm proposing an option to do the fsync at the end of
> the PUT (or maybe even the whole operation) in a thread(pool).
Then again this way you make it again "asynchronous" (since it is/could be
waiting in s
Hello!
On Fri, Mar 02, 2018 at 08:47:17PM +0100, Valery Kholodkov wrote:
> On 02-03-18 17:06, Maxim Dounin wrote:
> >>> The question here is - why you want the file to be on disk, and
> >>> not just in a buffer? Because you expect the server to die in a
> >>> few seconds without flushing the fil
On 02-03-18 17:06, Maxim Dounin wrote:
The question here is - why you want the file to be on disk, and
not just in a buffer? Because you expect the server to die in a
few seconds without flushing the file to disk? How probable it
is, compared to the probability of the disk to die? A more
relia
Hello!
On Fri, Mar 02, 2018 at 10:12:02AM +0100, Nagy, Attila wrote:
> On 02/28/2018 03:08 PM, Maxim Dounin wrote:
> > The question here is - why you want the file to be on disk, and
> > not just in a buffer? Because you expect the server to die in a
> > few seconds without flushing the file to
On 03/02/2018 11:42 AM, Aziz Rozyev wrote:
man page quote is related to the Valery’s argument that fsync wont affect
performance, forget it.
Of course it affects performance. But as for how much: it depends on
many factors. It's possible to build servers where the overall effect
will be neglig
Atilla,
man page quote is related to the Valery’s argument that fsync wont affect
performance, forget it.
It’s nonsense because you’re trying to solve the reliability problem at the
different level,
it has been multiple times suggested here already by maxim and Paul, that it’s
better
to inves
On 02/28/2018 10:41 PM, Aziz Rozyev wrote:
Without fsyncing file's data and metadata a client will receive a positive
reply before data has reached the storage, thus leaving non-zero probability
that states of two systems involved into a web transaction end up inconsistent.
I understand why o
On 02/28/2018 03:08 PM, Maxim Dounin wrote:
The question here is - why you want the file to be on disk, and
not just in a buffer? Because you expect the server to die in a
few seconds without flushing the file to disk? How probable it
is, compared to the probability of the disk to die? A more
On 02/28/2018 11:33 PM, Peter Booth wrote:
This discussion is interesting, educational, and thought provoking.
Web architects
only learn “the right way” by first doing things “the wrong way” and
seeing what happens.
Attila and Valery asked questions that sound logical, and I think
there's valu
On 02/28/2018 11:04 AM, Aziz Rozyev wrote:
While it’s not clear why one may need to flush the data on each http operation,
I can imagine to what performance degradation that may lead of.
I store data on HTTP servers in a distributed manner and have a catalog
of where each file is. If I get back
You can also apply online:
https://angel.co/qubiq-digital-b-v/jobs
That's more 2018-nish.
On 01-03-18 13:24, Valery Kholodkov wrote:
I admire your wise approach to this discussion, as well your technical
expertise! I see the value in people who know the right way, but I see
the value in people
I admire your wise approach to this discussion, as well your technical
expertise! I see the value in people who know the right way, but I see
the value in people who dare to explore and want learning the right way.
Coincidently, at Qubiq Labs we're looking for that kind of kick-ass
Systems and
This discussion is interesting, educational, and thought provoking. Web
architects
only learn “the right way” by first doing things “the wrong way” and seeing
what happens.
Attila and Valery asked questions that sound logical, and I think there's value
in exploring
what would happen if their s
here is a synthetic test on vm, not perfect, but representative:
[root@nginx-single ~]# dd if=/dev/zero of=/writetest bs=8k count=30960
30960+0 records in
30960+0 records out
253624320 bytes (254 MB) copied, 0.834861 s, 304 MB/s
[root@nginx-single ~]# dd if=/dev/zero of=/writetest bs=8k count=3
Valery,
may you please suggest how you came to the conclusion that
“fsync simply instructs OS to ensure consistency of a file"?
As far as understand simply instructing OS staff come at no cost, right?
> Without fsyncing file's data and metadata a client will receive a positive
> reply before
Not waiting for fsync to complete makes calling fsync pointless, waiting for
fsync is blocking, thread based or otherwise.
The only midway solution is to implement fsync as a cgi, ea. a none-blocking
(background)fc call in combination with an OS resource lock.
Posted at Nginx Forum:
https://forum
On 28-02-18 15:08, Maxim Dounin wrote:
What do you mean by a reliable server?
I want to make sure when the HTTP operation returns, the file is on the
disk, not just in a buffer waiting for an indefinite amount of time to
be flushed.
This is what fsync is for.
The question here is - why you want
It's completely clear why someone would need to flush file's data and
metadata upon a WebDAV PUT operation. That is because many architectures
expect a PUT operation to be completely settled before a reply is returned.
Without fsyncing file's data and metadata a client will receive a
positive
Hello!
On Wed, Feb 28, 2018 at 10:30:08AM +0100, Nagy, Attila wrote:
> On 02/27/2018 02:24 PM, Maxim Dounin wrote:
> >
> >> Now, that nginx supports running threads, are there plans to convert at
> >> least DAV PUTs into it's own thread(pool), so make it possible to do
> >> non-blocking (from ngi
While it’s not clear why one may need to flush the data on each http operation,
I can imagine to what performance degradation that may lead of.
if it’s not a some kind of funny clustering among nodes, I wouldn't care much
where actual data is, RAM still should be much faster, than disk I/O.
br,
On 02/27/2018 02:24 PM, Maxim Dounin wrote:
Now, that nginx supports running threads, are there plans to convert at
least DAV PUTs into it's own thread(pool), so make it possible to do
non-blocking (from nginx's event loop PoV) fsync on the uploaded file?
No, there are no such plans.
(Also, t
Hello!
On Tue, Feb 27, 2018 at 11:21:23AM +0100, Nagy, Attila wrote:
> I would like to make sure when a WebDAV (with ngx_http_dav) PUT returns
> the file is reliably on the storage. This needs an fsync() on the file.
> It would be easy to put that into the module, but it would block the
> whole
Hi,
I would like to make sure when a WebDAV (with ngx_http_dav) PUT returns
the file is reliably on the storage. This needs an fsync() on the file.
It would be easy to put that into the module, but it would block the
whole nginx process.
Now, that nginx supports running threads, are there pla
28 matches
Mail list logo