[ANNOUNCE] Three things.

2019-08-29 Thread Daniel Phillips
Hi folks, how's it going? Over here, we have been rather busy lately, and for the last five years or so to be honest. Today it is my pleasure to be able to announce three GPL open source projects: 1) Shardmap Shardmap is the next generation directory index developed for Tux3, and which we are

Re: [FYI] tux3: Core changes

2015-07-31 Thread Daniel Phillips
On Friday, July 31, 2015 5:00:43 PM PDT, Daniel Phillips wrote: Note: Hirofumi's email is clear, logical and speaks to the question. This branch of the thread is largely pointless, though it essentially says the same thing in non-technical terms. Perhaps your next response should be to Hirofumi

Re: [FYI] tux3: Core changes

2015-07-31 Thread Daniel Phillips
On Friday, July 31, 2015 3:27:12 PM PDT, David Lang wrote: On Fri, 31 Jul 2015, Daniel Phillips wrote: On Friday, July 31, 2015 11:29:51 AM PDT, David Lang wrote: ... you weren't asking about any particular feature of Tux, you were asking if we were still willing to push out stuff

Re: [FYI] tux3: Core changes

2015-07-31 Thread Daniel Phillips
On Friday, July 31, 2015 11:29:51 AM PDT, David Lang wrote: We, the Linux Community have less tolerance for losing people's data and preventing them from operating than we used to when it was all tinkerer's personal data and secondary systems. So rather than pushing optimizations out to

Re: [FYI] tux3: Core changes

2015-07-31 Thread Daniel Phillips
On Friday, July 31, 2015 11:29:51 AM PDT, David Lang wrote: If you define this as "loosing our mojo", then yes we have. A pity. There remains so much to do that simply will not get done in the absence of mojo. Regards, Daniel -- To unsubscribe from this list: send the line "unsubscribe

Re: [FYI] tux3: Core changes

2015-07-31 Thread Daniel Phillips
On Friday, July 31, 2015 8:37:35 AM PDT, Raymond Jennings wrote: Returning ENOSPC when you have free space you can't yet prove is safer than not returning it and risking a data loss when you get hit by a write/commit storm. :) Remember when delayed allocation was scary and unproven, because

Re: [FYI] tux3: Core changes

2015-07-31 Thread Daniel Phillips
On Friday, July 31, 2015 5:00:43 PM PDT, Daniel Phillips wrote: Note: Hirofumi's email is clear, logical and speaks to the question. This branch of the thread is largely pointless, though it essentially says the same thing in non-technical terms. Perhaps your next response should be to Hirofumi

Re: [FYI] tux3: Core changes

2015-07-31 Thread Daniel Phillips
On Friday, July 31, 2015 11:29:51 AM PDT, David Lang wrote: If you define this as loosing our mojo, then yes we have. A pity. There remains so much to do that simply will not get done in the absence of mojo. Regards, Daniel -- To unsubscribe from this list: send the line unsubscribe

Re: [FYI] tux3: Core changes

2015-07-31 Thread Daniel Phillips
On Friday, July 31, 2015 8:37:35 AM PDT, Raymond Jennings wrote: Returning ENOSPC when you have free space you can't yet prove is safer than not returning it and risking a data loss when you get hit by a write/commit storm. :) Remember when delayed allocation was scary and unproven, because

Re: [FYI] tux3: Core changes

2015-07-31 Thread Daniel Phillips
On Friday, July 31, 2015 11:29:51 AM PDT, David Lang wrote: We, the Linux Community have less tolerance for losing people's data and preventing them from operating than we used to when it was all tinkerer's personal data and secondary systems. So rather than pushing optimizations out to

Re: [FYI] tux3: Core changes

2015-07-31 Thread Daniel Phillips
On Friday, July 31, 2015 3:27:12 PM PDT, David Lang wrote: On Fri, 31 Jul 2015, Daniel Phillips wrote: On Friday, July 31, 2015 11:29:51 AM PDT, David Lang wrote: ... you weren't asking about any particular feature of Tux, you were asking if we were still willing to push out stuff

Re: Tux3 Report: How fast can we fail?

2015-05-27 Thread Daniel Phillips
On 05/27/2015 02:39 PM, Pavel Machek wrote: > On Wed 2015-05-27 11:28:50, Daniel Phillips wrote: >> On Tuesday, May 26, 2015 11:41:39 PM PDT, Mosis Tembo wrote: >>> On Tue, May 26, 2015 at 6:03 PM, Pavel Machek wrote: >>> >>>> >>>>> We ident

Re: [FYI] tux3: Core changes

2015-05-27 Thread Daniel Phillips
On 05/27/2015 02:37 PM, Pavel Machek wrote: > On Wed 2015-05-27 11:09:25, Daniel Phillips wrote: >> On Wednesday, May 27, 2015 12:41:37 AM PDT, Pavel Machek wrote: >>> On Fri 2015-05-15 02:38:33, Daniel Phillips wrote: >>>> On 05/14/2015 08:06 PM, Rik van Riel wrot

Re: Tux3 Report: How fast can we fail?

2015-05-27 Thread Daniel Phillips
On Tuesday, May 26, 2015 11:41:39 PM PDT, Mosis Tembo wrote: On Tue, May 26, 2015 at 6:03 PM, Pavel Machek wrote: We identified the following quality metrics for this algorithm: 1) Never fails to detect out of space in the front end. 2) Always fills a volume to 100% before reporting out

Re: [FYI] tux3: Core changes

2015-05-27 Thread Daniel Phillips
On Wednesday, May 27, 2015 12:41:37 AM PDT, Pavel Machek wrote: On Fri 2015-05-15 02:38:33, Daniel Phillips wrote: On 05/14/2015 08:06 PM, Rik van Riel wrote: ... Umm. Why do you think it is only issue for executable files? I meant: files with code in them, that will be executed. Please

Re: [FYI] tux3: Core changes

2015-05-27 Thread Daniel Phillips
On Wednesday, May 27, 2015 12:41:37 AM PDT, Pavel Machek wrote: On Fri 2015-05-15 02:38:33, Daniel Phillips wrote: On 05/14/2015 08:06 PM, Rik van Riel wrote: ... Umm. Why do you think it is only issue for executable files? I meant: files with code in them, that will be executed. Please

Re: Tux3 Report: How fast can we fail?

2015-05-27 Thread Daniel Phillips
On Tuesday, May 26, 2015 11:41:39 PM PDT, Mosis Tembo wrote: On Tue, May 26, 2015 at 6:03 PM, Pavel Machek pa...@ucw.cz wrote: We identified the following quality metrics for this algorithm: 1) Never fails to detect out of space in the front end. 2) Always fills a volume to 100% before

Re: Tux3 Report: How fast can we fail?

2015-05-27 Thread Daniel Phillips
On 05/27/2015 02:39 PM, Pavel Machek wrote: On Wed 2015-05-27 11:28:50, Daniel Phillips wrote: On Tuesday, May 26, 2015 11:41:39 PM PDT, Mosis Tembo wrote: On Tue, May 26, 2015 at 6:03 PM, Pavel Machek pa...@ucw.cz wrote: We identified the following quality metrics for this algorithm: 1

Re: [FYI] tux3: Core changes

2015-05-27 Thread Daniel Phillips
On 05/27/2015 02:37 PM, Pavel Machek wrote: On Wed 2015-05-27 11:09:25, Daniel Phillips wrote: On Wednesday, May 27, 2015 12:41:37 AM PDT, Pavel Machek wrote: On Fri 2015-05-15 02:38:33, Daniel Phillips wrote: On 05/14/2015 08:06 PM, Rik van Riel wrote: ... Umm. Why do you think it is only

Re: [FYI] tux3: Core changes

2015-05-26 Thread Daniel Phillips
On 05/26/2015 02:36 PM, Rik van Riel wrote: > On 05/26/2015 04:22 PM, Daniel Phillips wrote: >> On 05/26/2015 02:00 AM, Jan Kara wrote: >>> So my opinion is: Don't fork the page if page_count is elevated. You can >>> just wait for the IO if you need stable pa

Re: [FYI] tux3: Core changes

2015-05-26 Thread Daniel Phillips
On 05/26/2015 02:00 AM, Jan Kara wrote: > On Tue 26-05-15 01:08:56, Daniel Phillips wrote: >> On Tuesday, May 26, 2015 12:09:10 AM PDT, Jan Kara wrote: >>> E.g. video drivers (or infiniband or direct IO for that matter) which >>> have buffers in user memory (may be mma

Re: [FYI] tux3: Core changes

2015-05-26 Thread Daniel Phillips
Hi Sergey, On 05/26/2015 03:22 AM, Sergey Senozhatsky wrote: > > Hello, > > is it possible to page-fork-bomb the system by some 'malicious' app? Not in any new way. A page fork can happen either in the front end, where it has to wait for memory like any other normal memory user, or in the

Re: [FYI] tux3: Core changes

2015-05-26 Thread Daniel Phillips
On Monday, May 25, 2015 11:13:46 PM PDT, David Lang wrote: I'm assuming that Rik is talking about whatever has the reference to the page via one of the methods that he talked about. This would be a good moment to provide specifics. Regards, Daniel -- To unsubscribe from this list: send the

Re: [FYI] tux3: Core changes

2015-05-26 Thread Daniel Phillips
On Tuesday, May 26, 2015 12:09:10 AM PDT, Jan Kara wrote: E.g. video drivers (or infiniband or direct IO for that matter) which have buffers in user memory (may be mmapped file), grab references to pages and hand out PFNs of those pages to the hardware to store data in them... If you fork a

Re: [FYI] tux3: Core changes

2015-05-26 Thread Daniel Phillips
On Monday, May 25, 2015 11:04:39 PM PDT, David Lang wrote: if the page gets modified again, will that cause any issues? what if the page gets modified before the copy gets written out, so that there are two dirty copies of the page in the process of being written? David Lang How is the

Re: [FYI] tux3: Core changes

2015-05-26 Thread Daniel Phillips
On Monday, May 25, 2015 11:04:39 PM PDT, David Lang wrote: if the page gets modified again, will that cause any issues? what if the page gets modified before the copy gets written out, so that there are two dirty copies of the page in the process of being written? David Lang How is the

Re: [FYI] tux3: Core changes

2015-05-26 Thread Daniel Phillips
On Monday, May 25, 2015 11:13:46 PM PDT, David Lang wrote: I'm assuming that Rik is talking about whatever has the reference to the page via one of the methods that he talked about. This would be a good moment to provide specifics. Regards, Daniel -- To unsubscribe from this list: send the

Re: [FYI] tux3: Core changes

2015-05-26 Thread Daniel Phillips
On Tuesday, May 26, 2015 12:09:10 AM PDT, Jan Kara wrote: E.g. video drivers (or infiniband or direct IO for that matter) which have buffers in user memory (may be mmapped file), grab references to pages and hand out PFNs of those pages to the hardware to store data in them... If you fork a

Re: [FYI] tux3: Core changes

2015-05-26 Thread Daniel Phillips
On 05/26/2015 02:00 AM, Jan Kara wrote: On Tue 26-05-15 01:08:56, Daniel Phillips wrote: On Tuesday, May 26, 2015 12:09:10 AM PDT, Jan Kara wrote: E.g. video drivers (or infiniband or direct IO for that matter) which have buffers in user memory (may be mmapped file), grab references to pages

Re: [FYI] tux3: Core changes

2015-05-26 Thread Daniel Phillips
Hi Sergey, On 05/26/2015 03:22 AM, Sergey Senozhatsky wrote: Hello, is it possible to page-fork-bomb the system by some 'malicious' app? Not in any new way. A page fork can happen either in the front end, where it has to wait for memory like any other normal memory user, or in the backend,

Re: [FYI] tux3: Core changes

2015-05-26 Thread Daniel Phillips
On 05/26/2015 02:36 PM, Rik van Riel wrote: On 05/26/2015 04:22 PM, Daniel Phillips wrote: On 05/26/2015 02:00 AM, Jan Kara wrote: So my opinion is: Don't fork the page if page_count is elevated. You can just wait for the IO if you need stable pages in that case. It's slow but it's safe

Re: [FYI] tux3: Core changes

2015-05-25 Thread Daniel Phillips
On Monday, May 25, 2015 9:25:44 PM PDT, Rik van Riel wrote: On 05/21/2015 03:53 PM, Daniel Phillips wrote: On Wednesday, May 20, 2015 8:51:46 PM PDT, David Lang wrote: how do you prevent it from continuing to interact with the old version of the page and never see updates or have it's changes

Re: [FYI] tux3: Core changes

2015-05-25 Thread Daniel Phillips
On Monday, May 25, 2015 9:25:44 PM PDT, Rik van Riel wrote: On 05/21/2015 03:53 PM, Daniel Phillips wrote: On Wednesday, May 20, 2015 8:51:46 PM PDT, David Lang wrote: how do you prevent it from continuing to interact with the old version of the page and never see updates or have it's changes

Re: [FYI] tux3: Core changes

2015-05-21 Thread Daniel Phillips
On Wednesday, May 20, 2015 8:51:46 PM PDT, David Lang wrote: how do you prevent it from continuing to interact with the old version of the page and never see updates or have it's changes reflected on the current page? Why would it do that, and what would be surprising about it? Did you have a

[WIP][PATCH] tux3: preliminatry nospace handling

2015-05-21 Thread Daniel Phillips
Hi Josef, This is a rollup patch for preliminary nospace handling in Tux3, in line with my post here: http://lkml.iu.edu/hypermail/linux/kernel/1505.1/03167.html You still have ENOSPC issues. Maybe it would be helpful to look at what we have done. I saw a reproducible case with 1,000 tasks

[WIP][PATCH] tux3: preliminatry nospace handling

2015-05-21 Thread Daniel Phillips
Hi Josef, This is a rollup patch for preliminary nospace handling in Tux3, in line with my post here: http://lkml.iu.edu/hypermail/linux/kernel/1505.1/03167.html You still have ENOSPC issues. Maybe it would be helpful to look at what we have done. I saw a reproducible case with 1,000 tasks

Re: [FYI] tux3: Core changes

2015-05-21 Thread Daniel Phillips
On Wednesday, May 20, 2015 8:51:46 PM PDT, David Lang wrote: how do you prevent it from continuing to interact with the old version of the page and never see updates or have it's changes reflected on the current page? Why would it do that, and what would be surprising about it? Did you have a

Re: [FYI] tux3: Core changes

2015-05-20 Thread Daniel Phillips
On 05/20/2015 03:51 PM, Daniel Phillips wrote: > On 05/20/2015 12:53 PM, Rik van Riel wrote: >> How does tux3 prevent a user of find_get_page() from reading from >> or writing into the pre-COW page, instead of the current page? > > Careful control of the dirty bits (we have

Re: [FYI] tux3: Core changes

2015-05-20 Thread Daniel Phillips
On 05/20/2015 12:53 PM, Rik van Riel wrote: > On 05/20/2015 12:22 PM, Daniel Phillips wrote: >> On 05/20/2015 07:44 AM, Jan Kara wrote: >>> On Tue 19-05-15 13:33:31, David Lang wrote: > >>> Yeah, that's what I meant. If you create a function which manipulates &

Re: [FYI] tux3: Core changes

2015-05-20 Thread Daniel Phillips
On 05/20/2015 07:44 AM, Jan Kara wrote: > On Tue 19-05-15 13:33:31, David Lang wrote: >> On Tue, 19 May 2015, Daniel Phillips wrote: >> >>>> I understand that Tux3 may avoid these issues due to some other mechanisms >>>> it internally has but if pa

Re: [FYI] tux3: Core changes

2015-05-20 Thread Daniel Phillips
On 05/20/2015 07:44 AM, Jan Kara wrote: On Tue 19-05-15 13:33:31, David Lang wrote: On Tue, 19 May 2015, Daniel Phillips wrote: I understand that Tux3 may avoid these issues due to some other mechanisms it internally has but if page forking should get into mm subsystem, the above must work

Re: [FYI] tux3: Core changes

2015-05-20 Thread Daniel Phillips
On 05/20/2015 12:53 PM, Rik van Riel wrote: On 05/20/2015 12:22 PM, Daniel Phillips wrote: On 05/20/2015 07:44 AM, Jan Kara wrote: On Tue 19-05-15 13:33:31, David Lang wrote: Yeah, that's what I meant. If you create a function which manipulates page cache, you better make it work

Re: [FYI] tux3: Core changes

2015-05-20 Thread Daniel Phillips
On 05/20/2015 03:51 PM, Daniel Phillips wrote: On 05/20/2015 12:53 PM, Rik van Riel wrote: How does tux3 prevent a user of find_get_page() from reading from or writing into the pre-COW page, instead of the current page? Careful control of the dirty bits (we have two of them, one each

Re: [FYI] tux3: Core changes

2015-05-19 Thread Daniel Phillips
Hi Jan, On 05/19/2015 07:00 AM, Jan Kara wrote: > On Thu 14-05-15 01:26:23, Daniel Phillips wrote: >> Hi Rik, >> >> Our linux-tux3 tree currently currently carries this 652 line diff >> against core, to make Tux3 work. This is mainly by Hirofumi, except >> the fs

Re: [FYI] tux3: Core changes

2015-05-19 Thread Daniel Phillips
Hi Jan, On 05/19/2015 07:00 AM, Jan Kara wrote: On Thu 14-05-15 01:26:23, Daniel Phillips wrote: Hi Rik, Our linux-tux3 tree currently currently carries this 652 line diff against core, to make Tux3 work. This is mainly by Hirofumi, except the fs-writeback.c hook, which is by me. The main

Re: [FYI] tux3: Core changes

2015-05-18 Thread Daniel Phillips
On 05/17/2015 07:20 PM, Rik van Riel wrote: > On 05/17/2015 09:26 AM, Boaz Harrosh wrote: >> On 05/14/2015 03:59 PM, Rik van Riel wrote: >>> The issue is that things like ptrace, AIO, infiniband >>> RDMA, and other direct memory access subsystems can take >>> a reference to page A, which Tux3

Re: [FYI] tux3: Core changes

2015-05-18 Thread Daniel Phillips
On 05/17/2015 07:20 PM, Rik van Riel wrote: On 05/17/2015 09:26 AM, Boaz Harrosh wrote: On 05/14/2015 03:59 PM, Rik van Riel wrote: The issue is that things like ptrace, AIO, infiniband RDMA, and other direct memory access subsystems can take a reference to page A, which Tux3 clones into a

Re: [FYI] tux3: Core changes

2015-05-15 Thread Daniel Phillips
On 05/15/2015 01:09 AM, Mel Gorman wrote: > On Thu, May 14, 2015 at 11:06:22PM -0400, Rik van Riel wrote: >> On 05/14/2015 08:06 PM, Daniel Phillips wrote: >>>> The issue is that things like ptrace, AIO, infiniband >>>> RDMA, and other direct memory access sub

Re: [FYI] tux3: Core changes

2015-05-15 Thread Daniel Phillips
On 05/14/2015 08:06 PM, Rik van Riel wrote: > On 05/14/2015 08:06 PM, Daniel Phillips wrote: >>> The issue is that things like ptrace, AIO, infiniband >>> RDMA, and other direct memory access subsystems can take >>> a reference to page A, which Tux3 clones into a n

Re: [FYI] tux3: Core changes

2015-05-15 Thread Daniel Phillips
On 05/14/2015 08:06 PM, Rik van Riel wrote: On 05/14/2015 08:06 PM, Daniel Phillips wrote: The issue is that things like ptrace, AIO, infiniband RDMA, and other direct memory access subsystems can take a reference to page A, which Tux3 clones into a new page B when the process writes

Re: [FYI] tux3: Core changes

2015-05-15 Thread Daniel Phillips
On 05/15/2015 01:09 AM, Mel Gorman wrote: On Thu, May 14, 2015 at 11:06:22PM -0400, Rik van Riel wrote: On 05/14/2015 08:06 PM, Daniel Phillips wrote: The issue is that things like ptrace, AIO, infiniband RDMA, and other direct memory access subsystems can take a reference to page A, which

Re: [FYI] tux3: Core changes

2015-05-14 Thread Daniel Phillips
Hi Rik, Added Mel, Andrea and Peterz to CC as interested parties. There are probably others, please just jump in. On 05/14/2015 05:59 AM, Rik van Riel wrote: > On 05/14/2015 04:26 AM, Daniel Phillips wrote: >> Hi Rik, >> >> Our linux-tux3 tree currently currently carri

[FYI] tux3: Core changes

2015-05-14 Thread Daniel Phillips
Hi Rik, Our linux-tux3 tree currently currently carries this 652 line diff against core, to make Tux3 work. This is mainly by Hirofumi, except the fs-writeback.c hook, which is by me. The main part you may be interested in is rmap.c, which addresses the issues raised at the 2013 Linux Storage

[WIP] tux3: Optimized fsync

2015-05-14 Thread Daniel Phillips
elta_ref) sb->delta_pending++; wake_up_all(>delta_transition_wq); } + +#ifdef __KERNEL__ +#include "commit_fsync.c" +#endif diff --git a/fs/tux3/commit_fsync.c b/fs/tux3/commit_fsync.c new file mode 100644 index 000..9a59c59 --- /dev/null +++ b/fs/tux3/commit_f

Re: [FYI] tux3: Core changes

2015-05-14 Thread Daniel Phillips
Hi Rik, Added Mel, Andrea and Peterz to CC as interested parties. There are probably others, please just jump in. On 05/14/2015 05:59 AM, Rik van Riel wrote: On 05/14/2015 04:26 AM, Daniel Phillips wrote: Hi Rik, Our linux-tux3 tree currently currently carries this 652 line diff against

[WIP] tux3: Optimized fsync

2015-05-14 Thread Daniel Phillips
--git a/fs/tux3/commit_fsync.c b/fs/tux3/commit_fsync.c new file mode 100644 index 000..9a59c59 --- /dev/null +++ b/fs/tux3/commit_fsync.c @@ -0,0 +1,341 @@ +/* + * Optimized fsync. + * + * Copyright (c) 2015 Daniel Phillips + */ + +#include linux/delay.h + +static inline int fsync_pending(struct

[FYI] tux3: Core changes

2015-05-14 Thread Daniel Phillips
Hi Rik, Our linux-tux3 tree currently currently carries this 652 line diff against core, to make Tux3 work. This is mainly by Hirofumi, except the fs-writeback.c hook, which is by me. The main part you may be interested in is rmap.c, which addresses the issues raised at the 2013 Linux Storage

Re: Tux3 Report: How fast can we fail?

2015-05-13 Thread Daniel Phillips
Addendum to that post... On 05/12/2015 10:46 AM, I wrote: > ...For example, we currently > overestimate the cost of a rewrite because we would need to go poking > around in btrees to do that more accurately. Fixing that will be quite > a bit of work... Ah no, I was wrong about that, it will not

Re: xfs: does mkfs.xfs require fancy switches to get decent performance? (was Tux3 Report: How fast can we fsync?)

2015-05-13 Thread Daniel Phillips
On Wednesday, May 13, 2015 1:25:38 PM PDT, Martin Steigerwald wrote: Am Mittwoch, 13. Mai 2015, 12:37:41 schrieb Daniel Phillips: On 05/13/2015 12:09 PM, Martin Steigerwald wrote: ... Daniel, if you want to change the process of patch review and inclusion into the kernel, model an example

Re: xfs: does mkfs.xfs require fancy switches to get decent performance? (was Tux3 Report: How fast can we fsync?)

2015-05-13 Thread Daniel Phillips
On Wednesday, May 13, 2015 1:02:34 PM PDT, Jeremy Allison wrote: On Wed, May 13, 2015 at 12:37:41PM -0700, Daniel Phillips wrote: On 05/13/2015 12:09 PM, Martin Steigerwald wrote: ... Daniel, please listen to Martin. He speaks a fundamental truth here. As you know, I am also interested

Re: xfs: does mkfs.xfs require fancy switches to get decent performance? (was Tux3 Report: How fast can we fsync?)

2015-05-13 Thread Daniel Phillips
On 05/13/2015 12:09 PM, Martin Steigerwald wrote: > Daniel, what are you trying to achieve here? > > I thought you wanted to create interest for your filesystem and acceptance > for merging it. > > What I see you are actually creating tough is something different. > > Is what you see after you

Re: xfs: does mkfs.xfs require fancy switches to get decent performance? (was Tux3 Report: How fast can we fsync?)

2015-05-13 Thread Daniel Phillips
On 05/13/2015 06:08 AM, Mike Galbraith wrote: > On Wed, 2015-05-13 at 04:31 -0700, Daniel Phillips wrote: >> Third possibility: build from our repository, as Mike did. > > Sorry about that folks. I've lost all interest, it won't happen again. Thanks for your valuable contr

Re: xfs: does mkfs.xfs require fancy switches to get decent performance? (was Tux3 Report: How fast can we fsync?)

2015-05-13 Thread Daniel Phillips
On 05/13/2015 04:31 AM, Daniel Phillips wrote: Let me be the first to catch that arithmetic error > Let's say our delta size is 400MB (typical under load) and we leave > a "nice big gap" of 112 MB after flushing each one. Let's say we do > two thousand of those before de

Re: xfs: does mkfs.xfs require fancy switches to get decent performance? (was Tux3 Report: How fast can we fsync?)

2015-05-13 Thread Daniel Phillips
On 05/13/2015 12:25 AM, Pavel Machek wrote: > On Mon 2015-05-11 16:53:10, Daniel Phillips wrote: >> Hi Pavel, >> >> On 05/11/2015 03:12 PM, Pavel Machek wrote: >>>>> It is a fact of life that when you change one aspect of an intimately >>>>> inte

Re: xfs: does mkfs.xfs require fancy switches to get decent performance? (was Tux3 Report: How fast can we fsync?)

2015-05-13 Thread Daniel Phillips
On 05/13/2015 12:25 AM, Pavel Machek wrote: On Mon 2015-05-11 16:53:10, Daniel Phillips wrote: Hi Pavel, On 05/11/2015 03:12 PM, Pavel Machek wrote: It is a fact of life that when you change one aspect of an intimately interconnected system, something else will change as well. You have

Re: xfs: does mkfs.xfs require fancy switches to get decent performance? (was Tux3 Report: How fast can we fsync?)

2015-05-13 Thread Daniel Phillips
On 05/13/2015 04:31 AM, Daniel Phillips wrote: Let me be the first to catch that arithmetic error Let's say our delta size is 400MB (typical under load) and we leave a nice big gap of 112 MB after flushing each one. Let's say we do two thousand of those before deciding that we have enough

Re: xfs: does mkfs.xfs require fancy switches to get decent performance? (was Tux3 Report: How fast can we fsync?)

2015-05-13 Thread Daniel Phillips
On 05/13/2015 06:08 AM, Mike Galbraith wrote: On Wed, 2015-05-13 at 04:31 -0700, Daniel Phillips wrote: Third possibility: build from our repository, as Mike did. Sorry about that folks. I've lost all interest, it won't happen again. Thanks for your valuable contribution. Now we are seeing

Re: xfs: does mkfs.xfs require fancy switches to get decent performance? (was Tux3 Report: How fast can we fsync?)

2015-05-13 Thread Daniel Phillips
On Wednesday, May 13, 2015 1:25:38 PM PDT, Martin Steigerwald wrote: Am Mittwoch, 13. Mai 2015, 12:37:41 schrieb Daniel Phillips: On 05/13/2015 12:09 PM, Martin Steigerwald wrote: ... Daniel, if you want to change the process of patch review and inclusion into the kernel, model an example

Re: xfs: does mkfs.xfs require fancy switches to get decent performance? (was Tux3 Report: How fast can we fsync?)

2015-05-13 Thread Daniel Phillips
On 05/13/2015 12:09 PM, Martin Steigerwald wrote: Daniel, what are you trying to achieve here? I thought you wanted to create interest for your filesystem and acceptance for merging it. What I see you are actually creating tough is something different. Is what you see after you send

Re: xfs: does mkfs.xfs require fancy switches to get decent performance? (was Tux3 Report: How fast can we fsync?)

2015-05-13 Thread Daniel Phillips
On Wednesday, May 13, 2015 1:02:34 PM PDT, Jeremy Allison wrote: On Wed, May 13, 2015 at 12:37:41PM -0700, Daniel Phillips wrote: On 05/13/2015 12:09 PM, Martin Steigerwald wrote: ... Daniel, please listen to Martin. He speaks a fundamental truth here. As you know, I am also interested

Re: Tux3 Report: How fast can we fail?

2015-05-13 Thread Daniel Phillips
Addendum to that post... On 05/12/2015 10:46 AM, I wrote: ...For example, we currently overestimate the cost of a rewrite because we would need to go poking around in btrees to do that more accurately. Fixing that will be quite a bit of work... Ah no, I was wrong about that, it will not be a

Re: xfs: does mkfs.xfs require fancy switches to get decent performance? (was Tux3 Report: How fast can we fsync?)

2015-05-12 Thread Daniel Phillips
On 05/12/2015 03:35 PM, David Lang wrote: > On Tue, 12 May 2015, Daniel Phillips wrote: >> On 05/12/2015 02:30 PM, David Lang wrote: >>> You need to get out of the mindset that Ted and Dave are Enemies that you >>> need to overcome, they are >>> friendly

Re: xfs: does mkfs.xfs require fancy switches to get decent performance? (was Tux3 Report: How fast can we fsync?)

2015-05-12 Thread Daniel Phillips
On 05/12/2015 02:30 PM, David Lang wrote: > On Tue, 12 May 2015, Daniel Phillips wrote: >> Phoronix published a headline that identifies Dave Chinner as >> someone who takes shots at other projects. Seems pretty much on >> the money to me, and it ought to be obvious why he d

Re: xfs: does mkfs.xfs require fancy switches to get decent performance? (was Tux3 Report: How fast can we fsync?)

2015-05-12 Thread Daniel Phillips
On 05/12/2015 02:30 PM, David Lang wrote: > On Tue, 12 May 2015, Daniel Phillips wrote: >> Phoronix published a headline that identifies Dave Chinner as >> someone who takes shots at other projects. Seems pretty much on >> the money to me, and it ought to be obvious why he d

Re: xfs: does mkfs.xfs require fancy switches to get decent performance? (was Tux3 Report: How fast can we fsync?)

2015-05-12 Thread Daniel Phillips
On 05/12/2015 11:39 AM, David Lang wrote: > On Mon, 11 May 2015, Daniel Phillips wrote: >>> ...it's the mm and core kernel developers that need to >>> review and accept that code *before* we can consider merging tux3. >> >> Please do not say "we"

Tux3 Report: How fast can we fail?

2015-05-12 Thread Daniel Phillips
bus locked add is about 6 nanoseconds on my i5, and about ten times higher when contended. /* * Blurt v0.0 * * A trivial multitasking filesystem load generator * * Daniel Phillips, June 2015 * * to build: c99 -Wall blurt.c -oblurt * to run: blurt */ #include #include #include #include #

Re: Tux3 Report: How fast can we fsync?

2015-05-12 Thread Daniel Phillips
bus locked add is about 6 nanoseconds on my i5, and about ten times higher when contended. /* * Blurt v0.0 * * A trivial multitasking filesystem load generator * * Daniel Phillips, June 2015 * * to build: c99 -Wall blurt.c -oblurt * to run: blurt */ #include #include #include #include #

Re: xfs: does mkfs.xfs require fancy switches to get decent performance? (was Tux3 Report: How fast can we fsync?)

2015-05-12 Thread Daniel Phillips
On 05/12/2015 02:03 AM, Pavel Machek wrote: > On Mon 2015-05-11 19:34:34, Daniel Phillips wrote: >> On 05/11/2015 04:17 PM, Theodore Ts'o wrote: >>> and another way that people >>> doing competitive benchmarking can screw up and produce misleading >>> numbe

Re: xfs: does mkfs.xfs require fancy switches to get decent performance? (was Tux3 Report: How fast can we fsync?)

2015-05-12 Thread Daniel Phillips
On Monday, May 11, 2015 10:38:42 PM PDT, Dave Chinner wrote: > I think Ted and I are on the same page here. "Competitive > benchmarks" only matter to the people who are trying to sell > something. You're trying to sell Tux3, but By "same page", do you mean "transparently obvious about

Re: xfs: does mkfs.xfs require fancy switches to get decent performance? (was Tux3 Report: How fast can we fsync?)

2015-05-12 Thread Daniel Phillips
On Monday, May 11, 2015 10:38:42 PM PDT, Dave Chinner wrote: I think Ted and I are on the same page here. Competitive benchmarks only matter to the people who are trying to sell something. You're trying to sell Tux3, but By same page, do you mean transparently obvious about obstructing

Re: Tux3 Report: How fast can we fsync?

2015-05-12 Thread Daniel Phillips
* * A trivial multitasking filesystem load generator * * Daniel Phillips, June 2015 * * to build: c99 -Wall blurt.c -oblurt * to run: blurt basename steps tasks */ #include unistd.h #include stdlib.h #include stdio.h #include fcntl.h #include sys/wait.h #include errno.h #include sys/types.h #include

Re: xfs: does mkfs.xfs require fancy switches to get decent performance? (was Tux3 Report: How fast can we fsync?)

2015-05-12 Thread Daniel Phillips
On 05/12/2015 11:39 AM, David Lang wrote: On Mon, 11 May 2015, Daniel Phillips wrote: ...it's the mm and core kernel developers that need to review and accept that code *before* we can consider merging tux3. Please do not say we when you know that I am just as much a we as you are. Merging

Tux3 Report: How fast can we fail?

2015-05-12 Thread Daniel Phillips
* * A trivial multitasking filesystem load generator * * Daniel Phillips, June 2015 * * to build: c99 -Wall blurt.c -oblurt * to run: blurt basename steps tasks */ #include unistd.h #include stdlib.h #include stdio.h #include fcntl.h #include sys/wait.h #include errno.h #include sys/types.h #include

Re: xfs: does mkfs.xfs require fancy switches to get decent performance? (was Tux3 Report: How fast can we fsync?)

2015-05-12 Thread Daniel Phillips
On 05/12/2015 03:35 PM, David Lang wrote: On Tue, 12 May 2015, Daniel Phillips wrote: On 05/12/2015 02:30 PM, David Lang wrote: You need to get out of the mindset that Ted and Dave are Enemies that you need to overcome, they are friendly competitors, not Enemies. You are wrong about Dave

Re: xfs: does mkfs.xfs require fancy switches to get decent performance? (was Tux3 Report: How fast can we fsync?)

2015-05-12 Thread Daniel Phillips
On 05/12/2015 02:30 PM, David Lang wrote: On Tue, 12 May 2015, Daniel Phillips wrote: Phoronix published a headline that identifies Dave Chinner as someone who takes shots at other projects. Seems pretty much on the money to me, and it ought to be obvious why he does it. Phoronix turns any

Re: xfs: does mkfs.xfs require fancy switches to get decent performance? (was Tux3 Report: How fast can we fsync?)

2015-05-12 Thread Daniel Phillips
On 05/12/2015 02:30 PM, David Lang wrote: On Tue, 12 May 2015, Daniel Phillips wrote: Phoronix published a headline that identifies Dave Chinner as someone who takes shots at other projects. Seems pretty much on the money to me, and it ought to be obvious why he does it. Phoronix turns any

Re: xfs: does mkfs.xfs require fancy switches to get decent performance? (was Tux3 Report: How fast can we fsync?)

2015-05-12 Thread Daniel Phillips
On 05/12/2015 02:03 AM, Pavel Machek wrote: On Mon 2015-05-11 19:34:34, Daniel Phillips wrote: On 05/11/2015 04:17 PM, Theodore Ts'o wrote: and another way that people doing competitive benchmarking can screw up and produce misleading numbers. If you think we screwed up or produced

Re: xfs: does mkfs.xfs require fancy switches to get decent performance? (was Tux3 Report: How fast can we fsync?)

2015-05-11 Thread Daniel Phillips
Hi David, On 05/11/2015 05:12 PM, David Lang wrote: > On Mon, 11 May 2015, Daniel Phillips wrote: > >> On 05/11/2015 03:12 PM, Pavel Machek wrote: >>>>> It is a fact of life that when you change one aspect of an intimately >>>>> interconnected system

Re: xfs: does mkfs.xfs require fancy switches to get decent performance? (was Tux3 Report: How fast can we fsync?)

2015-05-11 Thread Daniel Phillips
On 05/11/2015 04:17 PM, Theodore Ts'o wrote: > On Tue, May 12, 2015 at 12:12:23AM +0200, Pavel Machek wrote: >> Umm, are you sure. If "some areas of disk are faster than others" is >> still true on todays harddrives, the gaps will decrease the >> performance (as you'll "use up" the fast areas

Re: xfs: does mkfs.xfs require fancy switches to get decent performance? (was Tux3 Report: How fast can we fsync?)

2015-05-11 Thread Daniel Phillips
Hi Pavel, On 05/11/2015 03:12 PM, Pavel Machek wrote: >>> It is a fact of life that when you change one aspect of an intimately >>> interconnected system, >>> something else will change as well. You have naive/nonexistent free space >>> management now; when you >>> design something workable

Re: xfs: does mkfs.xfs require fancy switches to get decent performance? (was Tux3 Report: How fast can we fsync?)

2015-05-11 Thread Daniel Phillips
Hi Pavel, On 05/11/2015 03:12 PM, Pavel Machek wrote: It is a fact of life that when you change one aspect of an intimately interconnected system, something else will change as well. You have naive/nonexistent free space management now; when you design something workable there it is going

Re: xfs: does mkfs.xfs require fancy switches to get decent performance? (was Tux3 Report: How fast can we fsync?)

2015-05-11 Thread Daniel Phillips
Hi David, On 05/11/2015 05:12 PM, David Lang wrote: On Mon, 11 May 2015, Daniel Phillips wrote: On 05/11/2015 03:12 PM, Pavel Machek wrote: It is a fact of life that when you change one aspect of an intimately interconnected system, something else will change as well. You have naive

Re: xfs: does mkfs.xfs require fancy switches to get decent performance? (was Tux3 Report: How fast can we fsync?)

2015-05-11 Thread Daniel Phillips
On 05/11/2015 04:17 PM, Theodore Ts'o wrote: On Tue, May 12, 2015 at 12:12:23AM +0200, Pavel Machek wrote: Umm, are you sure. If some areas of disk are faster than others is still true on todays harddrives, the gaps will decrease the performance (as you'll use up the fast areas more

Re: Tux3 Report: How fast can we fsync?

2015-05-02 Thread Daniel Phillips
On Friday, May 1, 2015 6:07:48 PM PDT, David Lang wrote: On Fri, 1 May 2015, Daniel Phillips wrote: On Friday, May 1, 2015 8:38:55 AM PDT, Dave Chinner wrote: Well, yes - I never claimed XFS is a general purpose filesystem. It is a high performance filesystem. Is is also becoming more

Re: Tux3 Report: How fast can we fsync?

2015-05-02 Thread Daniel Phillips
On Friday, May 1, 2015 6:07:48 PM PDT, David Lang wrote: On Fri, 1 May 2015, Daniel Phillips wrote: On Friday, May 1, 2015 8:38:55 AM PDT, Dave Chinner wrote: Well, yes - I never claimed XFS is a general purpose filesystem. It is a high performance filesystem. Is is also becoming more

Re: Tux3 Report: How fast can we fsync?

2015-05-01 Thread Daniel Phillips
On Friday, May 1, 2015 8:38:55 AM PDT, Dave Chinner wrote: > > Well, yes - I never claimed XFS is a general purpose filesystem. It > is a high performance filesystem. Is is also becoming more relevant > to general purpose systems as low cost storage gains capabilities > that used to be considered

Re: Tux3 Report: How fast can we fsync?

2015-05-01 Thread Daniel Phillips
On Friday, May 1, 2015 8:38:55 AM PDT, Dave Chinner wrote: Well, yes - I never claimed XFS is a general purpose filesystem. It is a high performance filesystem. Is is also becoming more relevant to general purpose systems as low cost storage gains capabilities that used to be considered the

Re: xfs: does mkfs.xfs require fancy switches to get decent performance? (was Tux3 Report: How fast can we fsync?)

2015-04-30 Thread Daniel Phillips
Hi Ted, On 04/30/2015 07:57 AM, Theodore Ts'o wrote: > This is one of the reasons why I find head-to-head "competitions" > between file systems to be not very helpful for anything other than > benchmarketing. It's almost certain that the benchmark won't be > "fair" in some way, and it doesn't

Re: xfs: does mkfs.xfs require fancy switches to get decent performance? (was Tux3 Report: How fast can we fsync?)

2015-04-30 Thread Daniel Phillips
On 04/30/2015 07:33 AM, Mike Galbraith wrote: > Well ok, let's forget bad blood, straw men... and answering my question > too I suppose. Not having any sexy IO gizmos in my little desktop box, > I don't care deeply which stomps the other flat on beastly boxen. I'm with you, especially the

Re: xfs: does mkfs.xfs require fancy switches to get decent performance? (was Tux3 Report: How fast can we fsync?)

2015-04-30 Thread Daniel Phillips
On 04/30/2015 07:28 AM, Howard Chu wrote: > Daniel Phillips wrote: >> >> >> On 04/30/2015 06:48 AM, Mike Galbraith wrote: >>> On Thu, 2015-04-30 at 05:58 -0700, Daniel Phillips wrote: >>>> On Thursday, April 30, 2015 5:07:21 AM PDT, Mike Galbraith wrot

  1   2   3   4   5   6   7   8   9   10   >