Re: Tux3 Report: Initial fsck has landed

2013-03-19 Thread Daniel Phillips
On Tue, Mar 19, 2013 at 9:04 PM, David Lang da...@lang.hm wrote: On Wed, 20 Mar 2013, Martin Steigerwald wrote: Am Dienstag, 29. Januar 2013 schrieb Daniel Phillips: On Mon, Jan 28, 2013 at 5:40 PM, Theodore Ts'o ty...@mit.edu wrote: On Mon, Jan 28, 2013 at 04:20:11PM -0800, Darrick J. Wong

Re: Tux3 Report: Initial fsck has landed

2013-03-20 Thread Daniel Phillips
On Tue, Mar 19, 2013 at 11:54 PM, Rob Landley r...@landley.net wrote: I'm confused, http://tux3.org/ lists a bunch of dates from 5 years ago, then nothing. Is this project dead or not? Not. We haven't done much about updating tux3.org lately, however you will find plenty of activity here:

Re: kernel merge

2013-03-22 Thread Daniel Phillips
On Tue, Mar 19, 2013 at 1:52 AM, Raymond Jennings shent...@gmail.com wrote: What I've heard so far about tux3 is very promising. When can consideration be given to merging it into the mainline linux kernel? For starters it's a great way to increase the testing base, and I'm actually

Re: Tux3 Report: Faster than tmpfs, what?

2013-05-11 Thread Daniel Phillips
Hi Dave, Thanks for the catch - I should indeed have noted that modified dbench was used for this benchmark, thus amplifying Tux3's advantage in delete performance. This literary oversight does not make the results any less interesting: we beat Tmpfs on that particular load. Beating tmpfs at

Tux3 Report: Meet Shardmap, the designated successor of HTree

2013-06-18 Thread Daniel Phillips
Greetings all, From time to time, one may fortunate enough to be blessed with a discovery in computer science that succeeds at improving all four of performance, scalability, reliability and simplicity. Of these normally conflicting goals, simplicity is usually the most elusive. It is

Design note: Simplified implementation of free block tags

2013-06-24 Thread Daniel Phillips
Free tags Free tags in Tux3 will perforrm a similar function to Ext2's unallocated block counts. In Ext2, an unallocated block count is a 16 bit field in the group descriptor for each block group. Tux3 does not have group descriptors, but will use for a similar purpose a table of free tags, where

Re: [PATCH] Optimize wait_sb_inodes()

2013-06-26 Thread Daniel Phillips
Hi Dave, On Wed, Jun 26, 2013 at 9:47 PM, Dave Chinner da...@fromorbit.com wrote: You have your own wait code, that doesn't make what the VFS does unnecesary. Quite frankly, I don't trust individual filesystems to get it right - there's a long history of filesystem specific data sync problems

Re: [RFC] Tux3 for review

2014-05-16 Thread Daniel Phillips
On Friday, May 16, 2014 10:09:50 PM PDT, Martin Steigerwald wrote: Hi Daniel! Am Freitag, 16. Mai 2014, 17:50:59 schrieb Daniel Phillips: We would like to offer Tux3 for review for mainline merge. We have prepared a new repository suitable for pulling: At long last! Congrats for arriving

Re: [RFC] Tux3 for review

2014-05-20 Thread Daniel Phillips
Hi Dave, This is to address your concern about theoretical interaction between direct IO and Tux3 page fork. On Monday, May 19, 2014 10:41:40 PM PDT, I wrote: Except that Direct IO impacts on the design of the page forking code (because of how things like get_user_pages() need to be aware of

Re: [RFC] Tux3 for review

2014-05-23 Thread Daniel Phillips
Hi Dongsu, On Thursday, May 22, 2014 2:52:27 AM PDT, Dongsu Park wrote: First of all, thank you for trying to merge it to mainline. Maybe I cannot say the code is clean enough, but basically the filesystem seems to work at least. Thank you for confirming that. We test Tux3 extensively so we

Re: Tux3 influence

2014-07-24 Thread Daniel Phillips
Hi Sachar, On Wednesday, July 23, 2014 4:43:35 AM PDT, you wrote: In the past few month I developed Funex, a new (FUSE-based, GPL) file-system. Although still in very early alpha stages, the basic functionality seams to work fine. During the development process I discovered again and again that

Re: xfs: does mkfs.xfs require fancy switches to get decent performance? (was Tux3 Report: How fast can we fsync?)

2015-04-30 Thread Daniel Phillips
On Wednesday, April 29, 2015 5:20:08 PM PDT, Dave Chinner wrote: It's easy to be fast on empty filesystems. XFS does not aim to be fast in such situations - it aims to have consistent performance across the life of the filesystem. In this case, ext4, btrfs and tux3 have optimal allocation

Re: Tux3 Report: How fast can we fsync?

2015-04-30 Thread Daniel Phillips
On Wednesday, April 29, 2015 8:50:57 PM PDT, Mike Galbraith wrote: On Wed, 2015-04-29 at 13:40 -0700, Daniel Phillips wrote: That order of magnitude latency difference is striking. It sounds good, but what does it mean? I see a smaller difference here, maybe because of running under KVM

Re: Tux3 Report: How fast can we fsync?

2015-04-30 Thread Daniel Phillips
On Wednesday, April 29, 2015 6:46:16 PM PDT, Dave Chinner wrote: I measured fsync performance using a 7200 RPM disk as a virtual drive under KVM, configured with cache=none so that asynchronous writes are cached and synchronous writes translate into direct writes to the block device. Yup, a

Re: Tux3 Report: How fast can we fsync?

2015-05-01 Thread Daniel Phillips
On Friday, May 1, 2015 8:38:55 AM PDT, Dave Chinner wrote: Well, yes - I never claimed XFS is a general purpose filesystem. It is a high performance filesystem. Is is also becoming more relevant to general purpose systems as low cost storage gains capabilities that used to be considered the

Re: Tux3 Report: How fast can we fsync?

2015-05-02 Thread Daniel Phillips
On Friday, May 1, 2015 6:07:48 PM PDT, David Lang wrote: On Fri, 1 May 2015, Daniel Phillips wrote: On Friday, May 1, 2015 8:38:55 AM PDT, Dave Chinner wrote: Well, yes - I never claimed XFS is a general purpose filesystem. It is a high performance filesystem. Is is also becoming more

Re: Tux3 Report: How fast can we fsync?

2015-04-30 Thread Daniel Phillips
On Thursday, April 30, 2015 2:17:55 PM PDT, James Cloos wrote: DP == Daniel Phillips dan...@phunq.net writes: DP you build userspace tools from the hirofumi-user branch In a fresh clone there is no hirofumi-user branch, only hirofumi and master: :; cat .git/packed-refs # pack-refs

Re: xfs: does mkfs.xfs require fancy switches to get decent performance? (was Tux3 Report: How fast can we fsync?)

2015-05-11 Thread Daniel Phillips
Hi David, On 05/11/2015 05:12 PM, David Lang wrote: On Mon, 11 May 2015, Daniel Phillips wrote: On 05/11/2015 03:12 PM, Pavel Machek wrote: It is a fact of life that when you change one aspect of an intimately interconnected system, something else will change as well. You have naive

Re: xfs: does mkfs.xfs require fancy switches to get decent performance? (was Tux3 Report: How fast can we fsync?)

2015-05-12 Thread Daniel Phillips
On Monday, May 11, 2015 10:38:42 PM PDT, Dave Chinner wrote: I think Ted and I are on the same page here. Competitive benchmarks only matter to the people who are trying to sell something. You're trying to sell Tux3, but By same page, do you mean transparently obvious about obstructing

Re: Tux3 Report: How fast can we fsync?

2015-05-12 Thread Daniel Phillips
* * A trivial multitasking filesystem load generator * * Daniel Phillips, June 2015 * * to build: c99 -Wall blurt.c -oblurt * to run: blurt basename steps tasks */ #include unistd.h #include stdlib.h #include stdio.h #include fcntl.h #include sys/wait.h #include errno.h #include sys/types.h #include

Re: xfs: does mkfs.xfs require fancy switches to get decent performance? (was Tux3 Report: How fast can we fsync?)

2015-05-13 Thread Daniel Phillips
On 05/13/2015 04:31 AM, Daniel Phillips wrote: Let me be the first to catch that arithmetic error Let's say our delta size is 400MB (typical under load) and we leave a nice big gap of 112 MB after flushing each one. Let's say we do two thousand of those before deciding that we have enough

Re: xfs: does mkfs.xfs require fancy switches to get decent performance? (was Tux3 Report: How fast can we fsync?)

2015-05-13 Thread Daniel Phillips
On 05/13/2015 06:08 AM, Mike Galbraith wrote: On Wed, 2015-05-13 at 04:31 -0700, Daniel Phillips wrote: Third possibility: build from our repository, as Mike did. Sorry about that folks. I've lost all interest, it won't happen again. Thanks for your valuable contribution. Now we are seeing

Re: xfs: does mkfs.xfs require fancy switches to get decent performance? (was Tux3 Report: How fast can we fsync?)

2015-05-12 Thread Daniel Phillips
On 05/12/2015 11:39 AM, David Lang wrote: On Mon, 11 May 2015, Daniel Phillips wrote: ...it's the mm and core kernel developers that need to review and accept that code *before* we can consider merging tux3. Please do not say we when you know that I am just as much a we as you are. Merging

Re: xfs: does mkfs.xfs require fancy switches to get decent performance? (was Tux3 Report: How fast can we fsync?)

2015-05-12 Thread Daniel Phillips
On 05/12/2015 03:35 PM, David Lang wrote: On Tue, 12 May 2015, Daniel Phillips wrote: On 05/12/2015 02:30 PM, David Lang wrote: You need to get out of the mindset that Ted and Dave are Enemies that you need to overcome, they are friendly competitors, not Enemies. You are wrong about Dave

Re: Tux3 Report: How fast can we fail?

2015-05-13 Thread Daniel Phillips
Addendum to that post... On 05/12/2015 10:46 AM, I wrote: ...For example, we currently overestimate the cost of a rewrite because we would need to go poking around in btrees to do that more accurately. Fixing that will be quite a bit of work... Ah no, I was wrong about that, it will not be a

Re: [FYI] tux3: Core changes

2015-05-15 Thread Daniel Phillips
On 05/15/2015 01:09 AM, Mel Gorman wrote: On Thu, May 14, 2015 at 11:06:22PM -0400, Rik van Riel wrote: On 05/14/2015 08:06 PM, Daniel Phillips wrote: The issue is that things like ptrace, AIO, infiniband RDMA, and other direct memory access subsystems can take a reference to page A, which

Re: Blurt code in Github

2015-05-13 Thread Daniel Phillips
On 05/13/2015 07:07 AM, Elifarley Callado Coelho Cruz wrote: Where can I see the torture test results ? You mean, http://buildbot.tux3.org:8010/ ? I am not as familiar with it as I should be. Maybe it would be good to use Wercker ( which is free - http://wercker.com/ ) to make sure we

Re: [FYI] tux3: Core changes

2015-05-18 Thread Daniel Phillips
On 05/17/2015 07:20 PM, Rik van Riel wrote: On 05/17/2015 09:26 AM, Boaz Harrosh wrote: On 05/14/2015 03:59 PM, Rik van Riel wrote: The issue is that things like ptrace, AIO, infiniband RDMA, and other direct memory access subsystems can take a reference to page A, which Tux3 clones into a

Re: xfs: does mkfs.xfs require fancy switches to get decent performance? (was Tux3 Report: How fast can we fsync?)

2015-05-12 Thread Daniel Phillips
On 05/12/2015 02:03 AM, Pavel Machek wrote: On Mon 2015-05-11 19:34:34, Daniel Phillips wrote: On 05/11/2015 04:17 PM, Theodore Ts'o wrote: and another way that people doing competitive benchmarking can screw up and produce misleading numbers. If you think we screwed up or produced

Re: [FYI] tux3: Core changes

2015-05-20 Thread Daniel Phillips
On 05/20/2015 07:44 AM, Jan Kara wrote: On Tue 19-05-15 13:33:31, David Lang wrote: On Tue, 19 May 2015, Daniel Phillips wrote: I understand that Tux3 may avoid these issues due to some other mechanisms it internally has but if page forking should get into mm subsystem, the above must work

Re: Tux3 Report: How fast can we fail?

2015-05-28 Thread Daniel Phillips
On Thursday, May 28, 2015 5:55:18 AM PDT, Austin S Hemmelgarn wrote: On 2015-05-27 18:46, Daniel Phillips wrote: On 05/27/2015 02:39 PM, Pavel Machek wrote: On Wed 2015-05-27 11:28:50, Daniel Phillips wrote: ... I mentioned earlier, it seems to work pretty well in Tux3. But do user

Re: Tux3 Report: How fast can we fail?

2015-05-28 Thread Daniel Phillips
On Thursday, May 28, 2015 4:19:29 PM PDT, Andreas Karlsson wrote: On 05/28/2015 07:27 PM, Daniel Phillips wrote: Not doubting you, but how would overwriting files help you recover from disk full? One benefit I see is that monitoring tools based on round-robin databases like Munin and MRTG

Re: Tux3 Report: How fast can we fail?

2015-05-27 Thread Daniel Phillips
On 05/27/2015 02:39 PM, Pavel Machek wrote: On Wed 2015-05-27 11:28:50, Daniel Phillips wrote: On Tuesday, May 26, 2015 11:41:39 PM PDT, Mosis Tembo wrote: On Tue, May 26, 2015 at 6:03 PM, Pavel Machek pa...@ucw.cz wrote: We identified the following quality metrics for this algorithm: 1

Re: [FYI] tux3: Core changes

2015-05-27 Thread Daniel Phillips
On 05/27/2015 02:37 PM, Pavel Machek wrote: On Wed 2015-05-27 11:09:25, Daniel Phillips wrote: On Wednesday, May 27, 2015 12:41:37 AM PDT, Pavel Machek wrote: On Fri 2015-05-15 02:38:33, Daniel Phillips wrote: On 05/14/2015 08:06 PM, Rik van Riel wrote: ... Umm. Why do you think it is only

Re: [FYI] tux3: Core changes

2015-05-26 Thread Daniel Phillips
On Tuesday, May 26, 2015 3:13:02 AM PDT, Pavel Machek wrote: On Tue 2015-05-26 01:09:59, Daniel Phillips wrote: On Monday, May 25, 2015 11:13:46 PM PDT, David Lang wrote: I'm assuming that Rik is talking about whatever has the reference to the page via one of the methods that he talked about

Re: Tux3 Report: How fast can we fail?

2015-05-26 Thread Daniel Phillips
On Tuesday, May 26, 2015 3:03:26 AM PDT, Pavel Machek wrote: We identified the following quality metrics for this algorithm: 1) Never fails to detect out of space in the front end. 2) Always fills a volume to 100% before reporting out of space. 3) Allows rm, rmdir and truncate even when a

Re: [FYI] tux3: Core changes

2015-05-26 Thread Daniel Phillips
On 05/26/2015 02:00 AM, Jan Kara wrote: On Tue 26-05-15 01:08:56, Daniel Phillips wrote: On Tuesday, May 26, 2015 12:09:10 AM PDT, Jan Kara wrote: E.g. video drivers (or infiniband or direct IO for that matter) which have buffers in user memory (may be mmapped file), grab references to pages

Re: [FYI] tux3: Core changes

2015-05-26 Thread Daniel Phillips
Hi Sergey, On 05/26/2015 03:22 AM, Sergey Senozhatsky wrote: Hello, is it possible to page-fork-bomb the system by some 'malicious' app? Not in any new way. A page fork can happen either in the front end, where it has to wait for memory like any other normal memory user, or in the backend,

Re: [FYI] tux3: Core changes

2015-07-31 Thread Daniel Phillips
On Friday, July 31, 2015 5:00:43 PM PDT, Daniel Phillips wrote: Note: Hirofumi's email is clear, logical and speaks to the question. This branch of the thread is largely pointless, though it essentially says the same thing in non-technical terms. Perhaps your next response should be to Hirofumi

Re: [FYI] tux3: Core changes

2015-07-31 Thread Daniel Phillips
On Friday, July 31, 2015 11:29:51 AM PDT, David Lang wrote: If you define this as loosing our mojo, then yes we have. A pity. There remains so much to do that simply will not get done in the absence of mojo. Regards, Daniel ___ Tux3 mailing list

Re: [FYI] tux3: Core changes

2015-07-31 Thread Daniel Phillips
On Friday, July 31, 2015 11:29:51 AM PDT, David Lang wrote: We, the Linux Community have less tolerance for losing people's data and preventing them from operating than we used to when it was all tinkerer's personal data and secondary systems. So rather than pushing optimizations out to

Re: [FYI] tux3: Core changes

2015-07-31 Thread Daniel Phillips
On Friday, July 31, 2015 3:27:12 PM PDT, David Lang wrote: On Fri, 31 Jul 2015, Daniel Phillips wrote: On Friday, July 31, 2015 11:29:51 AM PDT, David Lang wrote: ... you weren't asking about any particular feature of Tux, you were asking if we were still willing to push out stuff

Re: [FYI] tux3: Core changes

2015-07-31 Thread Daniel Phillips
On Friday, July 31, 2015 11:29:51 AM PDT, David Lang wrote: We, the Linux Community have less tolerance for losing people's data and preventing them from operating than we used to when it was all tinkerer's personal data and secondary systems. So rather than pushing optimizations out to

Re: Trying Tux3; What am I doing wrong?

2015-08-05 Thread Daniel Phillips
Hi Masoud, Your md device is read-only for some reason. You need to fix that, then try your mount. You can also make a tux3 volume on a file and mount it using mount ... -oloop Regards, Daniel On Wed, Aug 5, 2015 at 9:53 AM, Masoud Sharbiani masoud.sharbi...@gmail.com wrote: So, hirofumi

Re: poke?

2016-12-15 Thread Daniel Phillips
Thanks for the pokes guys. Tux3 was only resting. I am doing some work on Shardmap at the moment, so we should be able to see some interesting results for directory scalability from that. Hirofumi is busy updating the tree to current upstream kernel. Regards, Daniel On Tue, Dec 13, 2016 at 9:22

Re: Tux3 performance: FUSE vs kernel

2016-12-23 Thread Daniel Phillips
On 12/23/2016 03:19 AM, Raymond Jennings wrote: Hey um, is there much performance difference between using tux3 on FUSE vs as a built-in kernel module? Yes, a huge difference. Fuse imposes a large (huge) overhead on the async operations that a kernel filesystem uses, and in the case of our

Re: online resize

2017-01-15 Thread Daniel Phillips
On 01/12/2017 06:54 AM, Raymond Jennings wrote: Does/will tux3 support online resize, both shrink and grow? Not yet, but is straightforward. Regards, Daniel ___ Tux3 mailing list Tux3@phunq.net http://phunq.net/mailman/listinfo/tux3

Re: Backgrounded deletion, dirty writeback, security, and you

2017-03-24 Thread Daniel Phillips
Hi Raymond, On 03/23/2017 01:01 PM, Raymond Jennings wrote: Having recently tangled with this issue on ext4, I had a few questions: 1. Does the backend that actually digests an unlink request know how to handle outstanding dirty blocks belonging to a now nonexisting file? If a file has

Re: Alive?

2017-03-19 Thread Daniel Phillips
Hi Raymond, Tux3.org is sufficient. Tux3.org is now hosted on phunq.net (my server). Yes, still alive, and busy. There is a code drop coming from Hirofumi in the near future, to sync up with current mainline. I am currently working on a new distributed lock manager, which hopefully will play

Re: fallocate support ...

2017-03-22 Thread Daniel Phillips
Hi Lars, On 03/21/2017 12:37 AM, Lars Segerlund wrote: Hi guys, I am doing some apps that preallocates files at known locations on disk, continous and the order ( placement ) is important, so I thought I¨d ask if there has been any thoughts about this on tux3 ? It¨s really a killer app

Tux3 update - Shardmap

2017-10-09 Thread Daniel Phillips
Hi all, This is just a quick note to let everybody know that Tux3 development is active once again. There are a small number of areas we need to focus on, to make Tux3 ready for general (experimental!) use. One of those is directory scaling, that is, Shardmap. Shardmap was conceived as a

Re: Progress report?

2018-04-04 Thread Daniel Phillips
On 2018-04-03 12:30 AM, Raymond Jennings wrote: Are you guys close to getting merged into mainline? I think it's high time that btrfs got a healthy dose of competition Hi Raymond, For the time being we will continue to develop out-of-tree, while continuing to track Linus's latest mainline