Re: [RFC v4+ hot_track 00/19] vfs: hot data tracking

2012-10-29 Thread Andi Kleen
zwu.ker...@gmail.com writes: TODO List: 1.) Need to do scalability or performance tests. You're changing some of the most performance critical code in the kernel. This step is absolutely not optional. -Andi -- a...@linux.intel.com -- Speaking for myself only -- To unsubscribe from this

Re: [GIT PULL v2] Update LZO compression

2012-10-03 Thread Andi Kleen
- Post the entire patch series to lkml for review (I'd like a cc please) Already happened, multiple people reviewed and tested. um, I would not consider Looks ok to me from a quick look. and I couldn't tell from the github view, but I assume you follow standard coding style. to

Re: [GIT PULL] Update LZO compression

2012-08-16 Thread Andi Kleen
If you think a little bit, I bet you could come up with a solution that operates at cacheline-aligned granularity, something that would be _even faster_ than simply fixing the code to do aligned accesses. Cache aligned compression is unlikely to compress anything at all. Compression

Re: [GIT PULL] Update LZO compression

2012-08-16 Thread Andi Kleen
I have locked the Allwinner A10 CPU in my Mele A2000 to 60 MHz using cpufreq-set, and ran your test. rnd.lzo is a 9 MB file from /dev/urandom compressed with lzo. There doesn't seem to be a significant difference between all three variants. I found that in compression benchmarks it depends

Re: [GIT PULL] Update LZO compression

2012-08-13 Thread Andi Kleen
On Tue, Aug 14, 2012 at 01:44:02AM +0200, Markus F.X.J. Oberhumer wrote: Hi all, as suggested on the mailing list I have converted the updated LZO code into git, so please pull my lzo-update branch from git://github.com/markus-oberhumer/linux.git lzo-update You can browse the branch

Re: [PATCH] Btrfs: use a slab for btrfs_dio_private

2012-08-07 Thread Andi Kleen
Josef Bacik jba...@fusionio.com writes: This is in the IO path, let's try to avoid latencies by having our own slab. Thanks, What latencies does this avoid? -Andi -- a...@linux.intel.com -- Speaking for myself only -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in

Re: [PATCH 1/3] Add the snappy-c compressor to lib v2

2012-02-14 Thread Andi Kleen
(BTW: If you're ever reworking this patch set, I'd like to make an ad hoc request for slightly different names for fs/btrfs/snappy.c and lib/snappy.c) Why? When building a x86 kernel, I get the following errors: CC [M] lib/snappy.o lib/snappy.c: In function 'snappy_init_env':

Re: [RFB] add LZ4 compression method to btrfs

2012-02-14 Thread Andi Kleen
Markus Lindberg marcuslindb...@gmail.com writes: Are you sure about these figures ? the difference seems too large. It's almost unbelievable. Yes, my benchmarks totally disagree with them. In my tests lz4 is generally slower than snappy-c. -Andi -- a...@linux.intel.com -- Speaking for

Re: ANN: linux-kernel-lzo-2.06.20120123 - update LZO to v2.06

2012-01-24 Thread Andi Kleen
On Mon, Jan 23, 2012 at 05:19:40PM +0100, Markus F.X.J. Oberhumer wrote: Hi, I've prepared a small package that updates the LZO version in the Linux kernel to LZO v2.06. I ran benchmarks on the new miniLZO and LZ4 on 64bit. LZ4 is generally slower than snappy/lzo in the micro benchmarks.

Re: Updated btrfs/crypto snappy interface ready for merging

2012-01-17 Thread Andi Kleen
It's because decompressing inline extents always fails. I've fixed it and will send the patch out in a new mail thread. Thanks for fixing. But seems there's bug in lib snappy code, which makes the decompressed data doesn't quite match the original data. Simply copy a file to a btrfs

Re: Updated btrfs/crypto snappy interface ready for merging

2012-01-17 Thread Andi Kleen
At first I saved emails and patched them in linux-btrfs git tree, and I got weired result. Then I pulled the snappy branch directly, and now nothing is wrong.. Sorry for the noise. np. Thanks for testing. -Andi -- a...@linux.intel.com -- Speaking for myself only. -- To unsubscribe from this

[PATCH] SNAPPY: Add dual GPL/BSD license to snappy module

2012-01-15 Thread Andi Kleen
From: Andi Kleen a...@linux.intel.com To avoid complaints from the module loader. I made it dual because the original compression code was BSD. Reported-by: Chris Mason Signed-off-by: Andi Kleen a...@linux.intel.com --- lib/snappy.c |2 ++ 1 files changed, 2 insertions(+), 0 deletions

Re: [RFC PATCH v2 0/3] Btrfs: apply the Probabilistic Skiplist on btrfs

2012-01-13 Thread Andi Kleen
For the btrfs extent cache it's unclear if just RCUing is a good fit anyways: some workloads are very write heavy and RCU only performs well if you have a lot more reads than writes. For write heavy RCUification usually slows it down. FWIW, I'm mentioning this out of self interest - I need a

Re: [RFC PATCH v2 0/3] Btrfs: apply the Probabilistic Skiplist on btrfs

2012-01-12 Thread Andi Kleen
Liu Bo liubo2...@cn.fujitsu.com writes: Here we choose extent_map firstly, since it is a read mostly thing, and the change is quite direct, all we need to do is a) to replace rbtree with skiplist, b) to add rcu support. And more details are in patch 2 and patch 3. I've done some simple

Updated btrfs/crypto snappy interface ready for merging

2012-01-12 Thread Andi Kleen
/kernel/git/dtor/input (2012-01-12 12:40:41 -0800) are available in the git repository at: git://git.kernel.org/pub/scm/linux/kernel/git/ak/linux-misc snappy Andi Kleen (3): Add the snappy-c compressor to lib v2 BTRFS: Add snappy support v2 Add snappy interface to crypto API

[PATCH 2/3] BTRFS: Add snappy support v2

2012-01-12 Thread Andi Kleen
From: Andi Kleen a...@linux.intel.com Add support in btrfs for snappy compression. This is generally a faster compressor than LZO (except for higher memory usage) with comparable compression and should imho replace it. This is based on the lzo code with minor modifications. This has been tested

[PATCH 1/3] Add the snappy-c compressor to lib v2

2012-01-12 Thread Andi Kleen
From: Andi Kleen a...@linux.intel.com This is a C port of the google snappy compressor. It has roughly comparable compression to LZO, but is significantly faster on many file types. For example it beats all other compressors on already compressed data. I ported the original C++ code over to C

[PATCH 3/3] Add snappy interface to crypto API

2012-01-12 Thread Andi Kleen
From: Andi Kleen a...@linux.intel.com Mainly so that ubifs can use it. Acked-by: Herbert Xu herb...@gondor.apana.org.au Signed-off-by: Andi Kleen a...@linux.intel.com --- crypto/Kconfig |9 + crypto/Makefile |1 + crypto/snappy.c | 99

Re: [patch 07/99] btrfs: Use mempools for extent_state structures

2011-11-28 Thread Andi Kleen
Jeff Mahoney je...@suse.com writes: The extent_state structure is used at the core of the extent i/o code for managing flags, locking, etc. It requires allocations deep in the write code and if failures occur they are difficult to recover from. We avoid most of the failures by using a

[PATCH 1/3] Add the snappy-c compressor to lib

2011-10-20 Thread Andi Kleen
From: Andi Kleen a...@linux.intel.com This is a C port of the google snappy compressor. It has roughly comparable compression to LZO, but is significantly faster on many file types. For example it beats all other compressors on already compressed data. I ported the original C++ code over to C

[PATCH 2/3] BTRFS: Add snappy support

2011-10-20 Thread Andi Kleen
From: Andi Kleen a...@linux.intel.com Add support in btrfs for snappy compression. This is based on the lzo code with minor modifications. The btrfs glue code could be significantly improved over LZO by exploiting some snappy features, but hasn't so far. Open: implement scatter-gather support

[PATCH 3/3] Add snappy interface to crypto API

2011-10-20 Thread Andi Kleen
From: Andi Kleen a...@linux.intel.com Mainly so that ubifs can use it. Snappy is a better compressor in the same niche as LZO. Only lightly tested so far. Experiences welcome. Cc: herb...@gondor.apana.org.au Cc: dedeki...@gmail.com Cc: adrian.hun...@intel.com Signed-off-by: Andi Kleen

Re: Honest timeline for btrfsck

2011-10-06 Thread Andi Kleen
Jeff Putney jeffrey.put...@gmail.com writes: http://en.wikipedia.org/wiki/Release_early,_release_often Well the other principle in free software you're forgetting is: It will be released when it's ready If you don't like Chris' ways to do releases you're free to write something on your own

Re: [PATCH v0 07/18] btrfs: generic data structure to build unique lists

2011-10-06 Thread Andi Kleen
Arne Jansen sensi...@gmx.net writes: ulist is a generic data structures to hold a collection of unique u64 values. The only operations it supports is adding to the list and enumerating it. It is possible to store an auxiliary value along with the key. The implementation is preliminary and

Re: [PATCH v0 03/18] btrfs: add nested locking mode for paths

2011-10-06 Thread Andi Kleen
Arne Jansen sensi...@gmx.net writes: This patch adds the possibilty to read-lock an extent even if it is already write-locked from the same thread. Subvolume quota needs this capability. Recursive locking is generally strongly discouraged, it causes all kinds of problems and tends to

Re: [PATCH v0 17/18] btrfs: add qgroup ioctls

2011-10-06 Thread Andi Kleen
Arne Jansen sensi...@gmx.net writes: + + if (copy_to_user(arg, sa, sizeof(*sa))) + ret = -EFAULT; + + if (trans) { + err = btrfs_commit_transaction(trans, root); + if (err !ret) + ret = err; + } It would seem safer to

Re: [PATCH v0 03/18] btrfs: add nested locking mode for paths

2011-10-06 Thread Andi Kleen
On Fri, Oct 07, 2011 at 12:44:30AM +0400, Andrey Kuzmin wrote: Perhaps you could just elaborate on needs this feature? In general, write lock gives one exclusive access, so the need for additional read (non-exclusive) lock does not appear easily understandable. Usually it's because the low

Re: [PATCH 1/7] BTRFS: Fix lseek return value for error

2011-09-19 Thread Andi Kleen
On Mon, Sep 19, 2011 at 03:30:02PM -0400, Chris Mason wrote: Excerpts from Andi Kleen's message of 2011-09-19 13:52:03 -0400: Thanks everyone, I've put Jeff's last version of this in my queue. Can you post the version you merged? The previous ones all had issues.

Re: [PATCH 1/7] BTRFS: Fix lseek return value for error

2011-09-17 Thread Andi Kleen
with an additional improvement if the offset is larger or equal to the file size, return -ENXIO in directly: if (offset = inode-i_size) { mutex_unlock(inode-i_mutex); return -ENXIO; } Except that is

Re: [PATCH 1/7] BTRFS: Fix lseek return value for error

2011-09-16 Thread Andi Kleen
On Fri, Sep 16, 2011 at 11:48:15AM -0400, Christoph Hellwig wrote: On Thu, Sep 15, 2011 at 04:06:47PM -0700, Andi Kleen wrote: From: Andi Kleen a...@linux.intel.com Introduced by 9a4327ca1f45f82edad7dc0a4e52ce9316e0950c I think this should go to Chris/Linus ASAP. But a slightly better

Re: [PATCH] BTRFS: Free inode mutex on lseek error

2011-08-20 Thread Andi Kleen
On Sat, Aug 20, 2011 at 08:08:25AM -0400, Josef Bacik wrote: On 08/19/2011 08:07 PM, Andi Kleen wrote: From: Andi Kleen a...@linux.intel.com Introduced with b26751575a9aa55fd6dbf3febde3ff06dfadc44f This has already been fixed by 9a4327ca1f45f82edad7dc0a4e52ce9316e0950c It's

[PATCH] BTRFS: Free inode mutex on lseek error

2011-08-19 Thread Andi Kleen
From: Andi Kleen a...@linux.intel.com Introduced with b26751575a9aa55fd6dbf3febde3ff06dfadc44f Cc: jo...@redhat.com Cc: chris.ma...@oracle.com Signed-off-by: Andi Kleen a...@linux.intel.com --- fs/btrfs/file.c | 14 -- 1 files changed, 8 insertions(+), 6 deletions(-) diff --git

Re: [RFC PATCH 4/4] btrfs: Moved repair code from inode.c to extent_io.c

2011-07-24 Thread Andi Kleen
Jan Schmidt list.bt...@jan-o-sch.net writes: Repair works that way: Whenever a read error occurs and we have more mirrors to try, note the failed mirror, and retry another. If we find a good one, check if we did note a failure earlier and if so, do not allow the read to complete until after

Re: [RFC PATCH 4/4] btrfs: Moved repair code from inode.c to extent_io.c

2011-07-24 Thread Andi Kleen
I wasn't clear enough on that: We only track read errors, here. Ans error correction can only happen on the read path. So if the write attempt fails, we can't go into a loop. Not in a loop, but you trigger more IO errors, which can be nasty if the IO error logging triggers more IO (pretty

Re: [PATCH v5 8/8] btrfs: new ioctls to do logical-inode and inode-path resolving

2011-07-21 Thread Andi Kleen
Jan Schmidt list.bt...@jan-o-sch.net writes: + +static long btrfs_ioctl_logical_to_ino(struct btrfs_root *root, + void __user *arg) +{ + int ret = 0; + int size; + u64 extent_offset; + struct btrfs_ioctl_logical_ino_args *loi; +

Re: [PATCH] Btrfs: initial online fsck support

2011-07-02 Thread Andi Kleen
The two big features of btrfs are self-healing and online fsck, those have to Are they? be implemented in kernel space. Why? There have been online fscks in user space in the past, e.g. the various schemes using LVM snapshots for ext* and other related work on the BSD FFS. I don't see any

Re: [PATCH] Btrfs: initial online fsck support

2011-07-01 Thread Andi Kleen
Li Zefan l...@cn.fujitsu.com writes: This is an initial version of online fsck. What it does is: - check the dir item and dir index pointing to a file. - check the structure of extents of a file. As furthur work, we should consider: - fix but not only check the structure of a file. -

Re: [PATCH] Btrfs: save preloaded extent_state's in a percpu cache V2

2011-06-21 Thread Andi Kleen
Josef Bacik jo...@redhat.com writes: When doing DIO tracing I noticed we were doing a ton of allocations, a lot of the time for extent_states. Some of the time we don't even use the prealloc'ed extent_state, it just get's free'd up. So instead create a per-cpu cache like the radix tree

Re: [GIT PULL] Btrfs updates

2011-06-12 Thread Andi Kleen
Chris Mason chris.ma...@oracle.com writes: Hi everyone, The for-linus branch of the btrfs unstable tree: git://git.kernel.org/pub/scm/linux/kernel/git/mason/btrfs-unstable.git for-linus Has our current queue of fixes. Josef's is the biggest pile, mostly in the allocator. Josef and I

Re: [BUG 3.0-rc1] oops during file removal, severe lock contention

2011-06-02 Thread Andi Kleen
Dave Chinner da...@fromorbit.com writes: Also, there is massive lock contention while running these workloads. perf top shows this for the create after about 5m inodes have been created: We saw pretty much the same thing in some simple tests on large systems (extent io tree locking and higher

Re: [PATCH 2/2] Btrfs: load the key from the dir item in readdir into a fake dentry

2011-05-26 Thread Andi Kleen
On Thu, May 26, 2011 at 03:02:42PM -0400, Josef Bacik wrote: + /* + * If this dentry needs lookup, don't set the referenced flag so that it + * is more likely to be cleaned up by the dcache shrinker in case of + * memory pressure. + */ + if

Re: [PATCH 1/2] fs: add a DCACHE_NEED_LOOKUP flag for d_flags

2011-05-20 Thread Andi Kleen
Putting them at the end of the cache LRU instead of the head would allow them to be dropped quickly under memory pressure. This still would fill up your memory for find /, potentially pushing out other stuff. -Andi -- a...@linux.intel.com -- Speaking for myself only. -- To unsubscribe from

Re: [PATCH 1/2] fs: add a DCACHE_NEED_LOOKUP flag for d_flags

2011-05-20 Thread Andi Kleen
On Fri, May 20, 2011 at 08:30:19PM -0400, Josef Bacik wrote: On 05/20/2011 05:31 PM, Andi Kleen wrote: Putting them at the end of the cache LRU instead of the head would allow them to be dropped quickly under memory pressure. This still would fill up your memory for find /, potentially

[PATCH] BTRFS: Remove unused node_lock

2011-05-17 Thread Andi Kleen
From: Andi Kleen a...@linux.intel.com 240f62c8756 replaced the node_lock with rcu_read_lock, but forgot to remove the actual lock in the data structure. Remove it here. Signed-off-by: Andi Kleen a...@linux.intel.com --- fs/btrfs/ctree.h |3 --- fs/btrfs/disk-io.c |1 - 2 files changed

Re: stratified B-trees

2011-03-23 Thread Andi Kleen
Karn Kallio tierplusplusli...@gmail.com writes: Are these stratified B-trees something which the btrfs project could use? The current b*tree is pretty much hardcoded in the disk format, so it would be hard to change in a compatible way. -Andi -- a...@linux.intel.com -- Speaking for myself

Re: [PATCH v3 3/6] btrfs: add scrub code and prototypes

2011-03-16 Thread Andi Kleen
Arne Jansen sensi...@gmx.net writes: + */ + mutex_lock(fs_info-scrub_lock); + atomic_inc(fs_info-scrubs_running); + mutex_unlock(fs_info-scrub_lock); It seems odd to protect an atomic_inc with a mutex. Is that done for some side effect? Otherwise you either don't need

Re: hunt for 2.6.37 dm-crypt+ext4 corruption? (was: Re: dm-crypt barrier support is effective)

2010-12-15 Thread Andi Kleen
I have a question though: the deactivation of multiple page-io submission support most likely only would affect bigger systems or also desktop systems (like mine) ? I think this is not a final fix, just a workaround. The problem with the other path still really needs to be tracked down. -Andi

Re: hunt for 2.6.37 dm-crypt+ext4 corruption? (was: Re: dm-crypt barrier support is effective)

2010-12-09 Thread Andi Kleen
512MB. 'free' reports 75MB, 419MB free. I originally noticed the problem on really real hardware (thinkpad T61p), however. If you can easily reproduce it could you try a git bisect? -Andi -- a...@linux.intel.com -- Speaking for myself only. -- To unsubscribe from this list: send the

Re: [dm-devel] hunt for 2.6.37 dm-crypt+ext4 corruption?

2010-12-05 Thread Andi Kleen
I've been using a kernel which is between 2.6.37-rc2 and -rc3 with a LUKS / dm-crypt / LVM / ext4 setup for my primary file systems, and I haven't observed any corruption for the last two weeks or so. It's on my todo list to upgrade to top of Linus's tree, but perhaps this is a useful

Re: [PATCH] Btrfs: add a disk info ioctl to get the disks attached to a filesystem

2010-09-30 Thread Andi Kleen
Kay Sievers kay.siev...@vrfy.org writes: Yeah, we thought about that too, but a btrfs mount does not show up as a block device, like md/dm, so there is no place for a slaves/ directory in /sys with the individual disks listed. How could be solve that? Create some fake blockdev for every btrfs

Re: [PATCH] x86_64/lib: improve the performance of memmove

2010-09-16 Thread Andi Kleen
On Thu, 16 Sep 2010 15:16:31 +0800 Miao Xie mi...@cn.fujitsu.com wrote: On Thu, 16 Sep 2010 08:48:25 +0200 (cest), Andi Kleen wrote: When the dest and the src do overlap and the memory area is large, memmove of x86_64 is very inefficient, and it led to bad performance, such as btrfs's

Re: [PATCH V2 1/3] lib: introduce some memory copy macros and functions

2010-09-08 Thread Andi Kleen
According to the data, the length of the most copies is =128. Thanks for the data. Large is easier to optimize than small, that's good. Could you also measure how many memsets need the backwards copy? (should be easy to add) If the number is small that needs backwards then the easiest fix

Re: [PATCH V2 1/3] lib: introduce some memory copy macros and functions

2010-09-08 Thread Andi Kleen
On Wed, 08 Sep 2010 20:57:07 +0800 Miao Xie mi...@cn.fujitsu.com wrote: On Wed, 8 Sep 2010 14:19:25 +0200 (cest), Andi Kleen wrote: According to the data, the length of the most copies is=128. Thanks for the data. Large is easier to optimize than small, that's good. Could you also

Re: [PATCH V2 1/3] lib: introduce some memory copy macros and functions

2010-09-02 Thread Andi Kleen
Miao Xie mi...@cn.fujitsu.com writes: Changes from V1 to V2: - change the version of GPL from version 2.1 to version 2 the kernel's memcpy and memmove is very inefficient. But the glibc version is quite fast, in some cases it is 10 times faster than the kernel version. So I Can you

Re: Poor read performance on high-end server

2010-08-08 Thread Andi Kleen
Jens Axboe ax...@kernel.dk writes: Also, I didn't see Chris mention this, but if you have a newer intel box you can use hw accellerated crc32c instead. For some reason my test box always loads crc32c and not crc32c-intel, so I need to do that manually. I have a patch for that, will post it

Re: Btrfs: broken file system design

2010-06-25 Thread Andi Kleen
Daniel Taylor daniel.tay...@wdc.com writes: As long as no object smaller than the disk block size is ever flushed to media, and all flushed objects are aligned to the disk blocks, there should be no real performance hit from that. The question is just how large such a block needs to be.

[PATCH] [13/23] BTRFS: Clean up unused variables -- nonbugs

2010-06-10 Thread Andi Kleen
-log.c | 15 +-- fs/btrfs/volumes.c|4 fs/btrfs/xattr.c |2 -- fs/btrfs/zlib.c |5 - Signed-off-by: Andi Kleen a...@linux.intel.com --- fs/btrfs/compression.c |2 -- fs/btrfs/ctree.c

[PATCH] [12/23] BTRFS: Clean up unused variables -- bugs

2010-06-10 Thread Andi Kleen
@vger.kernel.org Signed-off-by: Andi Kleen a...@linux.intel.com --- fs/btrfs/dir-item.c|2 +- fs/btrfs/extent-tree.c |3 +-- fs/btrfs/extent_io.c |2 ++ fs/btrfs/inode.c |6 +++--- fs/btrfs/relocation.c |4 +++- fs/btrfs/tree-log.c|2 +- 6 files changed, 11

Re: [Fwd: Re: Linking two files together][RFC]

2010-06-09 Thread Andi Kleen
Roberto Ragusa m...@robertoragusa.it writes: I hope that ideas about btrfs are not off-topic for this mailing list. The forwarded message below was written by me on fedora-users. The thread is about the ability to link two files in a manner similar to cat 1 2 3 rm 1 2 while avoiding any

Re: [PATCH] btrfs: check alloc return value before use handle and struct

2010-05-19 Thread Andi Kleen
Steven Liu lingjiujia...@gmail.com writes: diff --git a/fs/btrfs/dir-item.c b/fs/btrfs/dir-item.c index e9103b3..230a131 100644 --- a/fs/btrfs/dir-item.c +++ b/fs/btrfs/dir-item.c @@ -142,6 +142,8 @@ int btrfs_insert_dir_item(struct btrfs_trans_handle *trans, struct btrfs_root

Re: Poor interactive performance with I/O loads with fsync()ing

2010-04-11 Thread Andi Kleen
Has the reason for this been identified? Judging from the nature of metadata loads, it would seem that it should be substantially easier to implement fsync() efficiently. By design a copy on write tree fs would need to flush a whole tree hierarchy on a sync. btrfs avoids this by using a

Re: [PATCH V3 17/18] Btrfs: Full direct I/O and AIO read implementation.

2010-03-25 Thread Andi Kleen
On Wed, Mar 24, 2010 at 11:08:07PM -0400, jim owens wrote: Andi Kleen wrote: On Tue, Mar 23, 2010 at 05:40:00PM -0400, jim owens wrote: Andi Kleen wrote: With checksums enabled, uncompressed reads aligned on the 4k block are classic direct IO to user memory except at EOF

Re: [PATCH V3 17/18] Btrfs: Full direct I/O and AIO read implementation.

2010-03-23 Thread Andi Kleen
On Tue, Mar 23, 2010 at 05:40:00PM -0400, jim owens wrote: Andi Kleen wrote: One thing that stroke me while reading is that, except for the out of line no data checksum case, this isn't real classical zero copy direct IO because you always have to copy through some buffer. Uh

Re: [PATCH V3 17/18] Btrfs: Full direct I/O and AIO read implementation.

2010-03-22 Thread Andi Kleen
jim owens owens6...@gmail.com writes: Hi Jim, I read through large chunks of that patch. I don't claim to fully understand all the btrfs infrastructure details enough, so it wasn't really serious code review. One thing that stroke me while reading is that, except for the out of line no data

Re: file/extent checksums for dedup/sync...

2010-01-27 Thread Andi Kleen
Daniel J Blueman daniel.blue...@gmail.com writes: For purposes of data deduplication and data synchronisation, it would be a powerful tool to expose file data checksums. Since eg BTRFS uses the crc32c algorithm [1], it's possible to compute the file's overall CRC from the accumulation of the

Re: [RFC 00/12] btrfs core patches for direct I/O

2010-01-06 Thread Andi Kleen
jim owens jow...@hp.com writes: The existing core code for pagecache doesn't work for directio because the existing I/O routines depend on peeking inside a struct page to get the valid btrfs inode info. Sorry, no can do, we don't own that page. And rewriting the whole pagecache I/O stack to

Re: [PATCH] Fix possible pointer dereference

2009-10-06 Thread Andi Kleen
Diego Calleja dieg...@gmail.com writes: We should always check btrfs_alloc_path(). Some places BUG(), others return -ENOMEM, btrfs_insert_dir_item() seems like it can return safely. The problem is that all the callers don't handle errors. It doesn't make sense to fix it low-level currently

Re: [PATCH] Btrfs: potential NULL dereferences

2009-09-06 Thread Andi Kleen
Roel Kluin roel.kl...@gmail.com writes: Allocations may fail, prevent NULL dereferences. Signed-off-by: Roel Kluin roel.kl...@gmail.com --- In several sections of fs/btrfs code a kmalloc() occurs without a check whether it succeeded. this potentially leads to dereferences of a NULL

Re: [PATCH] btrfs: implement FS_IOC_GETFLAGS/SETFLAGS/GETVERSION

2009-04-25 Thread Andi Kleen
Christoph Hellwig h...@lst.de writes: Add support for the standard attributes set via chattr and read vis lsattr. Currently we store the attributes in the flags value in the btrfs inode, but I wonder whether we should split it into two so that we don't have to keep converting between the two

Re: gcc inlining heuristics was Re: [PATCH -v7][RFC]: mutex: implement adaptive spinning

2009-01-21 Thread Andi Kleen
GCC 4.3.2. Maybe i missed something obvious? The typical use case of restrict is to tell it that multiple given arrays are independent and then give the loop optimizer more freedom to handle expressions in the loop that accesses these arrays. Since there are no loops in the list functions

Re: gcc inlining heuristics was Re: [PATCH -v7][RFC]: mutex: implement adaptive spinning

2009-01-21 Thread Andi Kleen
The point is that the compiler is then free to do it. If things slow down after the compiler gets *more* information, then that is a problem with the compiler heuristics rather than the information we give it. The point was the -Os disables it typically then. (not always, compiler heuristics

Re: btrfs and swap files on SSD's ?

2009-01-20 Thread Andi Kleen
Dmitri Nikulin dniku...@gmail.com writes: On Wed, Jan 21, 2009 at 12:02 AM, Chris Mason chris.ma...@oracle.com wrote: There are patches to support swap over NFS that might make it safe to use on btrfs. At any rate, it is a fixable problem. FreeBSD has been able to run swap over NFS for as

Re: gcc inlining heuristics was Re: [PATCH -v7][RFC]: mutex: implement adaptive spinning

2009-01-19 Thread Andi Kleen
The problem with 'restrict' is that almost nobody uses it, and it does Also gcc traditionally didn't do a very good job using it (this might be better in the very latest versions). At least some of the 3.x often discarded this information. automatically. But it should work well as a way to

Re: gcc inlining heuristics was Re: [PATCH -v7][RFC]: mutex: implement adaptive spinning

2009-01-12 Thread Andi Kleen
On Mon, Jan 12, 2009 at 11:02:17AM -0800, Linus Torvalds wrote: Something at the back of my mind said aliasing. $ gcc linus.c -O2 -S ; grep subl linus.s subl$1624, %esp $ gcc linus.c -O2 -S -fno-strict-aliasing; grep subl linus.s subl$824, %esp That's with

gcc inlining heuristics was Re: [PATCH -v7][RFC]: mutex: implement adaptive spinning

2009-01-11 Thread Andi Kleen
On Sun, Jan 11, 2009 at 11:25:32AM -0800, Linus Torvalds wrote: On Sun, 11 Jan 2009, Andi Kleen wrote: The proposal was to use -fno-inline-functions-called-once (but the resulting numbers were not promising) Well, the _optimal_ situation would be to not need it, because gcc does

Re: gcc inlining heuristics was Re: [PATCH -v7][RFC]: mutex: implement adaptive spinning

2009-01-11 Thread Andi Kleen
On Sun, Jan 11, 2009 at 03:05:53PM -0800, Linus Torvalds wrote: On Sun, 11 Jan 2009, Linus Torvalds wrote: On Sun, 11 Jan 2009, Andi Kleen wrote: Was -- i think that got fixed in gcc. But again only in newer versions. I doubt it. People have said that about a million times

Re: gcc inlining heuristics was Re: [PATCH -v7][RFC]: mutex: implement adaptive spinning

2009-01-11 Thread Andi Kleen
On Sun, Jan 11, 2009 at 04:21:03PM -0800, Linus Torvalds wrote: On Mon, 12 Jan 2009, Andi Kleen wrote: so at least least for this case it works. Your case also doesn't work for me. So it looks like gcc didn't like something you did in your test program. I very intentionally used

Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact

2009-01-09 Thread Andi Kleen
I vote for the, get rid of the current inline, rename __always_inline to There is some code that absolutely requires inline for correctness, like the x86-64 vsyscall code. I would advocate to keep the explicit __always_inline at least there to make it very clear. inline, and then remove all

Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact

2009-01-09 Thread Andi Kleen
On Fri, Jan 09, 2009 at 10:28:01AM -0700, Matthew Wilcox wrote: On Fri, Jan 09, 2009 at 06:20:11PM +0100, Andi Kleen wrote: Also cc Honza in case he has comments (you might want to review more of the thread in the archives) I think this particular bug is already known and discussed: I

Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact

2009-01-09 Thread Andi Kleen
What happens when you say -fno-inline-functions-called-once? Does it disable inlining for those functions IN GENERAL, or just for the LARGE It does disable it in general, unless they're marked inline explicitely : The traditional gcc 2.x rules were: I Only inline what is marked inline (but

Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact

2009-01-09 Thread Andi Kleen
On Fri, Jan 09, 2009 at 08:10:20PM +0100, Richard Guenther wrote: On Fri, Jan 9, 2009 at 8:21 PM, Andi Kleen a...@firstfloor.org wrote: GCC 4.3 should be good in compiling the kernel with default -Os settings. It's unfortunately not. It doesn't inline a lot of simple asm() inlines

Re: [PATCH -v7][RFC]: mutex: implement adaptive spinning

2009-01-09 Thread Andi Kleen
I've done a finegrained size analysis today (see my other mail in this thread), and it turns out that on gcc 4.3.x the main (and pretty much only) inlining annotation that matters in arch/x86/include/asm/*.h is the onliner patch attached below, annotating constant_test_bit(). That's pretty

Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact

2009-01-09 Thread Andi Kleen
What's the cost/benefit of that 4%? Does it actually improve performance? Especially if you then want to keep DWARF unwind information in memory in order to fix up some of the problems it causes? At that point, you lost dwarf unwind information has nothing to do with this, it doesn't tell

Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact

2009-01-09 Thread Andi Kleen
I thought -Os actually disabled the basic-block reordering, doesn't it? Not in current gcc head no (just verified by stepping through) And I thought it did that exactly because it generates bigger code and much worse I$ patterns (ie you have a lot of conditional branch to other place

Re: [PATCH -v7][RFC]: mutex: implement adaptive spinning

2009-01-09 Thread Andi Kleen
On Sat, Jan 10, 2009 at 12:53:42AM +, Jamie Lokier wrote: Harvey Harrison wrote: Oh yeah, and figure out what actually breaks on alpha such that they added the following (arch/alpha/include/asm/compiler.h) #ifdef __KERNEL__ /* Some idiots over in linux/compiler.h thought inline

Re: [PATCH -v7][RFC]: mutex: implement adaptive spinning

2009-01-08 Thread Andi Kleen
On Thu, Jan 08, 2009 at 05:44:25PM -0800, H. Peter Anvin wrote: Harvey Harrison wrote: We might still try the second or third options, as i think we shouldnt go back into the business of managing the inline attributes of ~100,000 kernel functions. Or just make it clear that inline

Re: [PATCH -v7][RFC]: mutex: implement adaptive spinning

2009-01-08 Thread Andi Kleen
On Thu, Jan 08, 2009 at 07:42:48PM -0800, Linus Torvalds wrote: I actually often use noinline when developing code simply because it makes it easier to read oopses when gcc doesn't inline ever static (which it normally does if it only has a single caller) Yes. Gcc inlining is a total

Re: [PATCH -v5][RFC]: mutex: implement adaptive spinning

2009-01-07 Thread Andi Kleen
I appreciate this is sample code, but using __get_user() on non-userspace pointers messes up architectures which have separate user/kernel spaces (eg the old 4G/4G split for x86-32). Do we have an appropriate function for kernel space pointers? probe_kernel_address(). But it's slow. -Andi

Re: Btrfs for mainline

2009-01-02 Thread Andi Kleen
Chris Mason chris.ma...@oracle.com writes: On Wed, 2008-12-31 at 10:45 -0800, Andrew Morton wrote: On Wed, 31 Dec 2008 06:28:55 -0500 Chris Mason chris.ma...@oracle.com wrote: Hello everyone, Hi! I've done some testing against Linus' git tree from last night and the current btrfs

Re: Btrfs for mainline

2009-01-02 Thread Andi Kleen
On Fri, Jan 02, 2009 at 02:32:29PM -0500, Chris Mason wrote: If combination spinlocks/mutexes are really a win they should be in the generic mutex framework. And I'm still dubious on the hardcoded numbers. Sure, I'm happy to use a generic framework there (or help create one). They are

Re: Some very basic questions

2008-10-21 Thread Andi Kleen
Chris Mason [EMAIL PROTECTED] writes: Started interactively? I'm not entirely sure what that means, but in general when you ask the user a question about if/how to fix a corruption, they will have no idea what the correct answer is. While that's true today, I'm not sure it has to be true

Re: Data-deduplication?

2008-10-15 Thread Andi Kleen
On Wed, Oct 15, 2008 at 03:39:16PM +0200, Avi Kivity wrote: Andi Kleen wrote: Ray Van Dolson [EMAIL PROTECTED] writes: I recall their being a thread here a number of months back regarding data-deduplication support for bttfs. Did anyone end up picking that up and giving a go

Re: Data-deduplication?

2008-10-13 Thread Andi Kleen
Ray Van Dolson [EMAIL PROTECTED] writes: I recall their being a thread here a number of months back regarding data-deduplication support for bttfs. Did anyone end up picking that up and giving a go at it? Block level data dedup would be *awesome* in a Linux filesystem. It does wonders for

Re: [PATCH] Btrfs: avoid NULL deref after failed allocation

2008-10-02 Thread Andi Kleen
Jim Meyering [EMAIL PROTECTED] writes: However, in some places, the trend is to BUG_ON(!ptr), so I've done that, too. Even if it's a trend, it's wrong. Better don't add more. And also unnecessary because next reference will obviously oops anyways. -Andi -- To unsubscribe from this list:

Re: btrfs_tree_lock trylock

2008-09-08 Thread Andi Kleen
On Mon, Sep 08, 2008 at 10:02:30AM -0400, Chris Mason wrote: On Mon, 2008-09-08 at 15:54 +0200, Andi Kleen wrote: The idea is to try to spin for a bit to avoid scheduling away, which is especially important for the high levels. Most holders of the mutex let it go very quickly. Ok

Re: New feature Idea

2008-08-13 Thread Andi Kleen
Morey Roof [EMAIL PROTECTED] writes: I have been thinking about a new feature to start work on that I am interested in and I was hoping people could give me some feedback and ideas of how to tackle it. Anyways, I want to create a data deduplication system that can work in two different

Re: Btrfs v0.16 released

2008-08-08 Thread Andi Kleen
So, the mirroring turns a single large write into two large writes. Definitely not free, but always a fixed cost. Thanks for the explanation and the numbers. I see that's the advantage of copy-on-write that you can actually always cluster the metadata together and get always batched IO this

Re: Btrfs v0.16 released

2008-08-07 Thread Andi Kleen
Chris Mason [EMAIL PROTECTED] writes: Metadata is duplicated by default even on single spindle drives, Can you please say a bit how much that impacts performance? That sounds costly. -Andi -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to